перфектният ми.не английски

Easy Connect [the crawler]

Software Specification

The purpose of this document is to describe the required software development in order to be created a software instrument (a tool for crawling) that includes the following parts:

•    Gathering information for companies and making profiles of them with the products they offer. (Crawling)

•    User interface for managing – the crawler and the information which is gathered. 

•    Sending offers via e-mail.

•    Sorting the results.

Date:     20.09.2015
Version:    1.0
Document Owner:     todor.panayotov@gmail.com

 


 

Software Architecture:

As a solution we are using three different types of crawlers written on Perl/ Python or NodeJS.
Redis as a temporary storage while we are building the companies profiles.
NoSQL and Node Webkit for the User Interface.


•    The following Diagram 0-A explains how the tool should work on:

Diagram 0 A

Crawlers and back end:

This chapter describes why we need three types of crawlers and Redis as middle layer storage.
No matter from the chosen technology the logic should be as it’s shown on:
Diagrams 0-B, 0-C, 0-D    

Wide Range Crawler:

Diagram 0 B

Network Crawler:

Diagram 0 C

Website Crawler:

Diagram 0 D

Use cases:

General use case is shown on Diagram 0-E
But there can be and will be plenty and diverse use cases for managing the user interface.

Diagram 0 E

User Interface:

We use the Node Webkit as a technology for building the User Interface.
Based on the rule keep it simple, design should be intuitive and UX oriented.

Diagram 0 F



NB: On Diagram 0-F, I show only a Wire Frame, not actual design!  

Development:

Depends on human resources we may have various approaches:
In Diagram 0-G it is shown one example how the project can be plan and executed with 4 people.
•    Project Manager
•    UI/UX Designer with knowledge of Node Webkit
•    Backend Developer with knowledge of Perl/Python or NodeJS for the Crawlers
•    Data Base Developer with knowledge of NoSQL, Redis and Big Data technologies 

Diagram 0 G

Document Information:

   
Date:     20.09.2015
Version:    1.0
Document Owner:     todor.panayotov@gmail.com