Easy Connect [the crawler]
Software Specification
The purpose of this document is to describe the required software development in order to be created a software instrument (a tool for crawling) that includes the following parts:
• Gathering information for companies and making profiles of them with the products they offer. (Crawling)
• User interface for managing – the crawler and the information which is gathered.
• Sending offers via e-mail.
• Sorting the results.
Date: 20.09.2015
Version: 1.0
Document Owner: todor.panayotov@gmail.com
Software Architecture:
As a solution we are using three different types of crawlers written on Perl/ Python or NodeJS.
Redis as a temporary storage while we are building the companies profiles.
NoSQL and Node Webkit for the User Interface.
• The following Diagram 0-A explains how the tool should work on:
Diagram 0 A
Crawlers and back end:
This chapter describes why we need three types of crawlers and Redis as middle layer storage.
No matter from the chosen technology the logic should be as it’s shown on:
Diagrams 0-B, 0-C, 0-D
Wide Range Crawler:
Diagram 0 B
Network Crawler:
Diagram 0 C
Website Crawler:
Diagram 0 D
Use cases:
General use case is shown on Diagram 0-E
But there can be and will be plenty and diverse use cases for managing the user interface.
Diagram 0 E
User Interface:
We use the Node Webkit as a technology for building the User Interface.
Based on the rule keep it simple, design should be intuitive and UX oriented.
Diagram 0 F
NB: On Diagram 0-F, I show only a Wire Frame, not actual design!
Development:
Depends on human resources we may have various approaches:
In Diagram 0-G it is shown one example how the project can be plan and executed with 4 people.
• Project Manager
• UI/UX Designer with knowledge of Node Webkit
• Backend Developer with knowledge of Perl/Python or NodeJS for the Crawlers
• Data Base Developer with knowledge of NoSQL, Redis and Big Data technologies
Diagram 0 G
Document Information:
Date: 20.09.2015
Version: 1.0
Document Owner: todor.panayotov@gmail.com