SpiderFlow
The new generation crawler platform defines the crawler process in a graphical way, and the crawler can be completed without writing code.
SpiderFlow is an open source artifact that can complete data grabbing by dragging and dropping a flowchart. It can automatically process JS dynamic rendering pages, support external push to MySQL/MariaDB to store data, comes with scheduled tasks and proxy IP functions, and can implement advanced operations such as OCR recognition and email notification through plug-ins. SpiderFlow is especially suitable for teams who need to quickly build a data collection system. Although the project has stopped being updated, its maturity can fully meet daily needs. For scenarios that require complex anti-crawling strategies, it is recommended to use them in conjunction with the proxy IP service.
