High Frequency Trading data Scraper
Budget: $30 – $250 USD
I need a scraper that reliably pulls live trading data from a web based trading platform. The scraper should run as frequently as the target site and practical rate-limits allow "ideally near real time" and quietly recover from captchas, session time-outs, and any layout changes without manual fixes.
Because this data will feed other internal dashboards and models, I’m open to your recommendation on the optimal delivery mechanism. Whether that’s streaming the results into a lightweight database, pushing updates through an internal REST endpoint, or simply appending to time stamped CSVs, the key is low latency, easy downstream integration, and strong fault tolerance.
Solution must be well commented, maintainable code (Python with requests, Selenium, or Scrapy is fine, but I’m flexible) and include clear setup instructions
Once 100% complete and all bugs have been fixed, I’ll test it against live data; if it can maintain uninterrupted collection and accurate data over a full 24 hrs trading period, I’ll sign off.
Because this data will feed other internal dashboards and models, I’m open to your recommendation on the optimal delivery mechanism. Whether that’s streaming the results into a lightweight database, pushing updates through an internal REST endpoint, or simply appending to time stamped CSVs, the key is low latency, easy downstream integration, and strong fault tolerance.
Solution must be well commented, maintainable code (Python with requests, Selenium, or Scrapy is fine, but I’m flexible) and include clear setup instructions
Once 100% complete and all bugs have been fixed, I’ll test it against live data; if it can maintain uninterrupted collection and accurate data over a full 24 hrs trading period, I’ll sign off.
Related categories:
JavaScript
Python
Data Processing
Web Scraping
Software Architecture
Scrapy
Selenium
REST API