Real-Time Stock Price Crawler for MySQL
Budget: $250 – $750 USD
I'm seeking a proficient data engineer to develop a system that captures real-time stock prices from the Nasdaq and NYSE.
Nasdaq:3313 stocks
NYSE:1941 stocks
The data points of interest include:
- Current price
- Opening price
- Trading volume
- Ask price
- Bid Price
- Last trade price
The data needs to be stored in a MySQL database, with updates occurring every 30 seconds to 1 minute.
Ideal Skills and Experience:
- Proficiency in data crawling and web scraping
- Experience with MySQL and database management
- Familiarity with stock market data and terminology
- Ability to create efficient, reliable systems for data capture and storage
1. setup a website to automatically craw the prices of all stocks in NASDAQ and NYSE.
2. Use Laravel as framework
3. data source: https://finance.yahoo.com/quote/TSLA/ (one stock example)
where TSLA is the symbol of tesla stock.
4. get the newest price of every stock per 30 second(a minute?); save them to mysql database. (multi ip and UA/multi-threads/Data source site anti-crawler)
when you are blocked by the data source website(the target website which is crawled),you must solve/bypassing the blocking.
(record format:<STOCK SYMBOL>,<PER>,<DATE>,<TIME>,<OPEN>,<HIGH>,<LOW>,<CLOSE>,<VOL>?)
when the database is big than 100MB size ,then save datas to a new table.
5. You can use Python/Java as combination if needed to rise the efficiency of the code.
Attention please:
I only have a linux os hosting plan now.(no panel, no apache,no php,no mysql installed yet)
vCPU: 2x AMD EPYC 9004
Memory: 2048 MB
Harddisk: 40 GB NVMe SSD（RAID10）
Bandwidth: 2 TB/Month（2.5Gbps port）
IP：1 IPv4 + 1 IPv6
Nasdaq:3313 stocks
NYSE:1941 stocks
The data points of interest include:
- Current price
- Opening price
- Trading volume
- Ask price
- Bid Price
- Last trade price
The data needs to be stored in a MySQL database, with updates occurring every 30 seconds to 1 minute.
Ideal Skills and Experience:
- Proficiency in data crawling and web scraping
- Experience with MySQL and database management
- Familiarity with stock market data and terminology
- Ability to create efficient, reliable systems for data capture and storage
1. setup a website to automatically craw the prices of all stocks in NASDAQ and NYSE.
2. Use Laravel as framework
3. data source: https://finance.yahoo.com/quote/TSLA/ (one stock example)
where TSLA is the symbol of tesla stock.
4. get the newest price of every stock per 30 second(a minute?); save them to mysql database. (multi ip and UA/multi-threads/Data source site anti-crawler)
when you are blocked by the data source website(the target website which is crawled),you must solve/bypassing the blocking.
(record format:<STOCK SYMBOL>,<PER>,<DATE>,<TIME>,<OPEN>,<HIGH>,<LOW>,<CLOSE>,<VOL>?)
when the database is big than 100MB size ,then save datas to a new table.
5. You can use Python/Java as combination if needed to rise the efficiency of the code.
Attention please:
I only have a linux os hosting plan now.(no panel, no apache,no php,no mysql installed yet)
vCPU: 2x AMD EPYC 9004
Memory: 2048 MB
Harddisk: 40 GB NVMe SSD（RAID10）
Bandwidth: 2 TB/Month（2.5Gbps port）
IP：1 IPv4 + 1 IPv6