Python Web Scraper for Live Forex Data
Budget: $30 – $250 USD
I'm looking for a Python expert who can create a web scraper to capture live forex data. The focus will be primarily on major pairs such as EUR/USD and GBP/USD.
Since you're focusing on web scraping, a robust scraping pipeline must be developed to continually fetch and process data from TradingView or similar platforms.
Scraping Tools:
BeautifulSoup or Scrapy can be used to scrape HTML content from TradingView and other reference site.
Selenium may be required for websites that heavily rely on JavaScript to load content dynamically.
Data Sources:
https://www.babypips.com/ https://www.tradingview.com/
https://www.forexfactory.com/
https://www.dailyfx.com/ https://www.investopedia.com/
https://www.forex.com/
https://www.oanda.com/
https://www.fxstreet.com/
https://tradingeconomics.com/
https://mytradingskills.com/
www.tradingview.com
Chart patterns: Scrape the latest patterns such as head and shoulders, double top, triple bottom, etc.
Community trading ideas: Scrape posts and analysis shared by traders on TradingView.
Technical indicators: Collect data on moving averages, RSI, MACD, Bollinger Bands, and other popular indicators.
Additional sources to consider:
FXStreet for Forex news and economic calendar events.
ForexFactory for community discussion and sentiment analysis.
Investing.com for Forex rates and financial news.
Data Processing Pipeline:
Cleaning and Structuring Data: Once scraped, the data needs to be cleaned, parsed, and structured (e.g., in JSON or database format) for efficient access.
Database Integration: Use PostgreSQL or MongoDB to store large datasets from TradingView. MongoDB would be better for unstructured community posts, while PostgreSQL can handle structured chart data and indicators.
Real-time scraping: Ensure the scraper runs frequently or in near real-time, as Forex data updates rapidly. Use cron jobs to schedule scraping or implement event-driven scraping to capture real-time updates.
Ideal Skills:
- Proficiency in Python
- Experience with web scraping
- Familiarity with forex data
- Knowledge of major forex pairs
- Ability to deliver real-time data
Since you're focusing on web scraping, a robust scraping pipeline must be developed to continually fetch and process data from TradingView or similar platforms.
Scraping Tools:
BeautifulSoup or Scrapy can be used to scrape HTML content from TradingView and other reference site.
Selenium may be required for websites that heavily rely on JavaScript to load content dynamically.
Data Sources:
https://www.babypips.com/ https://www.tradingview.com/
https://www.forexfactory.com/
https://www.dailyfx.com/ https://www.investopedia.com/
https://www.forex.com/
https://www.oanda.com/
https://www.fxstreet.com/
https://tradingeconomics.com/
https://mytradingskills.com/
www.tradingview.com
Chart patterns: Scrape the latest patterns such as head and shoulders, double top, triple bottom, etc.
Community trading ideas: Scrape posts and analysis shared by traders on TradingView.
Technical indicators: Collect data on moving averages, RSI, MACD, Bollinger Bands, and other popular indicators.
Additional sources to consider:
FXStreet for Forex news and economic calendar events.
ForexFactory for community discussion and sentiment analysis.
Investing.com for Forex rates and financial news.
Data Processing Pipeline:
Cleaning and Structuring Data: Once scraped, the data needs to be cleaned, parsed, and structured (e.g., in JSON or database format) for efficient access.
Database Integration: Use PostgreSQL or MongoDB to store large datasets from TradingView. MongoDB would be better for unstructured community posts, while PostgreSQL can handle structured chart data and indicators.
Real-time scraping: Ensure the scraper runs frequently or in near real-time, as Forex data updates rapidly. Use cron jobs to schedule scraping or implement event-driven scraping to capture real-time updates.
Ideal Skills:
- Proficiency in Python
- Experience with web scraping
- Familiarity with forex data
- Knowledge of major forex pairs
- Ability to deliver real-time data