Cloud Computing – Web Scrapping – Sync Google Drive – Merge Excel files - Reports on Excel file – Create alerts of Trading View.
Budget: €250 – €750 EUR
Project Summary: Cloud Computing – Web Scrapping – Sync Google Drive – Merge Excel files - Reports on Excel file – Create alerts of Trading View.
1. Cloud Computing: Set up a cloud of 10 machines (preferably Amazon LightSail) for web scrapping. Install Microsoft Excel, Python, and Selenium.
2. **Web scraping: Extract statistical data from TradingView.com. Excel input files are running with Python, and at the end of the process, we generate one output file.
3. Sync Google Drive: Each virtual machine syncs with one Google Drive folder, where all generated output excel files are stored.
4. Merge Excell files: All output files (from all 10 synced Google Drive folders) merged into a “Master File.”
5. Reports on Excel : Reports are generated after a period of time (7 days) where a loop of web scrapping period finishes.
Every 7 days, we generate approximately 215 output files. Each file contains 11500 lines, which equals a total of 2.5 Million lines.
I understand that creating a report on excel with such a large amount of data is not efficient; I choose this method for the first version of this project for two reasons: 1. I need an interface to cross-check manually if the results of the report are correct, 2. The speed of generating the reports is not important. You can propose any other more efficient methods for this version of later versions of the project.
6. Create alerts of Trading View: Develop further an existing python script that can create alerts of TradingView.com using data from an excel file.
What is ready:
Python script for web scraping on TradingView.com runs smoothly and efficiently.
Python script for creating alerts on TradingView.com, needs further adjustments.
Input files (see attachment)
Output files (see attachment)
What is not ready:
Cloud computing
Sync Google Drive folders
Master File –Reports (see attachments)
1. Cloud Computing: Set up a cloud of 10 machines (preferably Amazon LightSail) for web scrapping. Install Microsoft Excel, Python, and Selenium.
2. **Web scraping: Extract statistical data from TradingView.com. Excel input files are running with Python, and at the end of the process, we generate one output file.
3. Sync Google Drive: Each virtual machine syncs with one Google Drive folder, where all generated output excel files are stored.
4. Merge Excell files: All output files (from all 10 synced Google Drive folders) merged into a “Master File.”
5. Reports on Excel : Reports are generated after a period of time (7 days) where a loop of web scrapping period finishes.
Every 7 days, we generate approximately 215 output files. Each file contains 11500 lines, which equals a total of 2.5 Million lines.
I understand that creating a report on excel with such a large amount of data is not efficient; I choose this method for the first version of this project for two reasons: 1. I need an interface to cross-check manually if the results of the report are correct, 2. The speed of generating the reports is not important. You can propose any other more efficient methods for this version of later versions of the project.
6. Create alerts of Trading View: Develop further an existing python script that can create alerts of TradingView.com using data from an excel file.
What is ready:
Python script for web scraping on TradingView.com runs smoothly and efficiently.
Python script for creating alerts on TradingView.com, needs further adjustments.
Input files (see attachment)
Output files (see attachment)
What is not ready:
Cloud computing
Sync Google Drive folders
Master File –Reports (see attachments)