Python Expert
Budget: $30 – $250 USD
This is project for scraping video metadata from Bilibili and processing the data into a MySQL database.
My project Workflow 1.Database Initialization (init_database.py) Establishes a connection to the database and creates the video_stats table if it doesn't already exist. The video_stats table is designed to store various video metrics like views, likes, comments, and rankings. It has a foreign key relationship with the bv_numbers table, ensuring data integrity.
2.BV Number Collection (get_bv_data.py) Scrapes the unique BV numbers (video IDs) from the Bilibili website using Selenium. Handles pagination and avoids duplicates by cross-referencing existing data in the bv_numbers table.
3.Video Data Fetching (get_bv_video_data.py) Retrieves detailed video metrics for the collected BV numbers. Implements dynamic IP rotation to bypass request limits or bans. Inserts the fetched data into the video_stats table in batches for efficiency.
4.Main Orchestration (main.py) Coordinates the entire workflow using Python’s multiprocessing module. Runs the BV number scraper and video data fetcher concurrently. Monitors process health and restarts processes if they terminate unexpectedly. Enforces a maximum runtime for the script to prevent it from running indefinitely.
5.Database Cleaning (clean_database.py) Resets the database by truncating the video_stats and bv_numbers tables, clearing all existing data
but there are 2 issues when i try crawl the data by my own code.
the first is errors during API calls or scraping.
the IP of my crawler will be blocked, and I will no longer be allowed to access the website to obtain data. the second is As I get more and more videos to crawl, the more time it takes to crawl each time, and the interval between the data tracked at each time becomes larger, and I can't guarantee that it will be around 5 minutes.
My project Workflow 1.Database Initialization (init_database.py) Establishes a connection to the database and creates the video_stats table if it doesn't already exist. The video_stats table is designed to store various video metrics like views, likes, comments, and rankings. It has a foreign key relationship with the bv_numbers table, ensuring data integrity.
2.BV Number Collection (get_bv_data.py) Scrapes the unique BV numbers (video IDs) from the Bilibili website using Selenium. Handles pagination and avoids duplicates by cross-referencing existing data in the bv_numbers table.
3.Video Data Fetching (get_bv_video_data.py) Retrieves detailed video metrics for the collected BV numbers. Implements dynamic IP rotation to bypass request limits or bans. Inserts the fetched data into the video_stats table in batches for efficiency.
4.Main Orchestration (main.py) Coordinates the entire workflow using Python’s multiprocessing module. Runs the BV number scraper and video data fetcher concurrently. Monitors process health and restarts processes if they terminate unexpectedly. Enforces a maximum runtime for the script to prevent it from running indefinitely.
5.Database Cleaning (clean_database.py) Resets the database by truncating the video_stats and bv_numbers tables, clearing all existing data
but there are 2 issues when i try crawl the data by my own code.
the first is errors during API calls or scraping.
the IP of my crawler will be blocked, and I will no longer be allowed to access the website to obtain data. the second is As I get more and more videos to crawl, the more time it takes to crawl each time, and the interval between the data tracked at each time becomes larger, and I can't guarantee that it will be around 5 minutes.