Need to Scrap google search data for the Keyword
Budget: $30 – $60 USD
1) Need to scrap google search data for the keywords (80000) using a high performance multi-threaded asynchronous scrapers (300 to 500 threads) using proxy server that can handle IP rotation (need your suggestion which server we can use).
Keywords are stored in a table, Each thread will pick 1 keyword and mark status as processed (1) and insert the start time and scrap 50 (configurable) sites from google position 20 to 70 (from –to position are configurable) and scrap following data
Title , Description , URl , Domain Name
Insert all data in Master table and also when all 50 sites are scraped update finish time in Keyword table and threads pick up next keyword available where status is not equal to 1
Data fields of Keyword Table (MySql)
Domain Name , Subdomain name , Keywords , Category 1, Category 2, Category 3, Category 4, Sites scraped , Status , Start time , Finish time.
Data Field of Master Data (MySql)
Domain Name , Subdomain name , Keywords , Category 1, Category 2, Category 3, Category 4, Scraped Title , Scraped Description , Scraped Url, Scraped Domain name, Account Id, User ID ,User Name , Expiry Date.
Domain Name , Subdomain name , Keywords , Category 1, Category 2, Category 3, Category 4 you will get from Keyword table.
Scraped Title , Scraped Description , Scraped Url , Scraped Domain name you will get using your script
Update account Id as 1, User Id as 9810822422, User Name as TTPL Update Expiry Date as 01-01-2050 for all keywords.
This is a one time activity to fill Master table for our research.
2) Need just a function (PHP) to get 50 urls (Google) for a keyword starting from position 20 in an array . (no scraping , nothing).
3) Need just a function (PHP) to get the google ranking for a keyword for a website.
Keywords are stored in a table, Each thread will pick 1 keyword and mark status as processed (1) and insert the start time and scrap 50 (configurable) sites from google position 20 to 70 (from –to position are configurable) and scrap following data
Title , Description , URl , Domain Name
Insert all data in Master table and also when all 50 sites are scraped update finish time in Keyword table and threads pick up next keyword available where status is not equal to 1
Data fields of Keyword Table (MySql)
Domain Name , Subdomain name , Keywords , Category 1, Category 2, Category 3, Category 4, Sites scraped , Status , Start time , Finish time.
Data Field of Master Data (MySql)
Domain Name , Subdomain name , Keywords , Category 1, Category 2, Category 3, Category 4, Scraped Title , Scraped Description , Scraped Url, Scraped Domain name, Account Id, User ID ,User Name , Expiry Date.
Domain Name , Subdomain name , Keywords , Category 1, Category 2, Category 3, Category 4 you will get from Keyword table.
Scraped Title , Scraped Description , Scraped Url , Scraped Domain name you will get using your script
Update account Id as 1, User Id as 9810822422, User Name as TTPL Update Expiry Date as 01-01-2050 for all keywords.
This is a one time activity to fill Master table for our research.
2) Need just a function (PHP) to get 50 urls (Google) for a keyword starting from position 20 in an array . (no scraping , nothing).
3) Need just a function (PHP) to get the google ranking for a keyword for a website.