Write Python email scraping script
Budget: $30 – $250 USD
I'm looking for an experienced data scraping / automation expert to write a Python script that finds a specific type of email address on a given URL. I want the script to specifically find the email address for a website / company's privacy team. You should implement Brightdata proxy to ensure the page(s) load correctly.
Proposed functionality of script:
1. New request is created by submitting URL / domain to the script (should be called from Python function)
2. Script uses Google Search API (https://rapidapi.com/apigeek/api/google-search3/) to find all relevant pages on the given domain. For example: "site:domain.com+privacy+policy" will help us to find the pages on the input URL/domain that are related to the privacy policy of the website.
3. For all links returned by RapidAPI, load URL and try to extract email address
4. If no email address is found on these topic-specific URLs, attempt to find general contact page using Google Search API. For example: "site:domain.com+contact+email"
5. Compile output of results and where multiple emails are found, prioritise order based on appearance of keywords in email address (e.g privacy@ / dataprotection@ ranks more highly than contact@ / support@)
5. Return a JSON response to the initial function, with list of email addresses and the URL they were found on
It's important the script can detect modified email address formats, for example:
email (at) domain.com
emailATdomain.com
email at domain.com
This Python library looks like it could be useful for this task:
https://github.com/Hanel32/NLP_Regex_Email_and_Phone_extraction/blob/master/SpamLord.py
Proposed functionality of script:
1. New request is created by submitting URL / domain to the script (should be called from Python function)
2. Script uses Google Search API (https://rapidapi.com/apigeek/api/google-search3/) to find all relevant pages on the given domain. For example: "site:domain.com+privacy+policy" will help us to find the pages on the input URL/domain that are related to the privacy policy of the website.
3. For all links returned by RapidAPI, load URL and try to extract email address
4. If no email address is found on these topic-specific URLs, attempt to find general contact page using Google Search API. For example: "site:domain.com+contact+email"
5. Compile output of results and where multiple emails are found, prioritise order based on appearance of keywords in email address (e.g privacy@ / dataprotection@ ranks more highly than contact@ / support@)
5. Return a JSON response to the initial function, with list of email addresses and the URL they were found on
It's important the script can detect modified email address formats, for example:
email (at) domain.com
emailATdomain.com
email at domain.com
This Python library looks like it could be useful for this task:
https://github.com/Hanel32/NLP_Regex_Email_and_Phone_extraction/blob/master/SpamLord.py