Password-Protected Site Spider and Scrape
Budget: $30 – $250 CAD
I need a complete offline backup of a password-protected website. Think of a professor rating website. I already have valid credentials; what I do not have is the time to automate the collection and organisation of roughly 8,000 HTML pages that contain mainly text and embedded tables (for example rating professors based on 10 criteria)--all following 3-4 templates.
Your job is to log in programmatically and:
• Crawl every reachable page, saving the raw HTML plus all linked assets (CSS, JS, images, etc.) into a neatly structured local folder so the site can be browsed offline. (All links should be made relative so no absolute links refer back to original site)
• Additionally, render each of those pages to an individual PDF (wkhtmltopdf, pdfkit, or similar are fine as long as the final output mirrors the on-screen layout).
• Lastly, parse the text content and every HTML table, then load both into 2-3 Excel worksheets. Use pandas, openpyxl or comparable libraries so the result is immediately usable and not just a dump.
Accuracy matters: the Excel file must preserve table structure, text encoding, and any special characters exactly as displayed on the site.
Let me know how you plan to handle session management along with an estimated turnaround time.
Your job is to log in programmatically and:
• Crawl every reachable page, saving the raw HTML plus all linked assets (CSS, JS, images, etc.) into a neatly structured local folder so the site can be browsed offline. (All links should be made relative so no absolute links refer back to original site)
• Additionally, render each of those pages to an individual PDF (wkhtmltopdf, pdfkit, or similar are fine as long as the final output mirrors the on-screen layout).
• Lastly, parse the text content and every HTML table, then load both into 2-3 Excel worksheets. Use pandas, openpyxl or comparable libraries so the result is immediately usable and not just a dump.
Accuracy matters: the Excel file must preserve table structure, text encoding, and any special characters exactly as displayed on the site.
Let me know how you plan to handle session management along with an estimated turnaround time.
Related categories:
Python
Data Processing
Excel
Web Scraping
HTML
Web Development
Web Crawling
Data Extraction
Data Analysis
Automation