Scrape 299 pages to spreadsheet
Budget: €30 – €250 EUR
I have 299 simple webpages that all share the same underlying layout. What I need is the content pulled from each page — primarily the bodycontent which is mainly headers and paragraphs, plus a handful of smaller fields that sit in the same section of every page.
You can view the pages here:
https://www.uva.nl/medewerkers-cursussen/medewerkers-cursussen.html (dutch list)
https://www.uva.nl/en/employee-courses/employee-courses.html (english list)
For each page I need the contents of these fields to be scarped and exported to a spreadsheet:
• Title <h1 class="pageheader__title">
• <meta name="language">
• <meta property="og:image">
• Lead <div class="c-lead lead">
• The contents of this div <div class="c-detailsummary__title visually-hidden"> in four separate fields these fields are named differently in the dutch and english pages
• <dt>Thema's</dt> or <dt>Themes</dt>
• <dt>Faculteit</dt> or <dt>Faculty</dt>
• <dt>Doelgroep</dt> or <dt>Target group</dt>
• <dt>Werkvorm</dt> or <dt>Type</dt>
• All other visible bodycontent with markup and hrefs
• All other visible bodycontent withOUT markup
I expect the output to look like the Excel file I added as an example.
I am the owner of the site but I currently have nobody that can do the dataexport from the database so we'll just scape it from the frontend.
You’re free to script this in Python (BeautifulSoup, Scrapy, Selenium, etc.) or any other tooling you’re comfortable with, as long as the final deliverable is a UTF-8 CSV or Excel file that I can open directly. I’ll consider the work complete once a manual spot-check of 10 random pages shows the extracted text and HTML align perfectly with what appears on the live site.
You can view the pages here:
https://www.uva.nl/medewerkers-cursussen/medewerkers-cursussen.html (dutch list)
https://www.uva.nl/en/employee-courses/employee-courses.html (english list)
For each page I need the contents of these fields to be scarped and exported to a spreadsheet:
• Title <h1 class="pageheader__title">
• <meta name="language">
• <meta property="og:image">
• Lead <div class="c-lead lead">
• The contents of this div <div class="c-detailsummary__title visually-hidden"> in four separate fields these fields are named differently in the dutch and english pages
• <dt>Thema's</dt> or <dt>Themes</dt>
• <dt>Faculteit</dt> or <dt>Faculty</dt>
• <dt>Doelgroep</dt> or <dt>Target group</dt>
• <dt>Werkvorm</dt> or <dt>Type</dt>
• All other visible bodycontent with markup and hrefs
• All other visible bodycontent withOUT markup
I expect the output to look like the Excel file I added as an example.
I am the owner of the site but I currently have nobody that can do the dataexport from the database so we'll just scape it from the frontend.
You’re free to script this in Python (BeautifulSoup, Scrapy, Selenium, etc.) or any other tooling you’re comfortable with, as long as the final deliverable is a UTF-8 CSV or Excel file that I can open directly. I’ll consider the work complete once a manual spot-check of 10 random pages shows the extracted text and HTML align perfectly with what appears on the live site.
Related categories:
PHP
Python
Data Processing
Excel
Web Scraping
HTML
Web Development
Scrapy
Data Extraction
BeautifulSoup