Web Scraping Expert Needed
Budget: $2 – $8 CAD
I need help to assemble a dataset through web crawling and scraping a public conference archive. In total, about 650 web pages need to be accessed and processed from text into structured data.
The final deliverables will be:
- A CSV dataset containing key information displayed on the web pages (the participant list, their affiliations, and their roles)
- The code that generated the data
I need help to assemble a dataset through web crawling and scraping a public conference archive. In total, about 650 web pages need to be accessed and processed from text into structured data.
To start, you will need to open the following 13 pages:
https://www.nber.org/conferences/summer-institute-2012
https://www.nber.org/conferences/summer-institute-2013
https://www.nber.org/conferences/summer-institute-2014
https://www.nber.org/conferences/summer-institute-2015
https://www.nber.org/conferences/summer-institute-2016
https://www.nber.org/conferences/summer-institute-2017
https://www.nber.org/conferences/summer-institute-2018
https://www.nber.org/conferences/summer-institute-2019
https://www.nber.org/conferences/summer-institute-2020
https://www.nber.org/conferences/summer-institute-2021
https://www.nber.org/conferences/summer-institute-2022
https://www.nber.org/conferences/summer-institute-2023
https://www.nber.org/conferences/summer-institute-2024
Each of those 13 pages contains the high-level outline for a conference in a given year. Each conference includes about 50 topics (e.g. Real Estate or Labor Economics), so the total task will involve collecting data from about 13*50 = 650 web pages. If you click the link to a particular topic’s schedule, you can see the full program. For example, the 2024 Real Estate meeting agenda is here: https://www.nber.org/conferences/si-2024-real-estate (and the link is accessible from the 2024 master schedule).
On the web page for each topic, you can see four groups of people:
1. The organizers (at the top)
2. The authors of particular papers
3. The discussants particular papers
4. At the bottom of the page is a list of participants
I would you to assemble a CSV dataset that tells me in each of the 13 years and in each topic meeting: who were the organizers, authors, discussants, and participants. Please include their names and affiliations.
The dataset should have the following fields:
- Year
- Topic
- Person
- Affiliation
- Role (Organizer, Author, Discussant, or Participant)
An example of some rows of the finished data set would be:
2024, Science of Science Funding, Alex Bell, GA State University, Discussant
2024, Real Estate, Alex Bell, GA State University, Author
…
In addition to the final assembled dataset, I would also like you to supply me with the code so that I can re-run it later if necessary (e.g., to extend it to future years). Python is preferred although I am open to other languages if that makes the task simpler for you.
The final deliverables will be:
- A CSV dataset containing key information displayed on the web pages (the participant list, their affiliations, and their roles)
- The code that generated the data
I need help to assemble a dataset through web crawling and scraping a public conference archive. In total, about 650 web pages need to be accessed and processed from text into structured data.
To start, you will need to open the following 13 pages:
https://www.nber.org/conferences/summer-institute-2012
https://www.nber.org/conferences/summer-institute-2013
https://www.nber.org/conferences/summer-institute-2014
https://www.nber.org/conferences/summer-institute-2015
https://www.nber.org/conferences/summer-institute-2016
https://www.nber.org/conferences/summer-institute-2017
https://www.nber.org/conferences/summer-institute-2018
https://www.nber.org/conferences/summer-institute-2019
https://www.nber.org/conferences/summer-institute-2020
https://www.nber.org/conferences/summer-institute-2021
https://www.nber.org/conferences/summer-institute-2022
https://www.nber.org/conferences/summer-institute-2023
https://www.nber.org/conferences/summer-institute-2024
Each of those 13 pages contains the high-level outline for a conference in a given year. Each conference includes about 50 topics (e.g. Real Estate or Labor Economics), so the total task will involve collecting data from about 13*50 = 650 web pages. If you click the link to a particular topic’s schedule, you can see the full program. For example, the 2024 Real Estate meeting agenda is here: https://www.nber.org/conferences/si-2024-real-estate (and the link is accessible from the 2024 master schedule).
On the web page for each topic, you can see four groups of people:
1. The organizers (at the top)
2. The authors of particular papers
3. The discussants particular papers
4. At the bottom of the page is a list of participants
I would you to assemble a CSV dataset that tells me in each of the 13 years and in each topic meeting: who were the organizers, authors, discussants, and participants. Please include their names and affiliations.
The dataset should have the following fields:
- Year
- Topic
- Person
- Affiliation
- Role (Organizer, Author, Discussant, or Participant)
An example of some rows of the finished data set would be:
2024, Science of Science Funding, Alex Bell, GA State University, Discussant
2024, Real Estate, Alex Bell, GA State University, Author
…
In addition to the final assembled dataset, I would also like you to supply me with the code so that I can re-run it later if necessary (e.g., to extend it to future years). Python is preferred although I am open to other languages if that makes the task simpler for you.