Scrape 10K-Post Blog Dataset
Budget: €30 – €250 EUR
I need a dependable scraper that can crawl an online blog of roughly ten thousand posts and pull down every entry, complete with any comments attached to each post. The final dataset must be delivered in a clean, well-structured XML file because that is my preferred working format, but feel free to include additional JSON or raw HTML copies if they come out of your workflow naturally—extra formats are a bonus, not a requirement.
Scope of work
• Crawl every live post on the site, following all pagination and in-site links that surface original articles.
• Capture full article content and pair the corresponding comments so they stay linked to the right post.
• Preserve each post’s core details (title, body, URL slug, and whatever standard metadata your tool picks up) inside the XML structure.
• Package the finished XML so it can be imported in a single run without manual tweaking.
Deliverables
1. The complete XML dataset of ~10 k posts, each nested with its comments.
2. The scraping script or notebook, documented well enough that I can rerun it later if the blog updates.
3. A short run-through of the steps taken and any dependencies required.
I’m happy with mainstream libraries such as Python-requests, BeautifulSoup, Scrapy or similar—use what lets you move fastest while keeping the output tidy. Let me know your estimated turnaround and any questions you have about the target domain, and we can get moving right away.
Scope of work
• Crawl every live post on the site, following all pagination and in-site links that surface original articles.
• Capture full article content and pair the corresponding comments so they stay linked to the right post.
• Preserve each post’s core details (title, body, URL slug, and whatever standard metadata your tool picks up) inside the XML structure.
• Package the finished XML so it can be imported in a single run without manual tweaking.
Deliverables
1. The complete XML dataset of ~10 k posts, each nested with its comments.
2. The scraping script or notebook, documented well enough that I can rerun it later if the blog updates.
3. A short run-through of the steps taken and any dependencies required.
I’m happy with mainstream libraries such as Python-requests, BeautifulSoup, Scrapy or similar—use what lets you move fastest while keeping the output tidy. Let me know your estimated turnaround and any questions you have about the target domain, and we can get moving right away.