Basic web scraper for apify.com
Budget: $250 – $750 USD
If you have previous experience with developing a script for a web scraper on Apify.com I would like to hear from you. Please state in your reply if you have worked with Apify.com before. I will only consider offer from developer that has previous experience developing against the Apify api and platform.
I need a basic web scraper that can scrape all pages on a WordPress website. It needs to find an element with a specific class or id and get the text content from that element. If that element is not found the grab the content of the body
As it will be used as part of an automated process it must entirely be operated using the CLI.
Additional requirements:
- Scrape the text content of pdf files.
- Report progress to a webhook at intervals
- Post the content in batches to a webhook
- Must be written in javascript or typescript
Specifications
===========
It takes 5 input parameters:
- starturl [required]
- contentidentifier (default: “”, type:string)
- maxpages (default: 10, zero = all pages, type:string)
- issitemap (default: false, type:string)
- batchsize (default: 25, type:integer)
crawling behavior
=============
-if the “issitemap” param is set to "true" only scrape the links on the sitemap. Otherwise, follow all links that point to the same domain as the starturl.
-Respect robot.txt
- If maxpages has been set to 0 we must crawl all pages. Otherwise only scrape the number of pages set by the maxpages parameter.
- I need the content of the element specified by the "contentidentifyer” parameter. For example, if
"ContentArea" is specified get the text inside the div/span that has that id or class
Suggested value format:
“tagname.classname” e.g. “div.contentAreaClassName”
“tagname.id” e.g. “div.contentAreaId”
- Clean up the output and strip all HTML
Output format
============
I need the output as a JSON:
[{
“url”: “<fully qualified url1 : https://example.com/folder/page1.html>”,
“content”: $scrapedContent1
},{
“url”: “<https://example.com/folder/page1.html>”,
“content”: $scrapedContent2
}]
Webhook: content
=================
A webhook needs to be called when a page has been scrapped. Instead of calling the webhook every time a page has been scraped, the content must be sent in batches. The size of the batch is set by the “batchsize” input parameter.
Development of these webhooks are not part of this task.
Webhook: Progress report
====================
Progress reports containing statistics about the progress of the crawling process to be sent to a webhook.
This is my first scraper and am I not entirely sure about my options in this regard, but on the wish list of info I would like to receive is:
- Number of pages indexed / Total number of pages found.
- Event: CRAWLER_RUN_STARTED
- Event: CRAWLER_RUN_SUCCEEDED
- Event: CRAWLER_RUN_FAILED
- Event: CRAWLER_RUN_TIMED_OUT
- Event: CRAWLER_RUN_ABORTED
- The total cost of the task when the crawl is over.
Please note, that this is my first experience with running an actor and with Apify, and will be happy for any suggestions you might have.
Best regards
Tony
I need a basic web scraper that can scrape all pages on a WordPress website. It needs to find an element with a specific class or id and get the text content from that element. If that element is not found the grab the content of the body
As it will be used as part of an automated process it must entirely be operated using the CLI.
Additional requirements:
- Scrape the text content of pdf files.
- Report progress to a webhook at intervals
- Post the content in batches to a webhook
- Must be written in javascript or typescript
Specifications
===========
It takes 5 input parameters:
- starturl [required]
- contentidentifier (default: “”, type:string)
- maxpages (default: 10, zero = all pages, type:string)
- issitemap (default: false, type:string)
- batchsize (default: 25, type:integer)
crawling behavior
=============
-if the “issitemap” param is set to "true" only scrape the links on the sitemap. Otherwise, follow all links that point to the same domain as the starturl.
-Respect robot.txt
- If maxpages has been set to 0 we must crawl all pages. Otherwise only scrape the number of pages set by the maxpages parameter.
- I need the content of the element specified by the "contentidentifyer” parameter. For example, if
"ContentArea" is specified get the text inside the div/span that has that id or class
Suggested value format:
“tagname.classname” e.g. “div.contentAreaClassName”
“tagname.id” e.g. “div.contentAreaId”
- Clean up the output and strip all HTML
Output format
============
I need the output as a JSON:
[{
“url”: “<fully qualified url1 : https://example.com/folder/page1.html>”,
“content”: $scrapedContent1
},{
“url”: “<https://example.com/folder/page1.html>”,
“content”: $scrapedContent2
}]
Webhook: content
=================
A webhook needs to be called when a page has been scrapped. Instead of calling the webhook every time a page has been scraped, the content must be sent in batches. The size of the batch is set by the “batchsize” input parameter.
Development of these webhooks are not part of this task.
Webhook: Progress report
====================
Progress reports containing statistics about the progress of the crawling process to be sent to a webhook.
This is my first scraper and am I not entirely sure about my options in this regard, but on the wish list of info I would like to receive is:
- Number of pages indexed / Total number of pages found.
- Event: CRAWLER_RUN_STARTED
- Event: CRAWLER_RUN_SUCCEEDED
- Event: CRAWLER_RUN_FAILED
- Event: CRAWLER_RUN_TIMED_OUT
- Event: CRAWLER_RUN_ABORTED
- The total cost of the task when the crawl is over.
Please note, that this is my first experience with running an actor and with Apify, and will be happy for any suggestions you might have.
Best regards
Tony