Apify Http Scraper

Job ID: 37120293

Budget: $250 – $750 USD

I need a simple PlaywrightCrawler for apify that can crawl public websites and return the scraped content in JSON format.

tl;dr:
- Scrape pdf files
- Report progress to a webhook at intervals
- Post the content in batches to a webhook

Specifications
===========
It takes 5 input parameters:
- starturl [required]
- contentidentifier (default: "", type:string)
- maxpages (default: 10, zero = all pages, type:string)
- issitemap (default: false, type:string)
- batchsize (default: 25, type:integer)
- token (type:string) [required]

crawling behavior
=============
- if the "issitemap" param is set to "true" only scrape the links on the sitemap. Otherwise, follow all links that point to the same domain as the starturl.
- Respect robot.txt
- If maxpages has been set to 0 we must crawl all pages. Otherwise only scrape the number of pages set by the maxpages parameter.
- I need the content of the element specified by the "contentidentifyer" parameter. For example, if "ContentArea" is specified get the text inside the div/span that has that id or class
Suggested value format:
"tagname.classname" e.g. "div.contentAreaClassName"
"tagname.id" e.g. "div.contentAreaId"
- Can you clean up the HTML and strip all HTML? When i created my own crawler the output was full of "\r\n"

Output format
============
I need the output as a JSON:
[{
"url": "<fully qualified url: https://example.com/folder/page1.html>",
"content": $scrapedContent1
},{
"url": "<https://example.com/folder/page1.html>",
"content": $scrapedContent2
}]

Webhooks
============
- The "token" input parameter contains a token you must add to the query string as part of the request from Apify to my webhooks on Azure Function.
for example: https://mywebhook.com?token={$token}

Webhook: content
=================
A webhook needs to be called when a page has been scrapped. Instead of calling the webhook every time a page has been scraped, the content must be sent in batches. The size of the batch is set by the "batchsize" input parameter.

Webhook: Progress report
====================
I need progress reports containing statistics about the progress of the crawling process to be sent to a webhook.

This is my first scraper and am I not entirely sure about my options in this regard, but on the wish list of info I would like to receive is:
- Number of pages indexed / Total number of pages found plus current cost
- Event: CRAWLER_STARTED
- Event: CRAWLER_RUN_SUCCEEDED
- Event: CRAWLER_RUN_FAILED
- Event: CRAWLER_RUN_TIMED_OUT
- Event: CRAWLER_RUN_ABORTED

Please note, that this is my first experience with running an actor and with apify and will be happy for any suggestions you might have.
Related categories: JavaScript Web Scraping