for Webcrawler "Broad webcrawler" Got 600k link seeds .dk domain, need to find other 900k .dk domain existing
Budget: $30 – $250 USD
for "broad webcrawler" needed.
I expect you got experience with an existing webcrawler framework.
Intro:
In Denmark there is something like 1.5mil .dk domains.
With the 600k .dk domain that I already got, I expect to be able to find some of the 900k domains missing.
I welcome suggestions on how to solve better.
Deliverables
A webcrawler that does can do the following:
Based on the 600k provided, crawl the internet for DK domains missing, and that is not part of the 600k seeds urls provided.
Only follow .DK domains
I should be able to start, continue and stop the webcrawler if needed.
I should be able to add more seeds, via CSV list or alike that needs to be crawled (with stop, start, continue crawl)
Should run in multiple threads for speed
Be able to keep running meaning no leaks of memory
Already crawled url's should not be crawled again (something like via sqlite or alike )
Help with implementation on my server, so it works.
I expect you got experience with an existing webcrawler framework.
Intro:
In Denmark there is something like 1.5mil .dk domains.
With the 600k .dk domain that I already got, I expect to be able to find some of the 900k domains missing.
I welcome suggestions on how to solve better.
Deliverables
A webcrawler that does can do the following:
Based on the 600k provided, crawl the internet for DK domains missing, and that is not part of the 600k seeds urls provided.
Only follow .DK domains
I should be able to start, continue and stop the webcrawler if needed.
I should be able to add more seeds, via CSV list or alike that needs to be crawled (with stop, start, continue crawl)
Should run in multiple threads for speed
Be able to keep running meaning no leaks of memory
Already crawled url's should not be crawled again (something like via sqlite or alike )
Help with implementation on my server, so it works.