docker for webarchive
Budget: $10 – $90 USD
We look for system admin with very good experiences on configuring services on top of docker.
Your job will be to
- update instances to latest version: https://github.com/jmvezic/keres in docker compose
- to provide a video explaining how
....- to add new websites for curating
....- how to do the review process (manually and automated)
....- to add use the search and the navigation options as end users
....- to add use the review process (automated and manually)
Goal is to have later offline browsable scraped content of named websites (URLs)
Additional requirements:
It shall be used deduplication to reduce required storage (after the first run, the scraper shall check for changes on the page and persist the changed pages in new folders)
https://github.com/internetarchive/heritrix3/wiki/Deduping-%28Duplication-Reduction%29
https://pywb.readthedocs.io/en/latest/manual/recorder.html#deduplication-filters
Milestones
MS1:
update docker compose to use latest stable versions
running locally on your docker environment (and ours)
MS2:
record video how to add new websites and how to curate them
MS3:
implement deduplication
MS4 (optional):
provide REST api call from java springboot to add new domains to the crawler + UI
MS5 (optional):
implement screenshoting of the visited pages
Your background is:
- multiple years of experience with docker/docker compose
- multiple years of experience with web scraping
If you are a good fit, you are open to get more tasks about implementing solutions fully on your own (e.g. with your team)
Budget?
will not be disclosed, place your best bid to get considered
What is next?
We will share you a NDA and afterwards a paid test task.
Payment?
- you estimate in a WBS (optimistic, expected, pessimistic, where optimistic < expected < pessimistic) after getting the task
- we discuss about clearances and effort
- we mutually agree to effort
- we assign you the task after mutually agreed
- you implement & delivery
- we pay
(basically the rules of freelancer)
Closed book vs open book?
We work only on open book.
Closed book means you are unwilling to define a WBS for the work and you add only a price tag to the task.
We are sorry we will not hire you in such a case!
Deliveries?
- in our on premise git (access will be granted to you)
- full sources
- maven
- libs, need prior confirm and we prefer to use mostly latest stable versions
- JDK 17 (mostly LTS)
- CI/CD works fine with checkstyle, pmd, spotbugs
- works on our side too (stage + production)
Your job will be to
- update instances to latest version: https://github.com/jmvezic/keres in docker compose
- to provide a video explaining how
....- to add new websites for curating
....- how to do the review process (manually and automated)
....- to add use the search and the navigation options as end users
....- to add use the review process (automated and manually)
Goal is to have later offline browsable scraped content of named websites (URLs)
Additional requirements:
It shall be used deduplication to reduce required storage (after the first run, the scraper shall check for changes on the page and persist the changed pages in new folders)
https://github.com/internetarchive/heritrix3/wiki/Deduping-%28Duplication-Reduction%29
https://pywb.readthedocs.io/en/latest/manual/recorder.html#deduplication-filters
Milestones
MS1:
update docker compose to use latest stable versions
running locally on your docker environment (and ours)
MS2:
record video how to add new websites and how to curate them
MS3:
implement deduplication
MS4 (optional):
provide REST api call from java springboot to add new domains to the crawler + UI
MS5 (optional):
implement screenshoting of the visited pages
Your background is:
- multiple years of experience with docker/docker compose
- multiple years of experience with web scraping
If you are a good fit, you are open to get more tasks about implementing solutions fully on your own (e.g. with your team)
Budget?
will not be disclosed, place your best bid to get considered
What is next?
We will share you a NDA and afterwards a paid test task.
Payment?
- you estimate in a WBS (optimistic, expected, pessimistic, where optimistic < expected < pessimistic) after getting the task
- we discuss about clearances and effort
- we mutually agree to effort
- we assign you the task after mutually agreed
- you implement & delivery
- we pay
(basically the rules of freelancer)
Closed book vs open book?
We work only on open book.
Closed book means you are unwilling to define a WBS for the work and you add only a price tag to the task.
We are sorry we will not hire you in such a case!
Deliveries?
- in our on premise git (access will be granted to you)
- full sources
- maven
- libs, need prior confirm and we prefer to use mostly latest stable versions
- JDK 17 (mostly LTS)
- CI/CD works fine with checkstyle, pmd, spotbugs
- works on our side too (stage + production)