Multi-Website Data Scraping
Budget: $30 – $250 USD
Hi!
We're looking for a robust crawler that scrapes a specific list of international patent databases written in Python 3.
List of patent databases we need scraped:
1. Europe: https://www.epo.org/en/searching-for-patents/technical/espacenet
2. Japan: https://www.j-platpat.inpit.go.jp/
3. PCT: https://www.wipo.int/patentscope/en/
4. China: https://english.cnipa.gov.cn/
5. Korea: https://www.kipo.go.kr/en/MainApp?c=1000
We're looking to break this project up into milestones, each of which correspond to one of the data sources listed above.
Technical requirements:
- Written in Python 3.12
- Deployable in the cloud on GCP Cloud Run (although you don't have to do the deployment - we will do that ourselves). The scrapers must be horizontally scalable and leverage only selenium in headless mode to ensure they are not incompatible with deployment on Cloud Run.
- The scrapers must save structured data in MongoDB Atlas, using the mongoengine ORM to interface with MongoDB. We have a specific mongoengine model that we will share that we need data stored in. Adding fields to that model as necessary to capture new fields not yet captured is expected.
- Every single field available must be scraped from every patent data source. A meaningful part of this project is investigating the data sources to identify all the data fields available.
- Scraping logic must be robust to variances within the target websites' HTML page structures without using generic try/except statements.
- The scraper must check for a potentially equivalent document already in MongoDB. Using a few different fields (not just a single unique id field), will be necessary to ensure robustness of idempotence for cases where one unique id might be missing from a record.
- We need the fields that are made available by each data source saved on a data-source specific dataclass Python model (i.e. WipoPatent). The scraper class would create instances of the dataclass, and then have a utility for writing the data contained by the dataclass to MongoDB using the generalized mongoengine MongoPatent model we have (as mentioned above).
This is Urgent situation,Please only apply for this project if you are Web scrapping expert
To avoid the scam, Please start with word "scrapping"!
We're looking for a robust crawler that scrapes a specific list of international patent databases written in Python 3.
List of patent databases we need scraped:
1. Europe: https://www.epo.org/en/searching-for-patents/technical/espacenet
2. Japan: https://www.j-platpat.inpit.go.jp/
3. PCT: https://www.wipo.int/patentscope/en/
4. China: https://english.cnipa.gov.cn/
5. Korea: https://www.kipo.go.kr/en/MainApp?c=1000
We're looking to break this project up into milestones, each of which correspond to one of the data sources listed above.
Technical requirements:
- Written in Python 3.12
- Deployable in the cloud on GCP Cloud Run (although you don't have to do the deployment - we will do that ourselves). The scrapers must be horizontally scalable and leverage only selenium in headless mode to ensure they are not incompatible with deployment on Cloud Run.
- The scrapers must save structured data in MongoDB Atlas, using the mongoengine ORM to interface with MongoDB. We have a specific mongoengine model that we will share that we need data stored in. Adding fields to that model as necessary to capture new fields not yet captured is expected.
- Every single field available must be scraped from every patent data source. A meaningful part of this project is investigating the data sources to identify all the data fields available.
- Scraping logic must be robust to variances within the target websites' HTML page structures without using generic try/except statements.
- The scraper must check for a potentially equivalent document already in MongoDB. Using a few different fields (not just a single unique id field), will be necessary to ensure robustness of idempotence for cases where one unique id might be missing from a record.
- We need the fields that are made available by each data source saved on a data-source specific dataclass Python model (i.e. WipoPatent). The scraper class would create instances of the dataclass, and then have a utility for writing the data contained by the dataclass to MongoDB using the generalized mongoengine MongoPatent model we have (as mentioned above).
This is Urgent situation,Please only apply for this project if you are Web scrapping expert
To avoid the scam, Please start with word "scrapping"!