Back-End Offline Scraping
Budget: $250 – $750 CAD
**Project Overview:**
The purpose of this mandate is to enhance the existing back-end scraping sub-process within the Master workflow. The targeted sub-process involves the reception and processing of data collected from a product page on any given merchant e-commerce website, sent by a user's browsing session from our front-end app via API call. Currently, the back-end automation script utilizes a Plugin to scrape live data from the merchant's website using the provided URL_key from data collection, employing advanced and generic parsers to create a Woo product with all necessary details.
**Issue Statement:**
The current process encounters occasional disruptions due to some merchant websites detecting the back-end scraping as bot activity, leading to the blocking of the Scraper plugin's IP address. Attempts to mitigate this issue using rotating proxies have proven to be unreliable.
**Solution Approach (About this Mandate):**
To address the identified issue, the proposed solution is to modify the existing sub-process scraping automation behaviour. The key adjustment involves the back-end receiving scrapped data from the front-end app as an HTML component. This modification aims to eliminate the online scraping sub-process performed by the back-end, thereby avoiding detection by merchant websites.
**Specific Modifications:**
- Introduce a new key, "prodpg_HTML," in the API call from the front-end app to the back-end. This key will carry the HTML component of the product page. (Note: this part of the mandate will be performed by our in-house dev team)
- The Scraper plugin will be adapted to utilize the "prodpg_HTML" key to perform offline scraping.
- The parsing algorithm of the Scraper plugin will be applied to the offline scraped data to extract the required details for creating the respective Woo product.
**Project Constraints:**
The enhancement made to the sub-process should not disrupt or adversely affect any other processes within the existing Master workflow.
**Timeline:**
The Freelancer is expected to provide a project timeline with milestones for the implementation of the proposed modifications. Regular updates on progress should be communicated to ensure timely completion.
**Deliverables:**
1. Modified back-end sub-process capable of receiving and processing HTML components for offline scraping.
2. Adapted Scraper plugin incorporating the new scraping approach.
3. Documentation detailing the implemented changes and any relevant instructions for maintenance.
The Freelancer must be a good communicator and a fluent English speaker. He/She will be provided with proper access and will be required to;
1. Update tasks progress and completion under Jira
2. Commit all development codes to BitBucket and plugin deployed to Staging
3. Commit daily work hours and provide prompt daily progress updates
4. Attend meetings on-demand to review progress or any other requirements
**Recommendation:**
1. Use an Object-Oriented approach.
2. Low Coupling - This makes your code more modular, flexible, and testable.
3. High Cohesion - This makes your code more readable, understandable, and maintainable.
4. Use Guzzle instead of cURL - Why use Guzzle instead of cURL
**Testing:**
We will be providing API keys including credentials and several HTML component samples for testing under Postman.
The Freelancer is responsible for thorough testing of the modified sub-process to ensure its effectiveness in addressing the identified issue. Testing should cover various scenarios and potential edge cases.
**Conclusion:**
This mandate aims to improve the reliability and efficiency of the back-end parsing sub-process while ensuring seamless integration with the existing Master workflow. The Freelancer's expertise and attention to detail are crucial for the successful implementation of these modifications.
The purpose of this mandate is to enhance the existing back-end scraping sub-process within the Master workflow. The targeted sub-process involves the reception and processing of data collected from a product page on any given merchant e-commerce website, sent by a user's browsing session from our front-end app via API call. Currently, the back-end automation script utilizes a Plugin to scrape live data from the merchant's website using the provided URL_key from data collection, employing advanced and generic parsers to create a Woo product with all necessary details.
**Issue Statement:**
The current process encounters occasional disruptions due to some merchant websites detecting the back-end scraping as bot activity, leading to the blocking of the Scraper plugin's IP address. Attempts to mitigate this issue using rotating proxies have proven to be unreliable.
**Solution Approach (About this Mandate):**
To address the identified issue, the proposed solution is to modify the existing sub-process scraping automation behaviour. The key adjustment involves the back-end receiving scrapped data from the front-end app as an HTML component. This modification aims to eliminate the online scraping sub-process performed by the back-end, thereby avoiding detection by merchant websites.
**Specific Modifications:**
- Introduce a new key, "prodpg_HTML," in the API call from the front-end app to the back-end. This key will carry the HTML component of the product page. (Note: this part of the mandate will be performed by our in-house dev team)
- The Scraper plugin will be adapted to utilize the "prodpg_HTML" key to perform offline scraping.
- The parsing algorithm of the Scraper plugin will be applied to the offline scraped data to extract the required details for creating the respective Woo product.
**Project Constraints:**
The enhancement made to the sub-process should not disrupt or adversely affect any other processes within the existing Master workflow.
**Timeline:**
The Freelancer is expected to provide a project timeline with milestones for the implementation of the proposed modifications. Regular updates on progress should be communicated to ensure timely completion.
**Deliverables:**
1. Modified back-end sub-process capable of receiving and processing HTML components for offline scraping.
2. Adapted Scraper plugin incorporating the new scraping approach.
3. Documentation detailing the implemented changes and any relevant instructions for maintenance.
The Freelancer must be a good communicator and a fluent English speaker. He/She will be provided with proper access and will be required to;
1. Update tasks progress and completion under Jira
2. Commit all development codes to BitBucket and plugin deployed to Staging
3. Commit daily work hours and provide prompt daily progress updates
4. Attend meetings on-demand to review progress or any other requirements
**Recommendation:**
1. Use an Object-Oriented approach.
2. Low Coupling - This makes your code more modular, flexible, and testable.
3. High Cohesion - This makes your code more readable, understandable, and maintainable.
4. Use Guzzle instead of cURL - Why use Guzzle instead of cURL
**Testing:**
We will be providing API keys including credentials and several HTML component samples for testing under Postman.
The Freelancer is responsible for thorough testing of the modified sub-process to ensure its effectiveness in addressing the identified issue. Testing should cover various scenarios and potential edge cases.
**Conclusion:**
This mandate aims to improve the reliability and efficiency of the back-end parsing sub-process while ensuring seamless integration with the existing Master workflow. The Freelancer's expertise and attention to detail are crucial for the successful implementation of these modifications.