n8n Workflow: Web Scraping, Airtable, and Shopify Sync
Budget: €2 – €8 EUR
Project Description:
I am looking for a freelancer to develop an automated workflow using n8n.com to scrape data from a WordPress-based website, store it in Airtable, and sync it with a Shopify store via its API.
---
Main Tasks:
1. Scraping Product Data:
- Visit the target website and extract the list of available brands from the sidebar menu (approximately 20 brands).
- Retrieve the URLs of the product showcase pages for each brand.
- Extract the URLs of all products listed within each brand's catalog (currently 240 product URLs in total).
- Visit each product's page and extract the following details:
- Product Image
- Description
- Technical Specifications
- Price
- Stock
availability
2. Data Storage and Processing:
- Save the extracted data, including images, in an Airtable database.
- Create new entries for products not already in the database and update existing ones.
- Maintain a change log table to track fluctuations in product price and availability over time.
3. Resilience and Error Handling:
- Ensure the workflow can resume from the exact point where it was interrupted in case of failure.
4. Shopify Integration:
- Sync the updated data to a Shopify store using its API.
- Create or update products in Shopify accordingly.
5. Workflow Trigger and Frequency:
- Allow the workflow to run periodically based on a configurable trigger in n8n.
6. Proxy Service Integration:
- Use Smartproxy.com or ScraperAPI.com to handle website requests.
- The freelancer must decide which service to use based on their technical evaluation and justify their choice.
- Ensure efficient scraping while avoiding rate limits or IP bans.
---
Required Experience:
- Web Scraping: Experience extracting structured data from public websites.
- n8n Workflows: Proven ability to design and implement workflows in n8n.com.
- Shopify API: Experience integrating with Shopify's API for product creation and updates.
- Airtable API: Familiarity with managing data in Airtable, including attachments for images.
- Programming Languages: Proficiency in JavaScript (Node.js) or Python.
---
Additional Information:
- The workflow should be robust and handle large datasets efficiently.
- Clear documentation of the process is required for future maintenance.
- Candidates should include examples of similar projects they have completed.
---
Special Note:
Please include in your proposal:
1. Your choice between Smartproxy.com and ScraperAPI.com, with a brief justification for why it is the best option for this project.
2. Any suggestions to improve the workflow for better performance or scalability.
3. An estimate of the hours required to complete the project, along with your hourly rate (this point is mandatory).
I am looking for a freelancer to develop an automated workflow using n8n.com to scrape data from a WordPress-based website, store it in Airtable, and sync it with a Shopify store via its API.
---
Main Tasks:
1. Scraping Product Data:
- Visit the target website and extract the list of available brands from the sidebar menu (approximately 20 brands).
- Retrieve the URLs of the product showcase pages for each brand.
- Extract the URLs of all products listed within each brand's catalog (currently 240 product URLs in total).
- Visit each product's page and extract the following details:
- Product Image
- Description
- Technical Specifications
- Price
- Stock
availability
2. Data Storage and Processing:
- Save the extracted data, including images, in an Airtable database.
- Create new entries for products not already in the database and update existing ones.
- Maintain a change log table to track fluctuations in product price and availability over time.
3. Resilience and Error Handling:
- Ensure the workflow can resume from the exact point where it was interrupted in case of failure.
4. Shopify Integration:
- Sync the updated data to a Shopify store using its API.
- Create or update products in Shopify accordingly.
5. Workflow Trigger and Frequency:
- Allow the workflow to run periodically based on a configurable trigger in n8n.
6. Proxy Service Integration:
- Use Smartproxy.com or ScraperAPI.com to handle website requests.
- The freelancer must decide which service to use based on their technical evaluation and justify their choice.
- Ensure efficient scraping while avoiding rate limits or IP bans.
---
Required Experience:
- Web Scraping: Experience extracting structured data from public websites.
- n8n Workflows: Proven ability to design and implement workflows in n8n.com.
- Shopify API: Experience integrating with Shopify's API for product creation and updates.
- Airtable API: Familiarity with managing data in Airtable, including attachments for images.
- Programming Languages: Proficiency in JavaScript (Node.js) or Python.
---
Additional Information:
- The workflow should be robust and handle large datasets efficiently.
- Clear documentation of the process is required for future maintenance.
- Candidates should include examples of similar projects they have completed.
---
Special Note:
Please include in your proposal:
1. Your choice between Smartproxy.com and ScraperAPI.com, with a brief justification for why it is the best option for this project.
2. Any suggestions to improve the workflow for better performance or scalability.
3. An estimate of the hours required to complete the project, along with your hourly rate (this point is mandatory).