Build a crawler system

The provider sets up a crawler system based on python and selenium. One instance of the software will be be the foundation to crawl unlimited furniture websites. The logic to crawl 2 websites are in the scope of this project. The websites are in the attachment to this project.

The following data shall be extracted or filled:

products table
- added (current time stamp)
- website_id
- website_url_id
- website_category_id
- website_manufacturer_id
- free_delivery (1 if true)
- strike_price
- price
- short_description (this is the headline of the item)
- url (this is in our case the click-through url)

product_photos table
- added
- product_id
- main (set to 1 if this is the first photo: with these websites, there is only one photo)

website_categories table
- added
- website_id
- category (we can only extract products from leaf categories)

website_manufacturers table
- added
- website_id
- manufacturer (this is in our case on moebel24.de underneath the price)

website_urls table
- added
- crawled
- website_id
- statuscode
- url

Exclusions
URLs that contain strings as stored in the website_exclusions table won`t be saved into the database.

Identifiers / xPath / Locator
The identifiers (i.e. xPath) for the data type to be extracted shall be saved to the database into the table website_identifiers and upon running the crawling process, it should read that information from the database first. The following numbers (I marked the fields currently needed with *) shall be stored into the field data_type, whereas the identifier for this corresponding data type is saved into the field identifier.

1 = meta title
2 = meta keywords
3 = meta description
4 = short description *
5 = description
6 = photos *
7 = manufacturer *
8 = category *
9 = delivery time *
10 = EAN
11 = order id
12 = price *
13 = strike price *
14 = free delivery *

The vendor is setting up the system by using rotating proxies for the crawling. Under no circumstances should the IP address of the server crawl the websites.