Competitor Pricing Scraping -- 2

Background
To stay competitive, we need better visibility into how our competitors price their digital products. Right now, pricing changes frequently across platforms, and manual checks are time-consuming and inconsistent. This project aims to automate the collection of competitor pricing data so we can make faster and more informed decisions.

Objective
Build a scraping system that regularly collects and updates competitor pricing for key digital products, so the team can:

1. Monitor price differences
2. Detect pricing changes quickly
3. Support pricing strategy and promotions

Scope of Products

The scraping should cover the following product categories:
1. Pulsa
2. Paket data
3. E-wallet top-up
4. PDAM
5. Electricity token (PLN)
6. BPJS

Other relevant bill payment products if available

Competitors

1. Major direct competitors offering similar products
2. Mobile apps and/or web platforms (to be finalized)

Data to Collect

For each product, collect:

1. Product type (e.g. Pulsa Telkomsel 10K, Token PLN 50K)
2. Nominal / package size
3. Price
4. Admin / service fee (if any)
5. Promo or discount info (if visible)
6. Timestamp (date & time of data collection)
7. Source (competitor name & platform)

Frequency

Daily scraping (preferred) Or configurable frequency depending on platform limitations

Output & Format

1. Structured data in CSV / Google Sheets / Database
2. Easy to compare prices across competitors
3. Historical data retained for trend analysis

Technical Notes

1. craping must comply with legal and ethical guidelines
2. Handle anti-bot protections where possible
3. Log failures and missing data
4. Scalable to add more competitors or products in the future