Python Automation for Cloud Pricing Data

We are a SaaS company that provides cloud cost optimization solutions to enterprise customers. Our platform relies on accurate pricing information from major cloud providers such as AWS and Azure.
Currently, pricing data is collected manually, which is time-consuming and prone to errors. We need an automated solution to collect pricing information, validate the website functionality, and generate reports highlighting any pricing changes.
Project Objective
We are looking for a Python Automation Engineer to design and develop a scalable automation framework that can:
Scrape cloud pricing information from AWS and Azure websites.
Validate the functionality of pricing calculators and related pages.
Detect changes in pricing data compared to previous executions.
Store collected data in a database.
Generate detailed execution reports and logs.
The solution should be modular, reusable, and easy to maintain.
Scope of Work
Phase 1 – Web Scraping
Develop a Python scraper to collect:
Instance Name
Instance Type
CPU
Memory
Operating System
Region
Hourly Price
Currency
Last Updated Time
The scraper should handle:
Pagination
Dynamic content
Loading delays
Missing values
Retry mechanisms
Store all collected data in SQLite.
Phase 2 – UI Automation
Using Selenium with the Page Object Model (POM), automate the following scenarios:
Verify the pricing page loads successfully.
Validate the search functionality.
Verify region selection.
Validate instance filtering.
Verify pricing information is displayed correctly.
Capture screenshots for failed test cases.
Phase 3 – Data Validation
Compare the newly scraped data with the previous day's data.
Identify:
New instances added.
Instances removed.
Price changes.
Duplicate records.
Missing records.
Generate a comparison report.
Phase 4 – Reporting
Produce:
HTML Test Report
CSV Report
Log File
Summary Report
The summary should include:
Total records scraped
Total test cases executed
Passed tests
Failed tests
Number of pricing changes
Execution time
Technical Requirements
Programming Language:
Python 3.x
Automation:
Selenium WebDriver
PyTest
Data Handling:
Pandas
BeautifulSoup
Requests
Database:
SQLite
Reporting:
PyTest HTML
Logging
Version Control:
Git