Operator Portal Data Pull & Processing (No UI)
Budget: $30 – $250 USD
Objective
Build a backend-only pipeline that signs into our operator portal, exports the needed reports, cleans and unifies the data, and stores it in a database ready for future use (e.g., payouts, performance analysis). No dashboard or frontend is required.
Scope (high level, non-technical)
Portal sign-in & data export
Log in to the operator portal with credentials we provide.
Navigate to reports for: drivers, deliveries/orders, earnings/wallet, penalties/bonuses.
Export/download files for a selected period (daily/weekly/monthly).
Data preparation
Merge multiple files into one consistent dataset.
Remove duplicates and fix basic formatting issues.
Standardize driver identifiers so the same driver matches across all reports.
Normalize dates/timezones and currency fields.
Keep an audit trail of source files and run logs.
Storage (database-ready postgresql)
Load the cleaned data into a relational database with clear tables .
Provide a simple data dictionary (column names, meanings, and relationships).
Refresh job
Provide a repeatable job/command to re-run the full process for any date range:
download → prepare → load.
Support daily/weekly/monthly schedules (cron-ready).
Basic error handling and summary logs (rows loaded, skipped, any issues).
Outputs for downstream use
Export clean CSVs for each table on demand.
Documentation
Setup and runbook (how to configure credentials, run the job, and change date ranges).
Data dictionary and notes on any renamed or missing columns.
Reconciliation steps (how to verify totals against the portal).
Quality & reconciliation
For a sample period, totals in the database/CSVs should match the portal’s figures within a reasonable margin.
Clear logs for each run (success, warnings, failures).
Inputs we will provide
Portal URL, credentials, and navigation steps.
Example exported files and the initial date range.
Business rules for targets and basic payout logic.
Out of scope
Any frontend/dashboard or visualization.
Real-time/live integrations with the portal.
Changes to the portal itself or complex forecasting/AI.
Success criteria
Run a single command for a chosen date range and get refreshed, cleaned data loaded into the database and exported as CSVs.
Data reconciles with the portal for the tested period.
Clear documentation so we can operate the pipeline independently.
Build a backend-only pipeline that signs into our operator portal, exports the needed reports, cleans and unifies the data, and stores it in a database ready for future use (e.g., payouts, performance analysis). No dashboard or frontend is required.
Scope (high level, non-technical)
Portal sign-in & data export
Log in to the operator portal with credentials we provide.
Navigate to reports for: drivers, deliveries/orders, earnings/wallet, penalties/bonuses.
Export/download files for a selected period (daily/weekly/monthly).
Data preparation
Merge multiple files into one consistent dataset.
Remove duplicates and fix basic formatting issues.
Standardize driver identifiers so the same driver matches across all reports.
Normalize dates/timezones and currency fields.
Keep an audit trail of source files and run logs.
Storage (database-ready postgresql)
Load the cleaned data into a relational database with clear tables .
Provide a simple data dictionary (column names, meanings, and relationships).
Refresh job
Provide a repeatable job/command to re-run the full process for any date range:
download → prepare → load.
Support daily/weekly/monthly schedules (cron-ready).
Basic error handling and summary logs (rows loaded, skipped, any issues).
Outputs for downstream use
Export clean CSVs for each table on demand.
Documentation
Setup and runbook (how to configure credentials, run the job, and change date ranges).
Data dictionary and notes on any renamed or missing columns.
Reconciliation steps (how to verify totals against the portal).
Quality & reconciliation
For a sample period, totals in the database/CSVs should match the portal’s figures within a reasonable margin.
Clear logs for each run (success, warnings, failures).
Inputs we will provide
Portal URL, credentials, and navigation steps.
Example exported files and the initial date range.
Business rules for targets and basic payout logic.
Out of scope
Any frontend/dashboard or visualization.
Real-time/live integrations with the portal.
Changes to the portal itself or complex forecasting/AI.
Success criteria
Run a single command for a chosen date range and get refreshed, cleaned data loaded into the database and exported as CSVs.
Data reconciles with the portal for the tested period.
Clear documentation so we can operate the pipeline independently.
Related categories:
Python
Data Processing
NoSQL Couch & Mongo
MySQL
HTML
Backend Development
Database Management