AWS Data Workflow Enhancement
Budget: $1,500 – $3,000 AUD
Project Overview
This project focuses on enabling a streamlined workflow from raw data ingestion to meaningful data visualization using AWS services. It will include S3 bucket setup, data cleanup automation, dashboard creation with Amazon QuickSight, and automation for continuous data refresh.
Phased Deliverables
Phase 1: S3 Bucket Creation
•
Create an Amazon S3 bucket for storing CSV files.
•
Apply security best practices, including:
o
Bucket policy with least-privilege access
o
IAM permissions for data access control
o
Enable versioning and server-side encryption (SSE-S3 or SSE-KMS)
Deliverables:
•
S3 bucket with all required security and permission settings
Phase 2: Data Clean-Up Process Configuration
•
Design and implement a process to clean and transform CSV data:
o
Split combined fields
o
Apply labels to specific fields
o
Remove or ignore unwanted/malformed records
•
Define schema for cleaned output
•
Deliver as an AWS Lambda-based solution or simple script, based on feasibility
Deliverables:
•
Clean-up script/function deployed and tested
•
Sample before/after dataset comparison
Phase 3: QuickSight Dashboard Configuration
•
Connect Amazon QuickSight to cleaned S3 data
•
Prepare dataset with calculated fields (e.g., scores, categories)
•
Build visualizations
Deliverables:
•
Interactive dashboard on QuickSight
•
QuickSight dataset and visuals
•
User access setup and sharing configuration
Phase 4: Automation
•
Automate end-to-end pipeline from new data upload to dashboard refresh:
o
Auto-trigger cleanup process on new file upload in S3
o
Auto-refresh QuickSight dataset via scheduled refresh or event-based triggers
Deliverables:
•
Automated workflow tested end-to-end
•
Documentation for adding new data
This project focuses on enabling a streamlined workflow from raw data ingestion to meaningful data visualization using AWS services. It will include S3 bucket setup, data cleanup automation, dashboard creation with Amazon QuickSight, and automation for continuous data refresh.
Phased Deliverables
Phase 1: S3 Bucket Creation
•
Create an Amazon S3 bucket for storing CSV files.
•
Apply security best practices, including:
o
Bucket policy with least-privilege access
o
IAM permissions for data access control
o
Enable versioning and server-side encryption (SSE-S3 or SSE-KMS)
Deliverables:
•
S3 bucket with all required security and permission settings
Phase 2: Data Clean-Up Process Configuration
•
Design and implement a process to clean and transform CSV data:
o
Split combined fields
o
Apply labels to specific fields
o
Remove or ignore unwanted/malformed records
•
Define schema for cleaned output
•
Deliver as an AWS Lambda-based solution or simple script, based on feasibility
Deliverables:
•
Clean-up script/function deployed and tested
•
Sample before/after dataset comparison
Phase 3: QuickSight Dashboard Configuration
•
Connect Amazon QuickSight to cleaned S3 data
•
Prepare dataset with calculated fields (e.g., scores, categories)
•
Build visualizations
Deliverables:
•
Interactive dashboard on QuickSight
•
QuickSight dataset and visuals
•
User access setup and sharing configuration
Phase 4: Automation
•
Automate end-to-end pipeline from new data upload to dashboard refresh:
o
Auto-trigger cleanup process on new file upload in S3
o
Auto-refresh QuickSight dataset via scheduled refresh or event-based triggers
Deliverables:
•
Automated workflow tested end-to-end
•
Documentation for adding new data