Build a data ingestion platform - Azure/GCP
Budget: $30 – $250 AUD
Design a data ingestion platform architecture and a POC document that reads data from multiple sources and ingest to blob. Then, from blob to azure sql table (using stored proc) to stage the data and from azure sql table to data warehouse as the landing. Sometimes, there may be schema changes may come from different sources. Hence, the platform should support and shouldn't break (eg: when SPs are running). The platform should support both real-time and batch feeds.