GCP Data Engineer Task

Scheduler 1 → Cloud Function 1 (Scraped - Enriched)
• Runs every 5 minutes
• Processes records where the Scraped timestamp is greater than the last processed time
• Updates ProcessedToEnrichedAt after successful processing
• Concurrency set to 1 to prevent overlap
Scheduler 2 - Cloud Function 2 (Enriched - Master & History)
• Runs every 5 minutes, offset by 2-3 minutes from Scheduler 1
• Processes records where TimestampProcessed is greater than the last processed time and ProcessedToMasterAt is null
• Updates Processed ToMasterAt after successful processing
• Concurrency set to 1 to prevent overlap