Data ingestion and processing from Pub/sub to Bigquery
Budget: €30 – €250 EUR
I'm trying to automate the ingestion of data from Pub/sub to Bigquery. I need help writing a cloud function to subscribe to a pub/sub, do some processing on that data to apply a certain logic, and then insert the data into Bigquery. I intend to use cloud scheduler, cloud pub/sub, cloud functions and big query. Do not intend to use apache beam as the volume of data is very low. (<100 records per day)
Deliverables
Complete writing cloud function which does the following:
Picks up data from pub/sub.
Processes the data and compares it to data in another table in BigQuery.
Based on certain conditions and criteria, the cloud function will insert that record into a separate table with an additional column which describes the status as 'Met', 'Not Met', 'Partially Met'
Deliverables
Complete writing cloud function which does the following:
Picks up data from pub/sub.
Processes the data and compares it to data in another table in BigQuery.
Based on certain conditions and criteria, the cloud function will insert that record into a separate table with an additional column which describes the status as 'Met', 'Not Met', 'Partially Met'