[PYTHON & GCP EXPERT] Data ingestion and processing from Pub/sub to Bigquery -- 2
Budget: $50 – $100 USD
I'm trying to automate the ingestion of data from Pub/sub to Bigquery. I need help writing a cloud function to subscribe to a pub/sub, do some processing on that data to apply a certain logic, and then insert the data into Bigquery. I intend to use cloud scheduler, cloud pub/sub, cloud functions, and big query. Do not intend to use an apache beam as the volume of data is very low. (<100 records per day)
Deliverables
Complete writing cloud function which does the following:
1. Picks up data from pub/sub.
2. Processes the data and compares it to data in another table in BigQuery.
3. Based on certain conditions and criteria, the cloud function will insert that record into a separate table with an additional column that describes the status as 'Met', 'Not Met', 'Partially Met'
Please find project details in the attached excel file.
https://drive.google.com/file/d/1LMG4Xs9gN9gPyyFQ6VzjzVyHizF65W1x/view?usp=sharing
1. Data flow -> Pub/sub - Cloud function - Big Query
2. Flow logic - Every time data is pulled, the cloud function must use logic to map the ID, category, and status from the source table(highlighted in green) to Vul-CCF mapping, Vul-status to Control Health status, vul codename to app name with the environment and insert/update record in the destination table.
Deliverables
Complete writing cloud function which does the following:
1. Picks up data from pub/sub.
2. Processes the data and compares it to data in another table in BigQuery.
3. Based on certain conditions and criteria, the cloud function will insert that record into a separate table with an additional column that describes the status as 'Met', 'Not Met', 'Partially Met'
Please find project details in the attached excel file.
https://drive.google.com/file/d/1LMG4Xs9gN9gPyyFQ6VzjzVyHizF65W1x/view?usp=sharing
1. Data flow -> Pub/sub - Cloud function - Big Query
2. Flow logic - Every time data is pulled, the cloud function must use logic to map the ID, category, and status from the source table(highlighted in green) to Vul-CCF mapping, Vul-status to Control Health status, vul codename to app name with the environment and insert/update record in the destination table.