Google Pub/Sub ingestion
Budget: $750 – $1,500 USD
I need robust Python code written to ingest messages from Google Pub/Sub and write to Google Cloud Storage. Code needs to be simple and easy to follow.
- Ingest messages from Pub/Sub topic using Pull API
- Be able to run in parallel for N topics in N threads
- Code for fault tolerance so that if job fails, then it can be restarted and replayed from the last message where it left off so that duplicate messages are not received - process each message exactly once.
- Save a batch of messages from a topic with the ability to define batch size and then save as a Spark DataFrame and then write to Google Cloud storage as a file.
- Ingest messages from Pub/Sub topic using Pull API
- Be able to run in parallel for N topics in N threads
- Code for fault tolerance so that if job fails, then it can be restarted and replayed from the last message where it left off so that duplicate messages are not received - process each message exactly once.
- Save a batch of messages from a topic with the ability to define batch size and then save as a Spark DataFrame and then write to Google Cloud storage as a file.