Consolidate Scala Streaming Jobs Cluster
Budget: $30 – $75 CAD
I currently run four independent Scala streaming jobs on four separate clusters. I’d like to cut that overhead and run all jobs on a single job cluster sourced from my existing cluster pool.
Here is what I need from you:
• A clean, well-commented YAML definition that spins up one job cluster from the pool, then schedules the four jobs to run sequentially in a continuous loop (job 1 → job 2 → job 3 → job 4 → back to job 1, and so on).
• The jobs do not depend on, or share resources with, one another, so no inter-job data handling is required.
• If any job fails, it must restart immediately and resume its place in the loop without manual intervention. If it fails again, skip.
I want to drop your YAML into my deployment pipeline, hit run, and watch the jobs cycle endlessly on that single cluster. If something in your file doesn’t work out of the box, I’ll need you to adjust it until it does.
Let me know how you’ll structure the loop logic, and roughly how you plan to wire the automatic restart so I can review before we push to prod.
Here is what I need from you:
• A clean, well-commented YAML definition that spins up one job cluster from the pool, then schedules the four jobs to run sequentially in a continuous loop (job 1 → job 2 → job 3 → job 4 → back to job 1, and so on).
• The jobs do not depend on, or share resources with, one another, so no inter-job data handling is required.
• If any job fails, it must restart immediately and resume its place in the loop without manual intervention. If it fails again, skip.
I want to drop your YAML into my deployment pipeline, hit run, and watch the jobs cycle endlessly on that single cluster. If something in your file doesn’t work out of the box, I’ll need you to adjust it until it does.
Let me know how you’ll structure the loop logic, and roughly how you plan to wire the automatic restart so I can review before we push to prod.