Finalize Tradelytics Instrumentation & Scaling Model
Budget: $750 – $1,500 USD
Our internal team has pushed Tradelytics close to launch, but a few critical items remain before we can scale with confidence.
First, I need every existing data-pipeline component fully instrumented. That covers Python scripts, Talend jobs, AWS Lambdas, Step Functions, and Glue jobs. Logs, custom metrics, and traces should flow cleanly into CloudWatch (or an equivalent stack you propose) with consistent naming so dashboards and alarms light up without manual tweaking.
Second, all EC2 servers must carry the same observability baseline. Beyond today’s fleet, new instances created by the front-end Auto Scaling group should bootstrap themselves with this tooling automatically—think launch template, user-data script, or a lightweight agent baked into the AMI.
Third, a brand-new front-end database has been added. Please extend the same monitoring approach across all current databases so queries, replication lag, I/O, and storage growth are visible in one place.
Finally, I need a capacity and cost projection model. It should tell me, in plain numbers, how much compute, storage, network, and monthly spend we should expect when the platform serves 50, 100, 200, 500, and 1,000 users. Feel free to pull data from AWS pricing APIs, existing CloudWatch metrics, or any sizing heuristics that make sense. A Jupyter notebook or well-documented spreadsheet is fine as the deliverable, as long as I can tweak input assumptions later.
Acceptance criteria:
• Unified instrumentation active on all listed jobs and services, verified in our monitoring console.
• New EC2 instances show identical metrics within five minutes of launch.
• Front-end database metrics appear alongside existing database dashboards.
• Projection model clearly states resource counts and estimated monthly cost for each user tier and allows parameter changes without code edits.
If you have deep AWS observability experience and a knack for data-driven cost forecasting, this should be a straightforward engagement.
First, I need every existing data-pipeline component fully instrumented. That covers Python scripts, Talend jobs, AWS Lambdas, Step Functions, and Glue jobs. Logs, custom metrics, and traces should flow cleanly into CloudWatch (or an equivalent stack you propose) with consistent naming so dashboards and alarms light up without manual tweaking.
Second, all EC2 servers must carry the same observability baseline. Beyond today’s fleet, new instances created by the front-end Auto Scaling group should bootstrap themselves with this tooling automatically—think launch template, user-data script, or a lightweight agent baked into the AMI.
Third, a brand-new front-end database has been added. Please extend the same monitoring approach across all current databases so queries, replication lag, I/O, and storage growth are visible in one place.
Finally, I need a capacity and cost projection model. It should tell me, in plain numbers, how much compute, storage, network, and monthly spend we should expect when the platform serves 50, 100, 200, 500, and 1,000 users. Feel free to pull data from AWS pricing APIs, existing CloudWatch metrics, or any sizing heuristics that make sense. A Jupyter notebook or well-documented spreadsheet is fine as the deliverable, as long as I can tweak input assumptions later.
Acceptance criteria:
• Unified instrumentation active on all listed jobs and services, verified in our monitoring console.
• New EC2 instances show identical metrics within five minutes of launch.
• Front-end database metrics appear alongside existing database dashboards.
• Projection model clearly states resource counts and estimated monthly cost for each user tier and allows parameter changes without code edits.
If you have deep AWS observability experience and a knack for data-driven cost forecasting, this should be a straightforward engagement.