Design Robust STEM AI Benchmark
Budget: $30 – $250 USD
I’m building a new benchmark to measure how well frontier language models cope with genuine scientific workflow. Your role is to craft one self-contained, terminal-driven research task that feels exactly like real lab or data-science work—analysing raw data, running simulations, validating a hypothesis, comparing competing methods—rather than a polished textbook exercise. The task should force multi-step reasoning and code composition so thoroughly that today’s best models fail at least 80 % of the time, while an expert human (you) can still solve it reproducibly.
What I need from you
• A complete task package zipped together:
– instruction.md explaining the workflow, required inputs, expected outputs and the objective success criteria
– A fully reproducible Docker environment with every dependency and the dataset already inside (no outside downloads)
– solve.sh, an “oracle” reference solution that passes the local tests three-for-three
– A deterministic test suite, callable from the command line, that verifies objective success or failure without human judgment
– task.toml metadata so the benchmark harness can auto-discover and grade the task
Quality expectations
• Multi-step logic, genuine research flavour, and objective numerically-verifiable outputs
• No LLM-generated content anywhere in the task materials; everything must be authored by you
• The oracle must run cleanly inside the container on a fresh machine and reproduce identical results each time
Acceptance criteria
1. Running `./run_tests.sh` inside the Docker container returns all passes without flakiness.
2. Removing or altering any key step in solve.sh causes at least one test to fail.
3. Frontier models (GPT-4-Turbo or Gemini-3.5-Pro) fail the tests in more than 4 out of 5 blind trials.
Ideal background
You’re comfortable designing authentic research pipelines in biology, chemistry, physics, data science or machine learning, and you know your way around Python, Bash and Docker well enough to make everything turnkey.
If this sounds like a stimulating challenge, let’s talk through your proposed topic and dataset so you can start building.
What I need from you
• A complete task package zipped together:
– instruction.md explaining the workflow, required inputs, expected outputs and the objective success criteria
– A fully reproducible Docker environment with every dependency and the dataset already inside (no outside downloads)
– solve.sh, an “oracle” reference solution that passes the local tests three-for-three
– A deterministic test suite, callable from the command line, that verifies objective success or failure without human judgment
– task.toml metadata so the benchmark harness can auto-discover and grade the task
Quality expectations
• Multi-step logic, genuine research flavour, and objective numerically-verifiable outputs
• No LLM-generated content anywhere in the task materials; everything must be authored by you
• The oracle must run cleanly inside the container on a fresh machine and reproduce identical results each time
Acceptance criteria
1. Running `./run_tests.sh` inside the Docker container returns all passes without flakiness.
2. Removing or altering any key step in solve.sh causes at least one test to fail.
3. Frontier models (GPT-4-Turbo or Gemini-3.5-Pro) fail the tests in more than 4 out of 5 blind trials.
Ideal background
You’re comfortable designing authentic research pipelines in biology, chemistry, physics, data science or machine learning, and you know your way around Python, Bash and Docker well enough to make everything turnkey.
If this sounds like a stimulating challenge, let’s talk through your proposed topic and dataset so you can start building.
Related categories:
Python
Engineering
Chemical Engineering
Finite Element Analysis
Hadoop
Data Science
Docker
Data Analysis
Bash
AI Development