Small LLM ARC-AGI Prompt Tuning - 30/01/2026 05:58 EST
Budget: $100 – $500 USD
I’d like your help pushing a small language model toward record-level performance on the ARC-AGI benchmark by reproducing—and hopefully surpassing—the Evolutionary Test-Time Compute approach outlined in Jeremy Berman’s posts (https://jeremyberman.substack.com/p/how-i-got-a-record-536-on-arc-agi and follow-up). My main goal is stronger generalization, not just memorized accuracy, so the solution must show consistent gains across the public and hidden splits.
Everything will run in Python. You can pick whatever supporting libraries you like (PyTorch, JAX, NumPy, Hugging Face, etc.) as long as setup stays lightweight and the code is clearly documented.
Core scope
• Implement evolutionary prompt search: generate, mutate, rank, and select prompts on-the-fly against ARC tasks.
• Automate evaluation: scoring script should mirror ARC’s official rubric so results are directly comparable to the leaderboard.
• Track sample efficiency: log compute time, number of queries, and score progression so we can see where improvements come from.
• Deliver reproducible runs: a single command should download the data, load the model, run the search, and print the final score.
Acceptance criteria
•Re-running the notebook with y keys reproduces the accuracy numbers
•Plots clearly show defult vs tuned accuracy
• Codebase installs with `pip install -r requirements.txt` and runs on a single GPU.
• Clear README explaining how to tweak evolutionary parameters for further experimentation.
• document each expirements steps clearly
The sooner we can iterate, the better—I’m ready to review progress immediately and will test each checkpoint as you push commits.
Everything will run in Python. You can pick whatever supporting libraries you like (PyTorch, JAX, NumPy, Hugging Face, etc.) as long as setup stays lightweight and the code is clearly documented.
Core scope
• Implement evolutionary prompt search: generate, mutate, rank, and select prompts on-the-fly against ARC tasks.
• Automate evaluation: scoring script should mirror ARC’s official rubric so results are directly comparable to the leaderboard.
• Track sample efficiency: log compute time, number of queries, and score progression so we can see where improvements come from.
• Deliver reproducible runs: a single command should download the data, load the model, run the search, and print the final score.
Acceptance criteria
•Re-running the notebook with y keys reproduces the accuracy numbers
•Plots clearly show defult vs tuned accuracy
• Codebase installs with `pip install -r requirements.txt` and runs on a single GPU.
• Clear README explaining how to tweak evolutionary parameters for further experimentation.
• document each expirements steps clearly
The sooner we can iterate, the better—I’m ready to review progress immediately and will test each checkpoint as you push commits.