Design RLHF Failure Benchmark
Budget: €30 – €250 EUR
I want to commission a small-scale benchmark that reliably exposes where reinforcement-learning-from-human-feedback (RLHF) agents break down when the information they receive is partial, ambiguous, or internally inconsistent. The end goal is a reproducible environment that I can plug directly into my existing training pipeline to see whether new agents have truly learned to reason under uncertainty, or whether they succeed only when the data are clean and explicit.
Scope
• Create one self-contained environment or task suite that an RLHF agent must interact with to achieve a goal, yet cannot solve by brute-forcing rewards because critical cues are either missing, conflicting, or deliberately underspecified.
• Ensure the failure point highlights a specific limitation—handling ambiguous or incomplete data—without penalising unrelated capabilities such as raw compute or action-space size.
• Provide clear evaluation scripts and metrics so I can automatically score agent performance and verify that current state-of-the-art models still struggle.
Deliverables
1. Problem description and theoretical motivation (PDF or Markdown).
2. Fully runnable environment (Python, Gym-compatible or similar) with step, reset, and render functions.
3. Reference baseline results showing the gap between random, standard RL, and RLHF agents.
4. Automated evaluation script that outputs a single numeric score plus error logs highlighting where the agent diverged.
5. Read-me with setup instructions and any required packages.
Acceptance Criteria
• A modern open-source RLHF agent (e.g., PPO-based with preference modelling) cannot surpass the baseline threshold you establish.
• Re-running your evaluation script on my machine reproduces the same failure pattern within ±5% of your reported scores.
• Code is clean, documented, and released under a permissive licence.
If you have prior examples of adversarial or robustness-focused RL environments, send them along with your proposal so I can gauge fit quickly.
Scope
• Create one self-contained environment or task suite that an RLHF agent must interact with to achieve a goal, yet cannot solve by brute-forcing rewards because critical cues are either missing, conflicting, or deliberately underspecified.
• Ensure the failure point highlights a specific limitation—handling ambiguous or incomplete data—without penalising unrelated capabilities such as raw compute or action-space size.
• Provide clear evaluation scripts and metrics so I can automatically score agent performance and verify that current state-of-the-art models still struggle.
Deliverables
1. Problem description and theoretical motivation (PDF or Markdown).
2. Fully runnable environment (Python, Gym-compatible or similar) with step, reset, and render functions.
3. Reference baseline results showing the gap between random, standard RL, and RLHF agents.
4. Automated evaluation script that outputs a single numeric score plus error logs highlighting where the agent diverged.
5. Read-me with setup instructions and any required packages.
Acceptance Criteria
• A modern open-source RLHF agent (e.g., PPO-based with preference modelling) cannot surpass the baseline threshold you establish.
• Re-running your evaluation script on my machine reproduces the same failure pattern within ±5% of your reported scores.
• Code is clean, documented, and released under a permissive licence.
If you have prior examples of adversarial or robustness-focused RL environments, send them along with your proposal so I can gauge fit quickly.