AI Coding Transcript Reviewer Needed -- 4
Budget: $2 – $8 USD
# Hiring: AI Pairwise Coding Transcript Reviewer (Remote)
We are looking for detail-oriented reviewers to evaluate AI coding assistant conversations for a research project.
This is **not a software engineering position**. Instead, you'll review pairs of AI responses and evaluate how well each model behaved during coding tasks using a structured rubric.
### Responsibilities
* Review pairwise AI coding transcripts.
* Evaluate model behavior rather than code correctness.
* Apply behavioral evaluation rubrics consistently.
* Write concise, evidence-based rationales.
* Compare two model responses and select the stronger one.
* Maintain high annotation quality and consistency.
### Ideal Candidate
* Strong analytical and critical thinking skills.
* Software engineering or computer science background preferred.
* Comfortable reading code (Python, JavaScript, TypeScript, Java, C++, etc.).
* Excellent written English.
* Able to distinguish between technical mistakes and behavioral issues.
* Careful attention to detail.
### You'll Need to Understand Topics Like
* Agentic Safety
* Scoping
* Honesty vs. Confidence
* Interaction
* Deference
* Verification
* Engineering workflow
* Severity calibration
Training materials and rubrics will be provided.
### Compensation
* Competitive pay based on experience and quality.
* Remote work.
* Flexible schedule.
### To Apply
Please send:
1. A brief introduction.
2. Your software engineering or coding experience.
3. Any AI evaluation or annotation experience.
4. Your availability (hours per week).
5. Why you'd be a good fit for behavioral evaluation work.
Applicants who demonstrate strong reasoning and consistent rubric application will receive priority.
Only candidates with excellent attention to detail should apply.
We are looking for detail-oriented reviewers to evaluate AI coding assistant conversations for a research project.
This is **not a software engineering position**. Instead, you'll review pairs of AI responses and evaluate how well each model behaved during coding tasks using a structured rubric.
### Responsibilities
* Review pairwise AI coding transcripts.
* Evaluate model behavior rather than code correctness.
* Apply behavioral evaluation rubrics consistently.
* Write concise, evidence-based rationales.
* Compare two model responses and select the stronger one.
* Maintain high annotation quality and consistency.
### Ideal Candidate
* Strong analytical and critical thinking skills.
* Software engineering or computer science background preferred.
* Comfortable reading code (Python, JavaScript, TypeScript, Java, C++, etc.).
* Excellent written English.
* Able to distinguish between technical mistakes and behavioral issues.
* Careful attention to detail.
### You'll Need to Understand Topics Like
* Agentic Safety
* Scoping
* Honesty vs. Confidence
* Interaction
* Deference
* Verification
* Engineering workflow
* Severity calibration
Training materials and rubrics will be provided.
### Compensation
* Competitive pay based on experience and quality.
* Remote work.
* Flexible schedule.
### To Apply
Please send:
1. A brief introduction.
2. Your software engineering or coding experience.
3. Any AI evaluation or annotation experience.
4. Your availability (hours per week).
5. Why you'd be a good fit for behavioral evaluation work.
Applicants who demonstrate strong reasoning and consistent rubric application will receive priority.
Only candidates with excellent attention to detail should apply.