Health AI Conversation Evaluator
- Toronto, ON, Canada
Remote
Compensation: $20–$160/hour
Applications open in a new tab.
Job details
- Pay
- Compensation: $20–$160/hour
- Workplace
- Remote
- Posting language
- English
Original job description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
Position: Generalist Annotator - Health AI Conversation Quality Evaluation
Type: Contract
Compensation: $20–$160/hour
Location: Remote
Role Responsibilities
- Evaluate the relevance and clarity of AI responses. Ensure the AI answers the question in understandable language for non-experts.
- Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
- Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
- Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
- Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.
Qualifications
Must-Have
- Strong written English skills.
- Ability to explain judgments in two to four specific sentences.
- Careful reading to notice omissions in responses.
- Comfort with applying a detailed rubric consistently.
- Reliable availability for time-boxed batches.
Preferred
- Experience in annotation, human-feedback, evaluation, or content review.
- Background in explaining technical concepts to non-experts.
- Experience with rubric-based grading or quality assurance.
Important
- Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
- For any help or support, reach out to: support@mercor.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
More about this job
Responsibilities
The Generalist Annotator at Mercor will review AI health conversations and judge whether answers are clear, relevant and honest. They will write short justifications for lower ratings and explain what went wrong and how to improve it.
Requirements
Candidates need strong written English, careful reading to spot omissions, and comfort applying a detailed rubric consistently. Annotation, evaluation or explaining technical topics to non-experts is preferred, and AI tools may not be used to write comments.
Skills
- Rubric-based evaluation
- Written English
- Critical reading
- Annotation
- Quality assurance
- Technical explanation for non-experts
- Judgment and justification writing
- Time-boxed task completion
- Independent work
- Health information literacy
Categories
- Technology
- Healthcare
- Data & Analytics
Keywords
- annotation
- AI evaluation
- human feedback
- health AI
- rubric grading
- quality assurance
- content review
- conversation evaluation
- sycophancy detection
- written English
- data labeling
- remote contract
- AI training data
- LLM evaluation
- RLHF
- technical writing
- clear communication
- health information
Languages
- English