AI Conversation Research & Evaluation Specialist
- Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Posting language
- English
- Posted
Original job description
About Gramian
Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.
About The Role
We are seeking experienced Subject Matter Experts (SMEs)in psychology, social work, counseling, qualitative research, human-computer interaction (HCI), communication, conflict resolution, coaching, and related advisory fields to support AI conversation evaluation research. You will review conversations between people and AI models, assess whether sufficient context was gathered before recommendations were made, identify missed opportunities for clarification, and provide structured feedback to improve conversational quality. The ideal candidate has at least 3-5 years of relevant professional experience, strong analytical skills, and the ability to distinguish information that could materially change a recommendation from details that are less relevant.
Key Responsibilities
Candidates must have at least 3-5 years of relevant direct professional experience in one or more of the following domains:2. Counseling, Therapy, or Psychology3. Qualitative Research, HCI, or Communication4. Mediation, Coaching, Student Services, Organizational Consulting, or Related Advisory Fields
Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.
About The Role
We are seeking experienced Subject Matter Experts (SMEs)in psychology, social work, counseling, qualitative research, human-computer interaction (HCI), communication, conflict resolution, coaching, and related advisory fields to support AI conversation evaluation research. You will review conversations between people and AI models, assess whether sufficient context was gathered before recommendations were made, identify missed opportunities for clarification, and provide structured feedback to improve conversational quality. The ideal candidate has at least 3-5 years of relevant professional experience, strong analytical skills, and the ability to distinguish information that could materially change a recommendation from details that are less relevant.
Key Responsibilities
- Review AI conversations involving real-world interpersonal situations and everyday decision-making
- Identify missing context and recommend relevant follow-up questions that could improve the quality of AI responses
- Evaluate whether gathering additional information would materially change an AI-generated recommendation
- Distinguish between load-bearing information that affects a decision and non-essential or merely descriptive details
- Conduct qualitative reviews and document findings using structured evaluation guidelines
- Assess whether conversations adequately consider relevant personal, psychological, social, financial, practical, and organizational factors
- Identify missed opportunities for clarification, deeper understanding, and information gathering
- Review conversations involving relationships, work, education, family, wellbeing, planning, and resource constraints
- Collaborate with research teams to ensure consistent, evidence-based assessments
- Maintain high standards of accuracy, clarity, and consistency while meeting project deadlines
Candidates must have at least 3-5 years of relevant direct professional experience in one or more of the following domains:2. Counseling, Therapy, or Psychology3. Qualitative Research, HCI, or Communication4. Mediation, Coaching, Student Services, Organizational Consulting, or Related Advisory Fields
- Licensed Social Work
- Master's degree in Social Work (MSW) or an equivalent graduate-level qualification
- LCSW, LICSW, LMSW, or an equivalent applicable social work license or credential
- Experience in client assessment, case review, interviewing, counseling, or related social-work practice
- Master's degree in a relevant discipline with an LMFT, LPC, LCPC, or equivalent counseling or therapy license or credential; or
- PhD or PsyD in Clinical, Counseling, Social, or Industrial-Organizational Psychology
- Relevant experience in counseling, therapy, psychological assessment, interviewing, case review, or other client-facing professional work
- MA, PhD, or equivalent advanced expertise in Human-Computer Interaction, Communication, Anthropology, Sociology, Qualitative Research, or a closely related discipline
- Experience conducting qualitative research, interviews, user research, conversation analysis, thematic analysis, or qualitative coding
- Ability to analyze conversations systematically and identify missing context, communication patterns, and opportunities for clarification
- Relevant professional qualification, credential, or demonstrated subject-matter expertise
- At least 5 years of direct advisory or client-facing experience for candidates qualifying through this pathway
- Experience in coaching, mediation, student advising, organizational consulting, or comparable professional work
- Demonstrated ability to gather context, understand individual or organizational needs, and provide informed recommendations
More about this job
Responsibilities
Review AI conversations about interpersonal situations and everyday decisions, identifying missing context, useful follow-up questions, and information that could materially affect recommendations. Document structured qualitative assessments of conversational quality and collaborate with research teams to maintain accurate, consistent, evidence-based evaluations.
Requirements
Candidates need at least 3–5 years of relevant professional experience, with at least 5 years required for the advisory or client-facing pathway. Qualifications include relevant graduate-level education or professional credentials, such as social work or counseling licensure, advanced expertise in a related research discipline, or demonstrated advisory expertise, along with strong analytical and contextual assessment skills.
Skills
- Conversation Evaluation
- Qualitative Research
- Contextual Analysis
- Client Assessment
- Interviewing
- Counseling
- Psychological Assessment
- Conversation Analysis
- Thematic Analysis
- Qualitative Coding
- Critical Thinking
- Communication
- Coaching
- Mediation
- Evidence-Based Assessment
- Structured Documentation
Visa sponsorship
Not detected in the job text
Categories
- Science & Research
- Social Services
- Technology
- Consulting
- Healthcare
Keywords
- Artificial Intelligence
- AI Conversation Evaluation
- Subject Matter Experts
- Psychology
- Social Work
- Counseling
- Qualitative Research
- Human-Computer Interaction
- Communication
- Conflict Resolution
- Coaching
- Mediation
- Client Assessment
- Case Review
- Interviewing
- Conversation Analysis
- Thematic Analysis
- Qualitative Coding
- User Research
- Context Gathering
- Follow-Up Questions
- Recommendation Evaluation
- Evidence-Based Assessment
- Structured Evaluation Guidelines
- MSW
- LCSW
- LICSW
- LMSW
- LMFT
- LPC