Applied AI Engineer (Contract)
- Canada
Contract
Applications open in a new tab.
Job details
- Job type
- Contract
- Posting language
- English
- Posted
Original job description
About The Role
Upstream Works Software is hiring a contract Applied AI Engineer to contribute to the setup of our agentic AI evaluation and demo environment on Amazon Bedrock AgentCore.
Upstream Works is a Canadian software company with 25 years of experience in enterprise contact centre software, trusted by Cisco, Amazon Web Services and global partners.
Our platform brings AI into the contact centre, and this engagement will help us prove agent quality before it reaches customers.
You will own the hands on build: setting up AgentCore, creating a golden dataset and evaluation process, and delivering an end to end demo shaped around a real business use case. You will work closely with our Product Lead, who will define the business requirements the demo needs to meet.
Key Responsibilities
The work runs in three phases, each with a clear deliverable.
Setup AWS AgentCore
Required Qualifications
Upstream Works Software is hiring a contract Applied AI Engineer to contribute to the setup of our agentic AI evaluation and demo environment on Amazon Bedrock AgentCore.
Upstream Works is a Canadian software company with 25 years of experience in enterprise contact centre software, trusted by Cisco, Amazon Web Services and global partners.
Our platform brings AI into the contact centre, and this engagement will help us prove agent quality before it reaches customers.
You will own the hands on build: setting up AgentCore, creating a golden dataset and evaluation process, and delivering an end to end demo shaped around a real business use case. You will work closely with our Product Lead, who will define the business requirements the demo needs to meet.
Key Responsibilities
The work runs in three phases, each with a clear deliverable.
Setup AWS AgentCore
- Lead the setup of Amazon Bedrock AgentCore in our AWS environment, including Runtime, Observability and Evaluations.
- Configure secure access, permissions and logging following AWS best practices.
- Document the setup so our team can maintain and extend it.
- Work with our Product Lead and subject matter experts to define realistic scenarios, expected responses and the steps the agent should take.
- Build and version a golden dataset that reflects real contact centre interactions, using anonymized or synthetic data.
- Design the evaluation approach, including success criteria, built in and custom evaluators, and a repeatable process for regression testing as the agent changes.
- Report on agent quality in a way business stakeholders can understand.
- Build a working agent and connect it to the tools and data the use case needs.
- Shape the demo around the business requirements defined with our Product Lead, and iterate based on feedback.
- Prepare and present the demo to internal stakeholders.
Required Qualifications
- 3 or more years of hands on software or AI engineering experience, including building at least one LLM based application or agent that reached production or real users
- Hands on experience with Amazon Bedrock and Amazon Bedrock AgentCore
- Strong Python skills, including APIs, SDKs and data handling
- Experience building AI agents with tool or function calling, retrieval, and at least one agent framework (for example, Strands, LangGraph, CrewAI or similar)
- Experience designing evaluations for non deterministic AI systems, such as golden datasets, success criteria, LLM as a judge, and regression testing
- Working knowledge of core AWS services, including IAM, CloudWatch, Lambda and S3
- Ability to turn business requirements into a working technical solution with limited direction
- Excellent written and verbal English, and the ability to explain technical results to non-technical stakeholders
- Availability for regular meetings during Eastern Time business hours
- Experience with conversational AI or voice agents (for example, Amazon Connect, Cisco or Genesys)
- AWS certifications, such as AWS Certified AI Practitioner, Machine Learning Engineer Associate or Solutions Architect Associate
- Experience with Model Context Protocol (MCP) or AgentCore Gateway
- Experience setting up evaluation in CI/CD pipelines
- Experience handling sensitive customer data, including anonymization and privacy requirements
More about this job
Responsibilities
Set up and document Amazon Bedrock AgentCore, including secure access, observability, and evaluation capabilities. Develop a versioned golden dataset and repeatable evaluation process, then build, refine, and present an end-to-end agent demo based on business requirements.
Requirements
Requires at least three years of hands-on software or AI engineering experience, including a production or real-user LLM application, plus experience with Amazon Bedrock AgentCore, Python, AWS, agent frameworks, and AI evaluation. Candidates must be able to translate business needs into technical solutions, communicate results clearly in English, and attend regular meetings during Eastern Time business hours.
Skills
- Amazon Bedrock AgentCore
- Python
- AWS
- AI Agent Development
- Tool And Function Calling
- Retrieval-Augmented Generation
- AI Evaluation
- Golden Dataset Development
- Regression Testing
- IAM
- CloudWatch
- Lambda
- S3
- API And SDK Integration
- Data Handling
- Stakeholder Communication
Visa sponsorship
Not detected in the job text
Categories
- Technology
- Software
- Engineering
- Data & Analytics
Keywords
- Amazon Bedrock
- Amazon Bedrock AgentCore
- AWS
- Python
- LLM
- AI Agents
- Strands
- LangGraph
- CrewAI
- Tool Calling
- Function Calling
- Retrieval
- Golden Dataset
- AI Evaluation
- LLM As A Judge
- Regression Testing
- IAM
- CloudWatch
- AWS Lambda
- Amazon S3
- Amazon Connect
- Cisco
- Genesys
- Model Context Protocol
- AgentCore Gateway
- CI/CD
- Conversational AI
- Voice Agents
- Data Anonymization
- Contact Centre Software