GenAI DevOps Engineer / AI Engineer
- Canada
Contract
Applications open in a new tab.
Job details
- Job type
- Contract
- Posting language
- English
- Posted
Original job description
On LLMOp/DevOps they are looking for people who have setup and operated the Agentcore runtime environment.
DevOps
MUST
- AI-DLC practices
- Agentic code generation (Claude Code SDK, Kiro CLI) with human-in-the-loop checkpoints
- GitHub Actions: reusable workflows, environments and approvals, OIDC federation to AWS
- Container image builds and Amazon ECR (AgentCore Runtime requires ARM64 images)
- AgentCore Runtime deployment (versions, endpoints, rollback) and VPC mode
- VPC endpoints / PrivateLink and security groups
- Event-driven automation: EventBridge, Lambda, Step Functions
- Terraform
- IAM for CI/CD
SHOULD
- AgentCore Gateway rate limits and PortKey admin API (kill switch)
- Game-day practice with audit evidence
- Image signing (Cosign)
- Python
COULD
- Strands Agents SDK
- Backstage templates
- ROSA / OpenShift networking (later phase)
- AI eng
MUST
- AgentCore Evaluations and Bedrock evaluation
- Golden-dataset design and curation with SMEs
- LLM-as-judge design and calibration against human labels
- Integrating evaluations as a CI gate
- Langfuse / OpenTelemetry
- Python
SHOULD
- RAGAS or equivalent
- Trajectory evaluation
- Understanding of agentic red-teaming results (to feed them into the gate)
COULD
- Bedrock Guardrails
- Sampling design for continuous evaluation
- Strands Agents SDK
More about this job
Responsibilities
Set up and operate the AgentCore runtime environment, including CI/CD workflows, ARM64 container builds, deployments, networking, security, and rollback procedures. Build and maintain evaluation processes for AI agents, including curated golden datasets, calibrated LLM-as-judge evaluations, and CI gates informed by red-team results.
Requirements
The role requires hands-on experience operating AgentCore Runtime and applying AI-DLC practices, agentic code generation, AWS CI/CD, infrastructure as code, and secure deployment practices. It also requires experience with AgentCore and Bedrock evaluations, golden-dataset curation, LLM-as-judge calibration, and integrating evaluations into CI.
Skills
- AgentCore Runtime
- AI-DLC Practices
- Agentic Code Generation
- GitHub Actions
- AWS
- Amazon ECR
- Docker Container Builds
- VPC Networking
- Terraform
- IAM
- EventBridge
- Lambda
- Step Functions
- AgentCore Evaluations
- Amazon Bedrock
- Python
Visa sponsorship
Not detected in the job text
Categories
- Technology
- Software
- Engineering
- Data & Analytics
- Security & Safety
Keywords
- LLMOps
- DevOps
- AI Engineering
- AgentCore Runtime
- AI-DLC
- Agentic Code Generation
- Claude Code SDK
- Kiro CLI
- GitHub Actions
- OIDC Federation
- AWS
- Amazon ECR
- ARM64
- VPC
- PrivateLink
- EventBridge
- AWS Lambda
- AWS Step Functions
- Terraform
- IAM
- AgentCore Gateway
- Portkey Admin API
- Cosign
- Python
- Strands Agents SDK
- AgentCore Evaluations
- Amazon Bedrock Evaluations
- Golden-Dataset Design
- LLM-as-Judge
- Langfuse