Senior Machine Learning Systems Engineer

We are looking for an experienced Senior Machine Learning Systems Engineer to optimize large language model (LLM) inference performance across AWS Trainium platforms, including Trainium2 and Trainium3.

The ideal candidate will have strong hands-on experience in Python, PyTorch, LLM inference frameworks, and accelerator performance optimization. This role focuses on improving inference latency, throughput, scalability, and cost efficiency for production-grade AI workloads.

Key Responsibilities:

• Optimize LLM inference performance, focusing on decode latency, throughput, and serving costs.
• Work with AWS Trainium2/Trainium3, AWS Neuron SDK, and vLLM on Neuron.
• Implement and evaluate speculative decoding and Multi-Token Prediction (MTP) for Qwen-family models.
• Profile, benchmark, and optimize production LLM workloads across GPU and Trainium platforms.
• Optimize prefill/decode performance, continuous batching, KV-cache, and tensor/expert parallelism.
• Support long-context inference workloads (100K+ tokens) and multi-chip scaling.
• Work with advanced AI architectures, including Mixture of Experts (MoE) and State Space Models.
• Develop benchmarking reports, technical documentation, implementation guides, and customer runbooks.
• Collaborate with customer engineering teams to deliver measurable performance improvements.

Required Skills & Experience:

• 5+ years of experience in Machine Learning or Systems Engineering.
• 2+ years of hands-on experience with production LLM inference or serving.
• Strong programming experience in Python and PyTorch.
• Hands-on experience with LLM serving frameworks such as vLLM, TensorRT-LLM, SGLang, or Triton Inference Server.
• Strong understanding of LLM inference concepts, including prefill, decode, continuous batching, and KV-cache management.
• Experience with performance profiling, benchmarking, and accelerator workload optimization.
• Knowledge of attention mechanisms, Mixture of Experts (MoE), tensor parallelism, and expert parallelism.
• Strong analytical, problem-solving, and technical communication skills.

Preferred Qualifications:

• Experience with AWS Trainium, Inferentia, AWS Neuron SDK, NxD Inference, or Neuron Kernel Interface (NKI).
• Experience with speculative decoding, Medusa, EAGLE, or Multi-Token Prediction (MTP).
• Kernel development experience using NKI, Triton, or CUDA.
• Experience with Mamba, State Space Models, or large Mixture of Experts architectures.
• Knowledge of FP8/INT8 quantization and model accuracy validation.
• Previous experience in customer-facing engineering, consulting, or technical enablement.