Observability Engineer || Fulltime || Toronto, ON - Onsite

Job Title: Observability Engineer

Location: Toronto, ON

Work Model: Onsite – 4 Days per Week

About the Role

We are seeking an experienced Observability Engineer to join the Enterprise Kubernetes Platform team at a leading financial services organization.

The ideal candidate will be responsible for designing, implementing, and maintaining enterprise-scale observability solutions across 50+ production Kubernetes clusters. This role focuses on metrics, logging, distributed tracing, and intelligent alerting to ensure the reliability, availability, and performance of mission-critical applications.

The successful candidate will have strong hands-on experience with modern observability technologies, Kubernetes platforms, GitOps, and Infrastructure as Code (IaC). Experience with AI/ML-driven monitoring, predictive alerting, and self-healing infrastructure is an added advantage.

Key Responsibilities

1. Observability Platform Engineering

•

Design, deploy, configure, and maintain enterprise observability platforms using Prometheus, Grafana, Thanos, and Loki.

•

Manage observability components across development, QA, UAT, production, and disaster recovery (DR) environments.

•

Implement scalable, highly available observability architectures across multiple Kubernetes clusters, data centers, and cloud environments.

•

Manage long-term metrics retention and storage using cloud object storage solutions.

•

Perform platform upgrades, troubleshoot issues, and optimize observability infrastructure for reliability and performance.

2. Metrics Collection & Monitoring

•

Design and implement Prometheus-based monitoring for Kubernetes infrastructure, containerized applications, and platform services.

•

Configure Kubernetes ServiceMonitors and PodMonitors to enable automated metrics discovery and collection.

•

Develop and maintain Prometheus alerting and recording rules to support proactive monitoring and incident detection.

•

Implement multi-cluster metrics federation, aggregation, and centralized monitoring.

•

Optimize metrics cardinality, storage efficiency, retention policies, and query performance.

3. Logging, Alerting & Performance Optimization

•

Implement centralized logging solutions using Loki and integrate logs with metrics and tracing data for effective troubleshooting.

•

Build and maintain Grafana dashboards to monitor platform health, application performance, and service reliability.

•

Configure actionable alerts and notification mechanisms while minimizing false positives and alert fatigue.

•

Troubleshoot observability issues and support root cause analysis of production incidents.

•

Establish monitoring standards and best practices for platform engineering teams and application developers.

4. Automation & Platform Reliability

•

Automate observability platform deployments and configuration management using GitOps principles and Infrastructure as Code.

•

Support consistent, repeatable deployments and configuration changes across environments.

•

Identify opportunities to improve monitoring through AI/ML-driven anomaly detection, predictive alerting, and automated remediation.

•

Contribute to platform reliability, operational efficiency, and continuous improvement initiatives.

Required Skills & Qualifications

•

Strong hands-on experience with Kubernetes and enterprise-scale observability platforms.

•

Proficiency in Prometheus and Grafana for metrics collection, visualization, monitoring, and alerting.

•

Practical experience with Thanos and Loki for scalable metrics storage, aggregation, and centralized logging.

•

Experience configuring ServiceMonitors, PodMonitors, Prometheus rules, and multi-cluster monitoring.

•

Strong understanding of GitOps workflows and Infrastructure as Code (IaC).

•

Experience with cloud object storage, metrics retention, and storage optimization.

•

Knowledge of distributed systems, production troubleshooting, high availability, and disaster recovery.

•

Ability to optimize monitoring performance, metrics cardinality, and alert quality.

Preferred Qualifications

•

Experience supporting observability platforms across large-scale Kubernetes environments in financial services or other regulated industries.

•

Familiarity with distributed tracing and OpenTelemetry.

•

Experience with AI/ML-based anomaly detection, predictive monitoring, and self-healing infrastructure.

•

Knowledge of SRE practices, SLIs, SLOs, and incident management.

•

Experience with multi-cloud or hybrid-cloud environments.

Ideal Candidate Profile

A hands-on Observability Engineer who can build, automate, scale, and maintain production-grade monitoring infrastructure across multiple Kubernetes clusters, ensuring reliable metrics collection, centralized logging, effective alerting, and proactive detection of platform issues.