Observability Engineer

We are seeking an experienced Observability Engineer to join our Enterprise Kubernetes Platform team at a leading financial services organization. In this role, you will own the complete observability stack across 50+ production Kubernetes clusters, delivering robust metrics, logging, tracing, and alerting capabilities to ensure exceptional reliability and performance for mission-critical applications.

This position bridges deep technical expertise in modern cloud-native observability tools with emerging AI/ML capabilities to build intelligent monitoring solutions, predictive alerting systems, and self-healing infrastructure.

What You'll Do

1. Observability Stack Ownership

• Architecture & Deployment: Design, deploy, and maintain enterprise-scale observability infrastructure, including Prometheus, Grafana, Thanos, Loki, and modern collection agents.
• GitOps & IaC: Manage all observability deployments using GitOps principles and Infrastructure as Code (IaC).
• Storage & Continuity: Implement long-term metrics storage solutions utilizing cloud object storage; maintain and upgrade components across Development, QA, UAT, Production, and DR environments.
• Multi-Cloud/Hybrid Setup: Configure distributed observability architectures spanning multiple data centers and cloud providers.

2. Metrics & Monitoring

• Strategy & Collection: Design and implement Prometheus monitoring strategies for Kubernetes infrastructure and containerized workloads using ServiceMonitors and PodMonitors.
• Alerting Strategy: Develop rules for intelligent alerting designed to minimize noise and false positives.
• Federation & Aggregation: Configure multi-cluster metrics federation and aggregation across clusters.
• Performance Optimization: Optimize metrics cardinality, storage efficiency, query performance, recording rules, and SLI/SLO calculations.

3. Dashboards & Visualization

• Grafana Management: Build comprehensive Grafana dashboards for infrastructure health, application performance, and business metrics.
• Dashboard-as-Code: Create reusable dashboard templates and libraries for internal development teams using code-driven approaches.
• Governance & Access: Configure multi-datasource setups, role-based access control (RBAC), and multi-tenancy in Grafana.
• Executive Metrics: Design executive-level dashboards with clear SLO/SLI tracking and business KPIs.

4. Logging Infrastructure

• Centralized Logging: Deploy and manage centralized logging solutions (Loki, ELK, Splunk, or similar).
• Log Collection Agents: Configure log collection agents (Promtail, Fluentd, Fluent Bit, Vector, etc.).
• Cost & Compliance: Design log retention policies that effectively balance operational needs, compliance requirements, and cloud costs.
• Log Analytics: Write and optimize LogQL/Lucene queries to extract actionable log-based insights.