We are seeking an experienced Observability Engineer to join our Enterprise Kubernetes Platform team at a leading financial services organization. In this role, you will own the complete observability stack across 50+ production Kubernetes clusters, delivering robust metrics, logging, tracing, and alerting capabilities to ensure exceptional reliability and performance for mission-critical applications.
This position bridges deep technical expertise in modern cloud-native observability tools with emerging AI/ML capabilities to build intelligent monitoring solutions, predictive alerting systems, and self-healing infrastructure.
What You'll Do
- Observability Stack Ownership
- Architecture & Deployment: Design, deploy, and maintain enterprise-scale observability infrastructure, including Prometheus, Grafana, Thanos, Loki, and modern collection agents.
- GitOps & IaC: Manage all observability deployments using GitOps principles and Infrastructure as Code (IaC).
- Storage & Continuity: Implement long-term metrics storage solutions utilizing cloud object storage; maintain and upgrade components across Development, QA, UAT, Production, and DR environments.
- Multi-Cloud/Hybrid Setup: Configure distributed observability architectures spanning multiple data centers and cloud providers.
- Metrics & Monitoring
- Strategy & Collection: Design and implement Prometheus monitoring strategies for Kubernetes infrastructure and containerized workloads using ServiceMonitors and PodMonitors.
- Alerting Strategy: Develop rules for intelligent alerting designed to minimize noise and false positives.
- Federation & Aggregation: Configure multi-cluster metrics federation and aggregation across clusters.
- Performance Optimization: Optimize metrics cardinality, storage efficiency, query performance, recording rules, and SLI/SLO calculations.
- Dashboards & Visualization
- Grafana Management: Build comprehensive Grafana dashboards for infrastructure health, application performance, and business metrics.
- Dashboard-as-Code: Create reusable dashboard templates and libraries for internal development teams using code-driven approaches.
- Governance & Access: Configure multi-datasource setups, role-based access control (RBAC), and multi-tenancy in Grafana.
- Executive Metrics: Design executive-level dashboards with clear SLO/SLI tracking and business KPIs.
- Logging Infrastructure
- Centralized Logging: Deploy and manage centralized logging solutions (Loki, ELK, Splunk, or similar).
- Log Collection Agents: Configure log collection agents (Promtail, Fluentd, Fluent Bit, Vector, etc.).
- Cost & Compliance: Design log retention policies that effectively balance operational needs, compliance requirements, and cloud costs.
- Log Analytics: Write and optimize LogQL/Lucene queries to extract actionable log-based insights.
