Kafka Platform Admin/Kafka Developer
Contract Term: 12 months, renewable
Work Location: Calgary, AB
Work Model: 5 days in office
We are looking for an experienced Kafka Platform Administrator / Kafka Developer with strong expertise in Apache Kafka, Kafka on-premises and cloud environments, and Red Hat OpenShift Service on AWS (ROSA). The ideal candidate should have hands-on experience in Kafka platform administration, cluster management, topic administration, connector development, and ongoing maintenance of existing integrations. The role requires a combination of platform engineering, development, operational support, and monitoring skills to ensure reliable, scalable, and highly available event-driven integration solutions.
Key Responsibilities:
- Administer, configure, and maintain Kafka platforms across on-premises, cloud, and Red Hat OpenShift Service on AWS (ROSA) environments.
- Manage Kafka clusters, brokers, topics, partitions, replication factors, retention policies, consumer groups, and access controls.
- Perform Kafka platform administration, including cluster health monitoring, capacity planning, performance tuning, and availability management.
- Deploy, configure, and support Kafka components and services across multiple environments, including development, testing, staging, and production.
- Develop, configure, deploy, and maintain Kafka connectors using Kafka Connect and related integration frameworks.
- Create new source and sink connectors to integrate Kafka with databases, enterprise applications, cloud services, and other external systems.
- Maintain and enhance existing connectors, troubleshoot connectivity and data ingestion issues, and implement fixes, upgrades, and performance improvements.
- Support connector lifecycle management, including configuration, deployment, validation, error handling, monitoring, and recovery.
- Manage Kafka topics and schemas, including topic creation, configuration changes, schema compatibility, and data retention requirements.
- Work with Kafka Connect, connector plugins, serialization formats, and data transformation mechanisms to support reliable data movement.
- Use Kafka CLI commands and administrative tools to manage topics, consumer groups, configurations, offsets, ACLs, and cluster operations.
- Troubleshoot producer and consumer issues, consumer lag, broker failures, connector task failures, rebalancing, and message delivery problems.
- Configure and maintain monitoring, alerting, and dashboards using Datadog to track Kafka cluster health, throughput, latency, consumer lag, connector status, and infrastructure utilization.
- Develop and maintain Datadog monitors, alerts, dashboards, and incident notifications to proactively identify platform and integration issues.
- Use GitHub for source control, connector configuration management, code reviews, versioning, and collaboration across development and operations teams.
- Support CI/CD pipelines and automated deployment processes for Kafka configurations, connector changes, and application releases.
- Provide L2/L3 production support, participate in incident management, perform root cause analysis (RCA), and implement preventive measures.
- Maintain operational documentation, runbooks, standard operating procedures, and platform administration guidelines.
- Collaborate with application developers, infrastructure teams, architects, security teams, and business stakeholders to deliver secure and reliable event-driven integration solutions.
- Follow best practices for Kafka security, platform governance, high availability, disaster recovery, and operational excellence.
Required Skills:
- Strong experience in middleware, messaging platforms, streaming technologies, platform administration, or integration development, with strong hands-on Kafka experience.
- Strong expertise in Apache Kafka administration and development across on-premises and cloud-based environments.
- Hands-on experience with Kafka on Red Hat OpenShift Service on AWS (ROSA), including deployment, configuration, operations, troubleshooting, and platform support.
- Experience administering Kafka clusters, brokers, topics, partitions, replication, retention settings, consumer groups, and access control lists (ACLs).
- Strong understanding of Kafka architecture, including producers, consumers, brokers, partitions, offsets, replication, and consumer group management.
- Practical experience with Kafka Connect, including connector deployment, configuration, troubleshooting, and lifecycle management.
- Experience developing or customizing Kafka source and sink connectors and maintaining existing connectors in production environments.
- Strong knowledge of connector configuration parameters, plugin management, connector tasks, error handling, retries, dead-letter queues, and offset management.
- Proficiency in Kafka CLI commands and administrative utilities for topic management, consumer group inspection, configuration changes, ACL management, and troubleshooting.
- Experience with command-line tools and scripting for platform administration, automation, and operational support.
- Hands-on experience with GitHub, including repositories, branching strategies, pull requests, code reviews, version control, and configuration management.
- Experience with Datadog monitoring, including dashboards, monitors, alerting, log analysis, metrics, and troubleshooting production incidents.
- Strong understanding of Kafka security, authentication, authorization, encryption, TLS, SASL, and access management.
- Experience troubleshooting Kafka performance issues, broker availability, connector failures, consumer lag, throughput bottlenecks, and message processing errors.
- Knowledge of REST APIs, JSON, Avro, XML, and other data serialization or integration formats.
- Experience working with relational databases, SQL, and enterprise systems for source/sink integrations.
- Understanding of Linux/Unix environments, networking fundamentals, DNS, certificates, and connectivity troubleshooting.
- Experience with CI/CD processes, deployment automation, configuration management, and environment promotion.
- Strong analytical, debugging, problem-solving, documentation, and communication skills.
Nice to Have:
- Experience with Red Hat OpenShift, Kubernetes, containerized workloads, and cloud-native platform operations.
- Exposure to Red Hat Streams for Apache Kafka, Strimzi, or managed Kafka offerings such as Amazon MSK and Confluent Cloud.
- Experience with Schema Registry, schema evolution, and compatibility management.
- Knowledge of Kafka ecosystem tools such as MirrorMaker 2, Kafka REST Proxy, and Kafka UI or equivalent administration tools.
- Experience integrating Kafka with Oracle, SQL Server, PostgreSQL, Salesforce, SAP, or other enterprise applications.
- Experience with Debezium, Change Data Capture (CDC), JDBC connectors, and database replication patterns.
- Knowledge of Python, Java, or shell scripting for connector development, automation, and troubleshooting.
- Experience with Jenkins, GitHub Actions, Azure DevOps, or other CI/CD tools.
- Exposure to AWS services, IAM, networking, secrets management, and cloud observability.
- Experience with disaster recovery, multi-cluster replication, high availability, and business continuity planning.
- Experience in enterprise production support, on-call operations, incident response, and service-level management.
- Kafka-related certifications or relevant Red Hat, cloud, or streaming platform certifications.
