We are seeking a talented and passionate Kafka Engineer to join our growing team in Bengaluru. This role will be crucial in designing, implementing, and maintaining our Kafka infrastructure, ensuring high availability, performance, and scalability. The ideal candidate will have a strong understanding of Kafka internals, experience with cluster management, and a proven track record of troubleshooting complex issues. Experience with Kubernetes is highly desirable.
Responsibilities:
Kafka Cluster Management:
Design, implement, and maintain highly available and scalable Kafka clusters.
Configure and optimize Kafka brokers, topics, and consumers.
Monitor cluster health, performance, and resource utilization.
Perform capacity planning
Kafka Development and Integration:
Develop and maintain Kafka producers and consumers for various applications.
Integrate Kafka with other systems and data pipelines.
Implement data serialization and schema management strategies.
Troubleshooting and Support:
Diagnose and resolve complex Kafka-related issues.
Provide timely support to development teams and other stakeholders.
Implement proactive monitoring and alerting to prevent issues.
Kubernetes (Desirable):
Deploy and manage Kafka on Kubernetes using operators like Strimzi.
Utilize Kubernetes features for scaling, monitoring, and managing Kafka deployments.
Troubleshoot Kafka deployments within a Kubernetes environment.
Required Skills and Experience:
Proven experience as a Kafka Engineer or similar role.
Strong understanding of Kafka architecture, internals, and best practices.
Experience with Kafka cluster management, monitoring, and troubleshooting.
Proficiency in Java, Scala, or Python for Kafka development.
Experience with Kafka Connect, Kafka Streams, or other Kafka ecosystem tools.
Familiarity with monitoring tools such as Prometheus, Grafana, or similar.
Strong understanding of Linux operating systems.
Excellent problem-solving and communication skills.
Preferred Skills and Experience:
Experience with Kubernetes and containerization technologies (Docker).
Experience with Kafka operators like Strimzi.
Experience with cloud platforms (AWS, Azure, GCP).
Experience with schema registry tools (e.g., Confluent Schema Registry).
Experience with CI/CD pipelines.
Knowledge of networking concepts.