Kafka Integration Consultant (Azure)

Quick Facts

• Role: System Integration Consultant (Kafka)

• Focus: Event-driven streaming architecture using Kafka in an Azure cloud environment

Description

You will support system integration tasks by designing and implementing modern, event-driven streaming solutions with Apache Kafka in an Azure Cloud environment. The role requires deep Kafka expertise (core concepts, Kafka Connect, Schema Registry, and stream processing) and the ability to integrate Kafka with Azure integration and infrastructure services. You will also contribute to migration efforts, including phased and parallel running, as well as secure networking and data flow connectivity.

Responsibilities

• Design and implement event-driven streaming architecture using Kafka

• Build integrations using Kafka core concepts (topics/partitions/consumer groups) and producers/consumers

• Use Kafka Connect connectors to integrate with external systems

• Manage data contracts via Schema Registry for backward compatibility

• Perform real-time transformations and enrichment with Kafka Streams or KSQLDB

• Connect and orchestrate Kafka within Azure (managed Kafka, integration services)

• Transform legacy data formats (e.g., XML/EDI) into modern formats (e.g., JSON/Avro)

• Support phased/parallel migration strategies

• Apply CI/CD and DevOps practices in Azure, using infrastructure as code

• Ensure secure access and networking for streaming data flows

Requirements

• Strong hands-on experience with Apache Kafka

• Ability to design and implement event-driven streaming architectures

• Deep knowledge of Kafka core: Topics, Partitions, Offsets, Consumer Groups, Producers, Consumers

• Experience with Kafka Connect and source/sink connectors

• Experience with Schema Registry and schema/data contract management (Avro, Protobuf, JSON Schema)

• Skills in Kafka Streams or KSQLDB for real-time stream processing

• Experience with managed Kafka in Azure (Azure Event Hubs with Kafka interface and/or Confluent Cloud on Azure)

• Experience with Azure integration services: Logic Apps, Azure Functions, API Management, Service Bus

• Strong programming skills (e.g., C#/.NET or Java/Scala/Python)

• Experience transforming data formats (XML/EDI to JSON/Avro)

• CI/CD and DevOps experience on Azure (Azure DevOps or GitHub Actions) and infrastructure as code (Terraform or Bicep)

• Understanding of migration strategies: Phased Migration and Parallel Running

• Knowledge of security and networking: Azure Key Vault, Managed Identities, VNet peering, Private Endpoints