Advanced Analytica Advanced Analytica: IBOM
← All connectors
Kafka logo

Destination connector

Kafka

Send governed data into Kafka with a clear, practical setup guide.

Version 0.1.13 community alpha

Kafka

Overview

Sync data to Kafka with the Airbyte Kafka destination. It writes each stream to the corresponding Kafka topic.

Prerequisites

  • If you use Airbyte Open Source with the Postgres source connector, upgrade your Airbyte platform to version v0.40.0-alpha or newer. Upgrade your Kafka connector to version 0.1.10 or newer.

Sync overview

Output schema

The connector outputs each stream to a Kafka topic.

This connector currently writes data only in JSON format. More formats (e.g. Apache Avro) will be supported in future.

Each record’s key contains the uuid assigned by Airbyte. Its value contains these four fields:

  • _airbyte_ab_id: a uuid that Airbyte assigns to each processed event.
  • _airbyte_emitted_at: a timestamp showing when the event was pulled from the data source.
  • _airbyte_data: a json blob containing the event data.
  • _airbyte_stream: the name of each record’s stream.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteNo
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

Get started

Requirements

To use the Kafka destination, you’ll need:

  • A Kafka cluster 1.0 or above.

Setup guide

Network access

Make sure Airbyte can access your Kafka brokers.

Permissions

Allow Airbyte to write messages to topics. Create these topics before writing to Kafka. Alternatively, enable auto.create.topics.enable in the brokers, although this isn’t recommended for production environments.

If you use dynamic topic names, you’ll probably need to enable auto.create.topics.enable. This prevents connection failures after source connector schema updates. Otherwise, a hardcoded topic name may be best.

Target topics

Use the topic_pattern configuration parameter to choose where messages are written. You can use a hardcoded, pre-defined topic. Alternatively, write dynamically to different topics based on their namespace or source stream.

To write all messages to one hardcoded topic, enter its name in the topic_pattern field. For example, setting topic_pattern to my-topic-name writes all messages from every stream and namespace to that topic.

To define output topics dynamically, use the {namespace} and {stream} pattern variables. These variables route messages to different topics using the values available when producing records. For example, set the topic_pattern parameter to airbyte_syncs/{namespace}/{stream}. Messages from namespace n1 and stream s1 will then be written to topic airbyte_syncs/n1/s1. Messages from s2 will be written to airbyte_syncs/n1/s2, and so on.

If you define output topics dynamically, consider enabling auto.create.topics.enable. This prevents connection failures after source connector schema updates. Otherwise, manually create Kafka topics as the source adds or updates them. Manual creation is recommended for production environments.

NOTICE: the StandardNameTransformer applies a naming convention transformation to the target topic name. This replaces some special characters.

Set up the Kafka destination

You now have the requirements to configure Kafka as a destination in the UI. You can configure the following Kafka destination parameters. Many are optional or have default values:

  • Bootstrap servers
  • Topic pattern
  • Test topic
  • Sync producer
  • Security protocol
  • SASL JAAS config
  • SASL mechanism
  • Client ID
  • ACKs
  • Enable idempotence
  • Compression type
  • Batch size
  • Linger ms
  • Max in flight requests per connection
  • Client DNS lookup
  • Buffer memory
  • Max request size
  • Retries
  • Socket connection setup timeout
  • Socket connection setup max timeout
  • Max block ms
  • Request timeout
  • Delivery timeout
  • Send buffer bytes
  • Receive buffer bytes

For more information, see the Kafka producer configs documentation site.

Secure Airbyte Cloud connections

Airbyte Cloud requires TLS encryption for Kafka broker connections. The Protocol field offers only SASL SSL (SASL_SSL), while SASL Mechanism offers only PLAIN. Connection checks and write setup reject other protocols or mechanisms before creating a Kafka producer. This includes configurations submitted through the API. The Kafka client uses TLS for the broker connection. It doesn’t fall back to plaintext if the TLS handshake fails.

On Airbyte Cloud, SASL JAAS Config must contain exactly one org.apache.kafka.common.security.plain.PlainLoginModule entry. Missing, malformed or unsupported JAAS configurations are rejected before connecting to Kafka.

Airbyte Open Source continues to support PLAINTEXT, SASL PLAINTEXT and SASL SSL without these Cloud-only restrictions. This connector doesn’t expose custom SSL truststore or client-certificate settings.

Confluent Cloud API-key authentication

To authenticate with a Confluent Cloud Kafka cluster:

  1. Set Bootstrap servers to your cluster’s bootstrap endpoint.

  2. Set Protocol to SASL SSL.

  3. Set SASL mechanism to PLAIN.

  4. Set SASL JAAS config to the following. Replace the placeholders with your Kafka API key and secret:

    org.apache.kafka.common.security.plain.PlainLoginModule required username="YOUR_KAFKA_API_KEY" password="YOUR_KAFKA_API_SECRET";

Your API key must have permission to write to the target topics. This destination writes JSON records. It doesn’t provide Avro serialisation or Schema Registry integration.

Namespace support

This destination supports namespaces. The {namespace} variable incorporates the namespace into the topic name in the topic_pattern configuration.

Changelog

Expand to review
VersionDatePull RequestSubject
0.1.132026-11-0491019Update Java CDK, Kafka libraries and security dependencies.
0.1.122026-11-0490997Enable on Airbyte Cloud with TLS-only Kafka broker connections; avoid logging SASL credentials.
0.1.112025-03-2856450Add support for other SASL Mechanisms when SASL_PLAINTEXT protocol is selected
0.1.102022-08-0415287Update Kafka destination to use outputRecordCollector to properly store state
0.1.92022-06-1713864Updated stacktrace format for any trace message errors
0.1.72022-04-1912134Add PLAIN Auth
0.1.62022-02-1510186Add SCRAM-SHA-512 Auth
0.1.52022-02-1410256Add -XX:+ExitOnOutOfMemoryError JVM option
0.1.42022-01-319905Fix SASL config read issue
0.1.32021-12-308809Update connector fields title/description
0.1.22021-09-146040Change spec.json and config parser
0.1.12021-07-305125Enable additionalPropertities in spec.json
0.1.02021-07-213746Initial Release
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.