Advanced Analytica Advanced Analytica: IBOM
← All connectors
Kafka logo

Source connector

Kafka

Set up Kafka as a source and move its data into your chosen destination.

Version 0.4.4 community alpha

Kafka

This page shows you how to set up the Kafka source connector.

Setup guide

Step 1: Set up Kafka

To use the Kafka source connector, you’ll need:

  • A Kafka cluster 1.0 or above
  • The Airbyte user must have permission to read messages from topics. Create these topics before reading from Kafka.

Step 2: Set up the Kafka source in Airbyte

You’ll need this information to configure the Kafka source:

  • Bootstrap Servers - A list of host/port pairs for establishing the initial connection to the Kafka cluster.
  • MessageFormat - How to deserialise messages. Choose either:
    • JSON
    • AVRO. If you choose avro, several other options are available:
      • Deserialization strategy - Set the [subject name strategy][subject-name-strategy] for choosing the schema from the registry when deserialising Avro binary
      • Schema Registry URL - The host/port for connecting to the schema registry server
      • Username and password, optional for authenticating to the schema registry server
  • Protocol - The Protocol used to communicate with brokers. If you use SASL, this includes how you authenticate to the brokers.
    • PLAINTEXT - No authentication or encryption. Available only on Airbyte Open Source
    • SASL PLAINTEXT - No encryption. A SASL JAAS Config is required for authentication. Available only on Airbyte Open Source
    • SASL SSL - Encrypted
      • SASL JAAS Config configures authentication to the brokers
      • SASL Mechanism appropriate to your brokers’ auth mechanism
      • OAUTHBEARER token endpoint URL - Optional. For SASL Mechanism OAUTHBEARER, set this to the token endpoint URL. This doesn’t apply to schema registry auth
  • Subscription Method - Manually assign a list of partitions or subscribe to topics matching a specified pattern for dynamically assigned partitions.
    • List of topic - Comma-separated topic:partitions. Each partition becomes a stream in the connection
    • Topic pattern - A pattern to use. Each matching topic becomes a stream in the connection

Set these optional fields according to your needs:

  • Client ID - An ID string passed to brokers when making requests. It adds a logical application name to server-side request logs. This helps you track request sources beyond their IP address and port. (e.g. airbyte-consumer)
  • Group ID - The Group ID distinguishes different consumer groups. (e.g. group.id)
  • Test Topic - The Topic used to test whether Airbyte can consume messages. (e.g. test.topic)
  • Polling Time - The time, in milliseconds, that the Kafka connector polls for messages during each sync

Secure Airbyte Cloud connections

Airbyte Cloud requires TLS encryption for Kafka broker connections. The Protocol field offers only SASL SSL (SASL_SSL). Connection checks, discovery and reads reject other protocols before connecting to Kafka. This includes configurations submitted through the API. The Kafka client uses TLS for the broker connection. It doesn’t fall back to plaintext if the TLS handshake fails.

On Airbyte Cloud, SASL JAAS Config must contain exactly one supported login module:

  • org.apache.kafka.common.security.plain.PlainLoginModule
  • org.apache.kafka.common.security.scram.ScramLoginModule
  • org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule
  • com.sun.security.auth.module.Krb5LoginModule
  • software.amazon.msk.auth.iam.IAMLoginModule

Use a login module that matches your selected SASL Mechanism. Missing, malformed or unsupported JAAS configurations are rejected before connecting to Kafka.

When using OAUTHBEARER on Airbyte Cloud, set OAUTHBEARER token endpoint URL to an HTTPS URL without embedded credentials or a fragment. HTTP URLs, local-file URLs and missing endpoints are rejected before connecting. Any OAuth endpoint allowlist configured by the operator remains enforced.

Airbyte Open Source still supports PLAINTEXT, SASL PLAINTEXT and SASL SSL without these Cloud-only restrictions.

Confluent Cloud API-key authentication

Configure authentication for a Confluent Cloud Kafka cluster:

  1. Set Bootstrap Servers to your cluster’s bootstrap endpoint.

  2. Set Protocol to SASL SSL.

  3. Set SASL Mechanism to PLAIN.

  4. Set SASL JAAS Config to the following, replacing the placeholders with your Kafka API key and secret:

    org.apache.kafka.common.security.plain.PlainLoginModule required username="YOUR_KAFKA_API_KEY" password="YOUR_KAFKA_API_SECRET";

The API key must have permission to read the configured topics. These settings configure Kafka broker authentication. Configure Schema Registry authentication separately when using Avro.

Airbyte Open Source

  1. Go to the Airbyte UI and click Sources in the left navigation bar. Click +new source in the top-right corner.
  2. On the Set up the source page, enter the Kafka connector name. Select Kafka from the Source type dropdown.
  3. Follow the Setup the Kafka source in Airbyte

Supported sync modes

The Kafka source connector supports these sync modes:

FeatureSupported?(Yes/No)Notes
Full Refresh SyncYes
Incremental - Append SyncYes
NamespacesNo

Supported formats

JSON - JSON value messages. Schema registry isn’t currently supported.

AVRO - Deserialises messages using the confluent API. See https://docs.confluent.io/platform/current/schema-registry/serdes-develop/serdes-avro.html.

IP allow list

If you use Airbyte Cloud and your organisation restricts access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Changelog

Expand to review
VersionDatePull RequestSubject
0.4.42026-11-0491018Update Java CDK, Kafka/Confluent libraries and security dependencies.
0.4.32026-11-0490996Enable on Airbyte Cloud with TLS-only Kafka broker connections; avoid logging SASL credentials.
0.4.22025-07-1062919Convert to new gradle build flow
0.4.12025-04-3059171Fixing vulnerabilities in dependencies.
0.4.02025-03-1855828Add configurations for AWS MSK IAM and fix JSON format
0.3.02025-02-1853231Add configurations for OAUTHBEARER SASL Mechanism
0.2.82025-02-0753221For AVRO MessageFormat, schema_registry_password is a secret
0.2.72025-01-1051480Use a non root base image
0.2.62024-12-1849907Use a base image: airbyte/java-connector-base:1.0.0
0.2.52024-06-1232538Fix empty airbyte data column
0.2.42024-02-1335229Adopt CDK 0.20.4
0.2.42024-01-2434453bump CDK version
0.2.32022-12-0619587Fix missing data before consumer is closed
0.2.22022-11-0418648Add missing record_count increment for JSON
0.2.12022-11-04This version was the same as 0.2.0 and was committed so using 0.2.2 next to keep versions in order
0.2.02022-08-2213864Added AVRO format support and Support for maximum records to process
0.1.72022-06-1713864Updated stacktrace format for any trace message errors
0.1.62022-05-2912903Add Polling Time to Specification (default 100 ms)
0.1.52022-04-1912134Add PLAIN Auth
0.1.42022-02-1510186Add SCRAM-SHA-512 Auth
0.1.32022-02-1410256Add -XX:+ExitOnOutOfMemoryError JVM option
0.1.22021-12-218865Fix SASL config read issue
0.1.12021-12-068524Update connector fields title/description
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.