Advanced Analytica Advanced Analytica: IBOM
← All connectors
Redpanda logo

Destination connector

Redpanda

Send governed data into Redpanda with a clear, practical setup guide.

Version 0.1.0 archived alpha

Redpanda

The Airbyte Redpanda destination connector syncs your data to Redpada. It writes each stream to the corresponding Redpanda topic.

Sync overview

Output schema

Each stream is written to a Redpanda topic.

Each Redpanda topic uses the following format: {namespace}_{stream}

This connector currently writes data only in JSON format. More formats, such as Apache Avro, will be supported in future.

Each record’s key contains the UUID assigned by Airbyte. Its value contains these three fields:

  • _airbyte_ab_id: a UUID assigned by Airbyte to each processed event.
  • _airbyte_emitted_at: a timestamp showing when the event was pulled from the data source.
  • _airbyte_data: a JSON blob containing the event data.

Data type mapping

Integration TypeAirbyte TypeNotes

Features

This section uses the following table format:

FeatureSupported?(Yes/No)Notes
Full Refresh SyncNo
Incremental - Append SyncYes
Incremental - Append + DedupedNo
NamespacesYes

Performance

With enough Redpanda nodes and partitions, the cluster should handle any connector load.

Get started

Requirements

  • The connector should be able to create topics using the AdminClient
  • Configuration options:
    • Bootstrap servers
    • Buffer Memory
    • Compression Type
    • Batch Size
    • Retries
    • Number of topic partitions
    • Topic replication factor
    • Socket Connection Setup Timeout
    • Socket Connection Setup Max Timeout

You can find more information in the Redpanda producer configs documentation site.

NOTE: SSL configurations aren’t available yet.

Changelog

Expand to review
VersionDatePull RequestSubject
0.1.02022-08-0518884Initial commit
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.