Advanced Analytica Advanced Analytica: IBOM
← All connectors
Kinesis logo

Destination connector

Kinesis

Send governed data into Kinesis with a clear, practical setup guide.

Version 0.1.5 archived alpha

Kinesis

Prerequisites

  • If you use the Postgres source connector with Airbyte Open Source, upgrade your platform to v0.40.0-alpha or newer. Then, upgrade your Kinesis connector to 0.1.4 or newer.

Sync overview

Output schema

Incoming Airbyte data uses JSON format. The partition key determines how data is sent across different stream shards. This connector maps incoming data from each namespace and stream to a unique Kinesis stream. Each Kinesis record contains the following JSON fields:

  • _airbyte_ab_id: A random UUID used as the partition key when sending data to different shards.
  • _airbyte_emitted_at: A timestamp showing when the event was received from the data source.
  • _airbyte_data: A JSON text or object containing the data received from the data source.

Features

FeatureSupportNotes
Full Refresh Sync❌
Incremental - Append Sync✅Incoming messages are streamed/appended to a Kinesis stream as they are received.
Incremental - Append + Deduped❌
Namespaces✅Namespaces will be used to determine the Kinesis stream name.

Performance considerations

Kinesis handles large volumes of real-time data by scaling streams with shards. However, you should understand the following Kinesis Quotas and Limits. Adjust the connector buffer size to match your data size and frequency.

Getting started

Requirements

  • The connector supports the latest Kinesis service version available when this page was written.
  • Configuration
    • Endpoint: The AWS Kinesis endpoint to connect to. If you don’t provide one, the default endpoint is used.
    • Region: The AWS Kinesis region to connect to. If you don’t provide one, the default region is used.
    • shardCount: The number of shards created for the stream. The shard count affects your stream’s throughput.
    • accessKey: The access key credential used to authenticate with the service.
    • privateKey: The private key credential used to authenticate with the service.
    • bufferSize: The buffer size used to increase throughput by sending data in one request.

Setup guide

Changelog

Expand to review
VersionDatePull RequestSubject
0.1.52022-09-2216952Add required config fields
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.