Advanced Analytica Advanced Analytica: IBOM
← All connectors
TiDB logo

Destination connector

TiDB

Send governed data into TiDB with a clear, practical setup guide.

Version 0.1.4 archived alpha

TiDB

TiDB (/’taɪdiːbi:/, “Ti” means Titanium) is an open-source, distributed NewSQL database for Hybrid Transactional and Analytical Processing (HTAP) workloads. It is MySQL compatible, with horizontal scalability, strong consistency and high availability. You can now start a free developer trial on TiDB Cloud.

This page shows you how to set up the TiDB destination connector.

Features

FeatureSupported?(Yes/No)Notes
Full Refresh SyncYes
Incremental - Append SyncYes
Incremental - Append + DedupedYes
NamespacesYes
SSH Tunnel ConnectionYes

Output schema

Each stream is written to its own TiDB table. Each table contains three columns:

  • _airbyte_ab_id: a UUID that Airbyte assigns to each processed event. Its TiDB column type is VARCHAR(256).
  • _airbyte_emitted_at: a timestamp showing when the event was pulled from the data source. Its TiDB column type is TIMESTAMP(6).
  • _airbyte_data: a JSON blob containing the event data. Its TiDB column type is JSON.

Get started

Requirements

To use the TiDB destination, you’ll need:

  • A TiDB database running v5.4.0 or later to sync data to TiDB with normalization.

Network access

Make sure Airbyte can access your TiDB database. For a database within a VPC, you may need to allow access from the IP exposing Airbyte.

Permissions

You need a user with CREATE, INSERT, SELECT, DROP, CREATE VIEW, ALTER permissions. Creating a dedicated Airbyte user is strongly recommended.

Run the following commands against your database to create a dedicated database user:

CREATE USER 'airbyte'@'%' IDENTIFIED BY 'your_password_here';

Then grant the user access to the relevant database:

GRANT CREATE, INSERT, SELECT, DROP, CREATE VIEW, ALTER ON <database name>.* TO 'airbyte'@'%';

Target database

TiDB doesn’t distinguish between a database and a schema. A database is effectively a schema containing all its tables. Choose an existing database or create one. It becomes the default database or schema for tables when the source doesn’t provide a namespace.

Set up the TiDB destination

Configure the following details in the TiDB destination:

  • Host
  • Port
  • Username
  • Password
  • Database
  • jdbc_url_params (Optional)

Note: When connecting to TiDB Cloud with TLS enabled, specify a TLS protocol in the JDBC parameters. For example, use enabledTLSProtocols=TLSv1.2 or enabledTLSProtocols=TLSv1.3.

Default JDBC URL parameters

  • useSSL=false (unless ssl is set to true)
  • requireSSL=false (unless ssl is set to true)
  • verifyServerCertificate=false (unless ssl is set to true)

Known limitations

The TiDB destination converts all identifier names (tables, schemas and columns) to lowercase.

Connect through an SSH tunnel

Airbyte can connect to a TiDB instance through an SSH tunnel. Use this when direct database access is unavailable or conflicts with your security policy. For example, the database may not have a public IP address.

An SSH tunnel connects Airbyte through an intermediate server, also called a bastion server. This server must have direct database access. Airbyte connects to the bastion, which then connects directly to the database server.

This feature needs extra configuration when you create the destination. The following steps explain each setting.

  1. Configure the destination fields as usual, except for SSH Tunnel Method.
  2. SSH Tunnel Method defaults to No Tunnel (a direct connection). To use an SSH tunnel, choose SSH Key Authentication or Password Authentication.
    1. Choose Key Authentication when using an RSA private key to establish the SSH tunnel. See below for details about generating this key.
    2. Choose Password Authentication when using a password to establish the SSH tunnel.
  3. SSH Tunnel Jump Server Host identifies the intermediate (bastion) server that Airbyte connects to. Enter a hostname or IP address.
  4. SSH Connection Port is the bastion server port used for the SSH connection. The default SSH port is 22. Keep this default unless you’ve explicitly changed the port.
  5. SSH Login Username is the username Airbyte uses to connect to the bastion server. This isn’t the TiDB username.
  6. If you’re using Password Authentication, set SSH Login Username to the previous step’s user password. If you’re using SSH Key Authentication, leave this blank. This isn’t the TiDB password. It’s the operating system user’s password that Airbyte uses to run commands on the bastion.
  7. If you’re using SSH Key Authentication, set SSH Private Key to the RSA Private Key used for the SSH connection. Include the full key file contents, starting with -----BEGIN RSA PRIVATE KEY----- and ending with -----END RSA PRIVATE KEY-----.

Changelog

Expand to review
VersionDatePull RequestSubject
0.1.42023-06-21#27555Reduce image size
0.1.32023-06-05#27025Internal code change for future development (install normalization packages inside connector)
0.1.22023-05-23#19109Enabled Append Dedub mode
0.1.12023-04-04#24604Support for destination checkpointing
0.1.02022-08-12#15592Added TiDB destination.
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.