Advanced Analytica Advanced Analytica: IBOM
← All connectors
MariaDB ColumnStore logo

Destination connector

MariaDB ColumnStore

Send governed data into MariaDB ColumnStore with a clear, practical setup guide.

Version 0.1.7 archived alpha

Mariadb Columnstore

Sync overview

Output schema

Each stream is written to its own MariaDB ColumnStore table. Each table contains three columns:

  • _airbyte_ab_id: a UUID that Airbyte assigns to each processed event. Its MariaDB ColumnStore column type is VARCHAR(256).
  • _airbyte_emitted_at: a timestamp showing when the event was pulled from the data source. Its MariaDB ColumnStore column type is TIMESTAMP.
  • _airbyte_data: a JSON blob containing the event data. Its MariaDB ColumnStore column type is LONGTEXT.

Features

FeatureSupported?(Yes/No)Notes
Full Refresh SyncYes
Incremental SyncYes
Replicate Incremental DeletesYes
SSL connectionNo
SSH Tunnel SupportYes

Performance considerations

Could this connector strain your database, API or another system under certain conditions? Consider high table or row counts. Identify the breaking point, such as 100mm> records. Explain how you can prevent this, perhaps with a read-only replica or frequent syncs.

Getting started

Requirements

To use the MariaDB ColumnStore destination, you must sync data to MariaDB ColumnStore 5.5.3 or above.

Network access

Make sure Airbyte can access your MariaDB ColumnStore database. For databases within a VPC, allow access from the IP exposing Airbyte.

Permissions

You need a MariaDB user with CREATE, INSERT, SELECT, DROP permissions. Creating a dedicated Airbyte user is highly recommended.

Target database

MariaDB ColumnStore doesn’t distinguish between a database and a schema. A database is essentially a schema containing all its tables. Choose an existing database or create one. This becomes the default database or schema for tables when the source provides no namespace.

Set up the MariaDB ColumnStore destination

Before setting up the MariaDB ColumnStore destination in Airbyte, set the local_infile system variable to true. Run the query SET GLOBAL local_infile = true . This is required because Airbyte uses LOAD DATA LOCAL INFILE to load data into the table.

You can now configure MariaDB ColumnStore as a destination in the UI. You’ll need this information:

  • Host
  • Port
  • Username
  • Password
  • Database

Connect through an SSH Tunnel

Airbyte can connect to a MariaDB instance through an SSH Tunnel. Use this when direct database access isn’t possible or breaches security policy. For example, the database might not have a public IP address.

With an SSH tunnel, Airbyte connects through an intermediate server, also called a bastion server. The bastion has direct database access. Airbyte connects to the bastion, which then connects directly to the database server.

This feature requires extra configuration when you create the destination. The following steps explain each configuration field.

  1. Configure every destination field as normal, except SSH Tunnel Method.
  2. SSH Tunnel Method defaults to No Tunnel (a direct connection). To use an SSH Tunnel, choose SSH Key Authentication or Password Authentication.
    1. Choose Key Authentication when using an RSA private key to establish the SSH Tunnel. See below for key-generation guidance.
    2. Choose Password Authentication when using a password to establish the SSH Tunnel.
  3. SSH Tunnel Jump Server Host identifies the intermediate (bastion) server that Airbyte connects to. Enter a hostname or IP Address.
  4. SSH Connection Port is the bastion server port used for the SSH connection. The default SSH port is 22. Keep this default unless you’ve explicitly changed the port.
  5. SSH Login Username is the username Airbyte uses to connect to the bastion server. This isn’t the MySQl username.
  6. If you’re using Password Authentication, set SSH Login Username to the previous step’s User password. If you’re using SSH Key Authentication, leave this blank. This isn’t the MariaDB password. It’s the OS-user password Airbyte uses to run commands on the bastion.
  7. If you’re using SSH Key Authentication, set SSH Private Key to the RSA Private Key used for the SSH connection. Include the full key file contents, starting with -----BEGIN RSA PRIVATE KEY----- and ending with -----END RSA PRIVATE KEY-----.

Changelog

Expand to review
VersionDatePull RequestSubject
0.1.72022-09-0716391Add custom JDBC parameters field
0.1.62022-06-1713864Updated stacktrace format for any trace message errors
0.1.52022-05-1712820Improved ‘check’ operation performance
0.1.42022-02-2510421Refactor JDBC parameters handling
0.1.32022-02-1410256Add -XX:+ExitOnOutOfMemoryError JVM option
0.1.22021-12-30#8809Update connector fields title/description
0.1.12021-12-01#8371Fixed incorrect handling “\n” in ssh key.
0.1.02021-11-15#7961Added MariaDB ColumnStore destination.
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.