Elasticsearch
Sync overview
Output schema
Elasticsearch is a Lucene-based search engine and a type of NoSQL storage.
It creates documents in an index, similar to a table in a relational database.
The output schema matches the source’s input schema.
Each source stream becomes a destination index.
For example, with a relational database source:
The database table name maps to the destination index.
The database table columns become fields in the destination document.
Each row becomes a document in the destination index.
Data type mapping
See Elastic documentation for detailed information about the field types
This section should map each connector data type to its corresponding Airbyte type. Airbyte currently uses the same types as JSONSchema. The most common data types are string, date-time, object, array, boolean, integer and number.
| Integration Type | Airbyte Type | Notes |
|---|---|---|
| text | string | more info |
| date | date-time | more info |
| object | object | more info |
| array | array | more info |
| boolean | boolean | more info |
| numeric | integer | more info |
| numeric | number | more info |
Performance
The connector uses batch or bulk writes. Large records may affect performance.
The connector should support variable batch sizes in future.
Supported sync modes
| Sync mode | Supported? |
|---|---|
| Full Refresh - Overwrite | Yes |
| Full Refresh - Append | Yes |
| Full Refresh - Overwrite + Deduped | No |
| Incremental Sync - Append | Yes |
| Incremental Sync - Append + Deduped | No |
Get started
Requirements
- Elasticsearch >= 7.x
- Configuration
- Endpoint URL [ex. https://elasticsearch.savantly.net:9423]
- Username [optional] (basic auth)
- Password [optional] (basic auth)
- CA certificate [optional]
- Api key ID [optional]
- Api key secret [optional]
- If you use authentication, your user needs permission to create missing indexes and/or
createdocuments.
CA certificate
You can fetch the CA certificate from /usr/share/elasticsearch/config/certs/http_ca.crt on the Elasticsearch server.
For a Docker-based Elasticsearch deployment, use:
docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt .
Here, es01 is the container’s name. For more details, visit https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html.
Setup
Enter the endpoint URL and select the authentication method. Then choose whether to use ‘upsert’ when indexing new documents.
Connect through an SSH tunnel
Airbyte can connect to an Elastic instance through an SSH tunnel. You may need this when direct access is unavailable or conflicts with your security policy. For example, the instance may lack a public IP address.
With an SSH tunnel, you configure Airbyte to connect through an intermediate server, also called a bastion server. The bastion must have direct access to the Elastic instance. Airbyte connects to the bastion, which then connects directly to the server.
This feature needs extra configuration when you create the source. The following steps explain each configuration value.
- Configure every source field as usual, except
SSH Tunnel Method. SSH Tunnel Methoddefaults toNo Tunnel(a direct connection). For an SSH tunnel, chooseSSH Key AuthenticationorPassword Authentication.- Choose
Key Authenticationwhen using an RSA private key to establish the SSH tunnel. See below for key-generation guidance. - Choose
Password Authenticationwhen using a password to establish the SSH tunnel.
- Choose
SSH Tunnel Jump Server Hostidentifies the intermediate (bastion) server that Airbyte connects to. Enter its hostname or IP address.SSH Connection Portis the bastion server port used for the SSH connection. The default SSH port is22. Keep this value unless you’ve explicitly changed the port.SSH Login Usernameis the username Airbyte uses to connect to the bastion server. It isn’t the TiDB username.- When using
Password Authentication, setSSH Login Usernameto the previous step’s user password. When usingSSH Key Authentication, this isn’t the TiDB password. It’s the OS-user password Airbyte uses to run commands on the bastion. - When using
SSH Key Authentication, setSSH Private Keyto the RSA Private Key used for the SSH connection. Include the complete key file, starting with-----BEGIN RSA PRIVATE KEY-----and ending with-----END RSA PRIVATE KEY-----.
Namespace support
This destination supports namespaces.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.2.0 | 2025-05-14 | 58586 | add path prefix |
| 0.1.6 | 2022-10-26 | 18341 | enforce ssl connection on cloud |
| 0.1.5 | 2022-10-24 | 18177 | add custom CA certificate processing |
| 0.1.4 | 2022-10-14 | 17805 | add SSH Tunneling |
| 0.1.3 | 2022-05-30 | 14640 | Include lifecycle management |
| 0.1.2 | 2022-04-19 | 11752 | Reduce batch size to 32Mb |
| 0.1.1 | 2022-02-10 | 1256 | Add ExitOnOutOfMemoryError connectors |
| 0.1.0 | 2021-10-13 | 7005 | Initial release. |