SurrealDB
SurrealDB is an end-to-end, cloud-native database designed for modern applications. These include web, mobile, serverless, Jamstack, backend and traditional applications. SurrealDB can simplify your database and API infrastructure. It can also reduce development time and help you build secure, performant apps quickly and cost-effectively.
Use this guide to set up the SurrealDB destination connector.
Output schema
SurrealDB writes each stream to its own table. Each table contains the following Airbyte columns:
_airbyte_raw_id: a uuid that Airbyte assigns to each processed event. The SurrealDB column type isstring. The connector uses this value as each record’s ID in the destination SurrealDB table._airbyte_extracted_at: a timestamp showing when Airbyte pulled the event from the data source. The SurrealDB column type isdatetime._airbyte_meta: additional information about the record. The SurrealDB column type isobject.
Each top-level field in the stream’s JSON schema becomes its own field. Each field is defined as an optional SurrealDB type.
The types are string, int, number, bool, object, array or datetime for date-time strings, and any for untyped fields.
SurrealDB reserves id for the record ID. Therefore, a source field named id is written as _airbyte_source_id.
Supported sync modes
| Sync mode | Supported? |
|---|---|
| Full Refresh - Overwrite | Yes |
| Full Refresh - Append | Yes |
| Full Refresh - Overwrite + Deduped | Yes |
| Incremental Sync - Append | Yes |
| Incremental Sync - Append + Deduped | Yes |
Getting started
Requirements
You need SurrealDB v2.2.0 or above to use the SurrealDB destination.
Network access
Airbyte must be able to access your SurrealDB database. If it’s within a VPC, allow access from the IP you use to expose Airbyte.
Permissions
You need a user with DEFINE TABLE, DEFINE INDEX, UPSERT, SELECT, REMOVE permissions. Creating a dedicated Airbyte user for this purpose is highly recommended.
Run the following commands against your database to create a dedicated database user:
DEFINE USER airbyte ON ROOT PASSWORD "your_password_here" ROLES OWNER;
Target database
SurrealDB doesn’t differentiate between a database and a schema. A database effectively acts as the schema containing all tables. Choose an existing database or create a new one. This becomes the default database or schema when the source doesn’t provide a namespace.
Set up the SurrealDB destination in Airbyte
Configure the following details for the SurrealDB destination:
- Host
- Port
- Username (
Tokenor a pair ofUsername/Passwordis required) - Password (
Tokenor a pair ofUsername/Passwordis required) - Namespace
- Database
- Token (
Tokenor a pair ofUsername/Passwordis required)
Known limitations
The SurrealDB destination converts all table, schema and column identifiers to lowercase.
Connect through an SSH tunnel
Airbyte can connect to a SurrealDB instance through an SSH tunnel. You might need this when direct database access is unavailable or conflicts with your security policy. For example, the database might not have a public IP address.
With an SSH tunnel, Airbyte connects to an intermediate server, also called a bastion server. This server has direct database access. Airbyte connects to the bastion, which then connects directly to the database server.
This feature needs additional configuration when you create the destination. The steps below explain each setting.
- Configure all destination fields as usual, except
SSH Tunnel Method. SSH Tunnel Methoddefaults toNo Tunnel, which means a direct connection. To use an SSH tunnel, chooseSSH Key AuthenticationorPassword Authentication.- Choose
Key Authenticationwhen using an RSA private key to establish the SSH tunnel. See below for more information about generating this key. - Choose
Password Authenticationwhen using a password to establish the SSH tunnel.
- Choose
SSH Tunnel Jump Server Hostidentifies the intermediate bastion server that Airbyte connects to. Enter a hostname or IP address.SSH Connection Portis the bastion server port used for the SSH connection. The default SSH port is22. Use this default unless you’ve explicitly changed the port.SSH Login Usernameis the username Airbyte uses to connect to the bastion server. This isn’t the SurrealDB username.- If you’re using
Password Authentication, setSSH Login Usernameto the previous step’s user password. If you’re usingSSH Key Authentication, leave this blank. This isn’t the SurrealDB password. It’s the OS-user password Airbyte uses to run commands on the bastion. - If you’re using
SSH Key Authentication, setSSH Private Keyto the RSA Private Key used for the SSH connection. Include the key file’s full contents, starting with-----BEGIN RSA PRIVATE KEY-----and ending with-----END RSA PRIVATE KEY-----.
Namespace support
This destination supports namespaces. SurrealDB doesn’t differentiate between a database and a schema. The configured database acts as the default namespace.