MongoDB
Prerequisites
- If you use Airbyte Open Source with the Postgres source connector, upgrade your platform to
v0.40.0-alphaor newer. Then upgrade your MongoDB connector to0.1.6or newer.
Output schema for destination-mongodb
Each stream is output to its own MongoDB collection. Each collection contains three fields:
_id: an identifier assigned to each processed document. Its MongoDB field type isString._airbyte_emitted_at: a timestamp showing when the event was pulled from the data source. Its MongoDB field type isTimestamp._airbyte_data: a JSON blob containing the event data. Its MongoDB field type isObject.
Supported sync modes
| Sync mode | Supported? |
|---|---|
| Full Refresh - Overwrite | Yes |
| Full Refresh - Append | Yes |
| Full Refresh - Overwrite + Deduped | No |
| Incremental Sync - Append | Yes |
| Incremental Sync - Append + Deduped | No |
Get started with Airbyte Cloud
Airbyte Cloud only supports connections to your MongoDB instance using TLS encryption. Otherwise, follow the open-source instructions below.
Get started with Airbyte Open Source
Requirements
To use the MongoDB destination, you’ll need:
- A MongoDB server
Permissions
You need a MongoDB user who can create collections and write documents. We recommend creating a dedicated Airbyte user for this purpose.
Target database
Choose an existing database or create one to store data synced from Airbyte.
Set up the MongoDB destination
You now have everything needed to configure MongoDB as a destination in the UI. You’ll need the following information:
- Standalone MongoDb instance
- Host: URL of the database
- Port: Port to use for connecting to the database
- TLS: indicates whether to create encrypted connection
- Replica Set
- Server addresses: the members of a replica set
- Replica Set: A replica set name
- MongoDb Atlas Cluster
- Cluster URL: URL of a cluster to connect to
- Database
- Username
- Password
For more information about the configuration parameters, see MongoDb Documentation.
Connect through an SSH tunnel
Airbyte can connect to a MongoDB instance through an SSH tunnel. You may need this when direct access isn’t possible or breaches security policy. For example, the instance may lack a public IP address.
With an SSH tunnel, you configure Airbyte to connect through an intermediate server, also called a bastion server. This server has direct access to MongoDB. Airbyte connects to the bastion, which then connects directly to the MongoDB server.
This feature requires extra configuration when you create the source. The steps below explain each setting.
- Configure every source field as usual, except
SSH Tunnel Method. SSH Tunnel Methoddefaults toNo Tunnel, which means a direct connection. For an SSH tunnel, chooseSSH Key AuthenticationorPassword Authentication.- Choose
Key Authenticationwhen using an RSA private key to establish the SSH tunnel. See below for key-generation guidance. - Choose
Password Authenticationwhen using a password to establish the SSH tunnel.
- Choose
SSH Tunnel Jump Server Hostidentifies the intermediate bastion server that Airbyte connects to. Enter its hostname or IP Address.SSH Connection Portis the bastion server port for the SSH connection. The default SSH port is22. Keep this value unless you’ve explicitly changed the port.SSH Login Usernameis the username Airbyte uses to connect to the bastion server. It isn’t the TiDB username.- If you use
Password Authentication, setSSH Login Usernameto the previous step’s User password. If you useSSH Key Authentication, enter the OS-user password, not the TiDB password. Airbyte uses this OS user to run commands on the bastion. - If you use
SSH Key Authentication, setSSH Private Keyto the RSA Private Key used for the SSH connection. Include the complete key file, starting with-----BEGIN RSA PRIVATE KEY-----and ending with-----END RSA PRIVATE KEY-----.
Naming conventions
The following information comes from the MongoDB Limits and Thresholds documentation.
Database name case sensitivity
MongoDB database names are case-insensitive, so names can’t differ only by character case.
Windows database name restrictions
For MongoDB deployments on Windows, database names can’t contain these characters: /. ”$<>:|?
Database names also can’t contain the null character.
Unix and Linux database name restrictions
For MongoDB deployments on Unix and Linux, database names can’t contain these characters: /. ”$
Database names also can’t contain the null character.
Database name length
Database names can’t be empty and must contain fewer than 64 characters.
Collection name restrictions
Collection names should start with an underscore or letter. They can’t:
- contain the $.
- be an empty string (e.g. "").
- contain the null character.
- begin with the system. prefix. (Reserved for internal use.)
Namespace support
This destination supports namespaces. The namespace maps to a MongoDB database.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.2.0 | 2023-06-27 | 27781 | License Update: Elv2 |
| 0.1.9 | 2022-11-08 | 18892 | Adds check for TLS flag |
| 0.1.8 | 2022-10-26 | 18280 | Adds SSH tunneling |
| 0.1.7 | 2022-09-02 | 16025 | Remove additionalProperties:false from spec |
| 0.1.6 | 2022-08-02 | 15211 | Fix standard mode |
| 0.1.5 | 2022-07-27 | 14561 | Change Airbyte Id from MD5 to SHA256 |
| 0.1.4 | 2022-02-14 | 10256 | (unpublished) Add -XX:+ExitOnOutOfMemoryError JVM option |
| 0.1.3 | 2021-12-30 | 8809 | Update connector fields title/description |
| 0.1.2 | 2021-10-18 | 6945 | Create a secure-only MongoDb destination |
| 0.1.1 | 2021-09-29 | 6536 | Destination MongoDb: added support via TLS/SSL |