Doris
destination-doris is built on Apache Doris stream load. It supports batch rollback and sends HTTP/HTTPS PUT requests.
Sync overview
Output schema
Each stream is written to its own table in Doris. Each table contains three columns:
_airbyte_ab_id: a UUID Airbyte assigns to each processed event. In Doris, the column type isVARCHAR(40)._airbyte_emitted_at: a timestamp showing when the event was pulled from the data source. In Doris, the column type isBIGINT._airbyte_data: a JSON blob containing the event data. In Doris, the column type isString.
Features
Use the following table format:
| Feature | Supported?(Yes/No) | Notes |
|---|---|---|
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Append + Deduped | No | it will soon be realized |
| For databases, WAL/Logical replication | Yes |
Performance considerations
Writes are processed in batches. Small record batches may affect performance. Importing multiple tables creates multiple Doris stream load transactions. Split imports as much as possible.
Getting started
Requirements
To use the Doris destination, you’ll need:
- A Doris server running version 0.14 or above
- Airbyte must be able to access your Doris FE HTTP port.
- Airbyte must be able to access your Doris database host.
- Your Doris user must have read and write permissions for specific tables.
Target database and tables
Choose a database to store data synced from Airbyte. Prepare the tables that will store this data. Match their column names and order as closely as possible.
Set up access parameters
- Host
- HttpPort
- QueryPort
- Username
- Password
- Database
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.1.0 | 2022-11-14 | 17884 | Initial Commit |