Advanced Analytica Advanced Analytica: IBOM
← All connectors
DuckDB logo

Destination connector

DuckDB

Send governed data into DuckDB with a clear, practical setup guide.

Version 0.6.0 community beta

DuckDB

:::caution

Local file-based DBs don’t work in Airbyte Cloud or Kubernetes. Use MotherDuck when running in Airbyte Cloud.

:::

Overview

DuckDB is an in-process SQL OLAP database management system. Use this destination locally to run analytics workloads across multiple smaller sources. These sources can include GitHub repos, social media, local CSVs or other local files. This destination writes data to the MotherDuck service or a file on the host’s local filesystem running Airbyte.

For file-based DBs, data is written to /tmp/airbyte_local by default. To change the location, update Airbyte’s LOCAL_ROOT environment variable.

Use with MotherDuck

This DuckDB destination works with MotherDuck.

Specify a MotherDuck database

To use a MotherDuck-hosted database as your destination, provide its database URI in the destination_path configuration option. Include the standard md: database prefix.

:::caution

Don’t include your API token in the md: connection string. Your token may then appear in execution logs. Use the MotherDuck API Key setting instead.

:::

Authenticate with MotherDuck

To authenticate, provide your MotherDuck Service Credential as the motherduck_api_key configuration option.

Sync overview

Output schema

Each table contains three columns:

  • _airbyte_ab_id: a uuid that Airbyte assigns to each processed event.
  • _airbyte_emitted_at: a timestamp showing when the event was pulled from the data source.
  • _airbyte_data: a json blob containing the event data.

Performance

Your filesystem’s write speed will constrain this integration’s performance.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

Use local database files

The destination_path always starts with /local, whether you specify it or not. Any directory nesting within local maps to the local mount.

By default, the LOCAL_ROOT env variable in the .env file is set to /tmp/airbyte_local.

Docker mounts the local mount onto LOCAL_ROOT. By default, this substitutes /local with /tmp/airbyte_local.

:::caution

Make sure Docker Desktop can access /tmp. On MacOS, it must also access /private because /tmp links to /private. Otherwise, it won’t work. Allow access through “File sharing” in Settings -> Resources -> File sharing -> add the one or two above folder, then select the “Apply & restart” button.

:::

Example

  • Set destination_path to /local/destination.duckdb.
  • Use the /tmp/airbyte_local default for the local mount.
  • All data will then be written to /tmp/airbyte_local/destination.duckdb.

Access replicated data files

If your Airbyte instance runs on your current computer, open your browser and enter file:///tmp/airbyte_local. You can then view the replicated data locally. Follow these steps if that approach fails or your Airbyte instance runs on a remote server:

  1. Access the scheduler container using docker exec -it airbyte-server bash.
  2. Go to the default local mount using cd /tmp/airbyte_local.
  3. Go to your specified replicated file directory using cd /{destination_path}.
  4. Run duckdb {filename} to access data in a specific database file.

You can also copy the output file to your host machine. The following command copies it to your current working directory:

docker cp airbyte-server:/tmp/airbyte_local/{destination_path} .

Note: If Airbyte runs on Windows with Docker backed by WSL2, follow similar steps. Alternatively, see this link.

Troubleshooting

Error message Request failed: (UNAVAILABLE, RPC 'GET_WELCOME_PACK')

This error may mean your DuckDB client version is incompatible with your database version. DuckDB Destination connector version 0.6.0 or later uses DuckDB 1.4.x. Your database must therefore be compatible with DuckDB 1.4.x. To fix this, manually upgrade your database or revert to an earlier DuckDB Destination connector version.

For guidance on migrating between DuckDB versions, see the DuckDB Migration Guide.

Namespace support

This destination supports namespaces. The namespace maps to a DuckDB schema.

Changelog

Expand to review
VersionDatePull RequestSubject
0.6.02026-01-0670221Upgrade DuckDB engine to v1.4.2 (LTS release). Requires Python 3.10+. Backwards compatible with databases created using DuckDB 0.10.x or higher.
0.5.12025-03-0755256Version bump to align Docker and Poetry versions
0.5.02025-03-0747861Upgrade DuckDB engine version to v1.2.1
0.4.262024-10-2947861Update dependencies
0.4.252024-10-2847070Update dependencies
0.4.242024-10-1246845Update dependencies
0.4.232024-10-0546463Update dependencies
0.4.222024-09-2846145Update dependencies
0.4.212024-09-2145800Update dependencies
0.4.202024-09-1445480Update dependencies
0.4.192024-09-0745288Update dependencies
0.4.182024-08-3144952Update dependencies
0.4.172024-08-2444739Update dependencies
0.4.162024-08-2244530Update test dependencies
0.4.152024-08-1744215Update dependencies
0.4.142024-08-1243755Update dependencies
0.4.132024-08-1043536Update dependencies
0.4.122024-08-0343151Update dependencies
0.4.112024-07-2742753Update dependencies
0.4.102024-07-2042233Update dependencies
0.4.92024-07-1341882Update dependencies
0.4.82024-07-1041521Update dependencies
0.4.72024-07-0941253Update dependencies
0.4.62024-07-0641014Update dependencies
0.4.52024-06-2740215Replaced deprecated AirbyteLogger with logging.Logger
0.4.42024-06-2540354Update dependencies
0.4.32024-06-2340224Update dependencies
0.4.22024-06-2139947Update dependencies
0.4.12024-06-0438959[autopull] Upgrade base image to v1.2.1
0.4.02024-05-30#37515Upgrade DuckDB engine version to v0.10.3.
0.3.62024-05-21#38486[autopull] base image + poetry + up_to_date
0.3.52024-04-23#37515Add resource requirements declaration to metatadat.yml.
0.3.42024-04-16#36715Improve ingestion performance using pyarrow inmem view for writing to DuckDB.
0.3.32024-04-07#36884Fix stale dependency versions in lock file, add CLI for internal testing.
0.3.22024-03-20#32635Instrument custom_user_agent to identify Airbyte-Motherduck connector usage.
0.3.12023-11-18#32635Upgrade DuckDB version to v0.9.2.
0.3.02022-10-23#31744Upgrade DuckDB version to v0.9.1. Required update for all MotherDuck users. Note, this is a BREAKING CHANGE for users who may have other connections using versions of DuckDB prior to 0.9.x. See the 0.9.0 release notes for more information and for upgrade instructions.
0.2.12022-10-20#30600Fix: schema name mapping
0.2.02022-10-19#29428Add support for MotherDuck. Upgrade DuckDB version to `v0.8“.
0.1.02022-10-1417494New DuckDB destination
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.