Advanced Analytica Advanced Analytica: IBOM
← All connectors
S3 Data Lake logo

Destination connector

S3 Data Lake

Send governed data into S3 Data Lake with a clear, practical setup guide.

Version 0.4.2 certified alpha

S3 Data Lake

Set up the S3 Data Lake destination connector using this guide.

This connector provides Airbyte’s official support for the Iceberg protocol on S3. It writes the Iceberg table format to S3 or a compatible storage backend. It uses a supported Iceberg catalog.

Prerequisites

You need two things to use the S3 Data Lake connector.

  1. An S3 storage bucket or S3-compatible storage backend.

  2. A supported Iceberg catalog. The connector currently supports these catalogs:

    • REST
    • AWS Glue
    • Nessie
    • Polaris

If you use Airbyte Cloud with IP-based access controls, add Airbyte’s IP addresses to your allowlist.

Set up the connector

Follow these steps to configure your S3 storage and Iceberg catalog permissions.

Set up S3 permissions

Create a bucket policy, then configure authentication.

:::note For S3-compatible storage, set the S3 Endpoint field to your storage endpoint, such as http://localhost:9000. Leave this field empty for Amazon S3. :::

Create a bucket policy

Create your bucket policy.

  1. Open the IAM console.

  2. In the IAM dashboard, select Policies > Create Policy.

  3. Select the JSON tab and paste the following JSON into the Policy editor. Replace the highlighted bucket name with your own.

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:ListAllMyBuckets",
            "s3:GetObject*",
            "s3:PutObject",
            "s3:PutObjectAcl",
            "s3:DeleteObject",
            "s3:ListBucket*"
          ],
          "Resource": [
            // highlight-next-line
            "arn:aws:s3:::YOUR_BUCKET_NAME/*",
            // highlight-next-line
            "arn:aws:s3:::YOUR_BUCKET_NAME"
          ]
        }
      ]
    }

    :::note Object-level permissions alone can’t authenticate. Include bucket-level permissions as shown in the preceding example. :::

  4. Click Next, give your policy a descriptive name, then click Create policy.

Authenticate {#authentication-s3}

In most cases, authenticate with an IAM user. With Airbyte Cloud and the Glue catalog, you can authenticate with an IAM role.

Authenticate with an IAM user (Self-Managed or Cloud, with any catalog)

Use a new or existing Access Key ID and Secret Access Key.

  1. In the IAM dashboard, click Users.

  2. For an existing IAM user, select the user, then click Add permissions > Add permission. To create a user, click Add users.

  3. Click Attach policies directly, then select the checkbox for your policy. Click Next > Add permissions.

  4. Click the Security credentials tab > Create access key. The AWS console asks you to select a use case and add optional access-key tags.

  5. Click Create access key and record your keys.

  6. In Airbyte, enter the keys in the connector’s AWS Access Key ID and AWS Secret Access Key fields.

Authenticate with an IAM role (Cloud with Glue catalog only)

:::note An Airbyte team member must enable S3 authentication with an IAM role. To use this feature, contact the Sales team. :::

  1. In the IAM dashboard, click Roles, then Create role.

  2. Choose the AWS account trusted entity type.

  3. Set up a trust relationship for the role. This lets the Airbyte instance’s AWS account assume the role. You must also specify an external ID. This secret key is known by both the trusting service, Airbyte, and the trusted role you’re creating. The ID prevents the “confused deputy” problem. Use your Airbyte workspace ID as the External ID. You can find it in your workspace page’s URL. Edit the trust relationship policy to include the external ID:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::094410056844:user/delegated_access_user"
                },
                "Action": "sts:AssumeRole",
                "Condition": {
                    "StringEquals": {
                        "sts:ExternalId": "{your-airbyte-workspace-id}"
                    }
                }
            }
        ]
    }
  4. Complete the role creation and save the Role ARN for later.

  5. Select Attach policies directly, then find and select the checkbox for your new policy. Click Next, then Add permissions.

  6. In Airbyte, select Glue as the catalog. Enter the Role ARN in the Role ARN field.

Set up the Iceberg catalog

Complete the remaining setup for your chosen catalog.

REST

Enter your REST catalog’s URI. You may also need to enter the default namespace.

AWS Glue

  1. Add a second statement to your existing S3 policy to grant Glue permissions. Glue actions require Glue resource ARNs, not S3 ARNs. They must therefore use a separate statement.

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:ListAllMyBuckets",
            "s3:GetObject*",
            "s3:PutObject",
            "s3:PutObjectAcl",
            "s3:DeleteObject",
            "s3:ListBucket*"
          ],
          "Resource": [
            "arn:aws:s3:::YOUR_BUCKET_NAME/*",
            "arn:aws:s3:::YOUR_BUCKET_NAME"
          ]
        },
        // highlight-start
        {
          "Effect": "Allow",
          "Action": [
            "glue:TagResource",
            "glue:UnTagResource",
            "glue:BatchCreatePartition",
            "glue:BatchDeletePartition",
            "glue:BatchDeleteTable",
            "glue:BatchGetPartition",
            "glue:CreateDatabase",
            "glue:CreateTable",
            "glue:CreatePartition",
            "glue:DeletePartition",
            "glue:DeleteTable",
            "glue:GetDatabase",
            "glue:GetPartition",
            "glue:GetPartitions",
            "glue:GetTable",
            "glue:GetTables",
            "glue:UpdateDatabase",
            "glue:UpdatePartition",
            "glue:UpdateTable"
          ],
          "Resource": [
            "arn:aws:glue:*:*:catalog",
            "arn:aws:glue:*:*:database/*",
            "arn:aws:glue:*:*:table/*/*"
          ]
        }
        // highlight-end
      ]
    }

    :::tip To restrict Glue access by region and account, replace the * wildcards in the Glue ARNs. Use your AWS region and account ID, such as arn:aws:glue:us-east-1:123456789012:catalog. :::

  2. Set the warehouse location option to s3://<bucket name>/path/within/bucket.

  3. If you use Airbyte Cloud with an IAM role, set the Role ARN option. Use the value you recorded while setting up authentication on S3.

  4. To replace an existing Glue table with an Airbyte-managed Iceberg table, drop the Glue table. Otherwise, you’ll encounter the error Input Glue table is not an iceberg table: <your table name>.

    Dropping Glue tables from the console may not immediately delete them. Wait for AWS to finish its background processing, or use the AWS API to drop all table versions.

  5. If you use AWS Lake Formation, grant these permissions through Lake Formation:

    1. Grant Data location_access on the S3 path.
    2. To let the connector create databases for you, also grant Create database on the catalog.
    3. As an advanced option, you can create the databases manually. To restrict the connector to those databases, grant Create table, Describe on them.

Nessie

Complete two steps to authenticate with Nessie.

  1. Set your Nessie catalog’s URI and an access token for catalog authentication.

  2. Set the Warehouse location option to s3://<bucket name>/path/within/bucket.

Polaris

Follow these steps to authenticate with Apache Polaris.

  1. Set up your Polaris catalog and create a principal with the required permissions. See the Apache Polaris documentation for detailed setup instructions.

  2. When you create a Polaris principal, you’ll receive OAuth credentials: Client ID and Client Secret. Keep these credentials secure.

  3. Grant the required privileges to your principal’s catalog role. Choose either:

    Option A: Grant the broad CATALOG_MANAGE_CONTENT privilege (recommended for simplicity):

    • This single privilege lets the connector manage tables and namespaces in the catalog

    Option B: Grant specific granular privileges:

    • TABLE_LIST - List tables in a namespace
    • TABLE_CREATE - Create new tables
    • TABLE_DROP - Delete tables
    • TABLE_READ_PROPERTIES - Read table metadata
    • TABLE_WRITE_PROPERTIES - Update table metadata
    • TABLE_WRITE_DATA - Write data to tables
    • NAMESPACE_LIST - List namespaces
    • NAMESPACE_CREATE - Create new namespaces
    • NAMESPACE_READ_PROPERTIES - Read namespace metadata
  4. Enter the following information in the Airbyte connector configuration:

    • Polaris Server URI: Your Polaris server’s base URL. For example: http://localhost:8181/api/catalog
    • Catalog Name: The catalog name you created in Polaris, for example quickstart_catalog
    • Client ID: The OAuth Client ID provided when you created the principal
    • Client Secret: The OAuth Client Secret provided when you created the principal
    • OAuth Scope: The authentication scope to request, using the format PRINCIPAL_ROLE:<role_name>. For example: PRINCIPAL_ROLE:catalog_admin
    • OAuth2 Server URI: The OAuth2 token endpoint, for example https://polaris.example.com/oauth/tokens. This field is currently optional. A future connector version will require it, so set it if you know it.
    • Default namespace: The namespace used for table identifiers when the destination namespace is “Destination-defined” or “Source-defined”
  5. Set the Warehouse location option to s3://<bucket name>/path/within/bucket.

  6. Make sure your Polaris catalog has the correct storage credentials for accessing your S3 bucket.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedYes
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedYes

Output schema

Iceberg schema generation

For each stream, Airbyte maps top-level fields to Iceberg fields. Airbyte writes objects and unions as string columns containing serialised JSON. Arrays retain their structure. If the source schema declares an array’s item type, Airbyte writes an Iceberg list column using the mapped item type. Arrays without a declared item type become string columns containing serialised JSON.

The following table shows the complete mapping between Airbyte and Iceberg types.

Airbyte typeIceberg type
BooleanBoolean
DateDate
IntegerLong
NumberDouble (String for primary keys)
StringString
Time with timezone*Time
Time without timezoneTime
Timestamp with timezone*Timestamp with timezone
Timestamp without timezoneTimestamp without timezone
ObjectString (JSON-serialized value)
Array with a declared item typeList of the mapped item type
Array without a declared item typeString (JSON-serialized value)
UnionString (JSON-serialized value)

*Airbyte converts the time with timezone and timestamp with timezone types to Coordinated Universal Time (UTC) before writing to the Iceberg file.

Manage schema evolution

The connector never rewrites existing Iceberg data files. Airbyte can therefore handle only specific source schema changes.

  • Adding or removing a column
  • Widening a column
  • Changing the primary key

Choose one of these options to manage schema evolution.

  • To handle unsupported schema changes automatically, use Full Refresh - Overwrite as your sync mode.

  • To handle unsupported changes individually, wait for a sync to fail. Then restore it using one of these options:

    • Edit your table schema directly in Iceberg.
    • Refresh your connection in Airbyte.
    • Clear your connection in Airbyte.

Naming

Like most Airbyte destination connectors, S3 Data Lake may modify identifiers for destination compatibility. These identifiers include stream names, namespaces and column names.

When you use AWS Glue, the connector:

Column names

By default, Iceberg column names match the source exactly, such as Foo.Bar or userId. Some query engines can’t read mixed-case or special-character column names through a Glue catalog.

Enable the Normalize Column Names option to normalise each column name before writing it. The connector applies these rules in order:

  1. Removes accents and other combining marks (café becomes cafe).
  2. Replaces whitespace and non-alphanumeric characters, except underscores, with an underscore (Foo.Bar becomes Foo_Bar).
  3. Converts the name to lowercase (Foo_Bar becomes foo_bar).
  4. Adds an underscore prefix if the result is a listed SQL reserved word. Common query engines reject these words as unquoted column names: constraint, current_date, current_time, current_timestamp, current_user, localtime, localtimestamp. For example, CURRENT_DATE becomes _current_date.

A numeric suffix makes names unique when they normalise to the same value. For example, ID and id become id and id_1. Airbyte’s _airbyte_* columns already meet these rules and are never renamed. The option applies to every catalog type. It only affects top-level column names, not keys within JSON-serialised object columns.

:::caution Changing the option on an existing connection Enabling Normalize Column Names after tables exist changes their column names. The connector won’t sync until you clear the stream’s data and run a full refresh. This refresh recreates the table with the new column names. Clear the affected streams before disabling the option too. :::

Deduplication

The connector supports deduplication using a merge-on-read strategy.

Number-type primary keys

For an Airbyte Number primary key, the connector stores an Iceberg StringType instead of DoubleType. This preserves correct deduplication because Iceberg identifier fields don’t support DoubleType. However, ordering and comparison become lexicographic rather than numeric, for example "9" > "10". You may need to adjust downstream queries that assume numeric comparison for these fields.

Primary key assumptions

The S3 Data Lake connector assumes one of these conditions is true:

  • The source never emits the same primary key twice during one sync attempt.
  • When a source emits the same primary key repeatedly, records always follow cursor order from oldest to newest.

If neither condition applies, older records may take precedence over newer records in Iceberg. If this occurs, use append or overwrite as your sync modes.

An unknown number of API source streams don’t meet these conditions. Airbyte knows Stripe and Monday don’t, and others probably don’t either.

Branching and data availability

Iceberg supports Git-like semantics over your data. The connector uses these semantics to make syncs resilient.

  • Each microbatch creates a new snapshot during every sync.

  • During truncate syncs, the connector writes refreshed data to a unique airbyte_staging_<uuid> branch. After a successful sync, it replaces the main branch with the staging branch. Most query engines target the main branch, so your data remains available during the truncate sync. At completion, the connector atomically swaps it to the new version. The connector removes the staging branch after a successful promotion. After failed syncs, it preserves the branch so retries can recover staged data.

Replace a branch

At the end of a stream sync, the connector replaces the current main branch with its unique staging branch. The connector intentionally avoids fast-forwarding to handle potential compaction issues better. Important Warning: This process removes changes made to the main branch outside Airbyte’s operations after the sync starts.

Compaction

:::caution Do not run compaction during a truncate refresh sync to prevent data loss. During a truncate refresh sync, the system deletes every file outside the latest generation. This includes:

  • Files without generation IDs (compacted files)
  • Files from previous generations

If compaction runs during the sync, it deletes current-generation files and causes data loss. The system identifies generations by parsing generation IDs from file names. :::

Considerations and limitations

This section explains known considerations and limitations when this Iceberg destination interacts with other products.

Snowflake

Airbyte marks older record versions as outdated using Iceberg row-level deletes. Snowflake doesn’t recognise native Iceberg row-level deletes for tables using external catalogs, such as Glue (see Snowflake’s docs). As a result, queries return every version of a record.

For example, this table contains three versions of the ‘Alice’ record.

idnameupdated_at_airbyte_extracted_at
1Alice2024-03-01 10:002024-03-01 10:10
1Alice2024-03-02 12:002024-03-02 12:10
1Alice2024-03-03 14:002024-03-03 14:10

To identify outdated records, generate a flag. Airbyte generates an airbyte_extracted_at metadata field to support this.

row_number() over (partition by {primary_key} order by {cursor}, _airbyte_extracted_at)) != 1 OR _ab_cdc_deleted_at IS NOT NULL as is_outdated;

You can now identify the latest ‘Alice’ record by checking whether is_outdated is false.

idnameupdated_at_airbyte_extracted_atrow_numberis_outdated
1Alice2024-03-01 10:002024-03-01 10:103True
1Alice2024-03-02 12:002024-03-02 12:102True
1Alice2024-03-03 14:002024-03-03 14:101False

Namespace support

This destination supports namespaces.

Changelog

Expand to review
VersionDatePull RequestSubject
0.4.22026-10-2289644Upgrade to Bulk CDK 1.1.9: fix critical CVEs in Apache MINA (mina-core 2.0.31), MINA SSHD 2.20.0 and BouncyCastle 1.86; netty 4.1.139, Hadoop 3.4.3, Parquet 1.18.1; AWS SDK 2.57.2; drop Jetty from the Hadoop client.
0.4.12026-10-0987621Upgrade to Bulk CDK 1.1.1.
0.4.02026-09-1585827Add normalize_column_names option: lowercase column names, replace non-alphanumeric characters with underscores, and prefix SQL reserved keywords.
0.3.532026-08-2484994Upgrade to Bulk CDK 1.0.25.
0.3.522026-06-2380349Remove awssdk:bundle fat jar to fix OOMKilled during CHECK operations
0.3.512026-06-1579123Update Apache Iceberg dependencies.
0.3.502026-06-0879112Use unique staging branches and clean them up after each sync.
0.3.492026-05-2078232Upgrade CDK to 1.0.13
0.3.482026-05-0477677Add configurable flush batch size for aggregate publishing.
0.3.472026-04-1776410Upgrade CDK to 1.0.9.
0.3.462026-03-3075628Upgrade CDK to 1.0.7: fix sort order handling during schema evolution.
0.3.452026-03-1374326Upgrade CDK to 1.0.5: Number-type primary keys are now stored as String (enabling dedup on numeric PKs); fix schema evolution when replacing identifier columns
0.3.442026-02-0972848Enable Speed.
0.3.432026-01-3072482Release on dataflow.
0.3.43-rc.12026-01-2972443Update to dataflow.
0.3.422026-01-1270205Implement support for scope and OAuth server URI attributes for Polaris catalog
0.3.412025-11-0769232Upgrade to Bulk CDK 0.1.69. Changes to handle changes in commit patterns
0.3.402025-11-0569133Upgrade to Bulk CDK 0.1.61.
0.3.392025-10-1568108Implement Polaris support
0.3.382025-10-0767005Fix: Treat empty string role_arn as null to prevent misleading config errors
0.3.372025-10-0767150Fix check operation to use unique table names, preventing conflicts with stale metadata and concurrent operations
0.3.362025-09-2566711CHECK operation uses configured default dataset instead of airbyte_test_namespace
0.3.352025-07-2363746Remove unnecessary properties from table
0.3.342025-07-1562952Update CDK version
0.3.332025-07-1162888Update CDK version to handle compaction issue when deleting files in a truncate refresh scenario
0.3.322025-07-0862852Fix metadata (revert accidental archiving)
0.3.312025-07-0762835Pin to latest CDK version 0.522
0.3.302025-07-0762105ReplaceBranch to staging from main instead of fast forwarding
0.3.292025-06-1361588Publish version to account for possible duplicate publishing in pipeline. Noop change. WARNING: THIS HAS A BUG. DO NOT USE.
0.3.282025-05-0759710CDK backpressure bugfix
0.3.272025-04-2158146Upgrade to latest CDK
0.3.262025-04-1758104Chore: Now passing a string around for the region
0.3.252025-04-1658085Internal refactoring
0.3.242025-03-2756435Bug fix: Correctly handle non-positive numbers.
0.3.232025-03-2556395Bug fix: Correctly coerce values inside nested arrays.
0.3.222025-03-2456355Upgrade to airbyte/java-connector-base:2.0.1 to be M4 compatible.
0.3.212025-03-22#56347Bugfix: stream start does not always await iceberg setup
0.3.202025-03-24#55849Internal refactoring
0.3.192025-03-19#55798CDK: Typing improvements
0.3.182025-03-18#55811CDK: Pass DestinationStream around vs Descriptor
0.3.172025-03-13#55737CDK: Pass DestinationRecordRaw around instead of DestinationRecordAirbyteValue
0.3.162025-03-13#55755Exclude number fields from identifier fields
0.3.152025-02-28#54724Certify connector
0.3.142025-02-14#53241New CDK interface; perf improvements, skip initial record staging
0.3.132025-02-14#53697Internal refactor
0.3.122025-02-12#53170Improve documentation, tweak error handling of invalid schema evolution
0.3.112025-02-12#53216Support arbitrary schema change in overwrite / truncate refresh / clear sync
0.3.102025-02-11#53622Enable the Nessie integration tests
0.3.92025-02-10#53165Very basic usability improvements and documentation
0.3.82025-02-10#52666Change the chunk size to 1.5Gb
0.3.72025-02-07#53141Adding integration tests around the Rest catalog
0.3.62025-02-06#53172Internal refactor
0.3.52025-02-06#53164Improve error message on null primary key in dedup mode
0.3.42025-02-05#53173Tweak spec wording
0.3.32025-02-05#53176Fix time_with_timezone handling (values are now adjusted to UTC)
0.3.22025-02-04#52690Handle special characters in stream name/namespace when using AWS Glue
0.3.12025-02-03#52633Fix dedup
0.3.02025-01-31#52639Make the database/namespace a required field
0.2.232025-01-27#51600Internal refactor
0.2.222025-01-22#52081Implement support for REST catalog
0.2.212025-01-27#52564Fix crash on stream with 0 records
0.2.202025-01-23#52068Add support for default namespace (/database name)
0.2.192025-01-16#51595Clarifications in connector config options
0.2.182025-01-15#51042Write structs as JSON strings instead of Iceberg structs.
0.2.172025-01-14#51542New identifier fields should be marked as required.
0.2.162025-01-14#51538Update identifier fields if incoming fields are different than existing ones
0.2.152025-01-14#51530Set AWS region for S3 bucket for nessie catalog
0.2.142025-01-14#50413Update existing table schema based on the incoming schema
0.2.132025-01-14#50412Implement logic to determine super types between iceberg types
0.2.122025-01-10#50876Add support for AWS instance profile auth
0.2.112025-01-10#50971Internal refactor in AWS auth flow
0.2.102025-01-09#50400Add S3DataLakeTypesComparator
0.2.92025-01-09#51022Rename all classes and files from Iceberg V2
0.2.82025-01-09#51012Rename/Cleanup package from Iceberg V2
0.2.72025-01-09#50957Add support for GLUE RBAC (Assume role)
0.2.62025-01-08#50991Initial public release.
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.