Advanced Analytica Advanced Analytica: IBOM
← All connectors
Snowflake logo

Destination connector

Snowflake

Send governed data into Snowflake with a clear, practical setup guide.

Version 5.2.2 certified generally available

import KeypairExample from ‘@site/static/_snowflake_keypair_generation.md’;

Snowflake

Set up the Snowflake destination connector by creating the required Snowflake entities and configuring the connector in the Airbyte UI. These entities include a warehouse, database, schema, user, and role.

Follow this guide to set up the Snowflake destination connector.

:::danger Username and password authentication is deprecated From version 5.0.0, username and password authentication is deprecated. It will be removed in a future release. Snowflake now recommends Key pair authentication or a programmatic access token for connections.

This change aligns with Snowflake’s deprecation of single-factor password sign-ins. Snowflake is enforcing strong authentication per account on a rolling basis between August and October 2026. Once enforced, password-only Airbyte logins to your account will fail.

If you currently use username and password authentication, see Snowflake Migration Guide. It explains how to migrate to key pair authentication or a programmatic access token. :::

Prerequisites

  • A Snowflake account with the ACCOUNTADMIN role. If your account doesn’t have the ACCOUNTADMIN role, ask your Snowflake administrator to set it up.

Setup

Step 1: Set up key pair authentication

Step 2: Create Airbyte entities in Snowflake

Create Airbyte-specific Snowflake entities to set up the destination connector. You need a warehouse, database, schema, service user, and role with OWNERSHIP permission. These entities let Airbyte write data into Snowflake, track related costs, and control permissions at a granular level.

Run the following script in a new Snowflake worksheet to create the entities:

  1. Log into your Snowflake account.

  2. Edit the script below if you want to rename the resources. Replace <public_key_value> with the RSA public key you generated in Step 1.

    Note: Follow the Snowflake identifier requirements when renaming resources.

-- set variables (these need to be uppercase)
set airbyte_role = 'AIRBYTE_ROLE';
set airbyte_username = 'AIRBYTE_USER';
set airbyte_warehouse = 'AIRBYTE_WAREHOUSE';
set airbyte_database = 'AIRBYTE_DATABASE';
set airbyte_schema = 'AIRBYTE_SCHEMA';

begin;

-- create Airbyte role
use role securityadmin;
create role if not exists identifier($airbyte_role);
grant role identifier($airbyte_role) to role SYSADMIN;

-- create Airbyte service user with key pair authentication
create user if not exists identifier($airbyte_username)
type = service
default_role = $airbyte_role
default_warehouse = $airbyte_warehouse;

grant role identifier($airbyte_role) to user identifier($airbyte_username);

-- assign the RSA public key to the service user
-- Replace <public_key_value> with the contents of your rsa_key.pub file (excluding the
-- -----BEGIN PUBLIC KEY----- and -----END PUBLIC KEY----- header/footer lines).
alter user identifier($airbyte_username) set rsa_public_key='<public_key_value>';

-- change role to sysadmin for warehouse / database steps
use role sysadmin;

-- create Airbyte warehouse
create warehouse if not exists identifier($airbyte_warehouse)
warehouse_size = xsmall
warehouse_type = standard
auto_suspend = 60
auto_resume = true
initially_suspended = true;

-- create Airbyte database
create database if not exists identifier($airbyte_database);

-- grant Airbyte warehouse access
grant USAGE
on warehouse identifier($airbyte_warehouse)
to role identifier($airbyte_role);

-- grant Airbyte database access
grant OWNERSHIP
on database identifier($airbyte_database)
to role identifier($airbyte_role);

commit;
  1. Run the script using the Worksheet page or Snowsight. In the Classic Console, select the All Queries checkbox. In Snowsight, select and highlight the entire query.

:::info Airbyte automatically creates the required schemas in your Snowflake destination database. To enable this, give the connection user CREATE SCHEMA privileges on the target database. You can also create schemas manually. The connection user must have OWNERSHIP privileges on those schemas. This lets the connection manage the tables and other objects required for the integration. :::

:::info Cost-saving tip Snowflake bills for compute by the second. The warehouse resumes and incurs charges each time Airbyte loads data. To reduce costs, use a dedicated X-Small warehouse with a one-minute auto-suspend timeout for Airbyte syncs. This is the smallest available compute tier. It shuts the warehouse down quickly after each load, minimising idle compute charges. The script above already configures these settings (warehouse_size = xsmall, auto_suspend = 60). :::

Step 3: Configure network policies

By default, Snowflake accepts connections from any computer or device IP address. A security administrator with the SECURITYADMIN role, or higher, can create a network policy. This policy can allow or deny one IP address or a list of addresses.

If you use Airbyte Cloud, add Airbyte’s IP addresses to your Snowflake network policy allowlist.

Run the SHOW PARAMETERS command to check for a network policy on your account or a specific user.

Account
SHOW PARAMETERS LIKE 'network_policy' IN ACCOUNT;
User
SHOW PARAMETERS LIKE 'network_policy' IN USER <username>;

For more information, see the official Snowflake documentation

Step 4: Choose a data loading method

Airbyte uses Snowflake’s Internal Stage to load data.

Airbyte creates internal stages in the internal schema (raw_data_schema, default airbyte_internal). The role needs CREATE STAGE only on that schema, not on the output schemas.

Ensure the database and schema have the USAGE privilege.

Step 5: Add Snowflake as an Airbyte destination

Open the Airbyte UI and add Snowflake as a destination. Authenticate with the private key generated in Step 1 or, if you prefer, a programmatic access token.

FieldDescription
HostThe host domain of the Snowflake instance, ending with snowflakecomputing.com. Use the format accountname.snowflakecomputing.com or, for accounts that use a region-based locator, accountname.region.cloud.snowflakecomputing.com. Example: accountname.us-east-2.aws.snowflakecomputing.com
RoleThe role you created in Step 2 for Airbyte to access Snowflake. Example: AIRBYTE_ROLE
WarehouseThe warehouse you created in Step 2 for Airbyte to sync data into. Example: AIRBYTE_WAREHOUSE
DatabaseThe database you created in Step 2 for Airbyte to sync data into. Example: AIRBYTE_DATABASE
SchemaThe default schema used as the target schema for all statements issued from the connection that do not explicitly specify a schema name.
UsernameThe service user you created in Step 2 to allow Airbyte to access the database. Example: AIRBYTE_USER
Authorization MethodHow Airbyte authenticates as the user above. Choose Key Pair Authentication (recommended) or Programmatic Access Token. Username and Password is deprecated; see the migration guide.
Private KeyFor Key Pair Authentication. The private key from the key pair you generated in Step 1. Paste the full contents of your rsa_key.p8 file, including the -----BEGIN ... PRIVATE KEY----- header and footer.
Passphrase (Optional)For Key Pair Authentication. The passphrase for the private key, if the key was generated with encryption. Leave blank if the key is unencrypted.
Programmatic Access TokenFor Programmatic Access Token authentication. The token secret generated for the user above. See Programmatic access token authentication.
CDC deletion modeWhether to execute CDC deletions as hard deletes or soft deletes. Hard deletes propagate source deletions to the destination. Soft deletes leave a tombstone record in the destination. Defaults to hard deletes.
JDBC URL Params (Optional)Additional properties to pass to the JDBC URL string when connecting to the database formatted as key=value pairs separated by the symbol &. Example: key1=value1&key2=value2&key3=value3
Legacy raw tables (Optional)Write the legacy raw tables format for backwards compatibility with older versions of this connector. See Output schema. The data format in _airbyte_data is fairly stable but there are no guarantees that other metadata columns will remain the same in future versions.
Airbyte Internal Table Dataset Name (Optional)The schema used for Airbyte’s internal tables. In legacy raw tables mode, the raw tables are stored in this schema. Defaults to airbyte_internal.
Trim Whitespace from String Fields (Optional)Whether Snowflake should trim leading and trailing whitespace from fields during data loading. Disable this option if leading or trailing whitespace in string fields is meaningful and should be preserved.
Data Retention Period (Optional)The number of days of Snowflake Time Travel to enable on tables. A nonzero value incurs increased storage costs in your Snowflake instance. Defaults to 1.
Decimal Data Type (Optional)Determines which Snowflake data type Airbyte uses for columns with the Airbyte number type: NUMBER(38,9) (recommended) or FLOAT (default). See Data type map for guidance on choosing between them.

Programmatic access token authentication

You can use a Snowflake programmatic access token (PAT) instead of key pair authentication. Airbyte presents the token to Snowflake in place of a password for the user entered in Username.

  1. Create a PAT for the Airbyte user in Snowflake:

    ALTER USER <user_name> ADD PROGRAMMATIC ACCESS TOKEN <token_name>
      ROLE_RESTRICTION = '<airbyte_role>'
      DAYS_TO_EXPIRY = <days>;

    Snowflake shows the token secret only when you create the token. Store it securely before closing the result.

  2. In Airbyte, select Programmatic Access Token as the authorisation method. Enter the Snowflake username in Username and paste the token secret into Programmatic Access Token.

Keep these points in mind when using a PAT:

  • Tokens expire. A PAT expires after the number of days set with DAYS_TO_EXPIRY. The default is 15 days, with a maximum of 365. When it expires, syncs fail until you create a new token and update the Programmatic Access Token field. Plan to rotate the token before expiry. Key pair authentication doesn’t have this limitation.
  • Network policy. By default, Snowflake requires a network policy on the user to generate or use a PAT. If you use Airbyte Cloud, ensure that policy allows Airbyte’s IP addresses. See Step 3.
  • Service users. For users with TYPE = SERVICE, Snowflake requires ROLE_RESTRICTION when you create the token. If an authentication policy restricts permitted methods, include PROGRAMMATIC_ACCESS_TOKEN in AUTHENTICATION_METHODS.

Output schema

By default, Airbyte outputs each stream to its own raw table in the airbyte_internal schema. You can override this using the Airbyte Internal Table Dataset Name setting. Airbyte also creates a final table with typed columns. Raw table contents aren’t deduplicated.

:::info By default, Airbyte creates permanent tables. To use transient tables, create a dedicated transient database for Airbyte. This makes every Airbyte-created table transient by default. It reduces the extra costs and storage associated with Fail-safe. However, you can’t recover transient tables after operational or system failures.

If you configure a custom Airbyte Internal Table Dataset Name internal schema, keep the default and internal schemas in matching database types. Both databases must be transient or both permanent. This prevents inconsistent data protection behaviour. For more information, see Working with Temporary and Transient Tables. :::

Raw table schema

The raw table contains these fields:

  • _airbyte_raw_id
  • _airbyte_generation_id
  • _airbyte_extracted_at
  • _airbyte_loaded_at
  • _airbyte_meta
  • _airbyte_data

_airbyte_data is a JSON blob containing the event data. See here for more information about the other fields.

:::info The contents of _airbyte_data are fairly stable. However, the raw table schema may change in future versions. :::

Final table schema

The final table includes these fields alongside the columns declared in your stream schema:

  • _AIRBYTE_RAW_ID
  • _AIRBYTE_GENERATION_ID
  • _AIRBYTE_EXTRACTED_AT
  • _AIRBYTE_LOADED_AT
  • _AIRBYTE_META

See here for more information about these fields.

Data type map

Airbyte typeSnowflake type
STRINGTEXT
STRING (BASE64)TEXT
STRING (BIG_NUMBER)TEXT
STRING (BIG_INTEGER)TEXT
NUMBERFLOAT (default) or NUMBER(38,9)
INTEGERNUMBER
BOOLEANBOOLEAN
STRING (TIMESTAMP_WITH_TIMEZONE)TIMESTAMP_TZ
STRING (TIMESTAMP_WITHOUT_TIMEZONE)TIMESTAMP_NTZ
STRING (TIME_WITH_TIMEZONE)TEXT
STRING (TIME_WITHOUT_TIMEZONE)TIME
DATEDATE
OBJECTOBJECT
ARRAYARRAY
UNIONVARIANT
UNKNOWNVARIANT

The Decimal Data Type setting controls the Snowflake data type used for columns with the Airbyte NUMBER decimal type:

  • NUMBER(38,9) (recommended): An exact fixed-point decimal type supporting up to 38 total digits. It supports up to 29 digits before the decimal point and 9 after it. Use it for financial data and other values requiring exact decimal representation within this precision and scale.
  • FLOAT (default): An approximate binary floating-point type with around 15 digits of precision. It supports magnitudes up to approximately 10^308. It suits scientific calculations, statistical models, probabilities, model scores, and uses requiring a wide numeric range. Small rounding differences must be acceptable.

Unaffected cases:

  • Legacy raw tables mode: This setting has no effect because there are no typed columns. All values are stored in the _airbyte_data VARIANT column.
  • Scale-0 database types, such as NUMERIC(10,0): Source connectors map these to the Airbyte INTEGER type. Snowflake always stores this type as NUMBER, so this setting doesn’t affect it.

:::warning Run a full refresh after changing this setting on an existing connection. Otherwise, Airbyte converts existing number columns in place during the next sync. FLOAT and NUMBER(38,9) have different range and precision characteristics. Conversion can therefore cause data loss, reduced precision, truncated values, or NULL values:

  • Switching to NUMBER(38,9): Stored FLOAT values with more than 29 digits before the decimal point become NULL during conversion. Snowflake converts the column in place, so Airbyte doesn’t process individual rows. Therefore, these changes don’t add a _airbyte_meta entry. Values originally loaded as FLOAT may already have lost precision. These rows are flagged TRUNCATED in _airbyte_meta and remain imprecise because conversion can’t restore unstored digits. Only a full refresh re-syncs the original source values.
  • Switching to FLOAT: Values silently become approximate again, and query results may differ slightly.

:::

Precision and size limits

Snowflake applies these precision limits to numeric types:

  • FLOAT: A standard 64-bit floating-point value.
  • NUMBER(38,9), when selected as the number data type: A maximum of 29 digits before the decimal point. Larger values become NULL and are flagged in _airbyte_meta. If your data contains values this large, use the FLOAT option instead. Values with more than 9 decimal places are truncated and flagged in _airbyte_meta.
  • NUMBER (INTEGER): A maximum of 38 digits.

Airbyte replaces values outside these type boundaries with NULL. It rounds values within the boundaries when they have excessive precision. In both cases, the _airbyte_meta column includes a changes entry recording the change.

Snowflake also applies size limits to text and semi-structured types:

  • VARCHAR: A maximum of 16 MB when UTF-8 encoded.
  • VARIANT, used for OBJECT, ARRAY, UNION, and UNKNOWN types: A maximum of 128 MB.

Values exceeding these size limits become NULL, and the _airbyte_meta column records the change.

Schema evolution

This connector supports automatic schema evolution. When the source schema changes, it adds new destination columns and modifies column types as needed. If a source column is removed, the connector retains its destination column and historical data. Subsequent rows use NULL for that column. The connector requires ALTER TABLE privileges on destination tables for this feature.

Supported sync modes

The Snowflake destination supports these sync modes:

Sync modeSupported
Full Refresh - OverwriteYes
Full Refresh - Overwrite + DedupedYes
Full Refresh - AppendYes
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedYes

:::note Deduplication doesn’t run in Legacy raw tables mode. Sync modes that normally deduplicate instead append every record to the raw table without deduplication. :::

Snowflake tutorials

After setting up the Snowflake destination connector, explore these Snowflake tutorials:

Troubleshooting

’Current role does not have permissions on the target schema’

If you receive the Current role does not have permissions on the target schema error, check the Snowflake destination SCHEMA. The role you’ve provided must have permission to access it. When creating a connection, you may be able to select Mirror source structure for the Destination namespace. With some default examples and tutorials, this can make the connection write to a PUBLIC schema.

For a quick fix, change your connection’s ‘Replication’ setting from Mirror source structure to Destination Default. Alternatively, grant the role the required permissions in your chosen namespace.

Namespace support

This destination supports namespaces. The namespace maps to a Snowflake schema.

Changelog

Expand to review
VersionDatePull RequestSubject
5.2.22026-10-3090225Create internal stages in the internal (raw_data_schema) schema instead of each output schema, so the role only needs CREATE STAGE on the internal schema.
5.2.12026-10-2289645Upgrade to Bulk CDK 1.1.9: fix critical CVEs in Apache MINA (mina-core 2.0.31), MINA SSHD 2.20.0 and BouncyCastle 1.86; netty 4.1.139; pin jsoup 1.23.2.
5.2.02026-10-0988417Add Programmatic Access Token as an authorization method. The token is passed to Snowflake in place of a password.
5.1.22026-10-0587720Allow custom domains in destination connections, consistent with Snowflake sources.
5.1.12026-10-0688143Fix HikariPool “Connection is not available” failures during the final flush by restoring bounded concurrency for Snowflake PUT/COPY operations.
5.1.02026-10-0187611Add optional Fusion S3 sync copies of loaded batches (off by default; enabled through environment configuration).
5.0.12026-09-0984953Add missing Airbyte meta columns (_airbyte_meta, _airbyte_generation_id) to tables created by connector versions prior to 3.10.0 when the AIRBYTE_DESTINATION_SNOWFLAKE_META_COLUMN_REPAIR env var is set to true (off by default), fixing “invalid identifier” sync failures after upgrading to 4.x.
5.0.02026-09-0285314Deprecate username/password authentication; key pair authentication is now the only recommended auth method. Username/password will be removed in a future release.
4.1.22026-08-2484979Upgrade CDK to 1.0.25 (prevents truncate-refresh retries from replacing a populated table with an empty one)
4.1.12026-08-1876313Handle ANSI reserved keywords as column names by prefixing with underscore
4.1.02026-08-0583713Add opt-in NUMBER(38,9) data type for number columns via the new “Decimal Data Type” option.
4.0.492026-08-0583740Upgrade CDK to 1.0.21
4.0.482026-07-2382272Columns removed from the source schema are now preserved in the destination table instead of being dropped to prevent unintentional data loss due to source schema changes.
4.0.472026-07-2382294Replace ALTER TABLE SWAP WITH with CREATE OR REPLACE TABLE CLONE COPY GRANTS to preserve table grants
4.0.462026-07-1582104Use CREATE TABLE IF NOT EXISTS for non-replace table creation to prevent accidental data loss
4.0.452026-07-1081530Restore NULL-safe primary-key matching in dedup MERGE to fix duplicate rows when composite PKs contain NULLs
4.0.442026-06-3081346Remove unnecessary NULL PK equality checks from merge SQL
4.0.432026-05-2078231Upgrade CDK to 1.0.13
4.0.422026-05-1477978Improve encrypted key-pair private key handling and error messages
4.0.412026-05-0777795Add option to preserve Snowflake whitespace
4.0.40-rc.12026-04-2776405Upgrade CDK to 1.0.9. Enable fast timestamp coercion. Progressive rollout.
4.0.402026-05-0177673Update the CDK to a version with default timestamp parsing
4.0.392026-03-1374715Drop temp table after successful upsert to prevent duplicate records
4.0.382026-02-2574041Upgrade CDK to 1.0.2 and base image to 2.0.4 for CVE patches
4.0.372026-02-0472854Internal interface changes
4.0.362026-01-2972417Handle exception in countTable correctly
4.0.352026-01-2672295Upgrade CDK to 0.2.0
4.0.342026-01-0971273Fix check operation in raw tables only mode.
4.0.332026-01-0971248Upgrade to CDK 0.1.99 — limit final aggregate parallelism.
4.0.322026-01-0871233Promoting release candidate 4.0.32-rc.2 to a main version.
4.0.32-rc.22025-12-1971007Fix Raw Mode regression: Change _airbyte_data back to SnowflakeType.VARIANT
4.0.32-rc.12025-12-1870903Upgrade to CDK 0.1.91; internal refactoring
4.0.312025-12-0870442Write VARIANT values correctly when underlying Airbyte type is a union
4.0.302025-11-2469842Update documentation about numeric value handling
4.0.292025-11-2469342Truncate NumberValues and IntegerValues with excessive precision instead of nullifying them
4.0.282025-11-1369245Upgrade to CDK 0.1.78
4.0.272025-11-1169285Remove spurious log messages
4.0.262025-11-1169117Upgrade to CDK 0.1.74 (internal code changes for schema evolution)
4.0.252025-11-0669226Improved additional statistics handling
4.0.242025-11-0569200Add support for observability metrics
4.0.232025-11-0369153Fix bug in connector spec
4.0.222025-11-0369147Upgrade to CDK 0.1.62 to improve handling of unrecognized JSON schema
4.0.212025-11-0369100Upgrade to CDK 0.1.61 to fix state index bug
4.0.202025-10-2968186Upgrade to CDK 0.1.59
4.0.192025-10-2868683Upgrade to CDK 0.1.58 (improve logging of unflushed states at end of sync)
4.0.182025-10-2868125Internal refactoring; upgrade to CDK 0.1.56
4.0.172025-10-2268585Wrap Snowflake permission errors as ConfigErrorException for better error handling
4.0.162025-10-2168516Ensure columns are marked as non-nullable in destination schema.
4.0.152025-10-2167153Implement new proto schema implementation
4.0.142025-10-1668148No user-facing changes. Internal code reorganization.
4.0.132025-10-1568106Revert cache change to fix performance issue.
4.0.122025-10-1568096Cache Airbyte metadata data column checks during formatting to improve performance.
4.0.112025-10-1467721Handle edge case in config migrator; improve test coverage.
4.0.102025-10-1367714Update check to handle QUOTED_IDENTIFIERS_IGNORE_CASE=true correctly.
4.0.92025-10-1067610Fix an issue with varchar size validation
4.0.82025-10-1067599Improve handling of heavily interleaved streams.
4.0.72025-10-0967590Use GZIP compression level 5 to improve performance. No longer explicitly CREATE FILE FORMAT.
4.0.62025-10-0967582Schema evolution supports dropping columns with lowercase characters in their name.
4.0.52025-10-0967575Revert inadvertent type changes.
4.0.42025-10-0867569Prevent crashes when a column value ends with \.
4.0.32025-10-0867556Improve performance and reduce cost by increasing maximum parallel processing.
4.0.22025-10-0767301Improve performance and reduce cost by caching SHOW COLUMNS output.
4.0.12025-10-0667106Removes unnecessary operations and allocations in value validator.
4.0.02025-10-0666747Upgrade to direct-load tables; add option for soft CDC deletes.
3.15.11-rc.22025-09-2966753Pin to CDK artifact
3.15.11-rc.12025-09-2566307Initial Direct Load RC release
3.15.102025-06-1361588Publish version to account for possible duplicate publishing in pipeline. Noop change.
3.15.92025-05-1660328Migrate to base 2.0.2.
3.15.82025-05-0259222Upgrade libraries for security patches.
3.15.72025-04-2459166Upgrade libraries for security patches.
3.15.62025-04-2458652Upgrade libraries for security patches.
3.15.52025-03-2456355Upgrade to airbyte/java-connector-base:2.0.1 to be M4 compatible.
3.15.42025-01-1051503Use a non root base image
3.15.32024-12-1849913Use a base image: airbyte/java-connector-base:1.0.0
3.15.22024-10-31#48070upgrade JDBC driver to 3.20.0
3.15.12024-10-20#46989add snowflake transaction wrapper for rollback support
3.15.02024-09-18#45437upgrade all dependencies
3.14.02024-09-18#45431truncate large records queries
3.13.02024-09-17#45422speed up metadata queries
3.12.02024-09-17#38585force UTF8 collation when creating schemas and tables
3.11.122024-09-12#45370fix a race condition in our orphanedThreadFilter
3.11.112024-08-20#44476Increase message parsing limit to 100mb
3.11.102024-08-22#44526Revert protocol compliance fix
3.11.92024-08-19#43367Add opt in using MERGE statement for upserts and deletes
3.11.82024-08-16#42505Fix bug in refreshes logic (already mitigated in platform, just fixing protocol compliance)
3.11.72024-08-09#43440remove contention on state table by deleting rows ony once every 100 updates
3.11.62024-08-09#43332bump Java CDK
3.11.52024-08-07#43348SnowflakeSqlGen cleanup to Kotlin string interpolation
3.11.42024-07-18#41940Update host regex to allow connecting to LocalStack Snowflake
3.11.32024-07-15#41968Don’t hang forever on empty stream list; shorten error message on INCOMPLETE stream status
3.11.22024-07-12#41674Upgrade to latest CDK
3.11.12024-07-08#41041Fix resume logic in truncate refreshes to prevent data loss
3.11.02024-06-25#39473Support for refreshes and resumable full refresh. WARNING: You must upgrade to platform 0.63.7 before upgrading to this connector version.
3.10.12024-06-11#39399Bug fix for _airbyte_meta not migrated in OVERWRITE mode
3.10.02024-06-10#39107_airbyte_meta and _airbyte_generation_id in Raw tables and final tables
3.9.12024-06-05#39135Improved error handling for Staging files
3.9.02024-05-23#38658Adapting to newer interfaces from #38107
3.8.42024-05-23#38632convert all tests to kotlin
3.8.32024-05-23#38586Bump CDK version
3.8.22024-05-22#38553Remove SwitchingDestination and AbstractJdbcDestination dependency in destination
3.8.12024-05-22#38568Adopt latest CDK
3.8.02024-05-08#37715Remove option for incremental typing and deduping
3.7.42024-05-07#38052Revert problematic optimization
3.7.32024-05-07#34612Adopt CDK 0.33.2
3.7.22024-05-06#37857Use safe executeMetadata call
3.7.12024-04-30#36910Bump CDK version
3.7.02024-04-08#35754Allow configuring data_retention_time_in_days; apply to both raw and final tables. Note: Existing tables will not be affected; you must manually alter them.
3.6.62024-03-26#36466Correctly hhandle instances with QUOTED_IDENTIFIERS_IGNORE_CASE enabled globally
3.6.52024-03-25#36461Internal code change (use published CDK artifact instead of source dependency)
3.6.42024-03-25#36396Handle instances with QUOTED_IDENTIFIERS_IGNORE_CASE enabled globally
3.6.32024-03-25#36452Remove Query timeout
3.6.22024-03-18#36240Hide oAuth config option
3.6.12024-03-07#35899Adopt CDK 0.23.18; Null safety check in state parsing
3.6.02024-03-06#35308Upgrade CDK; use utc tz for extracted_at; Migrate existing extracted_at to utc;
3.5.142024-02-22#35456Adopt CDK 0.23.0; Gather initial state upfront, reduce information_schema calls
3.5.132024-02-22#35569Fix logging bug.
3.5.122024-02-15#35240Adopt CDK 0.20.9
3.5.112024-02-12#35194Reorder auth options
3.5.102024-02-12#35144Adopt CDK 0.20.2
3.5.92024-02-12#35111Adopt CDK 0.20.1
3.5.82024-02-09#34574Adopt CDK 0.20.0
3.5.72024-02-08#34747Adopt CDK 0.19.0
3.5.62024-02-08#35027Upgrade CDK to version 0.17.1
3.5.52024-02-08#34502Reduce COPY frequency
3.5.42024-01-24#34451Improve logging for unparseable input
3.5.32024-01-25#34528Fix spurious check failure (UnsupportedOperationException: Snowflake does not use the native JDBC DV2 interface)
3.5.22024-01-24#34458Improve error reporting
3.5.12024-01-24#34501Internal code changes for Destinations V2
3.5.02024-01-24#34462Upgrade CDK to 0.14.0
3.4.222024-01-12#34227Upgrade CDK to 0.12.0; Cleanup unused dependencies
3.4.212024-01-10#34083Emit destination stats as part of the state message
3.4.202024-01-05#33948Skip retrieving initial table state when setup fails
3.4.192024-01-04#33730Internal code structure changes
3.4.182024-01-02#33728Add option to only type and dedupe at the end of the sync
3.4.172023-12-20#33704Update to java CDK 0.10.0 (no changes)
3.4.162023-12-18#33124Make Schema Creation Seperate from Table Creation
3.4.152023-12-13#33232Only run typing+deduping for a stream if the stream had any records
3.4.142023-12-08#33263Adopt java CDK version 0.7.0
3.4.132023-12-05#32326Use jdbc metadata for table existence check
3.4.122023-12-04#33084T&D SQL statements moved to debug log level
3.4.112023-11-14#32526Clean up memory manager logs.
3.4.102023-11-08#32125Fix compilation warnings.
3.4.92023-11-06#32026Add separate TRY_CAST transaction to reduce compute usage
3.4.82023-11-06#32190Further improve error reporting
3.4.72023-11-06#32193Adopt java CDK version 0.4.1.
3.4.62023-11-02#32124Revert merge statement
3.4.52023-11-02#31983Improve error reporting
3.4.42023-10-30#31985Delay upgrade deadline to Nov 7
3.4.32023-10-30#31960Adopt java CDK version 0.2.0.
3.4.22023-10-27#31897Further filtering on extracted_at
3.4.12023-10-27#31683Performance enhancement (switch to a merge statement for incremental-dedup syncs)
3.4.02023-10-25#31686Opt out flag for typed and deduped tables
3.3.02023-10-25#31520Stop deduping raw table
3.2.32023-10-17#31191Improve typing+deduping performance by filtering new raw records on extracted_at
3.2.22023-10-10#31194Deallocate unused per stream buffer memory when empty
3.2.12023-10-10#31083Fix precision of numeric values in async destinations
3.2.02023-10-09#31149No longer fail syncs when PKs are null - try do dedupe anyway
3.1.222023-10-06#31153Increase jvm GC retries
3.1.212023-10-06#31139Bump CDK version
3.1.202023-10-06#31129Reduce async buffer size
3.1.192023-10-04#31082Revert null PK checks
3.1.182023-10-01#30779Final table PK columns become non-null and skip check for null PKs in raw records (performance)
3.1.172023-09-29#30938Upgrade snowflake-jdbc driver
3.1.162023-09-28#30835Fix regression from 3.1.15 in supporting concurrent syncs with identical stream name but different namespace
3.1.152023-09-26#30775Increase async block size
3.1.142023-09-27#30739Fix column name collision detection
3.1.132023-09-19#30599Support concurrent syncs with identical stream name but different namespace
3.1.122023-09-21#30671Reduce async buffer size
3.1.112023-09-19#30592Internal code changes
3.1.102023-09-18#30546Make sure that the async buffer are flush every 5 minutes
3.1.92023-09-19#30319Support column names that are reserved
3.1.82023-09-18#30479Fix async memory management
3.1.72023-09-15#30491Improve error message display
3.1.62023-09-14#30439Fix a transient error
3.1.52023-09-13#30416Support ${ in stream name/namespace, and in column names
3.1.42023-09-12#30364Add log message
3.1.32023-08-29#29878Reenable incremental typing and deduping
3.1.22023-08-31#30020Run typing and deduping tasks in parallel
3.1.12023-09-05#30117Type and Dedupe at sync start and then every 6 hours
3.1.02023-09-01#30056Upcase final table names to allow case-insensitive references
3.0.22023-09-01#30121Improve performance on very wide streams by skipping TRY_CAST on strings
3.0.12023-08-27#30065Clearer error thrown when records are missing a primary key
3.0.02023-08-27#29783Destinations V2
2.1.72023-08-29#29949Destinations V2: Fix checking for empty table by ensuring upper-case DB names
2.1.62023-08-28#29878Destinations V2: Fix detection of existing table by ensuring upper-case DB names
2.1.52023-08-28#29903Destinations V2: Performance Improvement, Changing Metadata error array construction from ARRAY_CAT to ARRAY_CONSTRUCT_COMPACT
2.1.42023-08-28#29903Abort queries on crash
2.1.32023-08-25#29881Destinations v2: Only run T+D once at end of sync, to prevent data loss under async conditions
2.1.22023-08-24#29805Destinations v2: Don’t soft reset in migration
2.1.12023-08-23#29774Destinations v2: Don’t soft reset overwrite syncs
2.1.02023-08-21#29636Destinations v2: Several Critical Bug Fixes (cursorless dedup, improved floating-point handling, improved special characters handling; improved error handling)
2.0.02023-08-09#28894Remove support for Snowflake GCS/S3 loading method in favor of Snowflake Internal staging
1.3.32023-08-15#29461Changing a static constant reference
1.3.22023-08-11#29381Destinations v2: Add support for streams with no columns
1.3.12023-08-04#28894Destinations v2: Update SqlGenerator
1.3.02023-08-07#29174Destinations v2: early access release
1.2.102023-08-07#29188Internal code refactoring
1.2.92023-08-04#28677Destinations v2: internal code changes to prepare for early access release
1.2.82023-08-03#29047Avoid logging record if the format is invalid
1.2.72023-08-02#28976Fix composite PK handling in v1 mode
1.2.62023-08-01#28618Reduce logging noise
1.2.52023-07-24#28618Add hooks in preparation for destinations v2 implementation
1.2.42023-07-21#28584Install dependencies in preparation for destinations v2 work
1.2.32023-07-21#28345Pull in async framework minor bug fix for race condition on state emission
1.2.22023-07-14#28345Increment patch to trigger a rebuild
1.2.12023-07-14#28315Pull in async framework minor bug fix to avoid Snowflake hanging on close
1.2.02023-07-5#27935Enable Faster Snowflake Syncs with Asynchronous writes
1.1.02023-06-27#27781License Update: Elv2
1.0.62023-06-21#27555Reduce image size
1.0.52023-05-31#25782Internal scaffolding for future development
1.0.42023-05-19#26323Prevent infinite retry loop under specific circumstances
1.0.32023-05-15#26081Reverts splits bases
1.0.22023-05-05#25649Splits bases (reverted)
1.0.12023-04-29#25570Internal library update
1.0.02023-05-02#25739Removed Azure Blob Storage as a loading method
0.4.632023-04-27#25346Added FlushBufferFunction interface
0.4.612023-03-30#24736Improve behavior when throttled by AWS API
0.4.602023-03-30#24698Add option in spec to allow increasing the stream buffer size to 50
0.4.592023-03-23#23904Fail faster in certain error cases
0.4.582023-03-27#24615Fixed host validation by pattern on UI
0.4.56 (broken)2023-03-22#23904Added host validation by pattern on UI
0.4.542023-03-17#23788S3-Parquet: added handler to process null values in arrays
0.4.532023-03-15#24058added write attempt to internal staging Check method
0.4.522023-03-10#23931Added support for periodic buffer flush
0.4.512023-03-10#23466Changed S3 Avro type from Int to Long
0.4.492023-02-27#23360Added logging for flushing and writing data to destination storage
0.4.482023-02-23#22877Add handler for IP not in whitelist error and more handlers for insufficient permission error
0.4.472023-01-30#21912Catch “Create” Table and Stage Known Permissions and rethrow as ConfigExceptions
0.4.462023-01-26#20631Added support for destination checkpointing with staging
0.4.452023-01-25#21087Catch Known Permissions and rethrow as ConfigExceptions
0.4.442023-01-20#21087Wrap Authentication Errors as Config Exceptions
0.4.432023-01-20#21450Updated Check methods to handle more possible s3 and gcs stagings issues
0.4.422023-01-12#21342Better handling for conflicting destination streams
0.4.412022-12-16#20566Improve spec to adhere to standards
0.4.402022-11-11#19302Set jdbc application env variable depends on env - airbyte_oss or airbyte_cloud
0.4.392022-11-09#18970Updated “check” connection method to handle more errors
0.4.382022-09-26#17115Added connection string identifier
0.4.372022-09-21#16839Update JDBC driver for Snowflake to 3.13.19
0.4.362022-09-14#15668Wrap logs in AirbyteLogMessage
0.4.352022-09-01#16243Fix Json to Avro conversion when there is field name clash from combined restrictions (anyOf, oneOf, allOf fields).
0.4.342022-07-23#14388Add support for key pair authentication
0.4.332022-07-15#14494Make S3 output filename configurable.
0.4.322022-07-14#14618Removed additionalProperties: false from JDBC destination connectors
0.4.312022-07-07#13729Improve configuration field description
0.4.302022-06-24#14114Remove “additionalProperties”: false from specs for connectors with staging
0.4.292022-06-17#13753Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager
0.4.282022-05-18#12952Apply buffering strategy on GCS staging
0.4.272022-05-17#12820Improved ‘check’ operation performance
0.4.262022-05-12#12805Updated to latest base-java to emit AirbyteTraceMessages on error.
0.4.252022-05-03#12452Add support for encrypted staging on S3; fix the purge_staging_files option
0.4.242022-03-24#11093Added OAuth support (Compatible with Airbyte Version 0.35.60+)
0.4.222022-03-18#10793Fix namespace with invalid characters
0.4.212022-03-18#11071Switch to compressed on-disk buffering before staging to s3/internal stage
0.4.202022-03-14#10341Add Azure blob staging support
0.4.192022-03-11#10699Added unit tests
0.4.172022-02-25#10421Refactor JDBC parameters handling
0.4.162022-02-25#10627Add try catch to make sure all handlers are closed
0.4.152022-02-22#10459Add FailureTrackingAirbyteMessageConsumer
0.4.142022-02-17#10394Reduce memory footprint.
0.4.132022-02-16#10212Execute COPY command in parallel for S3 and GCS staging
0.4.122022-02-15#10342Use connection pool, and fix connection leak.
0.4.112022-02-14#9920Updated the size of staging files for S3 staging. Also, added closure of S3 writers to staging files when data has been written to an staging file.
0.4.102022-02-14#10297Halve the record buffer size to reduce memory consumption.
0.4.92022-02-14#10256Add ExitOnOutOfMemoryError JVM flag.
0.4.82022-02-01#9959Fix null pointer exception from buffered stream consumer.
0.4.72022-01-29#9745Integrate with Sentry.
0.4.62022-01-28#9623Add jdbc_url_params support for optional JDBC parameters
0.4.52021-12-29#9184Update connector fields title/description
0.4.42022-01-24#9743Fixed bug with dashes in schema name
0.4.32022-01-20#9531Start using new S3StreamCopier and expose the purgeStagingData option
0.4.22022-01-10#9141Fixed duplicate rows on retries
0.4.12021-01-06#9311Update сreating schema during check
0.4.02021-12-27#9063Updated normalization to produce permanent tables
0.3.242021-12-23#8869Changed staging approach to Byte-Buffered
0.3.232021-12-22#9039Added part_size configuration in UI for S3 loading method
0.3.222021-12-21#9006Updated jdbc schema naming to follow Snowflake Naming Conventions
0.3.212021-12-15#8781Updated check method to verify permissions to create/drop stage for internal staging; compatibility fix for Java 17
0.3.202021-12-10#8562Moving classes around for better dependency management; compatibility fix for Java 17
0.3.192021-12-06#8528Set Internal Staging as default choice
0.3.182021-11-26#8253Snowflake Internal Staging Support
0.3.172021-11-08#7719Improve handling of wide rows by buffering records based on their byte size rather than their count
0.3.152021-10-11#6949Each stream was split into files of 10,000 records each for copying using S3 or GCS
0.3.142021-09-08#5924Fixed AWS S3 Staging COPY is writing records from different table in the same raw table
0.3.132021-09-01#5784Updated query timeout from 30 minutes to 3 hours
0.3.122021-07-30#5125Enable additionalPropertities in spec.json
0.3.112021-07-21#3555Partial Success in BufferedStreamConsumer
0.3.102021-07-12#4713Tag traffic with airbyte label to enable optimization opportunities from Snowflake
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.