BigQuery
Set up the BigQuery destination connector by choosing a data loading method and configuring the connector in the Airbyte UI.
This page shows you how to set up the BigQuery destination connector.
Prerequisites
-
If you use Airbyte Open Source with the Postgres source connector, upgrade your Airbyte platform to version
v0.40.0-alphaor newer. Then upgrade your BigQuery connector to version1.1.14or newer. -
A BigQuery dataset to sync data to.
Note: BigQuery queries can only reference datasets in the same physical location. To combine synced data with other datasets, create those datasets in the same Google Cloud location. For more information, read Introduction to Datasets
-
You need a Google Cloud Service Account for Airbyte Cloud. It’s optional for Airbyte Open Source. Assign the
BigQuery UserandBigQuery Data Editorroles and the Service Account Key in JSON format. -
If you use Airbyte Cloud with IP-based access controls, add Airbyte’s IP addresses to your allowlist.
Setup guide
Step 1: Choose a data loading method
Use a Google Cloud Storage bucket (Recommended)
This loading method is recommended for production workloads. It uploads data to a GCS bucket, then uses a COPY job to load it into BigQuery. This provides better performance for large data volumes.
To use a Google Cloud Storage bucket:
- Create a Cloud Storage bucket with the
Protection Tools set to
noneorObject versioning. Make sure the bucket doesn’t have a retention policy. - Create an HMAC key and access ID.
- Grant the
Storage Object Adminrole to the Google Cloud Service Account. Use the same service account that you configure for BigQuery access in the BigQuery connector setup step. - Make sure the machine running Airbyte can access your Cloud Storage bucket. To check access, use the check connection tool in the UI.
Encrypt your bucket with a Google-managed encryption key, the default setting for new buckets.
Buckets using customer-managed encryption keys (CMEK) aren’t currently supported.
You can view this setting in the “Configuration” tab of your GCS bucket, in the
Encryption type row.
Use Batched Standard Inserts
This method offers a simpler setup without external staging. It uses the BigQuery SDK to stream data directly. It’s suitable for smaller data volumes or quick testing. BigQuery manages the staging files and deletes them automatically after loading finishes.
Step 2: Set up the BigQuery connector
- Log in to your Airbyte Cloud or Airbyte Open Source account.
- Click Destinations, then click + New destination.
- On the Set up the destination page, select BigQuery from the Destination type dropdown.
- Enter a name for the BigQuery connector.
- For Project ID, enter your Google Cloud project ID.
- For Dataset Location, select your BigQuery dataset’s location.
:::warning You can’t change the location later. :::
- For Default Dataset ID, enter the BigQuery Dataset ID.
- For Loading Method, select GCS Staging (Recommended) or Batched Standard Inserts.
- For Service Account Key JSON (Required for cloud, optional for open-source), enter the Google Cloud Service Account Key in JSON format.
:::note Copy everything in the Account Key JSON file, including the brackets. :::
- For Transformation Query Run Type (Optional), select interactive to have BigQuery run interactive query jobs, or select batch to have BigQuery run batch queries.
:::note Interactive queries run as soon as possible and count towards daily concurrent quotas and limits. Batch queries run when idle resources become available in the BigQuery shared resource pool. If a query hasn’t started within 24 hours, BigQuery changes its job priority to interactive. Batch queries don’t count towards your concurrent rate limit, so you can start many queries at once. :::
-
For Google BigQuery Client Chunk Size (Optional), keep the default value of 15 MiB. If the destination later has networking or memory management problems, try reducing the chunk size. The sync will run more slowly but will be more likely to succeed.
-
For Job Execution Project ID (Optional), leave the field empty to run BigQuery jobs in the project entered as Project ID. To use another project, enter its ID. For example, this keeps ingestion and analytics workloads from competing for the same project quotas.
:::note
Query, load and copy jobs then use the job project’s quotas.
Data still lands in datasets under Project ID.
The service account needs the
roles/bigquery.jobUser
role on the job project, alongside its existing roles on the dataset project.
:::
Supported sync modes
The BigQuery destination connector supports these sync modes:
| Sync mode | Supported? |
|---|---|
| Full Refresh - Overwrite | Yes |
| Full Refresh - Append | Yes |
| Full Refresh - Overwrite + Deduped | Yes |
| Incremental Sync - Append | Yes |
| Incremental Sync - Append + Deduped | Yes |
Output schema
The final table includes these fields, alongside the columns declared in your stream schema:
airbyte_raw_id_airbyte_generation_idairbyte_extracted_at_airbyte_meta
See here for more information about these fields.
BigQuery output tables use daily partitions based on the Time-unit column airbyte_extracted_at.
They are clustered by airbyte_extracted_at and the table Primary Keys.
Partition boundaries use UTC time.
Use a predicate filter with a WHERE clause to limit the partitions scanned.
Filters on the partitioning column prune partitions and reduce query costs.
Airbyte doesn’t enable Require partition filter, but you can toggle it by updating the produced tables.
Legacy Raw Tables schema
If you enable Legacy raw tables, the connector writes tables in this format.
By default, Airbyte writes each stream to its own raw table in the airbyte_internal dataset.
You can override this using the Airbyte Internal Table Dataset Name option.
Contents in the raw table are NOT deduplicated.
The raw table contains these fields:
_airbyte_raw_id_airbyte_generation_id_airbyte_extracted_at_airbyte_loaded_at_airbyte_meta_airbyte_data
_airbyte_data is a JSON blob containing the record’s data.
See here for more information about the other fields.
BigQuery naming conventions
Follow BigQuery Datasets Naming conventions.
Airbyte converts invalid characters to _ characters when writing data.
Datasets beginning with _ are hidden in the BigQuery Explorer panel.
Therefore, Airbyte prepends converted namespaces with n.
Data type map
| Airbyte type | BigQuery type |
|---|---|
| STRING | STRING |
| STRING (BASE64) | STRING |
| STRING (BIG_NUMBER) | STRING |
| STRING (BIG_INTEGER) | STRING |
| NUMBER | NUMERIC |
| INTEGER | INT64 |
| BOOLEAN | BOOL |
| STRING (TIMESTAMP_WITH_TIMEZONE) | TIMESTAMP |
| STRING (TIMESTAMP_WITHOUT_TIMEZONE) | DATETIME |
| STRING (TIME_WITH_TIMEZONE) | STRING |
| STRING (TIME_WITHOUT_TIMEZONE) | TIME |
| DATE | DATE |
| OBJECT | JSON |
| ARRAY | JSON |
Troubleshooting
Permission errors
The service account doesn’t have the correct permissions.
- Make sure the BigQuery service account has the
BigQuery UserandBigQuery Data Editorroles, or equivalent permissions. - If you select GCS staging mode, make sure the BigQuery service account can access the GCS bucket and path.
Alternatively, assign the
Cloud Storage Adminrole, which includes a superset of the required permissions.
The HMAC key is incorrect.
- Make sure the HMAC key belongs to the BigQuery service account. The service account must also have permission to access the GCS bucket and path.
HTTP 400 “Request had invalid euc header” during upload
If your sync fails with BigQueryException: 400 Bad Request and the message
Request had invalid euc header:
- This error comes from the Google BigQuery resumable-upload API and is usually temporary.
- Retry the sync. The error resolves on the next attempt in most cases.
- If every sync returns this error, the underlying causes can be complex.
Google doesn’t publicly document this specific error. These steps may help you narrow down the issue:
- If you self-manage Airbyte, check whether a proxy, VPN or firewall modifies HTTP headers on requests to
bigquery.googleapis.com. - Check whether the service account key has been rotated or revoked since you configured the connection.
- Try reducing Google BigQuery Client Chunk Size from the default 15 MiB to a smaller value, such as 5 MiB.
- Try running fewer concurrent syncs to your BigQuery instance or table. Contention may contribute to the error.
- If you self-manage Airbyte, check whether a proxy, VPN or firewall modifies HTTP headers on requests to
Concurrent script query quota exceeded
If your sync fails with:
BigQueryException: Quota exceeded: Your project_and_region exceeded quota for
concurrent script queries per project.
- This quota applies per project. Syncs share it with every other workload in that project, including analytics queries and Dataform pipelines.
- Set Job Execution Project ID to a separate project to run connector jobs against that project’s quota. Data continues to land in datasets under Project ID.
- Alternatively, reduce the number of streams or connections syncing concurrently to the project.
Load job timeouts
If your sync fails with Fail to complete a load job in big query:
- BigQuery load jobs have a 30-minute wait timeout. Very large batches or heavy contention in the BigQuery queue can exceed this limit.
- Concurrent syncs loading into the same BigQuery table aren’t supported and can also cause this timeout. See Stream uniqueness for details.
- Try reducing each sync’s volume by using incremental sync mode or fewer streams per connection.
Tutorials
After setting up the BigQuery destination connector, explore these BigQuery tutorials:
- Export Google Analytics data to BigQuery
- Load data from Facebook Ads to BigQuery
- Replicate Salesforce data to BigQuery
- Partition and cluster BigQuery tables with Airbyte and dbt
Namespace support
This destination supports namespaces. The namespace maps to a BigQuery dataset.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 3.2.3 | 2026-10-30 | 87606 | Retry transient BigQuery 5xx errors on all API calls, including job inserts and uploads. |
| 3.2.2 | 2026-10-22 | 89648 | Upgrade to Bulk CDK 1.1.9: fix critical CVEs in Apache MINA (mina-core 2.0.31), MINA SSHD 2.20.0 and BouncyCastle 1.86; netty 4.1.139, Hadoop 3.4.3, Parquet 1.18.1; AWS SDK 2.57.2; google-cloud-bigquery 2.73.0. |
| 3.2.1 | 2026-10-06 | 88144 | Fix Standard Inserts committing a partial batch to BigQuery when a sync fails or is cancelled. |
| 3.2.0 | 2026-10-01 | 87631 | Add optional Fusion S3 archiving for GCS Staging and Standard Inserts loads (off by default; enabled through environment configuration). |
| 3.1.2 | 2026-09-09 | [85804](https://github.com/airbytehq/airbyte/pull/85804 | Update loading method UI: reorder to show GCS Staging (Recommended) first, improve descriptions for both loading methods. |
| 3.1.1 | 2026-09-09 | 79178 | Classify BigQuery custom quota exceeded errors as config errors instead of system errors. |
| 3.1.0 | 2026-08-25 | 85041 | Add optional job_project_id field for BigQuery job quota isolation |
| 3.0.24 | 2026-08-24 | 84985 | Upgrade to Bulk CDK 1.0.25. |
| 3.0.23 | 2026-07-14 | 81550 | Use CREATE TABLE IF NOT EXISTS for non-replace table creation to prevent accidental data loss |
| 3.0.22 | 2026-07-10 | 81635 | Restore PK NULL equality checks |
| 3.0.21 | 2026-06-30 | 81346 | Remove unnecessary NULL PK equality checks from merge SQL |
| 3.0.20 | 2026-06-26 | 80932 | Handle BigQueryException in getGenerationId() for legacy tables without _airbyte_generation_id column. |
| 3.0.19 | 2026-05-21 | 78239 | Promoting release candidate 3.0.19-rc.1 to a main version. |
| 3.0.19-rc.1 | 2026-05-19 | 78239 | Upgrade CDK to 1.0.13. Progressive rollout. |
| 3.0.18 | 2026-03-31 | 75913 | Finalize upgrade BigQuery Cloud dependencies and CDK version |
| 3.0.18-rc.1 | 2026-03-27 | 75541 | Upgrade BigQuery Cloud dependencies and CDK version |
| 3.0.17 | 2026-01-28 | 72427 | Finalize upgrade CDK to 0.2.0 |
| 3.0.17-rc.1 | 2026-01-26 | 72296 | Upgrade CDK to 0.2.0 |
| 3.0.16 | 2025-11-25 | 67401 | Add backticks to column names in SQL generation to prevent syntax errors. |
| 3.0.15 | 2025-11-13 | 69307 | Handle out-of-range timestamps/times. |
| 3.0.14 | 2025-11-11 | 69231 | Upgrade to Bulk CDK 0.1.74. |
| 3.0.13 | 2025-11-05 | 69126 | Upgrade to Bulk CDK 0.1.61. |
| 3.0.12 | 2025-10-31 | 69083 | Fail loudly if Bigquery detects bad records. |
| 3.0.11 | 2025-10-28 | 68671 | Log record count per load job. |
| 3.0.10 | 2025-10-21 | 67153 | Implement new proto schema implementation |
| 3.0.9 | 2025-10-17 | 68152 | Update to new TableOperationsClient interface. |
| 3.0.8 | 2025-10-06 | 67078 | Remove memory limit for sync jobs to improve performance and resource utilization. |
| 3.0.7 | 2025-09-03 | 65905 | Promoting release candidate 3.0.7-rc.1 to a main version. |
| 3.0.7-rc.1 | 2025-08-27 | 65114 | Implement SOCKET+PROTO mode support. |
| 3.0.6 | 2025-07-24 | 63700 | Improve error reporting for Billing errors. |
| 3.0.5 | 2025-07-15 | 63312 | Pull in upstream fix to support null chars in GCS staging mode. |
| 3.0.4 | 2025-07-15 | 63327 | Improve error reporting for Billing errors. |
| 3.0.3 | 2025-07-02 | 62495 | Improve error reporting for misconfigured connections; improve support for complex types. |
| 3.0.2 | 2025-06-26 | 62106 | Improve error reporting during schema evolution. |
| 3.0.1 | 2025-06-26 | 62085 | Correctly handle stream names/namespaces and column names which start with a digit. |
| 3.0.0 | 2025-06-25 | 59752 | Upgrade to direct-load tables; add option for soft CDC deletes. |
| 2.12.4 | 2025-06-24 | 62045 | Promoting release candidate 2.12.4-rc.6 to a main version. |
| 2.12.4-rc.6 | 2025-06-24 | 62041 | 2.12.4 RC 6 More retries is better retries |
| 2.12.4-rc.5 | 2025-06-23 | 62016 | 2.12.4 RC 5 Handle records up to 20MiB |
| 2.12.4-rc.4 | 2025-06-23 | 62014 | 2.12.4 RC 4 Improve JSON schema parser’s behavior on invalid source schemas |
| 2.12.4-rc.3 | 2025-06-18 | 61702 | 2.12.4 RC 3 Fix an issue with streams with special chars on bulk loader |
| 2.12.4-rc.2 | 2025-06-18 | 61700 | 2.12.4 RC 2 (throw more informative error on invalid catalog) |
| 2.12.4-rc.1 | 2025-06-16 | 61637 | 2.12.4 RC 1 (theoretically equivalent to 2.12.0, but with fixed global state handling in CDK) |
| 2.12.3 | 2025-06-16 | 61648 | This is not the release you are looking for. identical to 2.10.2 |
| 2.12.1 | 2025-06-13 | 61588 | |
| 2.12.0 | 2025-06-06 | 61432 | Improve performance in GCS staging mode by writing GZIP-compressed files. |
| 2.11.4 | 2025-05-30 | 61018 | Always emit a useful error message when erroring during sync startup. |
| 2.11.3 | 2025-06-02 | 61321 | CHECK operation doesn’t need to drop the dataset anymore. |
| 2.11.2 | 2025-05-29 | 60986 | CHECK operation uses configured default dataset instead of airbyte_internal_test. |
| 2.11.1 | 2025-05-29 | 60973 | Improve error recovery when a stream’s sync mode and schema are both changed and the previous sync exited uncleanly. |
| 2.11.0 | 2025-05-27 | 60922 | Promoting release candidate 2.11.0-rc.7 to a main version. |
| 2.11.0-rc.7 | 2025-05-22 | 60865 | Add a random element to the check sync namespace |
| 2.11.0-rc.6 | 2025-05-22 | 60859 | Improve bigquery quota usage in standard inserts for streams with heavy load |
| 2.11.0-rc.5 | 2025-05-21 | 60841 | Improve behavior in syncs with many state messages |
| 2.11.0-rc.4 | 2025-05-19 | 60791 | Fix bug in detecting schema change when stream has no columns |
| 2.11.0-rc.3 | 2025-05-16 | 60324 | Fix bug in syncs with many state message |
| 2.11.0-rc.2 | 2025-05-15 | 60302 | Fix bugs in handling special characters / uppercase characters in column names |
| 2.11.0-rc.1 | 2025-05-08 | 59729 | Version 2.11.0 RC 1 |
| 2.10.2 | 2025-04-29 | 59142 | Add europe-north2 region. |
| 2.10.1 | 2025-04-14 | 57569 | Remove StringSubstitutor usage |
| 2.10.0 | 2025-04-02 | 56982 | Change default raw table partitioning scheme to be on _airbyte_extracted_at; upgrade CDK |
| 2.9.3 | 2025-03-24 | 56355 | Upgrade to airbyte/java-connector-base:2.0.1 to be M4 compatible. |
| 2.9.2 | 2025-01-10 | 51482 | Use a non root base image |
| 2.9.1 | 2024-12-18 | 49902 | Use a base image: airbyte/java-connector-base:1.0.0 |
| 2.9.0 | 2024-09-18 | 45433 | upgrade all dependencies |
| 2.8.9 | 2024-08-20 | 44476 | Increase message parsing limit to 100mb |
| 2.8.8 | 2024-08-22 | 44526 | Revert protocol compliance fix |
| 2.8.7 | 2024-08-15 | 42504 | Fix bug in refreshes logic (already mitigated in platform, just fixing protocol compliance) |
| 2.8.6 | 2024-07-30 | 42511 | Added a copy operation to validate copy permissions in the check function |
| 2.8.5 | 2024-07-22 | 42407 | Batched Standard Inserts is default loading mode |
| 2.8.4 | 2024-07-15 | 41968 | Don’t hang forever on empty stream list; shorten error message on INCOMPLETE stream status |
| 2.8.3 | 2024-07-12 | 41674 | Upgrade to latest CDK |
| 2.8.2 | 2024-07-08 | 41041 | Fix resume logic in truncate refreshes to prevent data loss |
| 2.8.1 | 2024-06-25 | 39379 | Removing requirement of a redundant permission bigquery.datasets.create permission |
| 2.8.0 | 2024-06-21 | 39904 | Convert all production code to kotlin |
| 2.7.1 | 2024-06-17 | 39526 | Internal code change for improved error reporting in case of source/platform failure (INCOMPLETE stream status / empty ConfiguredCatalog). |
| 2.7.0 | 2024-06-17 | 38713 | Support for refreshes and resumable full refresh. WARNING: You must upgrade to platform 0.63.7 before upgrading to this connector version. |
| 2.6.3 | 2024-06-10 | 38331 | Internal code changes in preparation for future feature release |
| 2.6.2 | 2024-06-07 | 38764 | Increase message length limit to 50MiB |
| 2.6.1 | 2024-05-29 | 38770 | Internal code change (switch to CDK artifact) |
| 2.6.0 | 2024-05-28 | 38359 | Propagate airbyte_meta from sources; add generation_id column |
| 2.5.1 | 2024-05-22 | 38591 | Bugfix to include forward-slash when cleaning up stage |
| 2.5.0 | 2024-05-22 | 38132 | Major rewrite of existing code, Adapting to CDK changes introduced in 38107 |
| 2.4.20 | 2024-05-13 | 38131 | Cleanup BigQueryWriteConfig and reuse StreamConfig; Adapt to StreamConfig signature changes |
| 2.4.19 | 2024-05-10 | 38125 | adopt latest CDK code |
| 2.4.18 | 2024-05-10 | 38111 | No functional changes, deleting unused code |
| 2.4.17 | 2024-05-09 | 38098 | Internal build structure change |
| 2.4.16 | 2024-05-08 | 37714 | Adopt CDK 0.34.0 |
| 2.4.15 | 2024-05-07 | 34611 | Adopt CDK 0.33.2 |
| 2.4.14 | 2024-02-25 | 37584 | Remove unused insecure dependencies from CDK |
| 2.4.13 | 2024-02-25 | 36899 | adopt latest CDK |
| 2.4.12 | 2024-03-04 | 35315 | Adopt CDK 0.23.11 |
| 2.4.11 | 2024-02-22 | 35569 | Fix logging bug. |
| 2.4.10 | 2024-02-15 | 35240 | Adopt CDK 0.20.9 |
| 2.4.9 | 2024-02-15 | 35285 | Adopt CDK 0.20.8 |
| 2.4.8 | 2024-02-12 | 35144 | Adopt CDK 0.20.2 |
| 2.4.7 | 2024-02-12 | 35111 | Adopt CDK 0.20.1 |
| 2.4.6 | 2024-02-09 | 34575 | Adopt CDK 0.20.0 |
| 2.4.5 | 2024-02-08 | 34745 | Adopt CDK 0.19.0 |
| 2.4.4 | 2024-02-08 | 35027 | Upgrade CDK to 0.17.1 |
| 2.4.3 | 2024-02-01 | 34728 | Upgrade CDK to 0.16.4; Notable changes from 0.14.2, 0.15.1 and 0.16.3 |
| 2.4.2 | 2024-01-24 | 34451 | Improve logging for unparseable input |
| 2.4.1 | 2024-01-24 | 34458 | Improve error reporting |
| 2.4.0 | 2024-01-24 | 34468 | Upgrade CDK to 0.14.0 |
| 2.3.31 | 2024-01-22 | #34023 | Combine DDL operations into a single execution |
| 2.3.30 | 2024-01-12 | #34226 | Upgrade CDK to 0.12.0; Cleanup dependencies |
| 2.3.29 | 2024-01-09 | #34003 | Fix loading credentials from GCP Env |
| 2.3.28 | 2024-01-08 | #34021 | Add idempotency ids in dummy insert for check call |
| 2.3.27 | 2024-01-05 | #33948 | Skip retrieving initial table state when setup fails |
| 2.3.26 | 2024-01-04 | #33730 | Internal code structure changes |
| 2.3.25 | 2023-12-20 | #33704 | Update to java CDK 0.10.0 (no changes) |
| 2.3.24 | 2023-12-20 | #33697 | Stop creating unnecessary tmp tables |
| 2.3.23 | 2023-12-18 | #33124 | Make Schema Creation Separate from Table Creation |
| 2.3.22 | 2023-12-14 | #33451 | Remove old spec option |
| 2.3.21 | 2023-12-13 | #33232 | Only run typing+deduping for a stream if the stream had any records |
| 2.3.20 | 2023-12-08 | #33263 | Adopt java CDK version 0.7.0 |
| 2.3.19 | 2023-12-07 | #32326 | Update common T&D interfaces |
| 2.3.18 | 2023-12-04 | #33084 | T&D SQL statements moved to debug log level |
| 2.3.17 | 2023-12-04 | #33078 | Further increase gcs COPY timeout |
| 2.3.16 | 2023-11-14 | #32526 | Clean up memory manager logs. |
| 2.3.15 | 2023-11-13 | #32468 | Further error grouping enhancements |
| 2.3.14 | 2023-11-06 | #32234 | Remove unused config option. |
| 2.3.13 | 2023-11-08 | #32125 | fix compiler warnings |
| 2.3.12 | 2023-11-08 | #32309 | Revert: Use Typed object for connection config |
| 2.3.11 | 2023-11-07 | #32147 | Use Typed object for connection config |
| 2.3.10 | 2023-11-07 | #32261 | Further improve error reporting |
| 2.3.9 | 2023-11-07 | #32112 | GCS staging mode: reduce flush frequency to use rate limit more efficiently |
| 2.3.8 | 2023-11-06 | #32026 | Move SAFE_CAST transaction to separate transactions |
| 2.3.7 | 2023-11-06 | #32190 | Further improve error reporting |
| 2.3.6 | 2023-11-06 | #32193 | Adopt java CDK version 0.4.1. |
| 2.3.5 | 2023-11-02 | #31983 | Improve error reporting |
| 2.3.4 | 2023-10-31 | #32010 | Add additional data centers. |
| 2.3.3 | 2023-10-30 | #31985 | Delay upgrade deadline to Nov 7 |
| 2.3.2 | 2023-10-30 | #31960 | Adopt java CDK version 0.2.0. |
| 2.3.1 | 2023-10-27 | #31529 | Performance enhancement (switch to a merge statement for incremental-dedup syncs) |
| 2.3.0 | 2023-10-25 | #31686 | Opt out flag for typed and deduped tables |
| 2.2.0 | 2023-10-25 | #31520 | Stop deduping raw table |
| 2.1.6 | 2023-10-23 | #31717 | Remove inadvertent Destination v2 check |
| 2.1.5 | 2023-10-17 | #30069 | Staging destination async |
| 2.1.4 | 2023-10-17 | #31191 | Improve typing+deduping performance by filtering new raw records on extracted_at |
| 2.1.3 | 2023-10-10 | #31358 | Stringify array and object types for type:string column in final table |
| 2.1.2 | 2023-10-10 | #31194 | Deallocate unused per stream buffer memory when empty |
| 2.1.1 | 2023-10-10 | #31083 | Fix precision of numeric values in async destinations |
| 2.1.0 | 2023-10-09 | #31149 | No longer fail syncs when PKs are null - try do dedupe anyway |
| 2.0.26 | 2023-10-09 | #31198 | Clarify configuration groups |
| 2.0.25 | 2023-10-09 | #31185 | Increase staging file upload timeout to 5 minutes |
| 2.0.24 | 2023-10-06 | #31139 | Bump CDK version |
| 2.0.23 | 2023-10-06 | #31129 | Reduce async buffer size |
| 2.0.22 | 2023-10-04 | #31082 | Revert null PK checks |
| 2.0.21 | 2023-10-03 | #31028 | Update timeout |
| 2.0.20 | 2023-09-26 | #30779 | Final table PK columns become non-null and skip check for null PKs in raw records (performance) |
| 2.0.19 | 2023-09-26 | #30775 | Increase async block size |
| 2.0.18 | 2023-09-27 | #30739 | Fix column name collision detection |
| 2.0.17 | 2023-09-26 | #30696 | Attempt unsafe typing operations with an exception clause |
| 2.0.16 | 2023-09-22 | #30697 | Improve resiliency to unclean exit during schema change |
| 2.0.15 | 2023-09-21 | #30640 | Handle streams with identical name and namespace |
| 2.0.14 | 2023-09-20 | #30069 | Staging destination async |
| 2.0.13 | 2023-09-19 | #30592 | Internal code changes |
| 2.0.12 | 2023-09-19 | #30319 | Improved testing |
| 2.0.11 | 2023-09-18 | #30551 | GCS Staging is first loading method option |
| 2.0.10 | 2023-09-15 | #30491 | Improve error message display |
| 2.0.9 | 2023-09-14 | #30439 | Fix a transient error |
| 2.0.8 | 2023-09-12 | #30364 | Add log message |
| 2.0.7 | 2023-08-29 | #29878 | Internal code changes |
| 2.0.6 | 2023-09-05 | #29917 | Improve performance by changing metadata error array construction from ARRAY_CONCAT to ARRAY_AGG |
| 2.0.5 | 2023-08-31 | #30020 | Run typing and deduping tasks in parallel |
| 2.0.4 | 2023-09-05 | #30117 | Type and Dedupe at sync start and then every 6 hours |
| 2.0.3 | 2023-09-01 | #30056 | Internal refactor, no behavior change |
| 2.0.2 | 2023-09-01 | #30120 | Improve performance on very wide streams by skipping SAFE_CAST on strings |
| 2.0.1 | 2023-08-29 | #29972 | Publish a new version to supersede old v2.0.0 |
| 2.0.0 | 2023-08-27 | #29783 | Destinations V2 |
| 1.10.2 | 2023-08-24 | #29805 | Destinations v2: Don’t soft reset in migration |
| 1.10.1 | 2023-08-23 | #29774 | Destinations v2: Don’t soft reset overwrite syncs |
| 1.10.0 | 2023-08-21 | #29636 | Destinations v2: Several Critical Bug Fixes (cursorless dedup, improved floating-point handling, improved special characters handling; improved error handling) |
| 1.9.1 | 2023-08-21 | #28687 | Under the hood: Add dependency on Java CDK v0.0.1. |
| 1.9.0 | 2023-08-17 | #29560 | Destinations v2: throw an error on disallowed column name prefixes |
| 1.8.1 | 2023-08-17 | #29522 | Migration BugFix - ensure raw dataset created |
| 1.8.0 | 2023-08-17 | #29498 | Fix checkpointing logic in GCS staging mode |
| 1.7.8 | 2023-08-15 | #29461 | Migration BugFix - ensure migration happens before table creation for GCS staging. |
| 1.7.7 | 2023-08-11 | #29381 | Destinations v2: Add support for streams with no columns |
| 1.7.6 | 2023-08-04 | #28894 | Destinations v2: Add v1 -> v2 migration Logic |
| 1.7.5 | 2023-08-04 | #29106 | Destinations v2: handle unusual CDC deletion edge case |
| 1.7.4 | 2023-08-04 | #29089 | Destinations v2: improve special character handling in column names |
| 1.7.3 | 2023-08-03 | #28890 | Internal code updates; improved testing |
| 1.7.2 | 2023-08-02 | #28976 | Fix composite PK handling in v1 mode |
| 1.7.1 | 2023-08-02 | #28959 | Destinations v2: Fix CDC syncs in non-dedup mode |
| 1.7.0 | 2023-08-01 | #28894 | Destinations v2: Open up early access program opt-in |
| 1.6.0 | 2023-07-26 | #28723 | Destinations v2: Change raw table dataset and naming convention |
| 1.5.8 | 2023-07-25 | #28721 | Destinations v2: Handle cursor change across syncs |
| 1.5.7 | 2023-07-24 | #28625 | Destinations v2: Limit Clustering Columns to 4 |
| 1.5.6 | 2023-07-21 | #28580 | Destinations v2: Create dataset in user-specified location |
| 1.5.5 | 2023-07-20 | #28490 | Destinations v2: Fix schema change detection in OVERWRITE mode when existing table is empty; other code refactoring |
| 1.5.4 | 2023-07-17 | #28382 | Destinations v2: Schema Change Detection |
| 1.5.3 | 2023-07-14 | #28345 | Increment patch to trigger a rebuild |
| 1.5.2 | 2023-07-05 | #27936 | Internal scaffolding change for future development |
| 1.5.1 | 2023-06-30 | #27891 | Revert bugged update |
| 1.5.0 | 2023-06-27 | #27781 | License Update: Elv2 |
| 1.4.6 | 2023-06-28 | #27268 | Internal scaffolding change for future development |
| 1.4.5 | 2023-06-21 | #27555 | Reduce image size |
| 1.4.4 | 2023-05-25 | #26585 | Small tweak in logs for clarity |
| 1.4.3 | 2023-05-17 | #26213 | Fix bug in parsing file buffer config count |
| 1.4.2 | 2023-05-10 | #25925 | Testing update. Normalization tests are now done in the destination container. |
| 1.4.1 | 2023-05-11 | #25993 | Internal library update |
| 1.4.0 | 2023-04-29 | #25570 | Internal library update. Bumping version to stay in sync with BigQuery-denormalized. |
| 1.3.4 | 2023-04-28 | #25588 | Internal scaffolding change for future development |
| 1.3.3 | 2023-04-27 | #25346 | Internal code cleanup |
| 1.3.1 | 2023-04-20 | #25097 | Internal scaffolding change for future development |
| 1.3.0 | 2023-04-19 | #25287 | Add parameter to configure the number of file buffers when GCS is used as the loading method |
| 1.2.20 | 2023-04-12 | #25122 | Add additional data centers |
| 1.2.19 | 2023-03-29 | #24671 | Fail faster in certain error cases |
| 1.2.18 | 2023-03-23 | #24447 | Set the Service Account Key JSON field to always_show: true so that it isn’t collapsed into an optional fields section |
| 1.2.17 | 2023-03-17 | #23788 | S3-Parquet: added handler to process null values in arrays |
| 1.2.16 | 2023-03-10 | #23931 | Added support for periodic buffer flush |
| 1.2.15 | 2023-03-10 | #23466 | Changed S3 Avro type from Int to Long |
| 1.2.14 | 2023-02-08 | #22497 | Fixed table already exists error |
| 1.2.13 | 2023-01-26 | #20631 | Added support for destination checkpointing with staging |
| 1.2.12 | 2023-01-18 | #21087 | Wrap Authentication Errors as Config Exceptions |
| 1.2.11 | 2023-01-18 | #21144 | Added explicit error message if sync fails due to a config issue |
| 1.2.9 | 2022-12-14 | #20501 | Report GCS staging failures that occur during connection check |
| 1.2.8 | 2022-11-22 | #19489 | Added non-billable projects handle to check connection stage |
| 1.2.7 | 2022-11-11 | #19358 | Fixed check method to capture mismatch dataset location |
| 1.2.6 | 2022-11-10 | #18554 | Improve check connection method to handle more errors |
| 1.2.5 | 2022-10-19 | #18162 | Improve error logs |
| 1.2.4 | 2022-09-26 | #16890 | Add user-agent header |
| 1.2.3 | 2022-09-22 | #17054 | Respect stream namespace |
| 1.2.1 | 2022-09-14 | #15668 | (bugged, do not use) Wrap logs in AirbyteLogMessage |
| 1.2.0 | 2022-09-09 | #14023 | (bugged, do not use) Cover arrays only if they are nested |
| 1.1.16 | 2022-09-01 | #16243 | Fix Json to Avro conversion when there is field name clash from combined restrictions (anyOf, oneOf, allOf fields) |
| 1.1.15 | 2022-08-22 | #15787 | Throw exception if job failed |
| 1.1.14 | 2022-08-03 | #14784 | Enabling Application Default Credentials |
| 1.1.13 | 2022-08-02 | #14801 | Fix multiple log bindings |
| 1.1.12 | 2022-08-02 | #15180 | Fix standard loading mode |
| 1.1.11 | 2022-06-24 | #14114 | Remove “additionalProperties”: false from specs for connectors with staging |
| 1.1.10 | 2022-06-16 | #13852 | Updated stacktrace format for any trace message errors |
| 1.1.9 | 2022-06-17 | #13753 | Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager |
| 1.1.8 | 2022-06-07 | #13579 | Always check GCS bucket for GCS loading method to catch invalid HMAC keys. |
| 1.1.7 | 2022-06-07 | #13424 | Reordered fields for specification. |
| 1.1.6 | 2022-05-15 | #12768 | Clarify that the service account key json field is required on cloud. |
| 1.1.5 | 2022-05-12 | #12805 | Updated to latest base-java to emit AirbyteTraceMessage on error. |
| 1.1.4 | 2022-05-04 | #12578 | In JSON to Avro conversion, log JSON field values that do not follow Avro schema for debugging. |
| 1.1.3 | 2022-05-02 | #12528 | Update Dataset location field description |
| 1.1.2 | 2022-04-29 | #12477 | Dataset location is a required field |
| 1.1.1 | 2022-04-15 | #12068 | Fixed bug with GCS bucket conditional binding |
| 1.1.0 | 2022-04-06 | #11776 | Use serialized buffering strategy to reduce memory consumption. |
| 1.0.2 | 2022-03-30 | #11620 | Updated spec |
| 1.0.1 | 2022-03-24 | #11350 | Improve check performance |
| 1.0.0 | 2022-03-18 | #11238 | Updated spec and documentation |
| 0.6.12 | 2022-03-18 | #10793 | Fix namespace with invalid characters |
| 0.6.11 | 2022-03-03 | #10755 | Make sure to kill children threads and stop JVM |
| 0.6.8 | 2022-02-14 | #10256 | Add -XX:+ExitOnOutOfMemoryError JVM option |
| 0.6.6 | 2022-02-01 | #9959 | Fix null pointer exception from buffered stream consumer. |
| 0.6.6 | 2022-01-29 | #9745 | Integrate with Sentry. |
| 0.6.5 | 2022-01-18 | #9573 | BigQuery Destination : update description for some input fields |
| 0.6.4 | 2022-01-17 | #8383 | Support dataset-id prefixed by project-id |
| 0.6.3 | 2022-01-12 | #9415 | BigQuery Destination : Fix GCS processing of Facebook data |
| 0.6.2 | 2022-01-10 | #9121 | Fixed check method for GCS mode to verify if all roles assigned to user |
| 0.6.1 | 2021-12-22 | #9039 | Added part_size configuration to UI for GCS staging |
| 0.6.0 | 2021-12-17 | #8788 | BigQuery/BiqQuery denorm Destinations : Add possibility to use different types of GCS files |
| 0.5.1 | 2021-12-16 | #8816 | Update dataset locations |
| 0.5.0 | 2021-10-26 | #7240 | Output partitioned/clustered tables |
| 0.4.1 | 2021-10-04 | #6733 | Support dataset starting with numbers |
| 0.4.0 | 2021-08-26 | #5296 | Added GCS Staging uploading option |
| 0.3.12 | 2021-08-03 | #3549 | Add optional arg to make a possibility to change the BigQuery client’s chunk\buffer size |
| 0.3.11 | 2021-07-30 | #5125 | Enable additionalPropertities in spec.json |
| 0.3.10 | 2021-07-28 | #3549 | Add extended logs and made JobId filled with region and projectId |
| 0.3.9 | 2021-07-28 | #5026 | Add sanitized json fields in raw tables to handle quotes in column names |
| 0.3.6 | 2021-06-18 | #3947 | Service account credentials are now optional. |
| 0.3.4 | 2021-06-07 | #3277 | Add dataset location option |