Advanced Analytica Advanced Analytica: IBOM
← All connectors
BigQuery logo

Destination connector

BigQuery

Send governed data into BigQuery with a clear, practical setup guide.

Version 3.2.3 certified generally available

BigQuery

Set up the BigQuery destination connector by choosing a data loading method and configuring the connector in the Airbyte UI.

This page shows you how to set up the BigQuery destination connector.

Prerequisites

Setup guide

Step 1: Choose a data loading method

This loading method is recommended for production workloads. It uploads data to a GCS bucket, then uses a COPY job to load it into BigQuery. This provides better performance for large data volumes.

To use a Google Cloud Storage bucket:

  1. Create a Cloud Storage bucket with the Protection Tools set to none or Object versioning. Make sure the bucket doesn’t have a retention policy.
  2. Create an HMAC key and access ID.
  3. Grant the Storage Object Admin role to the Google Cloud Service Account. Use the same service account that you configure for BigQuery access in the BigQuery connector setup step.
  4. Make sure the machine running Airbyte can access your Cloud Storage bucket. To check access, use the check connection tool in the UI.

Encrypt your bucket with a Google-managed encryption key, the default setting for new buckets. Buckets using customer-managed encryption keys (CMEK) aren’t currently supported. You can view this setting in the “Configuration” tab of your GCS bucket, in the Encryption type row.

Use Batched Standard Inserts

This method offers a simpler setup without external staging. It uses the BigQuery SDK to stream data directly. It’s suitable for smaller data volumes or quick testing. BigQuery manages the staging files and deletes them automatically after loading finishes.

Step 2: Set up the BigQuery connector

  1. Log in to your Airbyte Cloud or Airbyte Open Source account.
  2. Click Destinations, then click + New destination.
  3. On the Set up the destination page, select BigQuery from the Destination type dropdown.
  4. Enter a name for the BigQuery connector.
  5. For Project ID, enter your Google Cloud project ID.
  6. For Dataset Location, select your BigQuery dataset’s location.

:::warning You can’t change the location later. :::

  1. For Default Dataset ID, enter the BigQuery Dataset ID.
  2. For Loading Method, select GCS Staging (Recommended) or Batched Standard Inserts.
  3. For Service Account Key JSON (Required for cloud, optional for open-source), enter the Google Cloud Service Account Key in JSON format.

:::note Copy everything in the Account Key JSON file, including the brackets. :::

  1. For Transformation Query Run Type (Optional), select interactive to have BigQuery run interactive query jobs, or select batch to have BigQuery run batch queries.

:::note Interactive queries run as soon as possible and count towards daily concurrent quotas and limits. Batch queries run when idle resources become available in the BigQuery shared resource pool. If a query hasn’t started within 24 hours, BigQuery changes its job priority to interactive. Batch queries don’t count towards your concurrent rate limit, so you can start many queries at once. :::

  1. For Google BigQuery Client Chunk Size (Optional), keep the default value of 15 MiB. If the destination later has networking or memory management problems, try reducing the chunk size. The sync will run more slowly but will be more likely to succeed.

  2. For Job Execution Project ID (Optional), leave the field empty to run BigQuery jobs in the project entered as Project ID. To use another project, enter its ID. For example, this keeps ingestion and analytics workloads from competing for the same project quotas.

:::note Query, load and copy jobs then use the job project’s quotas. Data still lands in datasets under Project ID. The service account needs the roles/bigquery.jobUser role on the job project, alongside its existing roles on the dataset project. :::

Supported sync modes

The BigQuery destination connector supports these sync modes:

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedYes
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedYes

Output schema

The final table includes these fields, alongside the columns declared in your stream schema:

  • airbyte_raw_id
  • _airbyte_generation_id
  • airbyte_extracted_at
  • _airbyte_meta

See here for more information about these fields.

BigQuery output tables use daily partitions based on the Time-unit column airbyte_extracted_at. They are clustered by airbyte_extracted_at and the table Primary Keys. Partition boundaries use UTC time. Use a predicate filter with a WHERE clause to limit the partitions scanned. Filters on the partitioning column prune partitions and reduce query costs. Airbyte doesn’t enable Require partition filter, but you can toggle it by updating the produced tables.

Legacy Raw Tables schema

If you enable Legacy raw tables, the connector writes tables in this format.

By default, Airbyte writes each stream to its own raw table in the airbyte_internal dataset. You can override this using the Airbyte Internal Table Dataset Name option. Contents in the raw table are NOT deduplicated.

The raw table contains these fields:

  • _airbyte_raw_id
  • _airbyte_generation_id
  • _airbyte_extracted_at
  • _airbyte_loaded_at
  • _airbyte_meta
  • _airbyte_data

_airbyte_data is a JSON blob containing the record’s data. See here for more information about the other fields.

BigQuery naming conventions

Follow BigQuery Datasets Naming conventions.

Airbyte converts invalid characters to _ characters when writing data. Datasets beginning with _ are hidden in the BigQuery Explorer panel. Therefore, Airbyte prepends converted namespaces with n.

Data type map

Airbyte typeBigQuery type
STRINGSTRING
STRING (BASE64)STRING
STRING (BIG_NUMBER)STRING
STRING (BIG_INTEGER)STRING
NUMBERNUMERIC
INTEGERINT64
BOOLEANBOOL
STRING (TIMESTAMP_WITH_TIMEZONE)TIMESTAMP
STRING (TIMESTAMP_WITHOUT_TIMEZONE)DATETIME
STRING (TIME_WITH_TIMEZONE)STRING
STRING (TIME_WITHOUT_TIMEZONE)TIME
DATEDATE
OBJECTJSON
ARRAYJSON

Troubleshooting

Permission errors

The service account doesn’t have the correct permissions.

  • Make sure the BigQuery service account has the BigQuery User and BigQuery Data Editor roles, or equivalent permissions.
  • If you select GCS staging mode, make sure the BigQuery service account can access the GCS bucket and path. Alternatively, assign the Cloud Storage Admin role, which includes a superset of the required permissions.

The HMAC key is incorrect.

  • Make sure the HMAC key belongs to the BigQuery service account. The service account must also have permission to access the GCS bucket and path.

HTTP 400 “Request had invalid euc header” during upload

If your sync fails with BigQueryException: 400 Bad Request and the message Request had invalid euc header:

  • This error comes from the Google BigQuery resumable-upload API and is usually temporary.
  • Retry the sync. The error resolves on the next attempt in most cases.
  • If every sync returns this error, the underlying causes can be complex. Google doesn’t publicly document this specific error. These steps may help you narrow down the issue:
    • If you self-manage Airbyte, check whether a proxy, VPN or firewall modifies HTTP headers on requests to bigquery.googleapis.com.
    • Check whether the service account key has been rotated or revoked since you configured the connection.
    • Try reducing Google BigQuery Client Chunk Size from the default 15 MiB to a smaller value, such as 5 MiB.
    • Try running fewer concurrent syncs to your BigQuery instance or table. Contention may contribute to the error.

Concurrent script query quota exceeded

If your sync fails with:

BigQueryException: Quota exceeded: Your project_and_region exceeded quota for
concurrent script queries per project.
  • This quota applies per project. Syncs share it with every other workload in that project, including analytics queries and Dataform pipelines.
  • Set Job Execution Project ID to a separate project to run connector jobs against that project’s quota. Data continues to land in datasets under Project ID.
  • Alternatively, reduce the number of streams or connections syncing concurrently to the project.

Load job timeouts

If your sync fails with Fail to complete a load job in big query:

  • BigQuery load jobs have a 30-minute wait timeout. Very large batches or heavy contention in the BigQuery queue can exceed this limit.
  • Concurrent syncs loading into the same BigQuery table aren’t supported and can also cause this timeout. See Stream uniqueness for details.
  • Try reducing each sync’s volume by using incremental sync mode or fewer streams per connection.

Tutorials

After setting up the BigQuery destination connector, explore these BigQuery tutorials:

Namespace support

This destination supports namespaces. The namespace maps to a BigQuery dataset.

Changelog

Expand to review
VersionDatePull RequestSubject
3.2.32026-10-3087606Retry transient BigQuery 5xx errors on all API calls, including job inserts and uploads.
3.2.22026-10-2289648Upgrade to Bulk CDK 1.1.9: fix critical CVEs in Apache MINA (mina-core 2.0.31), MINA SSHD 2.20.0 and BouncyCastle 1.86; netty 4.1.139, Hadoop 3.4.3, Parquet 1.18.1; AWS SDK 2.57.2; google-cloud-bigquery 2.73.0.
3.2.12026-10-0688144Fix Standard Inserts committing a partial batch to BigQuery when a sync fails or is cancelled.
3.2.02026-10-0187631Add optional Fusion S3 archiving for GCS Staging and Standard Inserts loads (off by default; enabled through environment configuration).
3.1.22026-09-09[85804](https://github.com/airbytehq/airbyte/pull/85804Update loading method UI: reorder to show GCS Staging (Recommended) first, improve descriptions for both loading methods.
3.1.12026-09-0979178Classify BigQuery custom quota exceeded errors as config errors instead of system errors.
3.1.02026-08-2585041Add optional job_project_id field for BigQuery job quota isolation
3.0.242026-08-2484985Upgrade to Bulk CDK 1.0.25.
3.0.232026-07-1481550Use CREATE TABLE IF NOT EXISTS for non-replace table creation to prevent accidental data loss
3.0.222026-07-1081635Restore PK NULL equality checks
3.0.212026-06-3081346Remove unnecessary NULL PK equality checks from merge SQL
3.0.202026-06-2680932Handle BigQueryException in getGenerationId() for legacy tables without _airbyte_generation_id column.
3.0.192026-05-2178239Promoting release candidate 3.0.19-rc.1 to a main version.
3.0.19-rc.12026-05-1978239Upgrade CDK to 1.0.13. Progressive rollout.
3.0.182026-03-3175913Finalize upgrade BigQuery Cloud dependencies and CDK version
3.0.18-rc.12026-03-2775541Upgrade BigQuery Cloud dependencies and CDK version
3.0.172026-01-2872427Finalize upgrade CDK to 0.2.0
3.0.17-rc.12026-01-2672296Upgrade CDK to 0.2.0
3.0.162025-11-2567401Add backticks to column names in SQL generation to prevent syntax errors.
3.0.152025-11-1369307Handle out-of-range timestamps/times.
3.0.142025-11-1169231Upgrade to Bulk CDK 0.1.74.
3.0.132025-11-0569126Upgrade to Bulk CDK 0.1.61.
3.0.122025-10-3169083Fail loudly if Bigquery detects bad records.
3.0.112025-10-2868671Log record count per load job.
3.0.102025-10-2167153Implement new proto schema implementation
3.0.92025-10-1768152Update to new TableOperationsClient interface.
3.0.82025-10-0667078Remove memory limit for sync jobs to improve performance and resource utilization.
3.0.72025-09-0365905Promoting release candidate 3.0.7-rc.1 to a main version.
3.0.7-rc.12025-08-2765114Implement SOCKET+PROTO mode support.
3.0.62025-07-2463700Improve error reporting for Billing errors.
3.0.52025-07-1563312Pull in upstream fix to support null chars in GCS staging mode.
3.0.42025-07-1563327Improve error reporting for Billing errors.
3.0.32025-07-0262495Improve error reporting for misconfigured connections; improve support for complex types.
3.0.22025-06-2662106Improve error reporting during schema evolution.
3.0.12025-06-2662085Correctly handle stream names/namespaces and column names which start with a digit.
3.0.02025-06-2559752Upgrade to direct-load tables; add option for soft CDC deletes.
2.12.42025-06-2462045Promoting release candidate 2.12.4-rc.6 to a main version.
2.12.4-rc.62025-06-24620412.12.4 RC 6 More retries is better retries
2.12.4-rc.52025-06-23620162.12.4 RC 5 Handle records up to 20MiB
2.12.4-rc.42025-06-23620142.12.4 RC 4 Improve JSON schema parser’s behavior on invalid source schemas
2.12.4-rc.32025-06-18617022.12.4 RC 3 Fix an issue with streams with special chars on bulk loader
2.12.4-rc.22025-06-18617002.12.4 RC 2 (throw more informative error on invalid catalog)
2.12.4-rc.12025-06-16616372.12.4 RC 1 (theoretically equivalent to 2.12.0, but with fixed global state handling in CDK)
2.12.32025-06-1661648This is not the release you are looking for. identical to 2.10.2
2.12.12025-06-1361588Publish version to account for possible duplicate publishing in pipeline. Noop change. WARNING: THIS HAS A BUG. DO NOT USE.
2.12.02025-06-0661432Improve performance in GCS staging mode by writing GZIP-compressed files.
2.11.42025-05-3061018Always emit a useful error message when erroring during sync startup.
2.11.32025-06-0261321CHECK operation doesn’t need to drop the dataset anymore.
2.11.22025-05-2960986CHECK operation uses configured default dataset instead of airbyte_internal_test.
2.11.12025-05-2960973Improve error recovery when a stream’s sync mode and schema are both changed and the previous sync exited uncleanly.
2.11.02025-05-2760922Promoting release candidate 2.11.0-rc.7 to a main version.
2.11.0-rc.72025-05-2260865Add a random element to the check sync namespace
2.11.0-rc.62025-05-2260859Improve bigquery quota usage in standard inserts for streams with heavy load
2.11.0-rc.52025-05-2160841Improve behavior in syncs with many state messages
2.11.0-rc.42025-05-1960791Fix bug in detecting schema change when stream has no columns
2.11.0-rc.32025-05-1660324Fix bug in syncs with many state message
2.11.0-rc.22025-05-1560302Fix bugs in handling special characters / uppercase characters in column names
2.11.0-rc.12025-05-0859729Version 2.11.0 RC 1
2.10.22025-04-2959142Add europe-north2 region.
2.10.12025-04-1457569Remove StringSubstitutor usage
2.10.02025-04-0256982Change default raw table partitioning scheme to be on _airbyte_extracted_at; upgrade CDK
2.9.32025-03-2456355Upgrade to airbyte/java-connector-base:2.0.1 to be M4 compatible.
2.9.22025-01-1051482Use a non root base image
2.9.12024-12-1849902Use a base image: airbyte/java-connector-base:1.0.0
2.9.02024-09-1845433upgrade all dependencies
2.8.92024-08-2044476Increase message parsing limit to 100mb
2.8.82024-08-2244526Revert protocol compliance fix
2.8.72024-08-1542504Fix bug in refreshes logic (already mitigated in platform, just fixing protocol compliance)
2.8.62024-07-3042511Added a copy operation to validate copy permissions in the check function
2.8.52024-07-2242407Batched Standard Inserts is default loading mode
2.8.42024-07-1541968Don’t hang forever on empty stream list; shorten error message on INCOMPLETE stream status
2.8.32024-07-1241674Upgrade to latest CDK
2.8.22024-07-0841041Fix resume logic in truncate refreshes to prevent data loss
2.8.12024-06-2539379Removing requirement of a redundant permission bigquery.datasets.create permission
2.8.02024-06-2139904Convert all production code to kotlin
2.7.12024-06-1739526Internal code change for improved error reporting in case of source/platform failure (INCOMPLETE stream status / empty ConfiguredCatalog).
2.7.02024-06-1738713Support for refreshes and resumable full refresh. WARNING: You must upgrade to platform 0.63.7 before upgrading to this connector version.
2.6.32024-06-1038331Internal code changes in preparation for future feature release
2.6.22024-06-0738764Increase message length limit to 50MiB
2.6.12024-05-2938770Internal code change (switch to CDK artifact)
2.6.02024-05-2838359Propagate airbyte_meta from sources; add generation_id column
2.5.12024-05-2238591Bugfix to include forward-slash when cleaning up stage
2.5.02024-05-2238132Major rewrite of existing code, Adapting to CDK changes introduced in 38107
2.4.202024-05-1338131Cleanup BigQueryWriteConfig and reuse StreamConfig; Adapt to StreamConfig signature changes
2.4.192024-05-1038125adopt latest CDK code
2.4.182024-05-1038111No functional changes, deleting unused code
2.4.172024-05-0938098Internal build structure change
2.4.162024-05-0837714Adopt CDK 0.34.0
2.4.152024-05-0734611Adopt CDK 0.33.2
2.4.142024-02-2537584Remove unused insecure dependencies from CDK
2.4.132024-02-2536899adopt latest CDK
2.4.122024-03-0435315Adopt CDK 0.23.11
2.4.112024-02-2235569Fix logging bug.
2.4.102024-02-1535240Adopt CDK 0.20.9
2.4.92024-02-1535285Adopt CDK 0.20.8
2.4.82024-02-1235144Adopt CDK 0.20.2
2.4.72024-02-1235111Adopt CDK 0.20.1
2.4.62024-02-0934575Adopt CDK 0.20.0
2.4.52024-02-0834745Adopt CDK 0.19.0
2.4.42024-02-0835027Upgrade CDK to 0.17.1
2.4.32024-02-0134728Upgrade CDK to 0.16.4; Notable changes from 0.14.2, 0.15.1 and 0.16.3
2.4.22024-01-2434451Improve logging for unparseable input
2.4.12024-01-2434458Improve error reporting
2.4.02024-01-2434468Upgrade CDK to 0.14.0
2.3.312024-01-22#34023Combine DDL operations into a single execution
2.3.302024-01-12#34226Upgrade CDK to 0.12.0; Cleanup dependencies
2.3.292024-01-09#34003Fix loading credentials from GCP Env
2.3.282024-01-08#34021Add idempotency ids in dummy insert for check call
2.3.272024-01-05#33948Skip retrieving initial table state when setup fails
2.3.262024-01-04#33730Internal code structure changes
2.3.252023-12-20#33704Update to java CDK 0.10.0 (no changes)
2.3.242023-12-20#33697Stop creating unnecessary tmp tables
2.3.232023-12-18#33124Make Schema Creation Separate from Table Creation
2.3.222023-12-14#33451Remove old spec option
2.3.212023-12-13#33232Only run typing+deduping for a stream if the stream had any records
2.3.202023-12-08#33263Adopt java CDK version 0.7.0
2.3.192023-12-07#32326Update common T&D interfaces
2.3.182023-12-04#33084T&D SQL statements moved to debug log level
2.3.172023-12-04#33078Further increase gcs COPY timeout
2.3.162023-11-14#32526Clean up memory manager logs.
2.3.152023-11-13#32468Further error grouping enhancements
2.3.142023-11-06#32234Remove unused config option.
2.3.132023-11-08#32125fix compiler warnings
2.3.122023-11-08#32309Revert: Use Typed object for connection config
2.3.112023-11-07#32147Use Typed object for connection config
2.3.102023-11-07#32261Further improve error reporting
2.3.92023-11-07#32112GCS staging mode: reduce flush frequency to use rate limit more efficiently
2.3.82023-11-06#32026Move SAFE_CAST transaction to separate transactions
2.3.72023-11-06#32190Further improve error reporting
2.3.62023-11-06#32193Adopt java CDK version 0.4.1.
2.3.52023-11-02#31983Improve error reporting
2.3.42023-10-31#32010Add additional data centers.
2.3.32023-10-30#31985Delay upgrade deadline to Nov 7
2.3.22023-10-30#31960Adopt java CDK version 0.2.0.
2.3.12023-10-27#31529Performance enhancement (switch to a merge statement for incremental-dedup syncs)
2.3.02023-10-25#31686Opt out flag for typed and deduped tables
2.2.02023-10-25#31520Stop deduping raw table
2.1.62023-10-23#31717Remove inadvertent Destination v2 check
2.1.52023-10-17#30069Staging destination async
2.1.42023-10-17#31191Improve typing+deduping performance by filtering new raw records on extracted_at
2.1.32023-10-10#31358Stringify array and object types for type:string column in final table
2.1.22023-10-10#31194Deallocate unused per stream buffer memory when empty
2.1.12023-10-10#31083Fix precision of numeric values in async destinations
2.1.02023-10-09#31149No longer fail syncs when PKs are null - try do dedupe anyway
2.0.262023-10-09#31198Clarify configuration groups
2.0.252023-10-09#31185Increase staging file upload timeout to 5 minutes
2.0.242023-10-06#31139Bump CDK version
2.0.232023-10-06#31129Reduce async buffer size
2.0.222023-10-04#31082Revert null PK checks
2.0.212023-10-03#31028Update timeout
2.0.202023-09-26#30779Final table PK columns become non-null and skip check for null PKs in raw records (performance)
2.0.192023-09-26#30775Increase async block size
2.0.182023-09-27#30739Fix column name collision detection
2.0.172023-09-26#30696Attempt unsafe typing operations with an exception clause
2.0.162023-09-22#30697Improve resiliency to unclean exit during schema change
2.0.152023-09-21#30640Handle streams with identical name and namespace
2.0.142023-09-20#30069Staging destination async
2.0.132023-09-19#30592Internal code changes
2.0.122023-09-19#30319Improved testing
2.0.112023-09-18#30551GCS Staging is first loading method option
2.0.102023-09-15#30491Improve error message display
2.0.92023-09-14#30439Fix a transient error
2.0.82023-09-12#30364Add log message
2.0.72023-08-29#29878Internal code changes
2.0.62023-09-05#29917Improve performance by changing metadata error array construction from ARRAY_CONCAT to ARRAY_AGG
2.0.52023-08-31#30020Run typing and deduping tasks in parallel
2.0.42023-09-05#30117Type and Dedupe at sync start and then every 6 hours
2.0.32023-09-01#30056Internal refactor, no behavior change
2.0.22023-09-01#30120Improve performance on very wide streams by skipping SAFE_CAST on strings
2.0.12023-08-29#29972Publish a new version to supersede old v2.0.0
2.0.02023-08-27#29783Destinations V2
1.10.22023-08-24#29805Destinations v2: Don’t soft reset in migration
1.10.12023-08-23#29774Destinations v2: Don’t soft reset overwrite syncs
1.10.02023-08-21#29636Destinations v2: Several Critical Bug Fixes (cursorless dedup, improved floating-point handling, improved special characters handling; improved error handling)
1.9.12023-08-21#28687Under the hood: Add dependency on Java CDK v0.0.1.
1.9.02023-08-17#29560Destinations v2: throw an error on disallowed column name prefixes
1.8.12023-08-17#29522Migration BugFix - ensure raw dataset created
1.8.02023-08-17#29498Fix checkpointing logic in GCS staging mode
1.7.82023-08-15#29461Migration BugFix - ensure migration happens before table creation for GCS staging.
1.7.72023-08-11#29381Destinations v2: Add support for streams with no columns
1.7.62023-08-04#28894Destinations v2: Add v1 -> v2 migration Logic
1.7.52023-08-04#29106Destinations v2: handle unusual CDC deletion edge case
1.7.42023-08-04#29089Destinations v2: improve special character handling in column names
1.7.32023-08-03#28890Internal code updates; improved testing
1.7.22023-08-02#28976Fix composite PK handling in v1 mode
1.7.12023-08-02#28959Destinations v2: Fix CDC syncs in non-dedup mode
1.7.02023-08-01#28894Destinations v2: Open up early access program opt-in
1.6.02023-07-26#28723Destinations v2: Change raw table dataset and naming convention
1.5.82023-07-25#28721Destinations v2: Handle cursor change across syncs
1.5.72023-07-24#28625Destinations v2: Limit Clustering Columns to 4
1.5.62023-07-21#28580Destinations v2: Create dataset in user-specified location
1.5.52023-07-20#28490Destinations v2: Fix schema change detection in OVERWRITE mode when existing table is empty; other code refactoring
1.5.42023-07-17#28382Destinations v2: Schema Change Detection
1.5.32023-07-14#28345Increment patch to trigger a rebuild
1.5.22023-07-05#27936Internal scaffolding change for future development
1.5.12023-06-30#27891Revert bugged update
1.5.02023-06-27#27781License Update: Elv2
1.4.62023-06-28#27268Internal scaffolding change for future development
1.4.52023-06-21#27555Reduce image size
1.4.42023-05-25#26585Small tweak in logs for clarity
1.4.32023-05-17#26213Fix bug in parsing file buffer config count
1.4.22023-05-10#25925Testing update. Normalization tests are now done in the destination container.
1.4.12023-05-11#25993Internal library update
1.4.02023-04-29#25570Internal library update. Bumping version to stay in sync with BigQuery-denormalized.
1.3.42023-04-28#25588Internal scaffolding change for future development
1.3.32023-04-27#25346Internal code cleanup
1.3.12023-04-20#25097Internal scaffolding change for future development
1.3.02023-04-19#25287Add parameter to configure the number of file buffers when GCS is used as the loading method
1.2.202023-04-12#25122Add additional data centers
1.2.192023-03-29#24671Fail faster in certain error cases
1.2.182023-03-23#24447Set the Service Account Key JSON field to always_show: true so that it isn’t collapsed into an optional fields section
1.2.172023-03-17#23788S3-Parquet: added handler to process null values in arrays
1.2.162023-03-10#23931Added support for periodic buffer flush
1.2.152023-03-10#23466Changed S3 Avro type from Int to Long
1.2.142023-02-08#22497Fixed table already exists error
1.2.132023-01-26#20631Added support for destination checkpointing with staging
1.2.122023-01-18#21087Wrap Authentication Errors as Config Exceptions
1.2.112023-01-18#21144Added explicit error message if sync fails due to a config issue
1.2.92022-12-14#20501Report GCS staging failures that occur during connection check
1.2.82022-11-22#19489Added non-billable projects handle to check connection stage
1.2.72022-11-11#19358Fixed check method to capture mismatch dataset location
1.2.62022-11-10#18554Improve check connection method to handle more errors
1.2.52022-10-19#18162Improve error logs
1.2.42022-09-26#16890Add user-agent header
1.2.32022-09-22#17054Respect stream namespace
1.2.12022-09-14#15668(bugged, do not use) Wrap logs in AirbyteLogMessage
1.2.02022-09-09#14023(bugged, do not use) Cover arrays only if they are nested
1.1.162022-09-01#16243Fix Json to Avro conversion when there is field name clash from combined restrictions (anyOf, oneOf, allOf fields)
1.1.152022-08-22#15787Throw exception if job failed
1.1.142022-08-03#14784Enabling Application Default Credentials
1.1.132022-08-02#14801Fix multiple log bindings
1.1.122022-08-02#15180Fix standard loading mode
1.1.112022-06-24#14114Remove “additionalProperties”: false from specs for connectors with staging
1.1.102022-06-16#13852Updated stacktrace format for any trace message errors
1.1.92022-06-17#13753Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager
1.1.82022-06-07#13579Always check GCS bucket for GCS loading method to catch invalid HMAC keys.
1.1.72022-06-07#13424Reordered fields for specification.
1.1.62022-05-15#12768Clarify that the service account key json field is required on cloud.
1.1.52022-05-12#12805Updated to latest base-java to emit AirbyteTraceMessage on error.
1.1.42022-05-04#12578In JSON to Avro conversion, log JSON field values that do not follow Avro schema for debugging.
1.1.32022-05-02#12528Update Dataset location field description
1.1.22022-04-29#12477Dataset location is a required field
1.1.12022-04-15#12068Fixed bug with GCS bucket conditional binding
1.1.02022-04-06#11776Use serialized buffering strategy to reduce memory consumption.
1.0.22022-03-30#11620Updated spec
1.0.12022-03-24#11350Improve check performance
1.0.02022-03-18#11238Updated spec and documentation
0.6.122022-03-18#10793Fix namespace with invalid characters
0.6.112022-03-03#10755Make sure to kill children threads and stop JVM
0.6.82022-02-14#10256Add -XX:+ExitOnOutOfMemoryError JVM option
0.6.62022-02-01#9959Fix null pointer exception from buffered stream consumer.
0.6.62022-01-29#9745Integrate with Sentry.
0.6.52022-01-18#9573BigQuery Destination : update description for some input fields
0.6.42022-01-17#8383Support dataset-id prefixed by project-id
0.6.32022-01-12#9415BigQuery Destination : Fix GCS processing of Facebook data
0.6.22022-01-10#9121Fixed check method for GCS mode to verify if all roles assigned to user
0.6.12021-12-22#9039Added part_size configuration to UI for GCS staging
0.6.02021-12-17#8788BigQuery/BiqQuery denorm Destinations : Add possibility to use different types of GCS files
0.5.12021-12-16#8816Update dataset locations
0.5.02021-10-26#7240Output partitioned/clustered tables
0.4.12021-10-04#6733Support dataset starting with numbers
0.4.02021-08-26#5296Added GCS Staging uploading option
0.3.122021-08-03#3549Add optional arg to make a possibility to change the BigQuery client’s chunk\buffer size
0.3.112021-07-30#5125Enable additionalPropertities in spec.json
0.3.102021-07-28#3549Add extended logs and made JobId filled with region and projectId
0.3.92021-07-28#5026Add sanitized json fields in raw tables to handle quotes in column names
0.3.62021-06-18#3947Service account credentials are now optional.
0.3.42021-06-07#3277Add dataset location option
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.