Advanced Analytica Advanced Analytica: IBOM
← All connectors
Google Cloud Storage (GCS) logo

Destination connector

Google Cloud Storage (GCS)

Send governed data into Google Cloud Storage (GCS) with a clear, practical setup guide.

Version 1.0.2 certified generally available

Google Cloud Storage (GCS)

Set up the Google Cloud Storage (GCS) destination connector with this guide.

The GCS destination writes each stream to one or more objects in a GCS bucket. Choose Avro, CSV, JSON Lines or Parquet as your output format. You can control the directory layout and object names with path templates. The connector authenticates with an HMAC key. It uses the GCS XML API, which is compatible with the Amazon S3 API.

:::info This page covers the Google Cloud Storage destination, which writes plain files. For an Apache Iceberg lakehouse on GCS, use the separate GCS Data Lake destination. :::

Prerequisites

  • A GCS bucket. Create a bucket or choose an existing one. Bucket retention policies and object holds must allow Airbyte to delete its objects. See Permissions.
  • A Google Cloud service account with an HMAC key. If your organisation uses the constraints/storage.restrictAuthTypes organisation policy, it must allow HMAC authentication.
  • Network access from the Airbyte deployment to https://storage.googleapis.com.

Set up the destination

Step 1: Create a service account and HMAC key

  1. Create a dedicated service account for Airbyte in the Google Cloud console.
  2. Create an HMAC key for that service account. Copy the Access ID and Secret immediately. Google Cloud doesn’t display the secret again.

Step 2: Grant permissions {#permissions}

Grant the service account the Storage Object User role (roles/storage.objectUser) on the bucket. A custom role must include at least these permissions:

storage.objects.create
storage.objects.get
storage.objects.list
storage.objects.delete
storage.multipartUploads.create
storage.multipartUploads.abort

Each permission supports a specific action:

PermissionUsed for
storage.objects.create, storage.multipartUploads.createWriting output objects. Objects are uploaded as multipart uploads.
storage.multipartUploads.abortCleaning up a multipart upload that fails part-way.
storage.objects.list, storage.objects.getDiscovering existing objects and reading their generation metadata so Full Refresh - Overwrite can remove stale data.
storage.objects.deleteDeleting the connection-check probe object and stale objects in Full Refresh - Overwrite syncs.

Step 3: Configure the destination in Airbyte

  1. In the Airbyte UI, go to Destinations, click New destination and select Google Cloud Storage (GCS).
  2. Enter a Destination name.
  3. Complete the fields described in Configuration.
  4. Click Set up destination. Airbyte checks the connection by uploading, listing and deleting a small probe object under the bucket path.

Configuration

FieldRequiredDefaultDescription
GCS Bucket NameYesName of the bucket that receives the data.
GCS Bucket PathYesPrefix inside the bucket under which all objects are written. A trailing / is optional.
GCS Bucket RegionNousLocation of the bucket. See GCS locations.
Authentication → HMAC Access IDYesAccess ID of the service account’s HMAC key.
Authentication → HMAC SecretYesSecret of the HMAC key.
Output FormatYesOne of Avro, CSV, JSON Lines, or Parquet, with format-specific options. See Output formats.
GCS Path FormatNo${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_Template for the object key below the bucket path. See GCS Path Format.
File Name PatternNo{part_number}{format_extension}Template for the final file name. See File Name Pattern.

Authentication

The connector supports HMAC key authentication only. It doesn’t support service account JSON keys, OAuth or Application Default Credentials. The connector accesses the GCS XML API through an S3-compatible client. Google Cloud errors therefore use S3 terminology, such as AccessDenied or SignatureDoesNotMatch. See Troubleshooting.

Encryption

The connector supports Google-managed and customer-managed encryption keys (CMEK). It doesn’t set an encryption key. Instead, it uses the bucket’s default encryption. Review this setting in the bucket’s Configuration tab under Encryption type.

Output layout

Each object key combines the bucket path with the resolved GCS Path Format and File Name Pattern:

<bucket-name>/<bucket-path>/<resolved GCS Path Format><resolved File Name Pattern>

The path format and file name are concatenated directly. If the path format doesn’t end with /, its final segment prefixes the file name. With the default settings, syncing the users stream in the public namespace produces objects such as:

my-bucket/landing/public/users/2026_10_30_1761820800000_0.csv.gz
↑         ↑       ↑      ↑     ↑          ↑             ↑ ↑
|         |       |      |     |          |             | format extension
|         |       |      |     |          |             part number
|         |       |      |     |          epoch milliseconds of the sync
|         |       |      |     sync date (YYYY_MM_DD)
|         |       |      stream name
|         |       namespace (omitted when the stream has none)
|         bucket path
bucket name

Each sync writes one or more part files per stream. Files roll over at roughly 200 MB compressed. Large streams therefore produce several part numbers from the same sync.

GCS Path Format

GCS Path Format defines the directory structure under the bucket path. Variables use the ${VARIABLE} syntax and are replaced when the sync starts. Consecutive / characters collapse, so an empty ${NAMESPACE} doesn’t create an empty directory.

VariableDescriptionExample
${NAMESPACE}Namespace of the stream (empty if none)public
${STREAM_NAME}Name of the streamusers
${YEAR}Year of the sync (UTC, four digits)2026
${MONTH}Month of the sync (UTC, zero-padded)10
${DAY}Day of the sync (UTC, zero-padded)30
${HOUR}Hour of the sync (UTC, zero-padded)09
${MINUTE}Minute of the sync (UTC, zero-padded)30
${SECOND}Second of the sync (UTC, zero-padded)45
${MILLISECOND}Millisecond of the sync (UTC)0123
${EPOCH}Milliseconds since the Unix epoch at the start of the sync1761820800000
${UUID}Random UUIDa1b2c3d4-e5f6-...
${SYNC_ID}Unique ID of the sync101

Leave the field empty to use the default template. Multiple syncs of the same stream never overwrite each other’s files. Either ${EPOCH} in the default template or the part number differs.

Example: Hive-style partitions

Create partitioned tables that Hive, Spark, BigQuery external tables and similar engines can read. End the template with / so each sync lands in its own partition directory:

${STREAM_NAME}/dt=${YEAR}-${MONTH}-${DAY}/

With the default File Name Pattern, you get:

my-bucket/landing/users/dt=2026-10-30/0.parquet
my-bucket/landing/users/dt=2026-10-30/1.parquet

The file name no longer includes a timestamp. For multiple daily append-mode runs, add {timestamp} or {sync_id} to the File Name Pattern:

{sync_id}_{part_number}{format_extension}

Example: flat layout without the stream name

Write all files for a stream directly under the bucket path. Use a template without directory separators, such as ${YEAR}_${MONTH}_${DAY}_. This layout requires one connection or bucket path per stream. Otherwise, streams share the same directory.

File Name Pattern

File Name Pattern defines the file name. Variables use the {variable} syntax.

VariableDescriptionExample
{date}Date of the sync in yyyy_MM_dd format2026_10_30
{date:yyyy_MM}Date of the sync in yyyy_MM format2026_10
{timestamp}Wall-clock time in milliseconds when the file is created1761820845123
{part_number}Part number of the file within the sync (0, 1, 2, …)0
{sync_id}Unique ID of the sync101
{format_extension}File extension including the compression suffix.csv.gz

Always include {part_number} to give each part file from the same sync a distinct name. Include {format_extension} so the extension matches the configured format. Leave the field empty to use the default pattern.

:::note Path format variables use ${VARIABLE}. File name variables use {variable}. :::

Sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

Each stream’s directory contains its complete data set. Treat the directory as the equivalent of a table.

  • Append modes adds new files containing only the new data.
  • Full Refresh - Overwrite writes the new files first. After a successful sync, it deletes the stream’s objects from earlier generations. The connector matches objects using the resolved path template, including any ${...} variables. It also uses the Airbyte generation ID stored in object metadata. This process removes stale files, even when the path contains date variables.

:::warning Full Refresh - Overwrite deletes objects matching the stream’s path template under the bucket path. Use a dedicated bucket or bucket path for Airbyte. Check your path configuration carefully to avoid deleting unrelated data. :::

Full Refresh - Overwrite never deletes objects written by connector versions before 1.0.0 because they lack generation metadata. Remove these objects manually after upgrading. See the migration guide.

Output formats

Every format includes these Airbyte metadata fields in each record:

FieldDescription
_airbyte_raw_idUUID assigned by Airbyte to the record.
_airbyte_extracted_atTime the record was extracted from the source, in Unix epoch milliseconds.
_airbyte_metaRecord metadata: sync_id and a changes array describing any modifications Airbyte made to the record (for example a field set to NULLED because it didn’t match the declared type).
_airbyte_generation_idGeneration of the stream the record belongs to. Full Refresh - Overwrite starts a new generation.

Avro

Apache Avro is a compact binary format with an embedded schema. The connector writes binary-encoded Avro object container files. It derives the Avro schema from the stream’s JSON schema. See JSON to Avro conversion for Airbyte’s JSON-to-Avro type mappings and current limitations.

Available compression codecs:

CodecOptions
No compression
deflateCompression level 0–9 (default 0). 0 is fastest with no compression, 9 is smallest and slowest.
bzip2
xzCompression level 0–9 (default 6). 0–3 are fast with medium compression; 4–6 are slower with high compression; 7–9 use larger dictionaries and more memory and only help for files above 8, 16, or 32 MiB respectively.
zstandardCompression level -5–22 (default 3). Negative levels are fast modes similar to lz4/snappy; levels above 9 are for archival use and levels above 18 use a lot of memory. Optionally include a checksum in each data block.
snappy

CSV

CSV output supports two flattening modes:

  • No flattening: the source record becomes a JSON string in the _airbyte_data column.
  • Root level flattening: each top-level source field becomes a separate column. Nested objects and arrays become JSON strings.
ColumnPresent whenDescription
_airbyte_raw_idAlwaysSee Output formats.
_airbyte_extracted_atAlways
_airbyte_metaAlwaysSerialized as a JSON string.
_airbyte_generation_idAlways
_airbyte_dataNo flatteningSource record serialized as a JSON string.
Source fieldsRoot level flatteningOne column per top-level source field.

For example, start with this source record:

{
  "user_id": 123,
  "name": {
    "first": "John",
    "last": "Doe"
  }
}

Without flattening:

_airbyte_raw_id_airbyte_extracted_at_airbyte_meta_airbyte_generation_id_airbyte_data
26d73cde-7eb1-4e1e-b7db-a4c03b4cf2061622135805000{"sync_id":10111,"changes":[]}11{"user_id":123,"name":{"first":"John","last":"Doe"}}

With root level flattening:

_airbyte_raw_id_airbyte_extracted_at_airbyte_meta_airbyte_generation_iduser_idname
26d73cde-7eb1-4e1e-b7db-a4c03b4cf2061622135805000{"sync_id":10111,"changes":[]}11123{"first":"John","last":"Doe"}

CSV files are GZIP-compressed by default (.csv.gz). Select No Compression in the format options to write plain .csv files.

JSON Lines (JSONL)

JSON Lines writes one JSON object per line. It supports the same No flattening and Root level flattening options as CSV. Without flattening, the source record sits under _airbyte_data. With root level flattening, the source fields sit beside the Airbyte metadata fields.

Example output without flattening for two source records:

{"_airbyte_raw_id":"26d73cde-7eb1-4e1e-b7db-a4c03b4cf206","_airbyte_extracted_at":1622135805000,"_airbyte_meta":{"sync_id":10111,"changes":[]},"_airbyte_generation_id":11,"_airbyte_data":{"user_id":123,"name":{"first":"John","last":"Doe"}}}
{"_airbyte_raw_id":"0a61de1b-9cdd-4455-a739-93572c9a5f20","_airbyte_extracted_at":1631948170000,"_airbyte_meta":{"sync_id":10112,"changes":[]},"_airbyte_generation_id":12,"_airbyte_data":{"user_id":456,"name":{"first":"Jane","last":"Roe"}}}

JSONL files are GZIP-compressed by default (.jsonl.gz). Select No Compression to write plain .jsonl files.

Fields absent from a source record are also absent from the output record. The connector doesn’t backfill them with null.

Parquet

Apache Parquet is a columnar format. Records are first converted to Avro using the same mapping as the Avro format. They’re then written as Parquet.

OptionTypeDefaultDescription
Compression CodecenumUNCOMPRESSEDOne of UNCOMPRESSED, SNAPPY, GZIP, LZO, BROTLI, LZ4, ZSTD.
Block Size (Row Group Size)integer (MB)128Size of a row group buffered in memory. Larger values improve read performance but use more memory while writing.
Max Padding Sizeinteger (MB)8Maximum padding used to align row groups; also the minimum row group size.
Page Sizeinteger (KB)1024Size of a data page, the smallest unit that must be read to access a record. Very small pages reduce compression efficiency.
Dictionary Page Sizeinteger (KB)1024Size of the dictionary page per column per row group when dictionary encoding is on.
Dictionary EncodingbooleantrueWhether to use dictionary encoding.

These options map to Parquet’s ParquetOutputFormat settings. See the Parquet configuration guide for recommended values.

Namespace support

This destination supports namespaces through the ${NAMESPACE} variable in GCS Path Format. For streams without a namespace, the variable becomes an empty string and omits the directory segment. Stream and namespace names are sanitised to characters that are safe in object keys.

Troubleshooting

Connection check fails with AccessDenied

The connector reports Google Cloud errors using S3 terminology because it uses the S3-compatible XML API. AccessDenied means the HMAC key’s service account lacks one of the bucket’s required permissions. The missing permission is usually storage.multipartUploads.create or storage.objects.delete. Grant Storage Object User on the bucket or add the missing permissions to your custom role.

Connection check fails with SignatureDoesNotMatch or InvalidAccessKeyId

The HMAC access ID or secret is incorrect, or the key belongs to another project. Check that you copied the secret exactly and that the key is active. If your organisation restricts authentication types, confirm the constraints/storage.restrictAuthTypes policy allows HMAC keys.

Objects appear directly under the bucket path

The GCS Path Format doesn’t end with / and contains no /. Its resolved value therefore becomes a file name prefix. Add / where you need a directory boundary, or clear the field to use the default template. Before connector version 1.0.2, clearing the field could also produce this layout. Upgrade to version 1.0.2 or later.

Old files remain after Full Refresh - Overwrite

Only objects matching the stream’s current path template and carrying generation metadata are deleted. Files remain if versions before 1.0.0 wrote them or they use another GCS Path Format. Remove them manually.

Upgrade to 1.0.0

Version 1.0.0 rewrites the connector on the Bulk Load CDK. It changes the default compression and output layout. Review the GCS migration guide before upgrading.

Performance

The connector uses tuned object-storage pipeline defaults and provides no performance-tuning settings. This behaviour matches the other Bulk CDK object-storage destinations. Throughput scales with the destination pod’s CPU and memory. For socket-mode syncs, it also scales with source read concurrency and the number of streams synced in parallel. In socket mode, the CDK automatically derives socket count, part size and upload parallelism from negotiated CPU limits.

Changelog

Expand to review
VersionDatePull RequestSubject
1.0.22026-10-3090272Promote to certified (GA). Treat blank gcs_path_format / file_name_pattern as the default patterns; declare allowedHosts in metadata; report connection-check failures as configuration errors with GCS/HMAC-specific guidance (90279).
1.0.12026-10-2289650Upgrade to Bulk CDK 1.1.9: fix critical CVEs in Apache MINA (mina-core 2.0.31), MINA SSHD 2.20.0 and BouncyCastle 1.86; netty 4.1.139, Hadoop 3.4.3, Parquet 1.18.1.
1.0.02026-07-1581376Migrate to the Bulk Load CDK for significantly higher throughput on large syncs through parallel, back-pressured multipart uploads and SOCKET/PROTOBUF data transfer with compatible sources. Streams larger than approximately 200 MB may produce multiple objects, CSV and JSONL output is GZIP-compressed by default, and Airbyte metadata field names change. See Upgrading to 1.0.0.
0.4.92025-03-2155906Use M4 Compatible base image.
0.4.82025-01-1051479Use a non root base image
0.4.72024-12-1849884Use a base image: airbyte/java-connector-base:1.0.0
0.4.62024-02-1535285Adopt CDK 0.20.8
0.4.52024-02-0834745Adopt CDK 0.19.0
0.4.42023-07-1428345Increment patch to trigger a rebuild
0.4.32023-07-0527936Internal code update
0.4.22023-06-3027891Internal code update
0.4.12023-06-2827268Internal code update
0.4.02023-06-2627725License Update: Elv2
0.3.02023-04-2825570Fix: all integer schemas should be converted to Avro longs
0.2.172023-04-2725346Internal code cleanup
0.2.162023-03-1723788S3-Parquet: added handler to process null values in arrays
0.2.152023-03-1023466Changed S3 Avro type from Int to Long
0.2.142022-11-2321682Add support for buckets with Customer-Managed Encryption Key
0.2.132023-01-1821087Wrap Authentication Errors as Config Exceptions
0.2.122022-10-1817901Fix logging to GCS
0.2.112022-09-0116243Fix Json to Avro conversion when there is field name clash from combined restrictions (anyOf, oneOf, allOf fields)
0.2.102022-08-0514801Fix multiple log bindings
0.2.92022-06-2414114Remove “additionalProperties”: false from specs for connectors with staging
0.2.82022-06-1713753Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager
0.2.72022-06-1413483Added support for int, long, float data types to Avro/Parquet formats.
0.2.62022-05-1712820Improved ‘check’ operation performance
0.2.52022-05-0412578In JSON to Avro conversion, log JSON field values that do not follow Avro schema for debugging.
0.2.42022-04-2212167Add gzip compression option for CSV and JSONL formats.
0.2.32022-04-2211795Fix the connection check to verify the provided bucket path.
0.2.22022-04-0511728Properly clean-up bucket when running OVERWRITE sync mode
0.2.12022-04-0511499Updated spec and documentation.
0.2.02022-04-0411686Use serialized buffering strategy to reduce memory consumption; compress CSV and JSONL formats.
0.1.222022-02-1210256Add JVM flag to exist on OOME.
0.1.212022-02-1210299Fix connection check to require only the necessary permissions.
0.1.202022-01-119367Avro & Parquet: support array field with unknown item type; default any improperly typed field to string.
0.1.192022-01-109121Fixed check method for GCS mode to verify if all roles assigned to user
0.1.182021-12-308809Update connector fields title/description
0.1.172021-12-218574Added namespace to Avro and Parquet record types
0.1.162021-12-208974Release a new version to ensure there is no excessive logging.
0.1.152021-12-038386Add new GCP regions
0.1.142021-12-017732Support timestamp in Avro and Parquet
0.1.132021-11-037288Support Json additionalProperties.
0.1.22021-09-125720Added configurable block size for stream. Each stream is limited to 10,000 by GCS
0.1.12021-08-265296Added storing gcsCsvFileLocation property for CSV format. This is used by destination-bigquery (GCS Staging upload type)
0.1.02021-07-164329Initial release.
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.