Advanced Analytica Advanced Analytica: IBOM
← All connectors
S3 logo

Destination connector

S3

Send governed data into S3 with a clear, practical setup guide.

Version 1.9.11 certified generally available

S3

Set up the S3 destination connector by following this guide.

Prerequisites

  1. Allow connections from Airbyte to your AWS S3/Minio S3 cluster if they use separate VPCs. If you’re using Airbyte Cloud with IP-based access controls, add Airbyte’s IP addresses to your allowlist.
  2. Enforce encryption of data in transit.
  3. Prepare an S3 bucket with one of these authentication methods:
    • Credentials

    • A Role ARN

    • An instance profile with host read/write permissions configured for EC2 or EKS

    • You always need these fields:

      • S3 Bucket Name
      • S3 Bucket Path
      • S3 Bucket Region
    • For STS Assume Role, provide:

      • Role ARN
    • For AWS credentials, provide:

      • Access Key ID
      • Secret Access Key
    • For an Instance Profile, you can omit the Access Key ID, Secret Access Key, and Role ARN.

Setup

Step 1: Set up S3

Sign in to your AWS account.

Prepare the S3 bucket you’ll use as the destination. See this to create an S3 bucket.

NOTE: Without TLS, the Amazon S3 connection silently reverts to an unencrypted connection. Airbyte recommends configuring every connection to use TLS/SSL because of AWS’s shared responsibility model

Create a bucket policy

  1. Open the IAM console.
  2. In the IAM dashboard, select Policies, then click Create Policy.
  3. Select the JSON tab, then paste the following JSON into the Policy editor. Replace the example bucket name with your bucket name.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:PutObjectAcl",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR_BUCKET_NAME/*",
        "arn:aws:s3:::YOUR_BUCKET_NAME"
      ]
    }
  ]
}

:::note Object-level permissions alone can’t currently authenticate the connection. Include the bucket-level permissions shown in the example above. :::

  1. Give your policy a descriptive name, then click Create policy.

Authentication option 1: Use an IAM Role (Most secure)

:::note An Airbyte team member must enable S3 authentication through an IAM role member. For more information about using this feature, contact the Sales team. :::

  1. In the IAM dashboard, click Roles, then Create role.
  2. Choose the appropriate trusted entity and attach the policy you created.
  3. Set up a trust relationship for the role. For the AWS account trusted entity, use your instance’s default AWS account. It will assume the role. To use External ID, set it through environment variables as export AWS_ASSUME_ROLE_EXTERNAL_ID="{your-external-id}". Update the trust relationship policy as follows:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::{your-aws-account-id}:user/{your-username}"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "{your-external-id}"
                }
            }
        }
    ]
}
  1. Choose the AWS account trusted entity type.
  2. Set up a trust relationship that lets the Airbyte instance’s AWS account assume this role. You must also specify an external ID. This secret key is known by both the trusting service, Airbyte, and the trusted role you’re creating. The ID prevents the “confused deputy” problem. Use your Airbyte workspace ID as the External ID. You can find it in your workspace page URL. Update the trust relationship policy to include the external ID:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::094410056844:user/delegated_access_user"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "{your-airbyte-workspace-id}"
                }
            }
        }
    ]
}
  1. Complete the role creation and note the Role ARN.
  2. Select Attach policies directly, then find and select your new policy. Click Next, then Add permissions.
Authentication option 2: Use an IAM User

Use an existing Access Key ID and Secret Access Key or create a new one.

  1. In the IAM dashboard, click Users. Select an existing IAM user or create one by clicking Add users.
  2. For an existing IAM user, open the Add permissions dropdown menu and select Add permissions. For a new user, you’ll reach the Permissions screen after choosing a name.
  3. Select Attach policies directly, then find and select your new policy. Click Next, then Add permissions.
  4. After creating your user, select the Security credentials tab and click Create access key. Select a use case and add optional tags to your access key when prompted. Click Create access key to generate the keys.

Step 2: Set up the S3 destination connector in Airbyte

For Airbyte Cloud:

  1. Log into your Airbyte Cloud account.
  2. In the left navigation bar, click Destinations. In the top-right corner, click + new destination.
  3. On the destination setup page, select S3 from the Destination type dropdown. Enter a name for this connector.
  4. Configure these fields:
    • Access Key Id
      • See this to generate an access key.
      • Airbyte recommends creating an Airbyte-specific user. This user needs read and write permissions to objects in the bucket.
    • Secret Access Key
      • Enter the key that corresponds to the key id above.
    • Role ARN
      • See this to create a role.
    • S3 Bucket Name
      • See this to create an S3 bucket.
    • S3 Bucket Path
      • Enter the bucket subdirectory where you want to sync the data.
    • S3 Bucket Region:
      • See here for all region codes.
    • S3 Path Format
      • Set the additional string format for storing data under S3 Bucket Path. The default value is ${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_.
    • S3 Endpoint
      • Leave this empty for AWS S3. For Minio S3, enter the S3 URL.
    • S3 Filename pattern
      • Set the file-name format for the S3 staging file(s). The supported placeholder combinations are {date}, {date:yyyy_MM}, {timestamp}, {part_number}, {sync_id}, and {format_extension}. Don’t use spaces or unsupported placeholders because they won’t be recognised.
  5. Click Set up destination.

For Airbyte Open Source:

  1. Go to your local Airbyte page.

  2. In the left navigation bar, click Destinations. In the top-right corner, click + new destination.

  3. On the destination setup page, select S3 from the Destination type dropdown. Enter a name for this connector.

  4. Configure these fields:

    • Access Key Id
      • See this to generate an access key.
      • See this to create an instanceprofile.
      • Airbyte recommends creating an Airbyte-specific user. This user needs read and write permissions to objects in the staging bucket.
      • Without an Access Key and Secret Access Key, authentication uses either STS Assume Role or the instanceprofile. STS Assume Role uses the Role ARN.
    • Secret Access Key
      • Enter the key that corresponds to the key id above.
      • Ensure the machine running Airbyte can access your S3 bucket.
      • Access depends on your networking setup.
      • See the AWS S3 documentation for guidance on configuring your S3’s access here.
      • For instance profile authentication, ensure the role can read and write to the bucket.
      • Use the UI’s check connection tool to verify that Airbyte can connect to your S3 bucket.
    • S3 Bucket Name
      • See this to create an S3 bucket.
    • S3 Bucket Path
      • Enter the subdirectory in the bucket above where you want to sync the data.
    • S3 Bucket Region
      • See here for all region codes.
    • S3 Path Format
      • Set the additional string format for storing data under S3 Bucket Path. The default value is ${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_.
    • S3 Endpoint
      • Leave this empty for AWS S3. For Minio S3, enter the S3 URL.
    • S3 Filename pattern
      • Set the file-name format for the S3 staging file(s). The supported placeholder combinations are {date}, {date:yyyy_MM}, {timestamp}, {part_number}, {sync_id}, and {format_extension}.
      • Don’t use spaces or unsupported placeholders because they won’t be recognised.
  5. Click Set up destination.

With the default S3 Path Format ${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, the full output data path is:

<bucket-name>/<source-namespace-if-exists>/<stream-name>/<upload-date>_<epoch>_<partition-id>.<format-extension>

For example:

testing_bucket/data_output_path/public/users/2021_01_01_1234567890_0.csv.gz
↑              ↑                ↑      ↑     ↑          ↑          ↑ ↑
|              |                |      |     |          |          | format extension
|              |                |      |     |          |          unique incremental part id
|              |                |      |     |          milliseconds since epoch
|              |                |      |     upload date in YYYY_MM_DD
|              |                |      stream name
|              |                source namespace (if it exists)
|              bucket path
bucket name

This naming pattern provides these benefits:

  1. Each stream has its own directory.
  2. You can sort the data output files by upload time.
  3. The upload time combines date and milliseconds, making it readable and unique.

You can further customise the bucket path with these available variables:

  • ${NAMESPACE}: Namespace where the stream originates, or the namespace configured through the connection namespace fields.
  • ${STREAM_NAME}: Stream name.
  • ${YEAR}: Year when the sync wrote the output data.
  • ${MONTH}: Month when the sync wrote the output data.
  • ${DAY}: Day when the sync wrote the output data.
  • ${HOUR}: Hour when the sync wrote the output data.
  • ${MINUTE} : Minute when the sync wrote the output data.
  • ${SECOND}: Second when the sync wrote the output data.
  • ${MILLISECOND}: Millisecond when the sync wrote the output data.
  • ${EPOCH}: Milliseconds since Epoch when the sync wrote the output data.
  • ${UUID}: Random uuid string.
  • ${SYNC_ID}: ID of the sync job that wrote the output data.

Note:

  • Multiple / characters in the S3 path collapse into one / character.

The stream name can include a prefix if you’ve configured one on the connection. A data sync can create multiple files because output files are partitioned by size. Each compressed file targets a size of 200MB or less.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

The Airbyte S3 destination syncs data to AWS S3 or Minio S3. It writes each stream to its own directory under the bucket.

⚠️ In “Full Refresh Overwrite Sync” mode, a successful sync retains data from the same generation. It deletes all data from previous generations. If a sync fails between generations, data from multiple generations can remain until a later sync succeeds. Each S3 object uses the x-amz-meta-ab-generation-id tag to identify its generation. Airbyte recommends a dedicated S3 resource for this sync. This reduces accidental data deletion caused by misconfiguration. ⚠️

Supported output schema

Each stream outputs to its own directory, based on your configuration. The stream’s complete datastore includes every output file in that directory. You can treat the directory as the equivalent of a database table.

  • In Full Refresh Sync mode, old output files are purged before new files are created.
  • In Incremental - Append Sync mode, new output files contain only the new data.

Avro

Apache Avro serialises data in a compact binary format. The Airbyte S3 Avro connector always uses the binary encoding and assumes every data record follows the same schema.

Configuration

Choose from these compression codecs:

  • No compression
  • deflate
    • Compression level
      • Range [0, 9]. The default is 0.
      • Level 0: no compression and fastest.
      • Level 9: best compression and slowest.
  • bzip2
  • xz
    • Compression level
      • Range [0, 9]. The default is 6.
      • Levels 0-3 are fast with medium compression.
      • Levels 4-6 are fairly slow with high compression.
      • Levels 7-9 work like level 6 but use larger dictionaries and need more memory. Presets 7, 8, and 9 waste memory unless the uncompressed file exceeds 8 MiB, 16 MiB, and 32 MiB, respectively.
  • zstandard
    • Compression level
      • Range [-5, 22]. The default is 3.
      • Negative levels are ‘fast’ modes similar to lz4 or snappy.
      • Levels above 9 are generally for archival purposes.
      • Levels above 18 use considerable memory.
    • Include checksum
      • Set this to true to include a checksum in each data block.
  • snappy

Data schema

An Airbyte data stream in JSON schema is first converted to an Avro schema. The JSON object is then converted to an Avro record. Data streams can come from any source, so JSON-to-Avro conversion has arbitrary rules and limitations. Learn more about source data conversion to Avro and the current limitations here.

CSV

Like most other Airbyte destination connectors, the output usually has three columns. These are a UUID, an emission timestamp, and the data blob. With CSV output, you can normalise (flatten) the data blob into multiple columns.

ColumnConditionDescription
_airbyte_raw_idAlways exists.A uuid assigned by Airbyte to each processed record.
_airbyte_extracted_atAlways exists.A timestamp representing when the event was extracted from the data source.
_airbyte_generation_idAlways exists.An integer id that increases with each new refresh.
_airbyte_metaAlways exists.A structured object containing metadata about the record.
_airbyte_dataWhen no normalization (flattening) is needed, all data resides under this column as a JSON blob.
root level fieldsWhen root level normalization (flattening) is selected, the root level fields are expanded.

The schema for _airbyte_meta is:

Field NameTypeDescription
changeslistA list of structured change objects.
sync_idintegerAn integer identifier for the sync job.

The schema for a change object is:

Field NameTypeDescription
fieldstringThe name of the field that changed.
changestringThe type of change (eg, NULLED, TRUNCATED).
reasonstringThe reason for the change, including its system of origin (ie, whether it was a source, destination, or platform error).

For example, consider this JSON object from a source:

{
  "user_id": 123,
  "name": {
    "first": "John",
    "last": "Doe"
  }
}

Without normalisation, the output CSV is:

_airbyte_raw_id_airbyte_extracted_at_airbyte_generation_id_airbyte_meta_airbyte_data
26d73cde-7eb1-4e1e-b7db-a4c03b4cf206162213580500011{"changes":[], "sync_id": 10111 }{ "user_id": 123, name: { "first": "John", "last": "Doe" } }

With root-level normalisation, the output CSV is:

_airbyte_raw_id_airbyte_extracted_at_airbyte_generation_id_airbyte_metauser_idname.firstname.last
26d73cde-7eb1-4e1e-b7db-a4c03b4cf206162213580500011{"changes":[], "sync_id": 10111 }123JohnDoe

You can compress output files. GZIP compression is the default. When you select compression, the output filename gains an extra extension (GZIP: .csv.gz).

JSON Lines (JSONL)

JSON Lines is a text format containing one JSON object per line. Each line uses this structure:

{
  "_airbyte_raw_id": "<uuid>",
  "_airbyte_extracted_at": "<timestamp>",
  "_airbyte_generation_id": "<generation-id>",
  "_airbyte_meta": "<json-meta>",
  "_airbyte_data": "<json-data-from-source>"
}

For example, consider these two JSON objects from a source:

[
  {
    "user_id": 123,
    "name": {
      "first": "John",
      "last": "Doe"
    }
  },
  {
    "user_id": 456,
    "name": {
      "first": "Jane",
      "last": "Roe"
    }
  }
]

They appear in the output file as follows:

{ "_airbyte_raw_id": "26d73cde-7eb1-4e1e-b7db-a4c03b4cf206", "_airbyte_extracted_at": "1622135805000", "_airbyte_generation_id": "11", "_airbyte_meta": { "changes": [], "sync_id": 10111 }, "_airbyte_data": { "user_id": 123, "name": { "first": "John", "last": "Doe" } } }
{ "_airbyte_raw_id": "0a61de1b-9cdd-4455-a739-93572c9a5f20", "_airbyte_extracted_at": "1631948170000", "_airbyte_generation_id": "12", "_airbyte_meta": { "changes": [], "sync_id": 10112 }, "_airbyte_data": { "user_id": 456, "name": { "first": "Jane", "last": "Roe" } } }

You can compress output files. GZIP compression is the default. When you select compression, the output filename gains an extra extension (GZIP: .jsonl.gz).

Parquet

Configuration

Configure the Parquet output with these settings:

ParameterTypeDefaultDescription
compression_codecenumUNCOMPRESSEDCompression algorithm. Available candidates are: UNCOMPRESSED, SNAPPY, GZIP, LZO, BROTLI, LZ4, and ZSTD.
block_size_mbinteger128 (MB)Block size (row group size) in MB. This is the size of a row group being buffered in memory. It limits the memory usage when writing. Larger values will improve the IO when reading, but consume more memory when writing.
max_padding_size_mbinteger8 (MB)Max padding size in MB. This is the maximum size allowed as padding to align row groups. This is also the minimum size of a row group.
page_size_kbinteger1024 (KB)Page size in KB. The page size is for compression. A block is composed of pages. A page is the smallest unit that must be read fully to access a single record. If this value is too small, the compression will deteriorate.
dictionary_page_size_kbinteger1024 (KB)Dictionary Page Size in KB. There is one dictionary page per column per row group when dictionary encoding is used. The dictionary page size works like the page size but for dictionary.
dictionary_encodingbooleantrueDictionary encoding. This parameter controls whether dictionary encoding is turned on.

These parameters relate to the ParquetOutputFormat. See Java doc for more details. Also see Parquet documentation for the recommended settings: 512 - 1024 MB block size and 8 KB page size.

Data schema

An Airbyte data stream in JSON schema is first converted to an Avro schema. The JSON object is then converted to an Avro record. Finally, the Avro record is output in Parquet format. Data streams can come from any source, so JSON-to-Avro conversion has arbitrary rules and limitations. Learn more about source data conversion to Avro and the current limitations here.

For correct operation, the user associated with “S3 Key Id” and “S3 Access Key” needs access to the bucket and its contents. Use these policies:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::YOUR_BUCKET_NAME/*",
        "arn:aws:s3:::YOUR_BUCKET_NAME"
      ]
    }
  ]
}

Limitations and troubleshooting

For connector limitations and S3 connector troubleshooting, see in our s3 troubleshooting guide.

Namespace support

This destination supports namespaces. The namespace forms part of the S3 output path.

Changelog

Expand to review
VersionDatePull RequestSubject
1.9.112026-10-2289642Upgrade to Bulk CDK 1.1.9: fix critical CVEs in Apache MINA (mina-core 2.0.31), MINA SSHD 2.20.0 and BouncyCastle 1.86; netty 4.1.139, Hadoop 3.4.3, Parquet 1.18.1.
1.9.102026-10-0187620Upgrade to Bulk CDK 1.1.1.
1.9.92026-09-0184993Upgrade to Bulk CDK 1.0.25.
1.9.82026-05-2278332Promoting release candidate 1.9.8-rc.1 to a main version.
1.9.8-rc.12026-05-2178240Upgrade CDK to 1.0.13. Progressive rollout.
1.9.72026-01-2672354Fix sync failures for sources with empty schemas by upgrading CDK to 0.2.1
1.9.62026-01-2672298Upgrade CDK to 0.2.0
1.9.52025-11-0469134Upgrade to Bulk CDK 0.1.61.
1.9.42025-10-2167153Implement new proto schema implementation
1.9.32025-10-0667078Remove memory limit for sync jobs to improve performance and resource utilization.
1.9.22025-09-1966523Fix Minio compatibility by forcing path-style bucket access.
1.9.12025-07-3164138Promoting release candidate 1.9.1-rc.1 to a main version.
1.9.1-rc.12025-07-2864101Prepare release candidate with speed mode fixes.
1.9.02025-07-2463759Promoting release candidate 1.9.0-rc.4 to a main version.
1.9.0-rc.42025-07-1763350Release candidate 4
1.9.0-rc.32025-07-1062890Release candidate 3
1.9.0-rc.22025-07-0762829Release candidate 2 with bug fixes
1.9.0-rc.12025-07-0262068Prepare to enable speed improvements
1.8.82025-06-2762127Identical to 1.8.6. Image was published from PR 62071.
1.8.72025-06-1361588Publish version to account for possible duplicate publishing in pipeline. Noop change. WARNING: THIS HAS A BUG. DO NOT USE.
1.8.62025-05-3060327IPC Metadata, internal refactors.
1.8.52025-05-1660327Fixes file partitioning out of bounds error.
1.8.42025-05-1460313Re-wires up time window trigger. Files path releases memory properly.
1.8.32025-05-1460287Update spec.
1.8.22025-05-0759721Legacy File Transfer Uses New CDK Interface
1.8.12025-05-0759710CDK backpressure bugfix
1.8.02025-04-3059168Promoting release candidate 1.8.0-rc.1 to a main version.
1.8.0-rc.12025-04-2959148Upgrade to latest CDK with files+records support
1.7.42025-04-2158146Upgrade to latest CDK
1.7.32025-04-1858140Upgrade to latest CDK
1.7.22025-04-0756391Internal code refactor
1.7.12025-04-0256974Nonfunctional change: pin cdk version
1.7.02025-04-0256974Release 1.7.0 release candidate
1.7.0-rc.12025-03-3156935Internal performance refactor
1.6.02025-03-2856458Do not drop trailing .0 from decimals in CSV/JSONL data
1.5.82025-03-2556398Internal CDK change to mark old Avro and Parquet formats as deprecated
1.5.72025-03-2456355Upgrade to airbyte/java-connector-base:2.0.1 to be M4 compatible.
1.5.62025-03-2455849Internal refactoring
1.5.52025-03-2055875Bugfix: Sync Can Hang on OOM
1.5.42025-03-0554695Nonfunctional changes to support performance testing
1.5.32025-03-0454661Nonfunctional changes to support performance testing
1.5.22025-02-2554661Nonfunctional cleanup; dropped unused staging code
1.5.12025-02-1153636Nonfunctional CDK version pin.
1.5.02025-02-1153632Promoting release candidate 1.5.0-rc.20 to a main version.
1.5.0-rc.202025-02-0453173Tweak spec wording
1.5.0-rc.192025-02-0453163Various fixes to truncate sync
1.5.0-rc.182025-01-2952703Force list call evaluation before making head calls
1.5.0-rc.172025-01-2952610Pin CDK 0.296
1.5.0-rc.162025-01-2952610Fix assume role behavior
1.5.0-rc.152025-01-2352103Make the connector use our non root base image.
1.5.0-rc.142025-01-2451600Internal refactor
1.5.0-rc.132025-01-2252076Test improvements.
1.5.0-rc.122025-01-2252072Bug fix: Configure OpenStreamTask concurrency to handle connection to reduce http connection errors.
1.5.0-rc.112025-01-1751051Input fully read before end-of-stream now correctly marked transient
1.5.0-rc.102025-01-1550960Bug fixes: tolerate repeated path variables; avro meta field schema matches old cdk
1.5.0-rc.92025-01-1050960Bug fixes: variables respected in bucket path; sync does not hang on streams w/o state
1.5.0-rc.82025-01-0850960Use airbyte/java-connector-base base image.
1.5.0-rc.72025-01-0951021Bug fix: Use CRT HTTP client to avoid OkHttp idle connection handling errors
1.5.0-rc.62025-01-0650954Bug fix: transient failure due to bug in generation tracker
1.5.0-rc.52025-01-0650954Bug fix: transient failure due to bug in filename clash prevention
1.5.0-rc.42025-01-0650954Bug fix: StreamLoader::close dispatched multiple times per stream
1.5.0-rc.32025-01-0650949Bug fix: parquet types/values nested in union of objects do not convert properly
1.5.0-rc.22025-01-0250857Migrate to Bulk Load CDK: cost reduction, perf increase, bug fix for filename clashes
1.4.02024-10-2346302add support for file transfer
1.3.02024-09-3046281fix tests
1.2.12024-09-2045700Improve resiliency to jsonschema fields
1.2.02024-09-1845402fix exception with columnless streams
1.1.02024-09-1845436upgrade all dependencies
1.0.52024-09-0545143don’t overwrite (and delete) existing files, skip indexes instead
1.0.42024-08-3044933Fix: Avro/Parquet: handle empty schemas in nested objects/lists
1.0.32024-08-2044476Increase message parsing limit to 100mb
1.0.22024-08-1944401Fix: S3 Avro/Parquet: handle nullable top-level schema
1.0.12024-08-1442579OVERWRITE MODE: Deletes deferred until successful sync.
1.0.02024-08-0842409Major breaking changes: new destination schema, change capture, Avro/Parquet improvements, bugfixes
0.1.152024-12-1849879Use a base image: airbyte/java-connector-base:1.0.0
0.6.72024-08-1143713Decreased memory ratio (0.7 -> 0.5) and thread allocation (5 -> 2) for async S3 uploads.
0.6.62024-08-0643343Use Kotlin 2.0.0
0.6.52024-08-0142405S3 parallelizes workloads, checkpoints, submits counts, support for generationId in metadata for refreshes.
0.6.42024-04-1642006remove unnecessary zookeeper dependency
0.6.32024-04-1538204convert all production code to kotlin
0.6.22024-04-1538204add assume role auth
0.6.12024-04-0837546Adapt to CDK 0.30.8;
0.6.02024-04-0836869Adapt to CDK 0.29.8; Kotlin converted code.
0.5.92024-02-2235569Fix logging bug.
0.5.82024-01-03#33924Add new ap-southeast-3 AWS region
0.5.72023-12-28#33788Thread-safe fix for file part names
0.5.62023-12-08#33263(incorrect filename format, do not use) Adopt java CDK version 0.7.0.
0.5.52023-12-08#33264Update UI options with common defaults.
0.5.42023-11-06#32193(incorrect filename format, do not use) Adopt java CDK version 0.4.1.
0.5.32023-11-03#32050(incorrect filename format, do not use) Adopt java CDK version 0.4.0. This updates filenames to include a UUID.
0.5.12023-06-26#27786Fix build
0.5.02023-06-26#27725License Update: Elv2
0.4.22023-06-21#27555Reduce image size
0.4.12023-05-18#26284Fix: reenable LZO compression for Parquet output
0.4.02023-04-28#25570Fix: all integer schemas should be converted to Avro longs
0.3.252023-04-27#25346Internal code cleanup
0.3.232023-03-30#24736Improve behavior when throttled by AWS API
0.3.222023-03-17#23788S3-Parquet: added handler to process null values in arrays
0.3.212023-03-10#23466Changed S3 Avro type from Int to Long
0.3.202023-02-23#21355Add root level flattening option to JSONL output.
0.3.192023-01-18#21087Wrap Authentication Errors as Config Exceptions
0.3.182022-12-15#20088New data type support v0/v1
0.3.172022-10-15#18031Fix integration tests to use bucket path
0.3.162022-10-03#17340Enforced encrypted only traffic to S3 buckets and check logic
0.3.152022-09-01#16243Fix Json to Avro conversion when there is field name clash from combined restrictions (anyOf, oneOf, allOf fields).
0.3.142022-08-24#15207Fix S3 bucket path to be used for check.
0.3.132022-08-09#15394Added LZO compression support to Parquet format
0.3.122022-08-05#14801Fix multiple log bindings
0.3.112022-07-15#14494Make S3 output filename configurable.
0.3.102022-06-30#14332Change INSTANCE*PROFILE to use AWSDefaultProfileCredential, which supports more authentications on AWS
0.3.92022-06-24#14114Remove “additionalProperties”: false from specs for connectors with staging
0.3.82022-06-17#13753Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager
0.3.72022-06-14#13483Added support for int, long, float data types to Avro/Parquet formats.
0.3.62022-05-19#13043Destination S3: Remove configurable part size.
0.3.52022-05-12#12797Update spec to replace markdown.
0.3.42022-05-04#12578In JSON to Avro conversion, log JSON field values that do not follow Avro schema for debugging.
0.3.32022-04-20#12167Add gzip compression option for CSV and JSONL formats.
0.3.22022-04-22#11795Fix the connection check to verify the provided bucket path.
0.3.12022-04-05#11728Properly clean-up bucket when running OVERWRITE sync mode
0.3.02022-04-04#116660.2.12 actually has breaking changes since files are compressed by default, this PR also fixes the naming to be more compatible with older versions.
0.2.132022-03-29#11496Fix S3 bucket path to be included with S3 bucket format
0.2.122022-03-28#11294Change to serialized buffering strategy to reduce memory consumption
0.2.112022-03-23#11173Added support for AWS Glue crawler
0.2.102022-03-07#10856check method now tests for listObjects permissions on the target bucket
0.2.72022-02-14#10318Prevented double slashes in S3 destination path
0.2.62022-02-1410256Add -XX:+ExitOnOutOfMemoryError JVM option
0.2.52022-01-13#9399Use instance profile authentication if credentials are not provided
0.2.42022-01-12#9415BigQuery Destination : Fix GCS processing of Facebook data
0.2.32022-01-11#9367Avro & Parquet: support array field with unknown item type; default any improperly typed field to string.
0.2.22021-12-21#8574Added namespace to Avro and Parquet record types
0.2.12021-12-20#8974Release a new version to ensure there is no excessive logging.
0.2.02021-12-15#8607Change the output filename for CSV files - it’s now bucketPath/namespace/streamName/timestamp_epochMillis_randomUuid.csv
0.1.162021-12-10#8562Swap dependencies with destination-jdbc.
0.1.152021-12-03#8501Remove excessive logging for Avro and Parquet invalid date strings.
0.1.142021-11-09#7732Support timestamp in Avro and Parquet
0.1.132021-11-03#7288Support Json additionalProperties.
0.1.122021-09-13#5720Added configurable block size for stream. Each stream is limited to 10,000 by S3
0.1.112021-09-10#5729For field names that start with a digit, a * will be appended at the beginning for theParquetandAvroformats.
0.1.102021-08-17#4699Added json config validator
0.1.92021-07-12#4666Fix MinIO output for Parquet format.
0.1.82021-07-07#4613Patched schema converter to support combined restrictions.
0.1.72021-06-23#4227Added Avro and JSONL output.
0.1.62021-06-16#4130Patched the check to verify prefix access instead of full-bucket access.
0.1.52021-06-14#3908Fixed defaultmax_padding_size_mbinspec.json.
0.1.42021-06-14#3908Added Parquet output.
0.1.32021-06-13#4038Added support for alternative S3.
0.1.22021-06-10#4029Fixed \_airbyte_emitted_atfield to be a UTC instead of local timestamp for consistency.
0.1.12021-06-09#3973AddedAIRBYTE_ENTRYPOINT in base Docker image for Kubernetes support.
0.1.02021-06-03#3672Initial release with CSV output.
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.