Advanced Analytica Advanced Analytica: IBOM
← All connectors
S3 logo

Source connector

S3

Set up S3 as a source and move its data into your chosen destination.

Version 4.17.1 certified generally available

S3

Use this guide to set up the S3 source connector and find reference information.

:::warning Cloud storage may incur egress costs. Egress means transferring data out of cloud storage, such as downloading files or accessing them elsewhere. For detailed information about egress costs, see the AWS S3 pricing guide. :::

Prerequisites

  • Access to the S3 bucket containing the files you want to replicate.
  • For private buckets, an AWS account that can grant permission to read from the bucket.

Setup guide

Step 1: Set up S3

Choose authentication from the Authentication dropdown when setting up the connector: IAM User, IAM Role or No Authentication. For a private bucket, create a policy with the required permissions:

Create a policy

  1. Sign in to your Amazon AWS account and open the IAM console.
  2. In the IAM dashboard, select Policies, then click Create Policy.
  3. Select the JSON tab, then paste the following JSON into the Policy editor. Replace the bucket name with yours:
{
    "Version": "2012-10-17",
    "Statement": [
        {
        "Effect": "Allow",
        "Action": [
                "s3:GetObject",
                "s3:ListBucket"
        ],
        "Resource": [
                "arn:aws:s3:::{your-bucket-name}/*",
                "arn:aws:s3:::{your-bucket-name}"
        ]
        }
    ]
}

:::note Object-level permissions alone can’t authenticate the connection. Include the bucket-level permissions shown in the example above. :::

  1. Give your policy a descriptive name, then click Create policy.

Option 1: Use an IAM User

  1. In the IAM dashboard, click Users. Select an existing IAM user or click Add users to create one.
  2. For an existing IAM user, open the Add permissions dropdown menu and select Add permissions. For a new user, selecting a name takes you to the Permissions screen.
  3. Select Attach policies directly, then find and select your new policy. Click Next, then Add permissions.
  4. After creating your user, select the Security credentials tab and click Create access key. Select a use case and add optional tags to your access key. Click Create access key to generate the keys.

:::caution Your Secret Access Key appears only once when created. Copy it and store it securely for future use. :::

For more information about managing your access keys, see the official AWS documentation.

Option 2: Use an IAM Role (Most secure)

:::note The OSS platform doesn’t support S3 authentication using an IAM role member. :::

:::note An Airbyte team member must enable S3 authentication using an IAM role member. For more information about using this feature, contact the Sales team. :::

Step 2: Set up the S3 connector in Airbyte

Airbyte Cloud

  1. Log into your Airbyte Cloud account.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select S3 from the Source type dropdown.
  4. Enter a name for the S3 connector.
  5. Enter the name of the Bucket containing the files you want to replicate.
  6. Choose an authentication method from the Authentication dropdown:
    • IAM User: Enter the AWS Access Key ID and AWS Secret Access Key.
    • IAM Role: Enter the AWS Role ARN.
    • No Authentication / Default AWS Credentials: Use this for public buckets or the default AWS credential chain (instance profile, IRSA, or environment variables).
  7. (Optional) Choose a delivery method for your data. You can find this option under Advanced.
  8. Add a stream:
    1. Give the stream a Name.
    2. In the Format box, use the dropdown menu to select the format of the files you want to replicate. Supported formats are CSV, Parquet, Avro and JSONL. Enable Optional fields in the Format box to enter more settings for the selected format. For details about these settings, see the File Format section below.
    3. (Optional) Enter the Globs, which determines which files are synced. This regular expression lets Airbyte match the specific files you want to replicate. To replicate every file in your bucket, use ** as the pattern. For more precise matching options, see the Globs section below.
    4. (Optional) Change the Days To Sync If History Is Full value. This controls the lookback window used to determine which files to sync when the state history is full. See the State section below for details.
    5. (Optional) To enforce a specific schema, enter an Input schema. By default, this value is {} and infers the schema from the file(s) you’re replicating. For details about providing a custom schema, see the User Schema section.
    6. (Optional) Select Schemaless to skip all record validation against a schema. When selected, the schema is {"data": "object"} and all downstream data is nested in a “data” field. This option suits records whose schema changes frequently.
    7. (Optional) Select a Validation Policy to control how Airbyte handles records that don’t match the schema. You can emit the record anyway, but fields absent from the schema may not reach the destination. Alternatively, skip the record or wait for the next discovery, which occurs within 24 hours.

You can leave other optional fields empty. See the S3 Provider Settings section below for more information about each field.

Airbyte Open Source

  1. Open the Airbyte Open Source dashboard.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select S3 from the Source type dropdown.
  4. Enter a name for the S3 connector.

Delivery method

Choose a delivery method for your data.

Preserve subdirectories in file paths

When enabled, this sends the subdirectory structure and source file names to the destination. Otherwise, files sync using names only. This option is ignored when file-based replication isn’t enabled.

Supported sync modes

The S3 source connector supports these sync modes:

FeatureSupported?
Full Refresh SyncYes
Incremental SyncYes
Replicate Incremental DeletesNo
Replicate Multiple Files (pattern matching)Yes
Replicate Multiple Streams (distinct tables)Yes
NamespacesNo

Supported streams

There are no predefined streams. Streams are based on your bucket’s contents.

File compression

CompressionSupported?
GzipYes
ZipYes
Bzip2Yes
LzmaNo
XzNo
SnappyNo

Share which specific compression formats you’d like supported next.

Globs

(In short: path pattern syntax using wcmatch.glob. GLOBSTAR and SPLIT flags are enabled.)

Use glob-style patterns to sync multiple files without specifying every file path. This lets you:

  • Reference many files with one pattern. For example, ** indicates every file in the bucket.
  • Reference future files that don’t exist yet (and therefore don’t have a specific path).

You must provide a path pattern. For complex directory layouts, you can provide multiple patterns separated by |.

Each path pattern starts from the bucket’s root. Don’t include the bucket name in the pattern(s).

Example patterns:

  • **: Match everything.
  • **/*.csv: Match all files with a specific extension.
  • myFolder/**/*.csv: Match all csv files anywhere under myFolder.
  • */**: Match everything at least one folder deep.
  • */*/*/**: Match everything at least three folders deep.
  • **/file.*|**/file: Match every file named “file” with any extension (or no extension).
  • x/*/y/*: Match all files in folder x -> any folder -> folder y.
  • **/prefix*.csv: Match all csv files with a specific prefix.
  • **/prefix*.parquet: Match all parquet files with a specific prefix.

Consider this specific example, which matches the following bucket layout:

myBucket
    -> log_files
    -> some_table_files
        -> part1.csv
        -> part2.csv
    -> images
    -> more_table_files
        -> part3.csv
    -> extras
        -> misc
            -> another_part1.csv

You want part1.csv, part2.csv and part3.csv, while excluding another_part1.csv for now. You can do this in several ways:

  • Use the single pattern **/part*.csv to select every csv file named “partX”.
  • For tighter matching, use the dual pattern some_table_files/*.csv|more_table_files/*.csv to select relevant files only from those exact folders.
  • Use *table_files/*.csv to achieve the same result with one pattern. However, unexpected new folders could cause future problems.
  • You can also use recursive wildcards. The pattern extras/**/*.csv selects csv files nested below “extras”, including “extras/misc/another_part1.csv”.

Path patterns offer several ways to reach the same result. Choose a clear pattern that can accommodate future directory changes.

State

For incremental syncs, Airbyte syncs files from oldest to newest. Each synced file, up to 10,000, becomes an entry in the connection state message’s “history” section. When history is full, older messages are removed. Airbyte then reads files modified within Days to Sync if History is Full days before the newest file’s date in history.

User schema

Provide a schema for more control over this stream’s output. Without one, Airbyte infers columns and datatypes from the first created file matching your path pattern and suffix. This approach will suit most cases. However, you may want to enforce a schema when:

:::note

Without a schema for a CSV file, Airbyte infers every column as a string.

:::

  • You only need a known subset of columns. Other columns remain included but are packed into the _ab_additional_properties map.
  • Your initial dataset has few records, so its automatic type inference may not represent future data.
  • You want to define types for every column deliberately.
  • You know which columns future data will add. Include them as core schema columns instead of placing them in the _ab_additional_properties map.

You may have another reason. Provide the schema as valid JSON using a map of {"column": "datatype"}. Each datatype must be one of:

  • string
  • number
  • integer
  • object
  • array
  • boolean
  • null

For example:

  • {"id": "integer", "location": "string", "longitude": "number", "latitude": "number"}
  • {"username": "string", "friends": "array", "information": "object"}

:::note

The S3 Source connector previously inferred schemas from every available file, then merged them into a superset schema. From version 2.0.0, schema inference uses only the first file found. The first file considered is the oldest one written to the prefix.

:::

S3 provider settings

  • Authentication: Select IAM User to provide an AWS Access Key ID and AWS Secret Access Key for a private bucket. Select IAM Role to provide an AWS Role ARN. Select No Authentication / Default AWS Credentials for public buckets or the default AWS credential chain.
  • Endpoint: This optional parameter enables non-Amazon S3 compatible services. Leave this field blank when using the default Amazon service.
  • Start Date: This optional parameter sets a UTC start date and time for data replication. Files not modified since this date and time won’t be replicated. Use the provided datepicker (recommended) or enter the date programmatically in the format YYYY-MM-DDTHH:mm:ssZ. Leave this field blank to replicate every file not excluded by Path Pattern and Path Prefix.

File format settings

CSV

CSV files are plain text, so specific reader options are often required for correct parsing. Apply these settings consistently whenever you create or export a CSV.

  • Header Definition: Controls how headers are defined. User Provided assumes the CSV has no header row and uses the provided headers. Autogenerated also assumes no header row. The CDK generates headers using f{i}, where i is the zero-based index. Otherwise, the CSV file’s header is used by default. To generate or provide column names for a CSV with headers, set “Skip rows before header” to ignore the header row.
  • Delimiter: CSV means Comma Separated Values, but the term also covers flat-file data separated by other characters. Use the delimiter field to specify the separator character. To use tab-delimiters, set this value to \t. The default value is ,.
  • Double Quote: Determines whether two quotes within a quoted CSV value represent one quote in the data. The default is True.
  • Encoding: Some data uses a different character set, typically for different alphabets. See the list of allowable encodings here. The default is utf8.
  • Escape Character: An escape character can prefix a reserved character to ensure correct parsing. A common choice is the backslash (\). For example, consider the following data:
Product,Description,Price
Jeans,"Navy Blue, Bootcut, 34\"",49.99

The backslash (\) directly precedes the second double quote ("). This shows that it isn’t the field’s closing quote. Instead, it’s a literal double quote included in the value, showing the jeans size in inches: 34".

Leave this field blank, the default option, to disable escaping.

  • False Values: A set of case-sensitive strings interpreted as false values.
  • Null Values: A set of case-sensitive strings interpreted as null values. To interpret ‘NA’ as null, enter ‘NA’ in this field.
  • Quote Character: Data values may contain reserved characters, such as a comma used as the delimiter. CSVs handle these values by wrapping them in defined quote characters, allowing correct parsing. The default is ".
  • Skip Rows After Header: The number of rows skipped after the header row.
  • Skip Rows Before Header: The number of rows skipped before the header row.
  • Strings Can Be Null: Controls whether strings can be interpreted as null values. If true, strings matching the null_values set are interpreted as null. If false, they’re interpreted as the strings themselves.
  • True Values: A set of case-sensitive strings interpreted as true values.

Parquet

Apache Parquet is a column-oriented data storage format in the Apache Hadoop ecosystem. It offers efficient compression and encoding for handling complex data in bulk. Partitioned parquet datasets aren’t currently supported. These settings are available:

  • Convert Decimal Fields to Floats: Controls whether decimal fields are converted to floats. This conversion loses precision, so it isn’t recommended.

Avro

The Avro parser uses the Fastavro library. These settings are available:

  • Convert Double Fields to Strings: Controls whether double fields are converted to strings. This is recommended for highly precise decimal numbers because floating-point handling can lose precision.

JSONL

JSONL parsing currently has no options.

Document File Type Format

The Document File Type Format extracts text from Markdown, TXT, PDF, Word and Powerpoint documents. When selected, the connector outputs the extracted text in one field named content. The document_key field stores a unique identifier for the processed file, which can serve as a primary key. The document content includes markdown formatting converted from the original format. Each file matching the glob pattern must be markdown (md), PDF (pdf), Word (docx) or Powerpoint (.pptx).

Each document produces one record. Large files can produce large records that don’t fit every destination. Each destination has different limits for string fields.

Parse with the Unstructured.io Python Library

This connector uses the open source Unstructured library for OCR and text extraction. It extracts from PDFs, MS Word files, embedded tables and images. Read about the parsing logic in the Unstructured docs. Learn about other Unstructured tools and services at www.unstructured.io.

IP allow list

If you use Airbyte Cloud and your organisation restricts access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Troubleshoot the connection test

Enter only the bucket name in Bucket. A pasted s3://bucket or https://bucket.s3.amazonaws.com is accepted and normalised to bucket. URLs containing a path are rejected, rather than having their path silently ignored. Move the path into the stream glob or path prefix to avoid reading unintended files.

The connection test identifies invalid access key IDs, signatures, missing buckets and unreachable endpoints. Follow the error to check the access key ID, secret access key, bucket name, region, endpoint URL or network allowlist. For access denied errors, grant the configured IAM user or role s3:ListBucket on the bucket. Grant s3:GetObject on its objects, then check the bucket policy.

A stream without matching files fails the connection test. Compare its globs and path prefix with the bucket’s object keys. If a file is found but can’t be parsed, check its contents against the selected format, encoding and stream format options. The optional Skip Full Check for Parquet setting still skips record parsing.

Changelog

Expand to review
VersionDatePull RequestSubject
4.17.12026-11-1091694Update dependencies
4.17.02026-10-2790132Declare the stepped setup form sections (Connect bucket, Choose files, Review & set up) in the spec and make CHECK section-aware through the check-section config key; the bucket section checks the bucket access only
4.16.42026-11-0390913Raise minimum supported Python version to 3.11
4.16.32026-10-3090252Explain S3 authentication, permission, endpoint, and parsing failures in the connection test; normalize bucket URLs and reject paths in the bucket field.
4.16.22026-11-0390722Update dependencies
4.16.12026-10-2089378Update dependencies
4.16.02026-10-1689048Improve setup form: required Authentication dropdown (IAM User / IAM Role / No Authentication) shown first, followed by Bucket and Streams; existing configs are migrated automatically
4.15.272026-10-1388785Update dependencies
4.15.262026-10-0688018Update dependencies
4.15.252026-09-2987324Update dependencies
4.15.242026-09-2286787Update dependencies
4.15.232026-09-1586234Update dependencies
4.15.222026-09-0885648Update dependencies
4.15.212026-09-0185248Update dependencies
4.15.202026-08-2585018Update dependencies
4.15.192026-08-1884738Update dependencies
4.15.182026-08-1184085Update dependencies
4.15.172026-08-0483624Update dependencies
4.15.162026-07-2883085Update dependencies
4.15.152026-07-2282237Bump base image to python-connector-base 4.1.1 (Python 3.13.14)
4.15.142026-07-2182568Update dependencies
4.15.132026-07-1481995Update dependencies
4.15.122026-07-0781447Update dependencies
4.15.112026-06-3081236Update dependencies
4.15.102026-06-2380620Update dependencies
4.15.92026-06-1680010Update dependencies
4.15.82026-06-1079599Fix CSV sync failures for files with trailing empty headers by upgrading CDK to v7.21.2
4.15.72026-06-0979485Update dependencies
4.15.62026-06-0278421Throttle per-slice state-message emission to once every 10 minutes to prevent the orchestrator’s state buffer OOMing on streams with many files. Final state is force-emitted at end of stream so the destination always sees the latest cursor.
4.15.52026-06-0278945Update dependencies
4.15.42026-05-1478081Updates the source-s3 Poetry lock file to use airbyte-cdk 7.19.1 for CSV column error.
4.15.32026-04-0876162Add opt-in skip_full_check_for_parquet stream option to skip the full parse check for Parquet files during CHECK, avoiding OOM on large files
4.15.22025-11-1169268Update dependencies
4.15.12025-11-0468844Update dependencies
4.15.02025-10-2968640Update dependencies
4.14.62025-10-2167222Update dependencies
4.14.52025-10-0867494Fix utf_8_sig encoding for zip files
4.14.42025-09-3060547Update dependencies
4.14.32025-09-1566023Update to CDK v7
4.14.22025-05-2260863chore(source-s3): bump base image to 4.0.1
4.14.12025-05-1058988Update dependencies
4.14.02025-05-0659685Promoting release candidate 4.14.0-rc.1 to a main version.
4.14.0-rc.12025-05-0557498Adapt file-transfer records to latest protocol, requires platform >= 1.7.0, destination-s3 >= 1.8.0
4.13.52025-04-1957994Update dependencies
4.13.42025-04-0557485Update dependencies
4.13.32025-03-2956791Update dependencies
4.13.22025-03-2252953Update dependencies
4.13.12025-03-1355694Fix bug where csv column name is ‘type’
4.13.02025-03-1255202Bump base image to 4.0.0 and also CDK
4.12.22025-02-1453684Added pendulum to the dependencies
4.12.12025-01-2552509Update dependencies
4.12.02025-01-2052030Promoting release candidate 4.12.0-rc.1 to a main version.
4.12.0-rc.12025-01-1551474Bump cdk to have preserve subdirectories (default) in copy raw files functionality
4.11.42025-01-1151370Update dependencies
4.11.32025-01-0450932Update dependencies
4.11.22024-12-2850739Update dependencies
4.11.12024-12-2149042Update dependencies
4.11.02024-12-1749824Increase file size limit to 1.5GB
4.10.22024-11-2548613Starting with this version, the Docker image is now rootless. Please note that this and future versions will not be compatible with Airbyte versions earlier than 0.64
4.10.12024-11-1248346Implement file-transfer capabilities
4.9.22024-11-0448259Update dependencies
4.9.12024-10-2947038Update dependencies
4.9.02024-10-1746973Promote releae candidate.
4.9.0-rc.12024-10-1446298Migrate to CDK v5
4.8.52024-10-1246511Update dependencies
4.8.42024-09-2846131Update dependencies
4.8.32024-09-2145757Update dependencies
4.8.22024-09-1445504Update dependencies
4.8.12024-09-0745257Update dependencies
4.8.02024-09-0344908Migrate to CDK v3
4.7.82024-08-3145009Update dependencies
4.7.72024-08-2444732Update dependencies
4.7.62024-08-1944380Update dependencies
4.7.52024-08-1243868Update dependencies
4.7.42024-08-1043667Update dependencies
4.7.32024-08-0343083Update dependencies
4.7.22024-07-2742814Update dependencies
4.7.12024-07-2042205Update dependencies
4.7.02024-07-1641934Update to 3.5.1 CDK
4.6.32024-07-1341934Update dependencies
4.6.22024-07-1041503Update dependencies
4.6.12024-07-0940067Update dependencies
4.6.02024-06-2639573Improve performance: update to Airbyte CDK 2.0.0
4.5.172024-06-0639214[autopull] Upgrade base image to v1.2.2
4.5.162024-05-2938674Avoid error on empty stream when running discover
4.5.152024-05-2038252Replace AirbyteLogger with logging.Logger
4.5.142024-05-0938090Bump python-cdk version to include CSV field length fix
4.5.132024-05-0337776Update airbyte-cdk to fix the discovery command issue
4.5.122024-04-1137001Update airbyte-cdk to flush print buffer for every message
4.5.112024-03-1436160Bump python-cdk version to include CSV tab delimiter fix
4.5.102024-03-1135955Pin transformers transitive dependency
4.5.92024-03-0635857Bump poetry.lock to upgrade transitive dependency
4.5.82024-03-0435808Use cached AWS client
4.5.72024-02-2334895Run incremental syncs with concurrency
4.5.62024-02-2135246Fixes bug that occurred when creating CSV streams with tab delimiter.
4.5.52024-02-1835392Add support filtering by start date
4.5.42024-02-1535055Temporarily revert concurrency
4.5.32024-02-1235164Manage dependencies with Poetry.
4.5.22024-02-0634930Bump CDK version to fix issue when SyncMode is missing from catalog
4.5.12024-02-0231701Add region support
4.5.02024-02-0134591Run full refresh syncs concurrently
4.4.12024-01-3034665Pin moto & CDK version
4.4.02024-01-1233818Add IAM Role Authentication
4.3.12024-01-0433937Prepare for airbyte-lib
4.3.02023-12-1433411Bump CDK version to auto-set primary key for document file streams and support raw txt files
4.2.42023-12-0633187Bump CDK version to hide source-defined primary key
4.2.32023-11-1632608Improve document file type parser
4.2.22023-11-2032677Only read files with “.zip” extension as zipped files
4.2.12023-11-1332357Improve spec schema
4.2.02023-11-0232109Fix docs; add HTTPS validation for S3 endpoint; fix coverage
4.1.42023-10-3031904Update CDK
4.1.32023-10-2531654Reduce image size
4.1.22023-10-2331383Add handling NoSuchBucket error
4.1.12023-10-1931601Base image migration: remove Dockerfile and use the python-connector-base image
4.1.02023-10-1731340Add reading files inside zip archive
4.0.52023-10-1631209Add experimental Markdown/PDF/Docx file format
4.0.42023-09-1830476Remove streams.*.file_type from source-s3 configuration
4.0.32023-09-1330387Bump Airbyte-CDK version to improve messages for record parse errors
4.0.22023-09-0728639Always show S3 Key fields
4.0.12023-09-0630217Migrate inference error to config errors and avoir sentry alerts
4.0.02023-09-0529757New version using file-based CDK
3.1.112023-08-3029986Add config error for conversion error
3.1.102023-08-2929943Add config error for arrow invalid error
3.1.92023-08-2329753Feature parity update for V4 release
3.1.82023-08-1729520Update legacy state and error handling
3.1.72023-08-1729505v4 StreamReader and Cursor fixes
3.1.62023-08-1629480update Pyarrow to version 12.0.1
3.1.52023-08-1529418Avoid duplicate syncs when migrating from v3 to v4
3.1.42023-08-1529382Handle legacy path prefix & path pattern
3.1.32023-08-0529028Update v3 & v4 connector to handle either state message
3.1.22023-07-2928786Add a codepath for using the file-based CDK
3.1.12023-07-2628730Add human readable error message and improve validation for encoding field when it empty
3.1.02023-06-2627725License Update: Elv2
3.0.32023-06-2327651Handle Bucket Access Errors
3.0.22023-06-2227611Fix start date
3.0.12023-06-2227604Add logging for file reading
3.0.02023-05-0225127Remove ab_additional column; Use platform-handled schema evolution
2.2.02023-05-1025937Add support for Parquet Dataset
2.1.42023-05-0125361Parse nested avro schemas
2.1.32023-05-0125706Remove minimum block size for CSV check
2.1.22023-04-1825067Handle block size related errors; fix config validator
2.1.12023-04-1825010Refactor filter logic
2.1.02023-04-1025010Add start_date field to filter files based on LastModified option
2.0.42023-03-2324429Call check with a little block size to save time and memory.
2.0.32023-03-1724178Support legacy datetime format for the period of migration, fix time-zone conversion.
2.0.22023-03-1624157Return empty schema if discover finds no files; Do not infer extra data types when user defined schema is applied.
2.0.12023-03-0623195Fix datetime format string
2.0.02023-03-1423189Infer schema based on one file instead of all the files
1.0.22023-03-0223669Made Advanced Reader Options and Advanced Options truly optional for CSV format
1.0.12023-02-2723502Fix error handling
1.0.02023-02-1723198Fix Avro schema discovery
0.1.322023-02-0722500Speed up discovery
0.1.312023-02-0822550Validate CSV read options and convert options
0.1.302023-01-2521587Make sure spec works as expected in UI
0.1.292023-01-1921604Handle OSError: skip unreachable keys and keep working on accessible ones. Warn a customer
0.1.282023-01-1021210Update block size for json file format
0.1.272022-12-0820262Check config settings for CSV file format
0.1.262022-11-0819006Add virtual-hosted-style option
0.1.242022-10-2818602Wrap errors into AirbyteTracedException pointing to a problem file
0.1.232022-10-1017800Deleted use_ssl and verify_ssl_cert flags and hardcoded to True
0.1.232022-10-1017991Fix pyarrow to JSON schema type conversion for arrays
0.1.222022-09-2817304Migrate to per-stream state
0.1.212022-09-2016921Upgrade pyarrow
0.1.202022-09-1216607Fix for reading jsonl files containing nested structures
0.1.192022-09-1316631Adjust column type to a broadest one when merging two or more json schemas
0.1.182022-08-0114213Add support for jsonl format files.
0.1.172022-07-2114911”decimal” type added for parquet
0.1.162022-07-1314669Fixed bug when extra columns apeared to be non-present in master schema
0.1.152022-05-3112568Fixed possible case of files being missed during incremental syncs
0.1.142022-05-2311967Increase unit test coverage up to 90%
0.1.132022-05-1112730Fixed empty options issue
0.1.122022-05-1112602Added support for Avro file format
0.1.112022-04-3012500Improve input configuration copy
0.1.102022-01-288252Refactoring of files’ metadata
0.1.92022-01-069163Work-around for web-UI, backslash - t converts to tab for format.delimiter field.
0.1.72021-11-087499Remove base-python dependencies
0.1.62021-10-156615 & 7058Memory and performance optimisation. Advanced options for CSV parsing.
0.1.52021-09-246398Support custom non Amazon S3 services
0.1.42021-08-135305Support of Parquet format
0.1.32021-08-045197Fixed bug where sync could hang indefinitely on schema inference
0.1.22021-08-025135Fixed bug in spec so it displays in UI correctly
0.1.12021-07-304990Fixed documentation url in source definition
0.1.02021-07-304990Created S3 source connector
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.