Advanced Analytica Advanced Analytica: IBOM
← All connectors
Azure Blob Storage logo

Source connector

Azure Blob Storage

Set up Azure Blob Storage as a source and move its data into your chosen destination.

Version 0.8.42 certified generally available

Azure Blob Storage

Use this page to set up and reference the Azure Blob Storage source connector.

:::info Cloud storage may incur egress costs. Egress is data transferred out of cloud storage, including downloaded files or files accessed elsewhere. For more information, see the Azure Blob Storage pricing guide. :::

:::warning Azure storage account retirements

Microsoft will retire general-purpose v1 (GPv1) and legacy Blob Storage (BlobStorage kind) accounts on October 13, 2026. Before then, check the account kind for the storage account you sync from and upgrade it to general-purpose v2 (GPv2). Microsoft will automatically migrate any remaining accounts to GPv2 after the retirement date. The upgrade retains your endpoint names and data, so you don’t need to change your Airbyte source configuration. However, migration timing can vary, access may be temporarily interrupted, and legacy Blob Storage accounts may be billed differently afterwards. See Microsoft’s GPv1 migration overview and legacy Blob Storage account migration overview.

:::

Prerequisites

Minimum permissions (role [Storage Blob Data Reader](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/storage#storage-blob-data-reader) ): ```json [ { "actions": [ "Microsoft.Storage/storageAccounts/blobServices/containers/read", "Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action" ], "notActions": [], "dataActions": [ "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read" ], "notDataActions": [] } ] ```

Set up the connector

1. Set up Azure Blob Storage

  • Create a storage account and grant roles details

:::warning Set up Access Control (IAM) to use Oauth2 or Client Credentials Authentication. The recommended role is Storage Blob Data Reader

Follow these steps to set up an IAM role:
  1. In Azure portal, select the Storage or Container you want to sync. Then open Access Control(IAM) -> Role Assignment Access Control (IAM)
  2. Click Add, then select Add role assignment from the dropdown list Add role assignment
  3. Search for Storage Blob Data Reader in the search box. Select the role, then click Next Search Role
  4. Select User, Group, or service principal, then click members. Select members so they appear in the table, then click Next Add Members
  5. (Optional) Add Conditions to restrict which role assignments a user can create.
  6. Click Review + Assign
:::
Follow these steps to set up a Service Principal to use the Client Credentials authentication method.

In Azure portal, open your Service Principal’s App Registration.

Note the Directory (tenant) ID and Application (client) ID shown in the Overview panel.

In the Manage / Certificates & secrets panel, click Client Secrets and create a new secret. Note the secret’s Value.

2. Set up the connector in Airbyte

Airbyte Cloud

  1. Log into your Airbyte Cloud account.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select Azure Blob Storage from the Source type dropdown.
  4. Enter a name for the Azure Blob Storage connector.
  5. Choose a delivery method for your data.
  6. Enter the name of your Azure Account.
  7. Enter your Tenant ID, then click Authenticate your Azure Blob Storage account.
  8. Sign in and authorise the Azure Blob Storage account.
  9. Enter the name of the Container containing the files you want to replicate.
  10. Add a stream
  11. Enter the File Type
  12. In the Format box, select the format of the files you want to replicate. Supported formats are CSV, Parquet, Avro and JSONL. Turn on Optional fields in the Format box to enter more settings for the selected format. For details about these settings, see the File Format section below.
  13. Give the stream a Name
  14. (Optional) To enforce a specific schema, enter a Input schema. The default value is {}, which infers the schema automatically from the replicated file(s). For details about providing a custom schema, see the User Schema section.
  15. Optionally, enter the Globs to control which files are synced. This regular expression lets Airbyte match the specific files to replicate. To replicate every file in your bucket, use ** as the pattern. For more precise matching options, see the Path Patterns section below.
  16. (Optional) Enter the endpoint to use for data replication.
  17. (Optional) Enter the start date for data replication.

For Airbyte Open Source:

  1. Open the Airbyte Open Source dashboard.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select Azure Blob Storage from the Source type dropdown.
  4. Enter a name for the Azure Blob Storage connector.
  5. Choose a delivery method for your data.
  6. Enter the name of your Azure Storage Account and container.
  7. Choose the Authentication method.
    1. For access through a Storage Account Key, choose Authenticate via Storage Account Key and enter the key.
    2. For access through a Service Principal, choose the Authenticate via Client Credentials.
    3. See above for setting IAM role bindings and getting the Service Principal’s app registration details
    4. Enter the app registration’s Directory (tenant) ID value from Azure Portal in the Tenant ID field.
    5. Enter the Application (client) ID from Azure Portal in the Client ID field. This is not the secret ID
    6. Enter the Secret Value from Azure Portal in the Client Secret field.
  8. Add a stream
    1. Enter the File Type
    2. In the Format box, select the format of the files you want to replicate. Supported formats are CSV, Parquet, Avro and JSONL. Turn on Optional fields in the Format box to enter more settings for the selected format. For details about these settings, see the File Format section below.
    3. Give the stream a Name
    4. (Optional) To enforce a specific schema, enter a Input schema. The default value is {}, which infers the schema automatically from the replicated file(s). For details about providing a custom schema, see the User Schema section.
    5. Optionally, enter the Globs to control which files are synced. This regular expression lets Airbyte match the specific files to replicate. To replicate every file in your bucket, use ** as the pattern. For more precise matching options, see the Path Patterns section below.
  9. (Optional) Enter the endpoint to use for data replication.
  10. (Optional) Enter the start date for data replication.

Supported sync modes

The Azure Blob Storage source connector supports these sync modes:

FeatureSupported?
Full Refresh SyncYes
Incremental SyncYes
Replicate Incremental DeletesNo
Replicate Multiple Files (pattern matching)Yes
Replicate Multiple Streams (distinct tables)Yes
NamespacesNo

Supported streams

File compression

CompressionSupported?
GzipYes
ZipNo
Bzip2Yes
LzmaNo
XzNo
SnappyNo

Share any specific compression formats you’d like supported next.

Path patterns

(In short: path pattern syntax uses wcmatch.glob. GLOBSTAR and SPLIT flags are enabled.)

Use glob-style patterns to sync multiple files without specifying every file path. This lets you:

  • Reference many files with one pattern. For example, ** indicates every file in the bucket.
  • Reference future files that don’t yet exist (and therefore don’t have a specific path).

You must provide a path pattern. For complex directory layouts, you can provide multiple patterns separated by |.

Each path pattern starts from the bucket’s root. Don’t include the bucket name in the pattern(s).

Example patterns include:

  • **: Match everything.
  • **/*.csv: Match every file with a specific extension.
  • myFolder/**/*.csv: Match every csv file anywhere under myFolder.
  • */**: Match everything at least one folder deep.
  • */*/*/**: Match everything at least three folders deep.
  • **/file.*|**/file: Match every file named “file” with any extension (or no extension).
  • x/*/y/*: Match every file in folder x -> any folder -> folder y.
  • **/prefix*.csv: Match every csv file with a specific prefix.
  • **/prefix*.parquet: Match every parquet file with a specific prefix.

Consider a specific example using this bucket layout:

myBucket
    -> log_files
    -> some_table_files
        -> part1.csv
        -> part2.csv
    -> images
    -> more_table_files
        -> part3.csv
    -> extras
        -> misc
            -> another_part1.csv

You want part1.csv, part2.csv and part3.csv, while excluding another_part1.csv. You can match them in several ways:

  • Match every csv file named “partX” with the single pattern **/part*.csv.
  • Use the dual pattern some_table_files/*.csv|more_table_files/*.csv to match relevant files only from those exact folders.
  • Use *table_files/*.csv to achieve the same result with one pattern. However, unexpected new folders could cause future problems.
  • Use recursive wildcards. The pattern extras/**/*.csv matches any csv files nested below “extras”, including “extras/misc/another_part1.csv”.

Path patterns offer several ways to achieve the same goal. Choose a clear pattern that accommodates future additions to the directory structure.

User schema

Provide a schema for greater control over the stream’s output. Without one, the connector infers columns and datatypes from the first created matching file. The file must match your path pattern and suffix. This approach will suit most cases, but you may want to enforce a schema when:

  • You only need a known subset of columns. Other columns remain included but are packed into the _ab_additional_properties map.
  • Your initial dataset has few records, so automatic type inference may not represent future data.
  • You want to define every column’s type deliberately.
  • You know which columns future data will add. You want them as core schema columns, not in the _ab_additional_properties map.

You may also have another reason. Provide the schema as valid JSON mapping {"column": "datatype"}, with each datatype set to one of:

  • string
  • number
  • integer
  • object
  • array
  • boolean
  • null

For example:

  • {"id": "integer", "location": "string", "longitude": "number", "latitude": "number"}
  • {"username": "string", "friends": "array", "information": "object"}

File format settings

CSV

CSV files are plain text, so correct parsing often requires specific reader options. Apply these settings consistently whenever you create or export a CSV.

  • Header Definition: Defines the headers. User Provided assumes the CSV has no header row and uses the provided headers. Autogenerated also assumes there isn’t a header row. The CDK generates headers using f{i}, where i is the index starting from 0. Otherwise, the default behaviour uses the CSV file’s header. To generate or provide column names for a CSV with headers, set “Skip rows before header” to ignore the header row.
  • Delimiter: CSV means Comma Separated Values, but the term also covers flat-file data that may use another separator. The delimiter field defines which character acts as the separator. To use tab-delimiters, set this value to \t. The default value is ,.
  • Double Quote: Determines whether two quotes within a quoted CSV value represent one quote in the data. The default is True.
  • Encoding: Some data uses another character set, typically when it includes different alphabets. See the list of allowable encodings here. The default is utf8.
  • Escape Character: Prefix a reserved character with an escape character to support correct parsing. A common choice is the backslash (\). Consider this data:
Product,Description,Price
Jeans,"Navy Blue, Bootcut, 34\"",49.99

The backslash (\) directly precedes the second double quote ("), showing that it isn’t the field’s closing quote. Instead, it’s a literal double quote included in the value. Here, it indicates the jeans’ size in inches: 34".

Leave this field blank, the default option, to disallow escaping.

  • False Values: A set of case-sensitive strings interpreted as false values.
  • Null Values: A set of case-sensitive strings interpreted as null values. To interpret ‘NA’ as null, enter ‘NA’ in this field.
  • Quote Character: Data values may contain reserved characters, such as a comma used as the delimiter. CSVs handle these characters by enclosing values in defined quote characters, supporting correct parsing. The default is ".
  • Skip Rows After Header: The number of rows to skip after the header row.
  • Skip Rows Before Header: The number of rows to skip before the header row.
  • Strings Can Be Null: Controls whether strings can be interpreted as null values. If true, strings matching the null_values set become null. If false, strings matching the null_values set remain unchanged.
  • True Values: A set of case-sensitive strings interpreted as true values.

Parquet

Apache Parquet is a column-oriented data storage format in the Apache Hadoop ecosystem. It offers efficient data compression and encoding for processing complex data in bulk. Partitioned parquet datasets aren’t currently supported. The following setting is available:

  • Convert Decimal Fields to Floats: Controls whether decimal fields are converted to floats. This conversion loses precision, so it isn’t recommended.

Avro

The Avro parser uses the Fastavro library. The following setting is available:

  • Convert Double Fields to Strings: Controls whether double fields are converted to strings. This is recommended for highly precise decimal numbers because floating-point handling can lose precision.

JSONL

JSONL parsing currently has no options.

Document File Type Format

The Document File Type Format extracts text from Markdown, TXT, PDF, Word and Powerpoint documents. When selected, the connector outputs extracted document text in one field named content. The document_key field stores a unique identifier for the processed file and can serve as a primary key. Document content includes markdown formatting converted from the original file format. Each file matching the glob pattern must be markdown (md), PDF (pdf), Word (docx) or Powerpoint (.pptx).

The connector emits one record for each document. Large files can produce records that exceed some destinations’ different string-field limits.

Parse with the Unstructured.io Python Library

This connector uses the open source Unstructured library for OCR and text extraction from PDFs and MS Word files. It also extracts text from embedded tables and images. Read about the parsing logic in the Unstructured docs. Learn about other Unstructured tools and services at www.unstructured.io.

IP allow list

If you use Airbyte Cloud and restrict access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Performance

Under normal use, the Azure Blob Storage connector shouldn’t encounter any Microsoft API limitations.

Raw file copy limits

When using the Copy raw files delivery method:

  • Maximum file size: 1.5 GB per file
  • Requires Airbyte version 1.2.0 or later (1.7.0+ for metadata support)
  • Only works with file-based destinations that support file transfer

Changelog

Expand to review
VersionDatePull RequestSubject
0.8.422026-11-1091353Update dependencies
0.8.412026-11-0390485Update dependencies
0.8.402026-10-2789808Update dependencies
0.8.392026-10-2089104Update dependencies
0.8.382026-10-1388506Update dependencies
0.8.372026-10-0687764Update dependencies
0.8.362026-09-2987067Update dependencies
0.8.352026-09-2286513Update dependencies
0.8.342026-09-1585952Update dependencies
0.8.332026-09-0885398Update dependencies
0.8.322026-09-0185241Update dependencies
0.8.312026-08-2585007Update dependencies
0.8.302026-08-1884471Update dependencies
0.8.292026-08-1183826Update dependencies
0.8.282026-08-0483347Update dependencies
0.8.272026-07-2882801Update dependencies
0.8.262026-07-2282241Bump base image to python-connector-base 4.1.1 (Python 3.13.14)
0.8.252026-07-2182311Update dependencies
0.8.242026-07-1481715Update dependencies
0.8.232026-07-0781437Update dependencies
0.8.222026-06-3080986Update dependencies
0.8.212026-06-2380365Update dependencies
0.8.202026-06-1679773Update dependencies
0.8.192026-06-0979224Update dependencies
0.8.182026-06-0278559Update dependencies
0.8.172026-04-2877147Update dependencies
0.8.162026-04-2175028Update dependencies
0.8.152026-03-1074507Update dependencies
0.8.142026-03-0374178Update dependencies
0.8.132026-02-1773438Update dependencies
0.8.122026-01-2772368Update dependencies
0.8.112026-01-2071921Update dependencies
0.8.102026-01-1471450Update dependencies
0.8.92026-01-0771000Enable ‘Copy raw files’ delivery method for file transfers
0.8.82025-12-1870804Update dependencies
0.8.72025-12-0370316Increase memory for check_connection to 4096Mi
0.8.62025-12-0270291Update dependencies
0.8.52025-11-2569910Update dependencies
0.8.42025-11-1869579Update dependencies
0.8.32025-11-1169269Update dependencies
0.8.22025-11-0469156Update dependencies
0.8.12025-10-2968376Update dependencies
0.8.02025-10-2768615Update dependencies
0.7.02025-10-2768663Promoting release candidate 0.7.0-rc.1 to a main version.
0.7.0-rc.12025-10-2168161Update to airbyte-cdk ^v7
0.6.162025-10-1468015Update dependencies
0.6.152025-10-0767171Update dependencies
0.6.142025-09-3066167Update dependencies
0.6.132025-08-2365326Update dependencies
0.6.122025-07-2663800Update dependencies
0.6.112025-07-1963472Update dependencies
0.6.102025-07-1263067Update dependencies
0.6.92025-07-0562534Update dependencies
0.6.82025-05-2760867Update dependencies
0.6.72025-05-2460675Update dependencies
0.6.62025-05-1059806Update dependencies
0.6.52025-05-0359318Update dependencies
0.6.42025-04-2658728Update dependencies
0.6.32025-04-1958295Update dependencies
0.6.22025-04-1257627Update dependencies
0.6.12025-04-0543869Update dependencies
0.6.02025-03-3156466Upgrading dependencies and CDK and Python to 3.11
0.5.02025-01-0250398Add client_credentials auth for Azure Service Principals
0.4.42024-06-0639275[autopull] Upgrade base image to v1.2.2
0.4.32024-05-2938701Avoid error on empty stream when running discover
0.4.22024-04-2337504Update specification
0.4.12024-04-2237467Fix start date filter
0.4.02024-04-0536825Add oauth 2.0 support
0.3.62024-04-0336542Use Latest CDK; add integration tests
0.3.52024-03-2636487Manage dependencies with Poetry.
0.3.42024-02-0634936Bump CDK version to avoid missing SyncMode errors
0.3.32024-01-3034681Unpin CDK version to make compatible with the Concurrent CDK
0.3.22024-01-3034661Pin CDK version until upgrade for compatibility with the Concurrent CDK
0.3.12024-01-1034084Fix bug for running check with document file format
0.3.02023-12-1433411Bump CDK version to auto-set primary key for document file streams and support raw txt files
0.2.52023-12-0633187Bump CDK version to hide source-defined primary key
0.2.42023-11-1632608Improve document file type parser
0.2.32023-11-1332357Improve spec schema
0.2.22023-10-3031904Update CDK to support document file types
0.2.12023-10-1831543Base image migration: remove Dockerfile and use the python-connector-base image
0.2.02023-10-1031336Migrate to File-based CDK. Add support of CSV, Parquet and Avro files
0.1.02023-02-1723222Initial release with full-refresh and incremental sync with JSONL files
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.