Advanced Analytica Advanced Analytica: IBOM
← All connectors
Microsoft SharePoint logo

Source connector

Microsoft SharePoint

Set up Microsoft SharePoint as a source and move its data into your chosen destination.

Version 0.10.5 community alpha

Microsoft SharePoint

Use this page to set up and reference the Microsoft SharePoint source connector.

Prerequisites

  • Application (client) ID
  • Directory (tenant) ID
  • Drive name
  • Folder Path
  • Client secrets

Setup guide

Set up Microsoft SharePoint

Airbyte Cloud

  1. You need an Log into your Airbyte Cloud account.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select Microsoft SharePoint from the Source type dropdown.
  4. Enter a name for the Microsoft SharePoint connector.
  5. Select Search Scope to specify where to search for files. Valid options are ‘ACCESSIBLE_DRIVES’, ‘SHARED_ITEMS’ and ‘ALL’. ‘ACCESSIBLE_DRIVES’ searches all SharePoint drives you can access. ‘SHARED_ITEMS’ searches shared items you can access. ‘ALL’ searches both and is the default value. Microsoft is retiring the Graph sharedWithMe API used by ‘SHARED_ITEMS’ and ‘ALL’ to find shared items. The API stops returning data after November 2026. To sync files shared from a SharePoint site, select ‘ACCESSIBLE_DRIVES’ and set Site URL to that site. This setting syncs the whole document library. You can’t sync files shared from another user’s OneDrive after the retirement.
  6. Enter Folder Path. Leave it empty to search every folder in the drives. This setting doesn’t apply to shared items.
  7. The OAuth2.0 authorisation method is selected by default. Click Authenticate your Microsoft SharePoint account, then sign in and authorise your Microsoft account.
  8. For Start Date, enter the date in YYYY-MM-DD format. Data added on or after this date will be replicated.
  9. Add a stream:
    1. Enter the File Type.
    2. In the Format box, select the file format you want to replicate from the dropdown menu. Supported formats are CSV, Parquet, Avro and JSONL. Toggle the Optional fields button in the Format box to enter more settings for your selected format. For a detailed breakdown of these settings, see the File Format section below.
    3. Give the stream a Name.
    4. Optionally, enter a Input schema to enforce a specific schema. By default, this value is {}, which infers the schema from the file(s) you’re replicating. For details about providing a custom schema, see the User Schema section.
    5. Optionally, enter the Globs to control which files are synced. This regular expression lets Airbyte match the specific files to replicate. To replicate every file in your bucket, use ** as the pattern. For more precise matching options, see the Path Patterns section below.
  10. Click Set up source.

Airbyte Open Source

  1. Go to the Airbyte Open Source dashboard.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select Microsoft SharePoint from the Source type dropdown.
  4. Enter a name for the Microsoft SharePoint connector.

Step 1: Set up the SharePoint application

Microsoft Graph API uses OAuth for authentication. Microsoft Graph provides granular permissions that control app access to resources such as users, groups and mail. When you sign in, you or an administrator can consent to the requested permissions. After consent, your app can access the resources and APIs it requested. For apps without a signed-in user, an administrator can pre-consent to permissions during installation.

Microsoft Graph provides two permission types:

  • Delegated permissions apply to apps with a signed-in user. The user or an administrator consents to the requested permissions. The app can then act as the signed-in user when calling Microsoft Graph. Non-administrative users can consent to some delegated permissions. Higher-privileged permissions require administrator consent.
  • Application permissions apply to apps without a signed-in user, including background services and daemons. Only an administrator can consent to application permissions.

This source requires Application permissions. Follow these instructions to create an app in the Azure portal. This process provides the client_id, client_secret and tenant_id required for the tap configuration file.

  1. Sign in to Azure Portal.
  2. Click the upper-left menu icon and select Azure Active Directory.
  3. Select App Registrations.
  4. Click New registration.
  5. Register an application:
    1. Name:
    2. Supported account types: Accounts in this organizational directory only
    3. Register (button)
  6. Record the client_id and tenant_id. The tap uses them for authentication and API integration.
  7. Select Certificates & secrets.
  8. Provide Description and Expires:
    1. Description: tap-microsoft-teams client secret
    2. Expires: 1-year
    3. Add
  9. Copy the client secret value. This value is the client_secret.
  10. Select API permissions:
    1. Click Add a permission.
  11. Select Microsoft Graph.
  12. Select Application permissions.
  13. Select the following permissions:
    1. Files
      • Files.Read.All
  14. Click Add permissions.
  15. Click Grant admin consent.

Step 2: Set up the connector in Airbyte

  1. Go to the Airbyte Open Source dashboard.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select Microsoft SharePoint from the Source type dropdown.
  4. Enter a name for the Microsoft SharePoint connector.
  5. Select Search Scope to specify where to search for files. Valid options are ‘ACCESSIBLE_DRIVES’, ‘SHARED_ITEMS’ and ‘ALL’. ‘ACCESSIBLE_DRIVES’ searches all SharePoint drives you can access. ‘SHARED_ITEMS’ searches shared items you can access. ‘ALL’ searches both and is the default value. Service Key Authentication can’t list items shared with you, so ‘SHARED_ITEMS’ isn’t supported. With Service Key Authentication, ‘ALL’ only searches accessible drives. To sync files shared from a SharePoint site, select ‘ACCESSIBLE_DRIVES’ and enter that site’s Site URL. This setting syncs the whole document library.
  6. Enter Folder Path. Leave it empty to search every folder in the drives. This setting doesn’t apply to shared items.
  7. Switch to Service Key Authentication.
  8. For User Practical Name, enter the UPN for your user.
  9. Enter Tenant ID, Client ID and Client secret.
  10. For Start Date, enter the date in YYYY-MM-DD format. Data added on or after this date will be replicated.
  11. Add a stream:
    1. Enter the File Type.
    2. In the Format box, select the file format you want to replicate from the dropdown menu. Supported formats are CSV, Parquet, Avro and JSONL. Toggle the Optional fields button in the Format box to enter more settings for your selected format. For a detailed breakdown of these settings, see the File Format section below.
    3. Give the stream a Name.
    4. Optionally, enter a Input schema to enforce a specific schema. By default, this value is {}, which infers the schema from the file(s) you’re replicating. For details about providing a custom schema, see the User Schema section.
    5. Optionally, enter the Globs to control which files are synced. This regular expression lets Airbyte match the specific files to replicate. To replicate every file in your bucket, use ** as the pattern. For more precise matching options, see the Path Patterns section below.
  12. Click Set up source.

Path patterns

(In short: path pattern syntax uses wcmatch.glob. GLOBSTAR and SPLIT flags are enabled.)

Use glob-style patterns to sync multiple files without specifying every file path. You can:

  • Reference many files with one pattern. For example, ** indicates every file in the folder.
  • Reference future files that don’t yet exist and therefore don’t have a specific path.

You must provide a path pattern. For complex directory layouts, you can provide multiple patterns separated by |.

Each path pattern starts from the folder’s root. Don’t include the root folder name in the pattern(s).

Example patterns:

  • **: Match everything.
  • **/*.csv: Match all files with a specific extension.
  • myFolder/**/*.csv: Match all csv files anywhere under myFolder.
  • */**: Match everything at least one folder deep.
  • */*/*/**: Match everything at least three folders deep.
  • **/file.*|**/file: Match every file called “file”, with any extension or no extension.
  • x/*/y/*: Match all files in sub-folder x -> any folder -> folder y.
  • **/prefix*.csv: Match all csv files with a specific prefix.
  • **/prefix*.parquet: Match all parquet files with a specific prefix.

Consider this folder layout. MyFolder is the root folder specified in the connector config, and all patterns are relative to it:

MyFolder
    -> log_files
    -> some_table_files
        -> part1.csv
        -> part2.csv
    -> images
    -> more_table_files
        -> part3.csv
    -> extras
        -> misc
            -> another_part1.csv

To include part1.csv, part2.csv and part3.csv, while excluding another_part1.csv, you have several options:

  • Use the single pattern **/part*.csv to include every csv file called “partX”.
  • For more precise matching, use the dual pattern some_table_files/*.csv|more_table_files/*.csv to include relevant files only from those exact folders.
  • Use *table_files/*.csv to achieve the same result with one pattern. However, unexpected new folders could cause future problems.
  • Use a recursive wildcard. The pattern extras/**/*.csv includes csv files nested below “extras”, such as “extras/misc/another_part1.csv”.

Path patterns offer several ways to achieve the same result. Choose a clear pattern that accounts for future additions to the directory structure.

User schema

For Avro, Jsonl, CSV or Parquet formats, you can provide a schema for the output stream. Note that this doesn’t apply to the Document file type format.

A schema gives you more control over the stream output. Without one, columns and datatypes are inferred from the first created file matching your path pattern and suffix. This approach will suit most cases. However, you may want to enforce a schema when:

  • You only need a known subset of columns. Other columns remain included but are packed into the _ab_additional_properties map.
  • Your initial dataset has few records, so automatic type inference might not represent future data.
  • You want to define types for every column.
  • You know which columns future data will add. You want them in the core schema, rather than the _ab_additional_properties map.

You may also have another reason. Provide the schema as valid JSON, using a map of {"column": "datatype"}. Each datatype must be one of:

  • string
  • number
  • integer
  • object
  • array
  • boolean
  • null

For example:

  • {"id": "integer", "location": "string", "longitude": "number", "latitude": "number"}
  • {"username": "string", "friends": "array", "information": "object"}

File format settings

CSV

CSV files are plain text, so you may need specific reader options to parse them correctly. Apply these settings consistently whenever a CSV is created or exported.

  • Header Definition: Defines the headers. User Provided assumes the CSV has no header row and uses the provided headers. Autogenerated also assumes there is no header row. The CDK generates headers using f{i}, where i is the index starting from 0. Otherwise, the default behaviour uses the CSV file’s header. To generate or provide column names for a CSV with headers, set the “Skip rows before header” option. This setting ignores the header row.
  • Delimiter: CSV means Comma Separated Values, but the term also covers flat-file data using other separators. The delimiter field specifies the separator character. To use tab-delimiters, set this value to \t. The default value is ,.
  • Double Quote: Determines whether two quotes within a quoted CSV value represent one quote in the data. The default is True.
  • Encoding: Some data uses another character set, typically when it contains different alphabets. See the list of allowable encodings here. The default is utf8.
  • Escape Character: Prefix a reserved character with an escape character to ensure correct parsing. A common escape character is the backslash (\). For example, consider the following data:
Product,Description,Price
Jeans,"Navy Blue, Bootcut, 34\"",49.99

The backslash (\) directly precedes the second double quote ("), showing that it isn’t the field’s closing quote. Instead, the parser includes it as a literal double quote. Here, it marks the jeans’ size in inches: 34".

Leave this field blank, the default option, to prevent escaping.

  • False Values: A set of case-sensitive strings to interpret as false values.
  • Null Values: A set of case-sensitive strings to interpret as null values. For example, enter ‘NA’ to interpret ‘NA’ as null.
  • Quote Character: Data values can contain reserved characters, such as a comma used as the delimiter. CSVs handle these characters by wrapping the value in defined quote characters, allowing correct parsing. The default is ".
  • Skip Rows After Header: The number of rows to skip after the header row.
  • Skip Rows Before Header: The number of rows to skip before the header row.
  • Strings Can Be Null: Controls whether strings can be interpreted as null values. If true, strings matching the null_values set are interpreted as null. If false, they’re interpreted as the original strings.
  • True Values: A set of case-sensitive strings to interpret as true values.

Parquet

Apache Parquet is a column-oriented data storage format in the Apache Hadoop ecosystem. Its data compression and encoding schemes support efficient bulk processing of complex data. Partitioned parquet datasets aren’t currently supported. The following setting is available:

  • Convert Decimal Fields to Floats: Controls whether decimal fields are converted to floats. This conversion loses precision, so it isn’t recommended.

Avro

The Avro parser uses the Fastavro library. The following setting is available:

  • Convert Double Fields to Strings: Controls whether double fields are converted to strings. This is recommended for highly precise decimal numbers because floating-point handling can lose precision.

JSONL

JSONL parsing currently has no options.

Document file type format

The Document file type format extracts text from Markdown, TXT, PDF, Word, Powerpoint and Google documents. When selected, the connector outputs the extracted text in one field named content. The document_key field contains a unique identifier for the processed file and can serve as a primary key. Document content contains markdown formatting converted from the original file format. Every file matching the defined glob pattern must be a markdown (md), PDF (pdf) or Docx (docx) file.

The connector emits one record for each document. Large files can produce records that exceed a destination’s string-field limits, which vary by destination.

Before parsing, the connector internally exports Google Document files to Docx format. It exports and parses Google Sheets, Google Slides and drawings as PDFs.

Copy raw files configuration

:::info

Raw file replication has the following requirements and limitations:

  • Supported Airbyte Versions:
    • Cloud: All Workspaces
    • OSS / Enterprise: v1.2.0 or later
  • Max File Size: 1GB per file
  • Supported Destinations:
    • S3: v1.4.0 or later

:::

Copy raw files without parsing their contents. Bits reach the destination exactly as they appeared in the source. This method is recommended for unstructured text, non-text and compressed files.

Format options don’t apply. Files are transferred to the file-based destination without parsing the underlying data.

Preserve subdirectories in file paths

When enabled, this option sends the subdirectory structure and source file names to the destination. Otherwise, files sync using only their names. This option is ignored when file-based replication isn’t enabled.

Multi-site support

Enter a url in the site URL field to access files from a specific sharepoint site. The site url must use the format https://<tenant_name>.sharepoint.com/sites/<site>. Leave the field empty to access files from the main site. To iterate through every sub-site, enter https://<tenant_name>.sharepoint.com/sites/ as the site url.

Supported sync modes

The Microsoft SharePoint source connector supports these sync modes:

FeatureSupported?(Yes/No)
Full Refresh SyncYes
Incremental SyncYes

Supported streams

There are no predefined streams. Streams depend on the file contents added on the Set up page.

Performance considerations

Normal Microsoft Graph requests limitation restrict the connector.

Data type map

Integration TypeAirbyte Type
stringstring
numbernumber
arrayarray
objectobject

IP allow list

If you use Airbyte Cloud and restrict access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Changelog

Expand to review
VersionDatePull RequestSubject
0.10.52026-10-2289687Keep syncing drive files and report a clear error when the deprecated Microsoft Graph sharedWithMe API stops returning shared items
0.10.42025-08-2165118Decertify connector
0.10.32025-07-1360562Update dependencies
0.10.22025-05-1059113Update dependencies
0.10.12025-05-0759711Fix edege case for unexcpeted uris.
0.10.02025-05-0759700Promoting release candidate 0.10.0-rc.1 to a main version.
0.10.0-rc.12025-05-0557507Adapt file-transfer records to latest protocol, requires platform >= 1.7.0, destination-s3 >= 1.8.0
0.9.32025-04-1958471Update dependencies
0.9.22025-04-1257920Update dependencies
0.9.12025-04-0557065Update dependencies
0.9.02025-04-0155912Provide ability to iterate all sharepoint sites
0.8.22025-03-2956712Update dependencies
0.8.12025-03-2256014Update dependencies
0.8.02025-03-1254658Provide ability to sync other sites than Main sharepoint site
0.7.22025-03-0855427Update dependencies
0.7.12025-03-0154749Update dependencies
0.7.02025-02-2754200Add advanced Oauth
0.6.12025-02-2245062Update dependencies
0.6.02025-02-2054140Implement file transfer mode to move raw files
0.5.22024-08-2445646Fix: handle wrong folder name
0.5.12024-08-2444660Update dependencies
0.5.02024-08-1942983Migrate to CDK v4.5.1
0.4.52024-08-1944382Update dependencies
0.4.42024-08-1243743Update dependencies
0.4.32024-08-1043565Update dependencies
0.4.22024-08-0343235Update dependencies
0.4.12024-07-2742704Update dependencies
0.4.02024-07-2542008Migrate to CDK v3.5.3
0.3.12024-07-2042143Update dependencies
0.3.02024-07-1642007Migrate to CDK v2.4.0
0.2.112024-07-1341688Update dependencies
0.2.102024-07-1041589Update dependencies
0.2.92024-07-0640917Update dependencies
0.2.82024-06-2640539Update dependencies
0.2.72024-06-2540357Update dependencies
0.2.62024-06-2440233Update dependencies
0.2.52024-06-2239987Update dependencies
0.2.42024-05-2938675Avoid error on empty stream when running discover
0.2.32024-04-1737372Make refresh token optional
0.2.22024-03-2836573Update QL to 400
0.2.12024-03-2236381Unpin CDK
0.2.02024-03-0635830Add fetching shared items
0.1.02024-01-2533537New source
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.