Advanced Analytica Advanced Analytica: IBOM
← All connectors
Snowflake Cortex logo

Destination connector

Snowflake Cortex

Send governed data into Snowflake Cortex with a clear, practical setup guide.

Version 0.2.31 community beta

Snowflake Cortex destination

Overview

This page shows you how to set up the Snowflake as a vector destination.

The setup has three parts:

  • Processing - split individual records into chunks that fit the context window. Choose which fields provide context and which provide supplementary metadata.
  • Embedding - convert text into a vector representation with a pre-trained model. OpenAI’s text-embedding-ada-002 and Cohere’s embed-english-light-v2.0 are currently supported. Hugging Face’s e5-base-v2 is coming soon.
  • Snowflake Connection - choose where to store the vectors. This configures a vector store using Snowflake tables with the VECTOR data type.

Prerequisites

To use the Snowflake Cortex destination, you’ll need:

  • An account with API access for OpenAI or Cohere, depending on your chosen embedding method
  • A Snowflake account that supports vector type columns

You’ll need this information to configure the destination:

  • Embedding service API Key - The API key for your OpenAI or Cohere account
  • Snowflake Account - Your Snowflake account name
  • Snowflake User - Your Snowflake user name
  • Snowflake Password - Your Snowflake account password
  • Snowflake Database - The Snowflake database name where you want to load data
  • Snowflake Warehouse - The Snowflake warehouse name to use
  • Snowflake Role - The Snowflake role name to use.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedYes

Data type mapping

Fields specified as metadata are stored in the document’s metadata object and can support filtering. Metadata fields can use these data types:

  • String
  • Number, either integer or floating point, converted to a 64-bit floating point
  • Booleans (true, false)
  • List of String

The destination ignores all other fields.

Configuration

Processing

The destination splits each record into text and metadata fields, as configured in the “Processing” section. It combines all text fields into one string, then splits that string into chunks of your configured length. If specified, metadata fields are stored unchanged with the embedded text chunks. You can use metadata fields for filtering, but not retrieval. They must contain strings, numbers or booleans. The destination ignores all other values. The total metadata saved for each entry can’t exceed 40kb. Chunking configuration options use the Langchain Python library.

You can access nested record fields with dot notation when specifying text fields. For example, user.name accesses the name field in the user object. You can also use wildcards to access every field in an object. For example, users.*.name accesses all names fields across every entry in the users array.

Chunk length is measured in tokens produced by the tiktoken library. The maximum is 8191 tokens, matching the maximum length supported by the text-embedding-ada-002 model.

Each document includes the stream name in the _ab_stream metadata field. When available, the record’s primary key identifies the document and prevents duplication when updated records are indexed. It’s added as the _ab_record_id metadata field.

Embedding

You can choose one of these embedding methods:

  1. OpenAI - using OpenAI API, the connector creates embeddings with the text-embedding-ada-002 model and 1536 dimensions. The speed of the OpenAI embedding API constrains this integration.

  2. Cohere - using the Cohere API, the connector creates embeddings with the embed-english-light-v2.0 model and 1024 dimensions.

For testing, you can also use the Fake embeddings integration. It generates random embeddings, letting you test a data pipeline without incurring embedding costs.

Indexing and data storage

To begin, sign up for Snowflake. Set a database and data warehouse before running the Snowflake Cortex destination. Each stream is indexed and stored in a table with the same name. The destination creates the table if it doesn’t exist. The table contains these columns:

  • document_id (string) - the document’s unique identifier, created by appending the primary keys in the stream schema
  • chunk_id (string) - the chunk’s unique identifier, created by appending the chunk number to the document_id
  • metadata (variant) - the document’s metadata, stored as key-value pairs
  • page_content (string) - the chunk’s text content
  • embedding (vector) - the chunk’s embedding, stored as a list of floats

Namespace support

This destination doesn’t support namespaces.

Changelog

Expand to review
VersionDatePull RequestSubject
0.2.312026-10-0788166Remove the ruff dev dependency so connector CI formats with the repository ruff version
0.2.302026-08-1384363Update the CDK to remediate CVE-2025-68664 in the langchain dependency
0.2.292026-07-0281385Upgrade pillow from 11.x to 12.3.0 to resolve security vulnerabilities GHSA-cfh3-3jmp-rvhc, GHSA-pwv6-vv43-88gr, GHSA-whj4-6x5x-4v2j, GHSA-xg8h-j46f-w952
0.2.282026-03-3175645Bump version to force registry update for supportLevel change to community
0.2.272025-10-2168344Update dependencies
0.2.262025-10-1463066Update dependencies
0.2.252025-05-1751743Update dependencies
0.2.242025-03-0154735Bump snowflake-connector-python from 3.12.2 to 3.13.1 in /airbyte-integrations/connectors/destination-snowflake-cortex
0.2.232025-01-1145786Starting with this version, the Docker image is now rootless. Please note that this and future versions will not be compatible with Airbyte versions earlier than 0.64
0.2.222024-09-1445489Update dependencies
0.2.212024-09-0745313Update dependencies
0.2.202024-08-3144982Update dependencies
0.2.192024-08-2444694Update dependencies
0.2.182024-08-2244530Update test dependencies
0.2.172024-08-1743898Update dependencies
0.2.162024-08-1043584Update dependencies
0.2.152024-08-0343093Update dependencies
0.2.142024-07-2742684Update dependencies
0.2.132024-07-2042263Update dependencies
0.2.122024-07-1341758Update dependencies
0.2.112024-07-1041368Update dependencies
0.2.102024-07-0941173Update dependencies
0.2.92024-07-0640836Update dependencies
0.2.82024-06-2940630Update dependencies
0.2.72024-06-2740215Replaced deprecated AirbyteLogger with logging.Logger
0.2.62024-06-2540468Update dependencies
0.2.52024-06-2340225Update dependencies
0.2.42024-06-2240047Update dependencies
0.2.32024-06-0438955[autopull] Upgrade base image to v1.2.1
0.2.22024-06-04#39092Fix writing when multiple chunks exist for a document.
0.2.12024-06-03#38830Add handling for unexpected/undefined state codes.
0.2.02024-05-30#38337Fix merge behavior when multiple chunks exist for a document. Includes additional refactoring and improvements.
0.1.22024-05-17#38327Fix chunking related issue.
0.1.12024-05-15#38206Bug fixes.
0.1.02024-05-13#37333Add support for Snowflake as a Vector destination.
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.