Starburst Galaxy destination guide
Overview
The Starburst Galaxy destination syncs your data to Starburst Galaxy great lake catalogs in Apache Iceberg table format. Each stream is written to a separate Iceberg table.
Data storage
Starburst Galaxy supports various object storages. However, this connector only supports Amazon S3.
Configuration
| Category | Parameter | Type | Notes |
|---|---|---|---|
| Starburst Galaxy | Hostname | string | Required. Located in the Connection info section of the view clusters pane in Starburst Galaxy. |
Port | string | Optional. Located in the Connection info section of the view clusters pane in Starburst Galaxy. Defaults to 443. | |
User | string | Required. Galaxy user found in the Connection info section of the view clusters pane in Starburst Galaxy. | |
Password | string | Required. Password for the specified Galaxy user. | |
Amazon S3 catalog | string | Required. Name of the Amazon S3 catalog created in the Galaxy domain. | |
Amazon S3 catalog schema | string | Optional. The default Starburst Galaxy Amazon S3 catalog schema where tables are written to if the source does not specify a namespace. Each data stream is written to a table in this schema. Defaults to public. | |
| Staging Object Store - Amazon S3 | Bucket name | string | Required. Name of the bucket where the staging data is stored. |
Bucket path | string | Required. Sets the subdirectory of the specified S3 bucket used for storing staging data. | |
Bucket region | string | Required. Sets the region of the specified S3 bucket. | |
Access key | string | Required. AWS/Minio credential. | |
Secret key | string | Required. AWS/Minio credential. | |
| General | Purge staging Iceberg table | boolean | Optional. Indicates that staging Iceberg table is purged after a data sync is complete. Enabled by default. Disable it for debugging purposes only. |
Staging files
S3
Your data streams are written to a temporary Iceberg table. They’re then loaded into the Amazon S3 Starburst Galaxy catalog in Iceberg table format.
The staging table is deleted after each completed sync if Purge staging Iceberg table is enabled.
Here’s an example of a full staging file path:
s3://<bucket-name>/<bucket-path>/<namespace/schema>/<temp Iceberg table name {_airbyte_tmp_random-three-chars_stream-name}>
For example:
s3://galaxy_bucket/data_output_path/test_schema/_airbyte_tmp_qey_user
↑ ↑ ↑ ↑
| | | temporary Iceberg table holding data
| | source namespace or provided schema name
| |
| bucket path
bucket name
Target Iceberg table
Your streams are synced to the Starburst Galaxy Amazon S3 catalog in Iceberg table format.
Output schema
Each table in the output schema contains these columns:
| Column | Type | Description |
|---|---|---|
_airbyte_ab_id | varchar | UUID. |
_airbyte_emitted_at | timestamp(6) | Data emission timestamp. |
| Data fields from the source stream | various | All the fields from the source stream will be populated as an individual column in the target table. |
_airbyte_additional_properties | map(varchar, varchar) | Additional properties. |
The Airbyte data stream’s JSON schema is converted to an Avro schema. The JSON object is then converted to an Avro record. The Avro record is written to a staging Iceberg table. Data streams can come from any data source. Therefore, the JSON-to-Avro conversion process has arbitrary rules and limitations. Learn more about how source data is converted to Avro.
Data type support
Learn more about Starburst Galaxy Iceberg type mapping.
Supported sync modes
| Sync mode | Supported? |
|---|---|
| Full Refresh - Overwrite | Yes |
| Full Refresh - Append | Yes |
| Full Refresh - Overwrite + Deduped | No |
| Incremental Sync - Append | Yes |
| Incremental Sync - Append + Deduped | No |
Getting started
Requirements
- Starburst Galaxy cluster. You’ll find the required credentials in the Connection info section of the view clusters page.
- A Starburst Galaxy S3 catalog created within the Galaxy domain, and attached to a running cluster.
- Credentials for S3 bucket.
- Grant S3 bucket location privileges to the role assigned to the user.
Namespace support
This destination supports namespaces. The namespace maps to a Starburst Galaxy schema.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.0.1 | 2023-03-28 | #24620 | Initial public release. |