Advanced Analytica Advanced Analytica: IBOM
← All connectors
Amazon SQS logo

Destination connector

Amazon SQS

Send governed data into Amazon SQS with a clear, practical setup guide.

Version 0.1.18 archived alpha

Amazon SQS

:::warning

This connector is archived. It’s no longer available in the Airbyte Cloud or Airbyte Open Source connector catalogs. It doesn’t receive updates or support from the Airbyte team. For more information, see Archived connectors.

:::

Overview

The Amazon SQS destination sends every source record to one Amazon SQS queue. It supports standard and FIFO queues.

Prerequisites

  • An Amazon SQS queue. See Create a queue in the AWS documentation.
  • An AWS IAM access key ID and secret access key for an IAM user with these queue permissions:
    • sqs:SendMessage
    • sqs:GetQueueAttributes

The connector always authenticates with an access key. The configuration form shows these fields as optional. However, connection checks and syncs fail if you don’t provide them.

Setup guide

Step 1: Create an IAM user and access key

  1. Create an IAM user for Airbyte in the AWS IAM console.

  2. Attach a policy granting the required permissions for your queue. For example:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": ["sqs:SendMessage", "sqs:GetQueueAttributes"],
          "Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>"
        }
      ]
    }
  3. Create an access key for the user, then save the access key ID and secret access key.

Step 2: Set up the destination in Airbyte

Provide these configuration details:

  • Queue URL: The queue’s full URL, for example https://sqs.eu-west-1.amazonaws.com/1234567890/example-queue. The connector treats it as a FIFO queue if the URL ends in .fifo.
  • AWS Region: The queue’s region, for example eu-west-1.
  • AWS IAM Access Key ID and AWS IAM Secret Key: The credentials from Step 1.
  • Message Delay (optional): The delay, in seconds, before consumers can receive each message. Amazon SQS accepts values from 0 to 900. Don’t set this for FIFO queues. See Amazon SQS message timers.
  • Message Body Key (optional): The top-level record field to use as the message body. See Use the Message Body Key.
  • Message Group Id: Required for FIFO queues and ignored for standard queues. See FIFO queues.

Message format

All streams are written to the same queue. The connector sends one message per record without batching messages.

By default, the message body contains the record’s data serialised as a JSON string. It doesn’t include the stream name or other Airbyte metadata. Each message has one message attribute, airbyte_emitted_at and a String attribute. This attribute contains the record’s source emission time in milliseconds since the Unix epoch.

Amazon SQS limits each message’s size. Records exceeding this limit fail to send. See Amazon SQS message quotas.

Use the Message Body Key

Set Message Body Key to use one top-level field’s value as the message body instead of the whole record. Nested fields aren’t supported. Use this setting to send text or XML messages.

The field’s value must be a string. The message fails if the value is an object, array or number. The sync fails if a record doesn’t contain the field or the field is null.

For example, consider this input record:

{
  "my_xml_field": "<something>value</something>"
}

Setting Message Body Key to my_xml_field creates this message body:

<something>value</something>

FIFO queues

When the queue URL ends in .fifo, the connector confirms it’s a FIFO queue and requires Message Group Id. Every message uses the same message group ID. Amazon SQS therefore delivers all records in order as one group.

The queue’s content-based deduplication setting determines how the connector handles deduplication:

  • Disabled: The connector generates a random deduplication ID for each message. As a result, Amazon SQS doesn’t deduplicate any records.
  • Enabled: Amazon SQS deduplicates messages by their body. It delivers identical message bodies only once within the five-minute deduplication interval.

Amazon SQS doesn’t support per-message delays for FIFO queues. Leave Message Delay empty and configure the delay on the queue instead.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteNo
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

Namespace support

This destination doesn’t support namespaces.

Changelog

Expand to review
VersionDatePull RequestSubject
0.1.182026-11-0390866Archive connector
0.1.172024-08-2244530Update test dependencies
0.1.162024-08-0343278Update dependencies
0.1.152024-07-2742795Update dependencies
0.1.142024-07-2042165Update dependencies
0.1.132024-07-1341733Update dependencies
0.1.122024-07-1041439Update dependencies
0.1.112024-07-0941237Update dependencies
0.1.102024-07-0640960Update dependencies
0.1.92024-06-2940622Update dependencies
0.1.82024-06-2740215Replaced deprecated AirbyteLogger with logging.Logger
0.1.72024-06-2640536Update dependencies
0.1.62024-06-2540461Update dependencies
0.1.52024-06-2240075Update dependencies
0.1.42024-06-0439070[autopull] Upgrade base image to v1.2.1
0.1.32024-05-2138493[autopull] base image + poetry + up_to_date
0.1.22024-03-05#35838Un-archive connector
0.1.12024-01-03#33924Add new ap-southeast-3 AWS region
0.1.02021-10-27#0000Initial version
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.