AMQP Private Preview

The Advanced Message Queuing Protocol (AMQP) is an open standard messaging protocol that enables applications to communicate asynchronously and reliably through message-oriented middleware.
Fivetran’s AMQP connector lets you securely consume messages from AMQP 1.0-compatible brokers such as Solace PubSub+ and RabbitMQ.

Features

Feature Name	Supported	Notes
Capture deletes
History mode
Custom data
Data blocking
Column hashing
Re-sync		Connection level. We fetch events which are present in the source during the resync.
API configurable
Priority-first sync
Fivetran data models
Private networking
Authorization via API

Supported deployment models

We support the SaaS and Hybrid deployment models for the connector.

You must have an Enterprise or Business Critical plan to use the Hybrid Deployment model.

Setup guide

Follow our step-by-step AMQP setup guide to connect your AMQP 1.0-compatible brokers with your destination using Fivetran connectors.

Schema information

We create a table in your destination using the queue name.
For each table, we create the following columns as per the AMQP properties:
- event_id (Primary Key)
- data
- message_id
- user_id
- to
- subject
- reply_to
- correlation_id
- content_type
- content_encoding
- absolute_expiry_time
- created_time
- group_id
- group_sequence
- reply_to_group_id
- application_properties
If you sync JSON messages, Fivetran delivers your data in either packed or unpacked format as following:
- For packed messages, we sync the data into the data column.
- Unpacked messages must be in JSON format. For all the first-level JSON elements, we create a separate column with the data_<element_name> name format.

Sync note

When we sync events from the AMQP queue, we acknowledge each message after successfully processing it. This ensures that the message is not re-delivered and is deleted from the queue once acknowledged.
We use an event’s created_time value to determine the sync window. Only events created before the sync start time are processed. The sync stops once the connector encounters events with a created_time later than the sync start time.
If the created_time is not available, we limit the sync to a maximum of 1 million events to prevent excessive processing.
A historical re-sync only processed messages currently available in the queue. Previously acknowledged (and therefore deleted) messages cannot be re-synced.