Consumer connections

A consumer connection is one of your organisation's apps reading your workspace's messages. Each connection gets its own queue and dead-letter queue on the bus, subscribed to your workspace's messages with the message types you choose. Nothing another workspace publishes ever reaches it.

Create a connection Available now

In the portal, open Consumer connections and press New connection. You need the developer role or above in the workspace.

Field	What it does
App	One of your organisation's consumer or skill apps. A producer app only publishes, so it isn't offered here: an app that both sends and reads is two apps, a producer and a consumer. With none, the form links to New app with Consumer chosen
Message types	What the queue receives: exact types (story.context), a family (telling.*) or * for everything. Up to 20
Topic prefix	Optional. Only messages whose envelope topic starts with it, for example som.story.. Every topic starts with som., so a prefix that doesn't, such as story.context, is refused: it would match nothing. Leave it empty for every topic
Sources	Optional. Only messages from the producers you tick (shown as app and organisation), or with the originating_system.system_ids you list. See Subscriptions and filters
Attempts before the dead-letter queue	Optional, 1 to 1000, default 5. How often a message may be received without being deleted before it moves to the dead-letter queue

The bus makes the queue, usually within a minute (at most about 10 minutes). Its status then changes from Setting up to Ready, and the page shows how many messages are waiting, in flight and in the dead-letter queue.

A workspace has at most 50 consumer connections.

Read over HTTPS Available now

No AWS account needed. Press Get client secret on the connection: you get its client id and secret (shown once), and your app receives, acknowledges and releases messages with the HTTPS pull API. Rotate or revoke the secret from the same page.

Read with AWS credentials Available now

A queue can also be read with AWS credentials, as an IAM role in your own AWS account. You register the role on the connection yourself, and it proves it is yours before the queue lets it in. You need the developer role or above in the workspace.

1. In the portal, open Consumer connections and press Add reader role on the connection. The connection must be active.
2. Enter the role's ARN, as the IAM console shows it, and press Register role. Only an IAM role is accepted; an assumed-role session ARN counts as its role. IAM users and account roots are not. The role shows as Awaiting proof, and the queue grants it nothing yet.
3. The portal shows a one-time challenge and a curl command, once. Within 30 minutes, run the command with the role's credentials (it signs the request with SigV4; curl 7.75 or later). The answer is "verified": true. If the challenge expired or got lost, press New challenge.
4. The bus then adds the role to the policy of the connection's queue and its dead-letter queue, usually within a minute (at most about 10 minutes). The role shows as Verified.
5. On your side, allow the role the same actions in its own IAM policy, on the two queue ARNs the portal shows (Copy ARN): sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility and sqs:GetQueueAttributes.

Those four actions, on that connection's queue and dead-letter queue, for that one role, are all the bus grants: the role can't send, purge or change the queues, or read any other connection's. It isn't granted sqs:GetQueueUrl: use the queue URL from the portal (Copy URL).

• One reader role per connection. A role may read several connections: register it on each.
• Remove role takes it out of both queue policies, usually within a minute (at most about 10 minutes). Remove it in the portal before you delete the role in AWS.
• Revoking the connection keeps the reader, as it keeps the queue, so the role can read what's left. Removing the connection deletes the reader with the queues.
• RND can also set readers on a queue for you (through Ask RND in the portal). The page shows how many RND has set; they are separate from your own reader role.

Then use the queue URL from the portal with any AWS SDK or the AWS CLI:

loop:
  ReceiveMessage(QueueUrl, MaxNumberOfMessages=10, WaitTimeSeconds=20, MessageAttributeNames=[All])
  for each message, in the order received:
      handle(message)          # idempotent on message_id
      DeleteMessage(receipt)   # only after handle() succeeded
  on error in handle(): don't delete; the message comes back after 60 seconds

aws sqs receive-message --region eu-west-1 --queue-url "$QUEUE_URL" \
  --max-number-of-messages 10 --wait-time-seconds 20 \
  --message-attribute-names All --attribute-names All
# process, then:
aws sqs delete-message --region eu-west-1 --queue-url "$QUEUE_URL" --receipt-handle "$HANDLE"

• The body is the producer's envelope, byte for byte. Only messages the gateway accepted arrive.

• Message attributes:

  Attribute	Value
  message_type	The envelope's message_type
  family	The payload's schema family (story-context, …), or unknown for a type with no SOM 1.0 schema
  topic	The envelope's topic
  correlation_id	The envelope's correlation_id
  producer_id	The connection that published it
  system_id	The envelope's originating_system.system_id
  som_version	The envelope's som_version

• In order per correlation_id. The FIFO message group is the workspace id and the correlation_id joined by #: read the correlation_id attribute rather than parsing the group.

• Visibility timeout: 60 seconds. A message you don't delete comes back after it. If handling takes longer, extend it with ChangeMessageVisibility before it runs out.

• A failing message holds up its story until it's deleted, or reaches the attempt limit and moves to the dead-letter queue. Other stories keep flowing.

• Messages wait 14 days, in the queue and in the dead-letter queue.

What a consumer must do with what it receives: Order, duplicates and snapshots.

Edit, revoke, restore, remove Available now

Action	What happens
Edit filters	Change the message types, topic prefix and sources in place. The queue, its messages, its dead-letter queue and its client secret stay. See Subscriptions and filters
Revoke	New messages stop reaching the queue. The queue and the messages already in it stay. The connection's client secret gets no tokens and the pull API refuses it until you restore the connection; read what's left with AWS credentials, or restore first
Restore	A revoked connection receives new messages again, in the same queue. Messages published while it was revoked are not delivered
Remove	The connection, its queue, its dead-letter queue and its client secret are deleted, with any messages in them. This can't be undone

Every create, filter change, revoke, restore, remove, reader change and client secret change is recorded in your workspace's audit log, with who did it and when. For a reader role that includes each new challenge and the role's own proof.

Monitoring

RND watches every active connection's queue: a message in the dead-letter queue, or a message that has waited more than 5 minutes, raises an alarm once the connection has a reader (a client secret, a verified reader role, or a reader RND set). RND may contact you if your consumer stops reading. The portal shows the queue depths at any time.

Your workspace gets an email too, once the connection has a reader: when its dead-letter queue grows, or when its oldest message has waited longer than your lag threshold (15 minutes unless you change it). See Notifications.

Known limits

• One reader role per connection, and only an IAM role. For an IAM user, or more than one role on a queue, ask RND to set it.
• Changing a connection's app isn't possible: create a connection for the other app. Its filters (message types, topic prefix, sources) you edit in place.

Next: HTTPS pull API.