Meaning of Queues and Topics in AWS
What's the difference between queues and topics in AWS? It's not as obvious as it seems to be...
Queues and topics are standard communication channels in messaging systems. How to use then in AWS, for example to implement the Competing Consumers pattern?
Let's consider a following scenario:
A REST endpoint (Amazon API Gateway) initiates a time expensive processing served by a serverless function (AWS Lambda). Not only because of the max timeout 30 seconds on the gateway it is not a good idea to process the request synchronously (see The Reactive Manifesto).
Let's build a queue of working tasks into the middle between the gateway and the worker function:
Request --> Queue --> Worker
Immediately after a request comes it is put into the queue and taken by a worker function to the processing.
To implement this in AWS the tasks queue is not a queue (SQS) at all, but a topic (SNS). Let's explain why.
In a standard non-serverless implementation will be workers listening on the queue and distributing the tasks among each other. It means the queue is a pull mechanism.
Serverless service creates as many worker instances as needed. So there is no need to keep tasks in a queue while they are processed immediately as put into the queue. Initialization of the worker function must be event-driven (a lambda function must never idle!) - implementing a push mechanism.
There is no change to register a lambda function to a queue, for such a usage there is another concept - topics:
Request --> Topic --> Worker
As the request comes a task is put into the topic and processed by the lambda function, elastically created on demand.
So, that's the meaning of queues and topics in AWS.
Happy clouding!