What’s event driven architecture ?
Event driven architectures enable the possibility of triggering and communicating between services, in response to an event, the event can be a changed state, an update, a new object in a database etc..
This replaces the traditional request/response pattern, where components actually direct commands to other components, for example when a customer pays his order online, the order component will ask the invoice component to create the invoice.
In an event driven model, the events are observable by other components, and these components will act according to the event.
For example when a customer orders a widget, the order component will create an event saying that the customer has ordered a widget, the invoice component will then create the invoice, and the reports component will create the sale report. As you can see nothing is directed between these components.
An event driven architecture has usually three key components : event producers, event routers, and event consumers. These components are independent and decoupled will ease scaling, deployments and updating.
In our case we will try to explain the services to use to create a serverless event driven architecture based on serverless services : SNS, SQS and AWS Lambda.
An Overview of AWS Messaging Services :
Messaging Services are key components in an event driven architecture, they are responsible for managing the observable events gathered from the producer and eventually routing it to the right consumer.
1 – Simple Notification Service (SNS) :
SNS is a pub/sub service or publish subscribe service messaging service, where the producer sends a message to the topic.
A topic has one or more subscribers, so when a message is sent to the topic, multiple subscribers can read that message. This is called a “fan-out” capability to send messages in parallel to multiple subscribers.
Subscription protocols include : HTTP(S), email, SQS, SMS, mobile push and AWS Lambda
| Resource | Default |
| Topics | Standard: 100,000 per accountFIFO: 1,000 per account |
| Subscriptions | Standard: 12,500,000 per topic For Kinesis Data Firehose delivery streams, 5 per topic, per subscription ownerFIFO: 100 per topic |
| Pending subscriptions | 5000 per account |
| Subscription filter policies | 200 per account |
| Maximum payload size | 256 KB |
2 – Simple Queue Service (SQS) :
SQS is a durable and scalable message queue, producers can push messages to the queue at virtually any rate.
The consumers poll the message from the SQS queue by using a poll component.
One more consumer can poll the queue to receive the messages, in the example below the consumer is a lambda function that polls the messages from the SQS queue.
SQS Queues guarantees at least on delivery for the message, based on visibility timeout, the message that has been delivered will be processed by the consumer, if the message is successfully processed it will be deleted from the queue, if not, when the visibility timeout is expired thee message will be visible in the queue and can be processed again.
| Resource | Default |
| Messages per queue (backlog) | The number of messages that an Amazon SQS queue can store is unlimited. |
| Message retention | By default, a message is retained for 4 days. The minimum is 60 seconds (1 minute). The maximum is 1,209,600 seconds (14 days). |
| Message Size | The minimum message size is 1 byte (1 character). The maximum is 262,144 bytes (256 KB). |
| Message visibility timeout | The default visibility timeout for a message is 30 seconds. The minimum is 0 seconds. The maximum is 12 hours. |
An Overview of AWS Lambda Invocation Models :
As defined by AWS, AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes.
In a Serverless event driven architecture, and in our case, the lambdas function will be the consumers of our messaging services.
Depending on which service is the producer (SQS or SNS) Lambda will be invoked based on a specific mechanism.
As you can see in the figure above, in the asynchronous and synchronous model lambda will be triggered directly by a service, for example SNS will trigger the lambda due on an event message.
Interaction between Lambda and messaging services (SQS, SNS)
1 – Amazon SNS to Lambda :
When SNS receives a message it notifies all the lambdas (in this case two lambdas) about the message received, the two lambdas A and B are subscribed to the SNS Topic.
Function C is also subscribed to the topic,but it’s filter policy doesn’t match the message payload, so it didn’t get the message.
FinOps Point : Subscription Filter Policies are very helpful when optimising lambda functions executions time and cost, in the above case, if function C accepted all the messages, we will need to invoke function C many times and applies filters within the function, which will result in useless computation, Filter policies on the other hand filter messages before sending it to lambdas functions.
2 – Amazon SQS to Lambda :
When the SQS queue is set as an event trigger for lambda, the poller within the lambda component will take the message from the queue.
In contrast to SNS, the lambda using it’s poller will go to get the message from the SQS Queue, with SNS the poller will use the internal lambda event queue to get the message.
SQS also supports batching, so you can receive multiple messages to process in parallel in the lambda functions.
3 – So what’s the difference between SQS and SNS ? :
| SQS | SNS | |
| Service | Queue Service | Notification/Messaging Service |
| Messages Persistance | From 1 minute to 14 Days | No message retention |
| Message Delivery | Guaranteed | Not guaranteed |
| Message consumption | Pull Mechanism | Push Mechanism |
4 – How to choose between SQS or SNS ?
There is no specific pattern to this, however here is some recommendations :
Choose SNS if:
- You would like to be able to publish and consume batches of messages.
- You would like to allow the same message to be processed in multiple ways.
- Multiple subscribers are needed.
Choose SQS if:
- You need a simple queue with no particular additional requirements.
- Decoupling two applications and allowing parallel asynchronous processing.
- Only one subscriber is needed.
Putting the pieces together : a ticketing system based on serverless event driven architecture :
As shown in the figure above, in this example we created a simple ordering system based on serverless event driven architecture.
The main constraint that we had to keep in mind when creating this architecture, is that this system is dedicated to a ticketing system, so order is important, that’s why we use SQS FIFO Queues for the tickets inventory and payments, in order to keep an up-to-date ticketing inventory and to respect the queue of our customers in the front end. Just like in a physical ticketing place, it’s a first come first served system.
The advantages of this type of architecture is that it can scale as the number of events increases, as the number of users in the front end increases and may reach a certain peak, the infrastructure will scale accordingly, which will give the customer an enhanced experience.
Conclusion :
At Pisquare we help our customers in the retail industry, to have an enhanced customer experience, by creating event driven architectures for different components such as ordering and stock management.
In most of the cases, the common problem is that events are not processed fastly, issues and gaps between availability of products on the website and in stock and slow user experience on the front due to a slow backend processing.
This is mainly due to services direceting commands to other services through API’s, which makes error handling harder, losing the order of actions when dealing with commands, a spike in errors in peak periods such as Black Friday or Christmas holidays. This results in a loss of customers and profits.
