This project aims to demonstrate how it is possible to handle each HTTP request asynchronously (e.g., using events on Kafka) on the server side while maintaining a synchronous HTTP interface.
This approach guarantees at-least-once event delivery and does not require breaking changes to the HTTP API (i.e. REST, GraphQL, etc)
In event-driven architectures (EDA) it is essential to ensure that events are published at least once, eventually.
The dual-write problem can undermine this guarantee. Example:
repository.Add(user) // state mutation
// error here due to ungraceful shutdown
eventBroker.Publish(UserCreatedEvent{User: user}) // or error here due to network issues or service unavailable
In this case, the application's state was changed, but the outside world was not notified of the event.
The application of retry logic is essential in order to ensure that the event is eventually published at least once.
However, it is not always straightforward to apply retry logic especially if we are in the context of a HTTP request and do not want to block the user indefinitely or cannot force the user to re-post the request.
There are several patterns that help us implement retry. One is the outbox pattern, which we have already seen in another project. Another one is making every HTTP request an async event.
In this project we have a demonstration of how we can apply the retry pattern by making sure that we handle any state mutation within an event consuming context instead of the HTTP request context.
This approach ease the application of the retry pattern because we can simply retry the event consumption easily. The only caveat is that the consuming of the event is idempotent.
So any HTTP request for state mutation, is immediately transformed into an event. This event is processed and the result is obtained asynchronously. Usually this architecture also involves the front-end client by implementing patterns such as Asynchronous Request-Reply or Websocket for asynchronous reading of the outcome.
In some cases, however, it may not be possible or desirable to change the behavior of front-end clients to be asynchronous. This project aims to demonstrate that this problem can be worked around by injecting asynchronous outcome handling into the HTTP handler itself, instead of the frontend client.
The HTTP handler turns the request into an event (e.g., HTTPRequestReceived
) and publishes it to Kafka, attaching a
unique ID for the request.
A Kafka consumer, which handles the event, performs the requested action and publishes an outcome event (
e.g. UserCreated
or UserDuplicated
), again attaching the unique ID for the request.
Another Kafka consumer, executed in the same process as the initial HTTP handler but in a separate goroutine, handles the outcome event and responds to the initial HTTP handler with the result. It can associate the outcome event with the initial HTTP request by using the unique ID.
At this point the HTTP handler can return the response to the HTTP client.
Make sure ports 8080
, 8082
, 3306
and 9093
are available.
Otherwise, you can change the ports in the docker-compose.override.yaml
file.
Start the services:
docker-compose up -d --build --force-recreate
Make sure that every service is up and running.
Send a request to the HTTP handler:
curl -X POST --location "http://localhost:8080" \
-H "Content-Type: application/json" \
-d "{
\"name\": \"Lorenzo Ranucci\"
}" \
-i
It should return a 201 Created
response code.
Sending again the same request should return a 409 Conflict
response code.
Sending a new request with a different name should return a 201 Created
response code.
Connect to http://localhost:8082 and watch messages in the Kafka topics.
Connect to mysql on localhost:3306 and run the following query:
SELECT *
FROM users;
You should see the users that you have created.
Note that the HTTP handler has 5 seconds timeout. If you send a request and wait more than 5 seconds, you should see
a 504 Gateway Timeout
response code. This could happen if you are debugging the code and you have set a breakpoint or
if you docker environment is slow.