Main Concepts
Varadhi is a reliable and durable MessageBus which provides a RESTful API for application integration. The entities which provides the messaging capabilities are Messages, Topics, Subscriptions and Queues.
Data is transferred from publisher to consumer in the form of a Message via message bus, i.e. messages are produced from the publisher applications and consumed by the consumer applications and message bus works as an intermediary. Thus a Message is a basic entity of any messaging system and it can contain anything which relates to publisher/consumer applications e.g. an event of interest, db change, async api call etc.
As Varadhi is a RESTBus on top of a message broker with a functionality of application integration, a Message in Varadhi generally translates to an API payload and its associated metadata. In Varadhi, message is a two part entity
- Payload: This is data or payload carried by the message in the form of raw bytes. Varadhi doesn’t understand and attach any semantics to payload, however this can change in future.
- Metadata: This is in the form of request headers. Request headers can be used to define how Varadhi should handle the message. For details on how Varadhi handles message headers, refer <>
For more details on message, refer
- Message Headers in Varadhi
- How to produce a message to a Varadh
Topic is a named entity (or resource) in Varadhi to which messages are produced or it can also be considered as a stream of messages. When a publisher application produces messages to Varadhi, it targets a specific Topic (or Queue). One or more consumer applications also subscribe to a specific topic for consuming the messages, and all of them are eligible to receive exactly the same set of messages.
Topic can be created under a Project and is identified by its full name
{project_name}/{topic_name}
Since the Project is globally unique, the fully qualified topic name is also globally unique for a given Varadhi deployment.
Topic has below main properties.
- Name: This is the name of the Topic and also acts as a Topic’s identifier. It is immutable. Topic’s full name “{project_name}/{topic_name}” is also globally unique.
- Grouped: If set, indicates this topic supports message ordering. Consumers will receive the messages published to the topic, in the order of produce. For more details on message ordering refer <>
- Capacity Policy: This includes resource allocation details for the Topic and defines how much storage and n/w capacity (produce throughput) is available for this topic. System can enforce guard rails, to throttle message produce to a topic when it crosses allocated capacity.
Topics enable below somewhat similar integration pattern over messaging
- Pub/Sub: Publish/Subscribe pattern allows services to communicate asynchronously which decouples message publisher services (Producers) from message consumer services (Subscribers).
- BroadCast: Topic allows more than one application to subscribe for the messages produced to a given topic. Thus Topic enables one to many communication pattern or a broadcast pattern.
- Choreography: Publisher’s responsibility is to produce a message to a given topic in a fire and forget manner. Once a message is produced, individual consumers receive and process messages independent of others and publishers. This is similar to the Choreography pattern with nothing centrally controlled.
For more details on Topic, refer
- How to create & manage a Topic
- How to produce a message to a Topic.
- How to consume a message from the Topic.
- Message Ordering
Subscription is a named resource in Varadhi which defines how messages from a topic are delivered to the consumer application. A consumer application needs a Subscription, which subscribes to the given topic and consumes messages published to it by delivering them to the endpoint specified in the Subscription entity. A given topic can have one or more subscriptions and each one of them is independent of the other.
In Varadhi Subscriptions are push based i.e. subscriptions are responsible for delivering the messages to the configured endpoint and track successful or failed delivery.
When a subscription delivers a message to a consumer application, message delivery can either succeed or fail. Delivery failure could be because of multiple reasons e.g. transient errors, consumer application degradation, authn/authz failures etc. These failures can be classified as below -
- Retriable failure (or, Soft failures): Depending on Subscription’s configured RetryPolicy, delivery can be auto re-attempted for some kind of failures. Applications can use this to recover from transient failures which they expect to message consumption to succeed on retry. Varadhi supports up to 3 retries for such failure. These redelivery are attempted by moving the messages to Subscription’s internal Retry Queues. For more details, refer <>
- Non retriable failure (or, Hard failures): If an application can not consume messages without taking some explicit actions, then these failures are considered as non retriable failure or hard failures. In such cases, Varadhi sends these messages to Subscription’s Dead Letter Queues from where application can ask to redeliver such messages once endpoint has been deemed fixed.
Note, for ordered consumption Varadhi ensures order is maintained even across retriable and non-retraible failures as well.
At high level, a Subscription’s message delivery can be seen as -
A subscription can be created under a Project and is identified by its full name.
{project_name}/{subscription_name}
Subscription has below main properties
- Name: This is the name of the Subscription and also acts as a Subscription’s identifier. It is immutable. Subscription’s full name “{project_name}/{subscription_name}” is also globally unique.
- Topic: This is the topic’s full name to which this subscription is subscribing. This is immutable.
- Endpoint: This specifies how a message should be delivered to the consumer application. This includes protocol and endpoint related details.
- RetryPolicy: Configures what kind of failures would be treated as retriable and how redelivery should be attempted.
- ConsumptionPolicy: Configurations on messages delivery behavior in terms of expected latencies, parallelism, preference for failure recovery, degradation configurations etc.
- Grouped: If subscribed Topic supports message ordering, a Subscription can choose to have either an ordered or unordered delivery of the messages. If set, ordered delivery will be performed.
For more details on Subscription, refer
- How to create and manage a Subscription
- Handling retriable and non retriable consumption failures.
- How to manage dead letter queue
- Lifecycle of a message in Varadhi
Queue is a named resource in Varadhi that encompasses only one topic-subscription pair and messages carry the endpoint information where it needs to be delivered. It internally has a topic to which publishers produce messages and an auto created subscription which delivers messages to subscribers. As subscriptions are auto created, users can’t create Subscriptions on queue.
Unlike subscription, queue doesn’t have a fixed subscribing application, instead message is associated with endpoint information to which it should be delivered. Hence Queue enables P2P communication pattern and a single message can only be delivered to a single subscribing application.
Callback: Queues optionally support call back feature. If configured, it enables the publisher to receive a response from the subscribing application when a message is delivered. This allows request/response kind of communication between publisher and subscriber. A response message will also be delivered asynchronously and has the same guarantees as a request message.
In all other aspects the queue is similar to Topic-subscription pair and would have similar properties & behavior w.r.to quota, throttling, message ordering, failure handling etc.
Queue can be created under a Project and is identified by its full name
{project_name}/{queue_name}
Since the Project is globally unique, the fully qualified queue name is also globally unique for a given Varadhi deployment.
Queue enable below integration pattern over messaging
- Point-to-Point (P2P): Each message published to the queue carries the endpoint information to which it should be delivered i.e. publisher decides the subscribing application for each message and Varadhi adheres to this contract.
- Request/Response: Optional callback feature of queue allows a publisher to receive a response on message delivery from the subscribing application. When enabled, this enables asynchronous request/response integration pattern between publisher and subscriber applications.
- Orchestration: Varadhi itself doesn’t provide orchestration flow, but it can enable an application to behave as a central orchestrator by combining both P2P and Request/Response patterns to orchestrate a flow. To execute a flow, it can send messages to individual participants in the flow and decide the next action/step based on the response received.
Use Varadhi Queues if the use case aligns with any one of the above patterns, otherwise prefer messaging via Topic and subscription.
For more details on Queue, refer
- Understanding the Queue callbacks.
- How to create and manage a Queue.
- How to configure callbacks on Queue.
Varadhi supports server side message filtering. Publishers produce a lot of messages to the topic but the subscriber might be interested in selected messages. Subscriber can configure a subscription with a filter to receive messages it is interested in. Subscription filters are evaluated on message headers associated with the message. A topic can have multiple subscriptions and each subscription can have its own filter.
A filter is a condition, which needs to evaluate to True for a message to be delivered to the subscription. Filter is evaluated only on the first delivery attempt for the message. Any subsequent delivery attempt (either due to Retry or Dead letter) will not evaluate the filter condition again for the same message.
Messages which don't match filter condition are considered delivered for all internal bookkeeping purposes.
Filters are only supported for Topic/Subscriptions and Queues don't have message filtering capabilities.
For more details on Filter, refer
- Filter condition and Configuring a Subscription filter.
- How does the Subscription filter work ?