How to: Set up pub/sub namespace consumer groups

You’ve set up Dapr’s pub/sub API building block, and your applications are publishing and subscribing to topics smoothly, using a centralized message broker. What if you’d like to perform simple A/B testing, blue/green deployments, or even canary deployments for your applications? Even with using Dapr, this can prove difficult.

Dapr solves multi-tenancy at-scale with its pub/sub namespace consumer groups construct.

Without namespace consumer groups

Let’s say you have a Kubernetes cluster, with two applications (App1 and App2) deployed to the same namespace (namespace-a). App1 publishes to a topic called order, while App2 subscribes to the topic called order. This will create two consumer groups, named after your applications (App1 and App2).

In order to perform simple testing and deployments while using a centralized message broker, you create another namespace with two applications of the same app-id, App1 and App2.

Dapr creates consumer groups using the app-id of individual applications, so the consumer group names will remain App1 and App2.

Both namespaces now run an App2, and Dapr names a consumer group after the app-id alone. The two subscribers therefore join the same consumer group, and a message broker delivers each message to only one member of a group. The two compete: a message published in namespace-a can be handed to the App2 in namespace-b, and a message published in namespace-b can be handed to the App2 in namespace-a. Neither namespace can rely on receiving its own messages, which is what the crossed-out delivery in the diagram above stands for.

To avoid this, you’d then need to have something “creep” into your code to change the app-id, depending on the namespace on which you’re running. This workaround is cumbersome and a significant painpoint.

With namespace consumer groups

Not only can Dapr allow you to change the behavior of a consumer group with a consumerID for your UUID and pod names, Dapr also provides a namespace construct that lives in the pub/sub component metadata. For example, using Redis as your message broker:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: pubsub
spec:
  type: pubsub.redis
  version: v1
  metadata:
  - name: redisHost
    value: localhost:6379
  - name: redisPassword
    value: ""
  - name: consumerID
    value: "{namespace}"

By configuring consumerID with the {namespace} value, you’ll be able to use the same app-id with the same topics from different namespaces.

In the diagram above, you have two namespaces, each with applications of the same app-id, publishing and subscribing to the same logical topic order. Dapr namespaces the underlying broker entities so the two namespaces do not interfere with each other.

Without you needing to change your code/app-id, the namespace consumer group allows you to:

• Add more namespaces
• Use the same topic names in your application code across namespaces
• Keep the same app-id across namespaces
• Have your entire deployment pipeline remain intact

Simply include the "{namespace}" consumer group construct in your component metadata. You don’t need to encode the namespace in the metadata. Dapr understands the namespace it is running in and completes the namespace value for you, like a dynamic metadata value injected by the runtime.

Demo

Watch this video for an overview on pub/sub multi-tenancy:

Next steps

• Learn more about configuring Pub/Sub components with multiple namespaces pub/sub namespaces.