Pub/sub messaging explained: subscriptions are the whole idea

Publish/subscribe delivers one message to every interested consumer rather than to one of them. A publisher sends to a topic; every subscription on that topic gets a copy; the publisher never knows how many there are.

The distinguishing concept is the subscription, and it is the part most explanations skip. A subscription determines whether a consumer that was offline receives what it missed, whether two instances of the same service each get a copy or share the work, and how much the fan-out actually costs you.

What we'll cover

• Fan-out, and how it differs from a queue
• What a subscription is
• Durable and non-durable
• Consumer groups and shared subscriptions
• Topics, filtering, and where to draw them
• What fan-out amplifies
• Adding a consumer later
• Testing something nobody owns end to end
• Using pub/sub with CodeWords
• What to get right first
• Frequently asked questions

Fan-out, and how it differs from a queue

A queue delivers each message to one consumer. Several consumers share the work, each item handled once. This is work distribution.

Pub/sub delivers each message to every subscription. Five subscriptions means five deliveries of the same message. This is information distribution.

The publisher is decoupled from the count. Adding a sixth consumer requires no change to the publisher, which is the main benefit.

Many systems do both. A topic with several subscriptions, each consumed by a group of workers sharing that subscription's messages, is fan-out across subscriptions and work distribution within each.

Choose by intent. "This task should be done once" is a queue. "Several things need to know" is a topic.

What a subscription is

The central concept, and being precise about it resolves most confusion.

A subscription is a named interest in a topic, held by the broker, with its own position in the stream.

It has its own state. Each subscription tracks what it has acknowledged independently, so a slow consumer does not hold up a fast one.

It exists independently of any connected consumer when durable, which is the property that matters most and is covered next.

It can have its own filter, receiving a subset of what the topic carries.

Creating one is a deliberate act, and forgetting to create it before publishing starts is the commonest reason a consumer "missed" messages that were never routed to it.

Durable and non-durable

The distinction that determines whether a consumer can be offline.

A non-durable subscription exists while a consumer is connected. Disconnect, and messages published meanwhile are gone for that consumer.

A durable subscription persists. The broker retains messages for it while nobody is connected, and the consumer receives them on reconnection.

Non-durable is right for genuinely ephemeral things — a live dashboard, a presence indicator — where old information has no value.

Durable is right for anything that matters. A consumer that must not miss events needs the broker holding them during a deployment, a restart, or an outage.

Durable subscriptions accumulate. A subscription nobody consumes grows until its retention or quota is reached, and an abandoned one is a slow resource leak that eventually affects the topic.

Clean up subscriptions you stop using, deliberately, because nothing else will.

Consumer groups and shared subscriptions

The arrangement that combines both models, and the source of the most common design error.

Several instances of one service should share a subscription, so each message is handled once by that service.

Several instances each with their own subscription means every instance processes every message — which is almost never what anybody wants and is easy to configure accidentally.

The symptom is duplicated side effects scaling with your instance count: three replicas, three emails.

Scale within a subscription, not by adding subscriptions.

Ordering is usually per partition or per key within a shared subscription, so parallel consumers break global ordering by design.

Check this first when a service does something several times. It is a configuration mistake rather than a delivery problem, and it is very common.

Topics, filtering, and where to draw them

A topic too broad means every consumer receives and discards most of what arrives, which wastes throughput and makes reasoning harder.

A topic too narrow produces hundreds of topics and a publisher that must know which one applies — reintroducing the coupling pub/sub was meant to remove.

Filter at the subscription where the broker supports it, so the broker discards rather than your consumer.

Draw topics around the kind of thing that happened, not around who wants it. A topic named after a consumer is a queue with extra steps.

Attributes on the message support filtering without multiplying topics, which is usually the right balance.

Expect to get this wrong initially, and changing it later means coordinating publishers and consumers, so it is worth some thought up front.

What fan-out amplifies

The cost side, which is straightforward arithmetic that surprises people anyway.

Delivery volume multiplies by subscription count. One publish to five subscriptions is five deliveries, billed and processed as five.

A badly behaved consumer affects the topic's resources, not only itself, when its subscription backs up.

Retries multiply too. A consumer failing and retrying adds load proportional to its own failure rate, on top of the fan-out.

Payload size matters more than in a queue, since every byte is sent as many times as there are subscriptions.

Keep messages small and let consumers fetch detail if they need it — bearing in mind that a fetch from every consumer is its own amplification against the source.

Model the cost at the subscription count you expect to reach, not the one you have.

Adding a consumer later

The benefit that justifies the architecture, and the constraint on it.

A new subscription requires no publisher change, which is the point.

What it can see depends on retention. A new subscription typically starts from the moment it was created, so it does not receive history unless the broker supports starting from an earlier position and the retention covers it.

Log-based systems allow replay from a position, which is what makes "add a consumer and process the last month" possible — and it is a property of the transport rather than of pub/sub generally.

Decide retention with that in mind, because it determines what a future consumer can be built from.

Create the subscription before you need it if you know a consumer is coming, so it accumulates messages while the consumer is being built.

Testing something nobody owns end to end

A topic with several independent consumers has no single owner, which changes how it should be verified.

The publisher cannot test the consumers. It does not know they exist, which is the property you wanted and the reason integration testing is awkward.

Contract tests on the message shape are the practical answer: the publisher asserts what it emits, each consumer asserts what it requires, and a breaking change fails somebody's suite rather than production.

Keep a register of subscriptions per topic even though the architecture does not require one, because "who will this change break" otherwise has no answer.

Test the consumer against a recorded message, including older versions, since replay and slow consumers both mean old shapes arrive.

Test the missing-field case explicitly, which is what an additive change looks like from a consumer that has not been updated.

Nobody tests the fan-out itself, and that is usually fine — the per-subscription behaviour is what matters and the broker's delivery is not your code.

Using pub/sub with CodeWords

Many consumers of an event are integrations rather than services. CodeWords connects to more than 3,000 integrations and builds the automation from a plain-language description.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which event, and what should result.
3. Describe the exceptions: a duplicate, one that arrives out of order, one that cannot be processed.
4. Connect the systems involved once and reuse the connections.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

What to get right first

Durable subscriptions for anything that matters, created before publishing begins. A non-durable subscription silently discards everything published while a consumer is restarting, and "we lost events during a deployment" is the most common and most preventable pub/sub failure.

Two habits make the difference. Share one subscription across the instances of a service rather than giving each its own, since the second configuration sends every message to every replica and produces duplicated side effects that scale with your instance count. And delete subscriptions you stop using, because an abandoned durable subscription accumulates indefinitely and eventually affects the topic rather than only itself.

Frequently asked questions

What is the difference between pub/sub and a queue?

A queue delivers each message to one consumer, distributing work. Pub/sub delivers each message to every subscription, distributing information. Many systems do both: fan-out across subscriptions, work sharing within each.

Why did my consumer miss messages while it was restarting?

Almost certainly a non-durable subscription, which only exists while a consumer is connected. Anything that must not miss events needs a durable subscription, where the broker retains messages during the gap.

Why is my service doing everything three times?

Each replica probably has its own subscription rather than sharing one. Scale within a subscription; separate subscriptions per instance means every instance processes every message, and the duplication scales with your replica count.

How many topics should I have?

Enough that consumers are not discarding most of what they receive, few enough that publishers do not need to know who wants what. Draw them around what happened, and use message attributes with subscription filters rather than multiplying topics.

What does fan-out cost?

Delivery volume times subscription count, and payload size matters more than in a queue because every byte is sent once per subscription. Model it at the subscription count you expect rather than the one you have.

Can a new consumer process past messages?

Only if the transport supports starting from an earlier position and retention covers it — that is a log-based property rather than a general one. Otherwise a new subscription starts from when it was created.

What happens to a subscription nobody consumes?

It accumulates until retention or quota is reached, and it consumes resources on the topic rather than only its own. Abandoned durable subscriptions are a slow leak that nothing cleans up automatically.

How do I test a topic with independent consumers?

Contract tests: the publisher asserts what it emits and each consumer asserts what it requires, so a breaking change fails a test suite rather than production. The publisher cannot test the consumers, which is the decoupling working as intended.

Should I keep a list of who subscribes to what?

Yes, even though the architecture does not require one. Without it, "who will this change break" has no answer, and the decoupling that makes adding consumers easy makes changing messages risky.

Related reading

• Message queues explained
• Event-driven architecture
• Dead letter queues