This is Architecture Series, Part 2. Last time we built an idempotency key to stop a double-charge. Today is the payoff: we’re going to see why duplicates are unavoidable in the first place, why a feature called “exactly-once delivery” gets advertised on slides but can’t actually exist, and why none of that should scare you once you understand one simple idea.

If you’ve ever used a job queue — Celery, SQS, RabbitMQ, Kafka, anything—this is the bug waiting for you. Your worker will run the same job twice.

Let’s understand why, and why that’s fine.

The setup: what a queue actually does

Queue Work and Acknowledgement Flow

A message queue is just a to-do list that lives between your services. One part of your system drops a job on the list (“send this email,” “charge this card,” “resize this image”). A worker picks the job up, does it, and tells the queue “done.” That last step has a name—the worker acknowledges the message, and it’s where the whole story happens.

The queue needs that acknowledgement because workers die. Servers crash, deployments happen, and networks blip. So the queue follows a rule that sounds obviously correct:

Don’t remove a job from the list until a worker confirms it’s finished. If nobody confirms, give the job to another worker.

This rule is the reason your queue is reliable. It’s also the reason it processes things twice. Watch closely.

The exact moment a job runs twice

Crash After Work Before ACK

Here’s a worker doing its job in the sensible order — do the work first, then acknowledge:

1. Worker grabs the job: “send the welcome email.”
2. Worker sends the email. ✅ The email is now in the customer’s inbox.
3. Worker turns to tell the queue “done!”… and crashes right here. Power cut, OOM kill, deploy, whatever.

The email was sent. But the worker died before it could acknowledge. So from the queue’s point of view, this job was never confirmed finished. The rule kicks in: give it to another worker. A second worker picks up the same job and sends the welcome email again. Your customer gets two emails.

Now you might think: “Fine, I’ll acknowledge first, then do the work.” That just breaks it the other way:

1. Worker grabs the job.
2. Worker acknowledges immediately — queue deletes the job.
3. Worker crashes before sending the email.

Now the job is gone forever and the email was never sent. You traded “sometimes twice” for “sometimes never.”

That’s the entire dilemma, and it has two names.

The three delivery semantics, in plain English

Delivery Semantics Comparison

• At-most-once: acknowledge before doing the work. Every job runs zero or one times. No duplicates — but if a worker crashes mid-job, that job is silently lost. Lowest overhead, used for things you can afford to drop (a single missed metrics ping, a non-critical notification).
• At-least-once: do the work, then acknowledge. Every job runs one or more times. Nothing is ever lost — but crashes cause duplicates, exactly like the double-email above. This is the workhorse. Kafka, RabbitMQ, and SQS all default to at-least-once.
• Exactly-once: every job runs precisely one time. No drops, no duplicates. This is what everyone wants… and at the level of delivering a message across a network, it doesn’t exist. Here’s why.

Why exactly-once delivery is a myth

Lost ACK Ambiguity

The reason is older than any of these tools — it’s a classic problem called the Two Generals Problem, and once you see it you can’t unsee it.

Imagine two people coordinating only by sending messengers across hostile territory, where any messenger might be captured. General A sends “attack at dawn.” Does B receive it? A can’t know unless B sends a confirmation back. But does that confirmation arrive? B can’t know unless A confirms the confirmation. And so on, forever. There is no number of messages that lets both sides be certain the other got the last one. The uncertainty never fully closes.

Your worker and your queue are those two generals, and the network is the hostile territory. When a worker finishes a job and the acknowledgement gets lost on the way back, the queue cannot tell the difference between “the worker finished but the ack vanished” and “the worker died without finishing.” Faced with that ambiguity, a reliable system has exactly one safe choice: assume it might not have finished, and redeliver.

So the duplicate isn’t a bug. It’s the correct behavior of any system that refuses to lose your data. The industry made a deliberate bargain:

Dropping a message is unforgivable. Delivering it twice is recoverable. So almost every reliable queue chooses at-least-once and hands the duplicate problem to you.

When a vendor advertises “exactly-once,” they’re not lying exactly — they’ve usually built at-least-once delivery plus automatic deduplication, which produces the effect of once. Which brings us to the actual solution, and it’s something you already built last time.

The real fix: at-least-once + an idempotent consumer = “effectively once”

At-Least-Once Plus Idempotent Consumer

You can’t stop the queue from delivering twice. So stop trying. Instead, make your worker idempotent — able to process the same job twice with the same result as processing it once. (That word, and that pattern, are exactly what we built in Article 1 - Idempotency . This is where it pays off.)

The recipe is the same shape as the idempotency key: give every job a stable unique ID, and have the worker remember which IDs it has already completed. A duplicate gets recognized and skipped.

def process_job(job):
    # job.id is stable: a retry of the SAME job carries the SAME id
    if AlreadyProcessed.objects.filter(job_id=job.id).exists():
        return  # we've done this one — skip silently

    do_the_actual_work(job)
    AlreadyProcessed.objects.create(job_id=job.id)

But — and this is the part the naive version gets wrong, just like last time — that exists() check has a race condition if two copies of the job run at the same instant. Both check, both see "not done," both proceed. The robust version uses the database as the referee, with a unique constraint on the job ID:

from django.db import IntegrityError, transaction

def process_job(job):
    try:
        with transaction.atomic():
            # Claim the job atomically. The DB guarantees one winner.
            AlreadyProcessed.objects.create(job_id=job.id)
            do_the_actual_work(job)
    except IntegrityError:
        # The job_id already exists — this is a duplicate delivery. Skip.
        return

This is the pattern real systems run at scale. Netflix tags each event with a unique ID and puts a unique constraint on it in the database — a duplicate delivery just causes a constraint violation that gets silently ignored. Uber does the same with INSERT ... ON CONFLICT and last-write-wins. No exotic "exactly-once" machinery — just at-least-once delivery and a consumer that shrugs off duplicates.

Unique Constraint Referees Duplicate Jobs

The mantra: the broker guarantees the message arrives at least once*. Your handler guarantees processing it twice does no harm. Put those together and the* effect is exactly-once — which is the only kind of exactly-once you can actually have.

The hard case: side effects you can’t take back

Idempotency for Irreversible Side Effects

The pattern above is easy when the work is a database write — you just dedupe on the ID. It gets harder when the job does something irreversible in the outside world: sending an email, charging a card, firing a webhook. You can’t “un-send” an email if you discover it was a duplicate after the fact.

Two practical moves for those:

• Push the idempotency to the thing you’re calling. Payment APIs like Stripe accept an idempotency key (Article 1 !). Pass your job ID as that key, and even if your worker runs twice, Stripe charges once and replays the original response on the second call. You’ve moved the dedup to the system that can actually enforce it.
• Record the side effect in the same transaction that marks the job done. If “send email” and “mark job complete” don’t commit together, a crash between them reopens the duplicate window. Bundle them so they succeed or fail as one unit.

These aren’t always perfectly solvable — sometimes “we sent one extra email once in a blue moon” is a cost you accept. The goal isn’t theoretical perfection; it’s making duplicates harmless for anything that matters, like money.

Further reading: Idempotency Explained Simply

Why this is “Architecture”

“At-least-once delivery” gets dropped into architecture diagrams as if it’s a setting you tick. What the phrase actually means is “this system will hand you duplicates, and dealing with them is your job now” — but it’s almost never said that plainly. So people pick a queue, see the reassuring word “reliable,” and discover months later that their welcome email went out three times because they never made the consumer idempotent.

And “exactly-once” is worse, because it sounds like the obviously-better option, the upgrade you’d pick if you could afford it. The truth — that it can’t exist at the delivery layer, and that the real-world version is just at-least-once wearing a nicer label — is a genuinely useful thing to know, and it’s the kind of thing that separates someone who’s read the marketing from someone who understands the system. Now you’re the second kind.

The 30-second takeaway

• A queue redelivers any job a worker doesn’t acknowledge, because the worker might have crashed before finishing — that’s what makes it reliable.
• At-most-once (ack first) can lose jobs; at-least-once (ack after) can duplicate them; exactly-once delivery can’t truly exist because of the Two Generals Problem (lost acks are ambiguous).
• Kafka, RabbitMQ, and SQS all default to at-least-once — duplicates are expected, not a bug.
• The fix is the same pattern as idempotency keys: give each job a stable ID, dedupe with a database unique constraint (insert-first, not check-first). At-least-once + idempotent consumer = “effectively once.”
• For irreversible side effects (emails, charges), push idempotency into the downstream API (e.g. Stripe keys) or record the effect in the same transaction that marks the job done.

Next in the series: The Retry Storm — how a well-meaning retry loop turns one slow server into a full outage, why naive retries are a denial-of-service attack on yourself, and what exponential backoff and jitter actually do.