Your Notification Was Sent Twice: Designing a Production-Grade Notification System

Introduction

The payment succeeded.

The order was stored correctly.

The customer received the confirmation SMS.

Then they received the same SMS again.

Same order. Same amount. Same notification.

From the customer's point of view, something looks wrong. They may reasonably wonder whether they were charged twice.

The payment system may be completely correct. The duplicate may have been created by the notification pipeline.

A very common failure sequence looks like this:

sequenceDiagram
    autonumber
    actor W1 as Worker 1
    participant Q as Message Queue
    participant P as SMS Provider
    actor W2 as Worker 2
 
    Q->>W1: Deliver Notification Message
    W1->>P: Call SMS API (Send confirmation)
    P-->>W1: Provider accepts SMS (200 OK)
    Note over W1: 💥 Worker crashes before queue ACK!
    Note over Q: Visibility timeout expires
    Q->>W2: Redeliver same Message
    W2->>P: Call SMS API again!
    Note over P: ⚠️ Customer receives duplicate SMS!

This is not an exotic edge case. It is a direct consequence of building a reliable asynchronous system across multiple failure boundaries.

A notification system looks simple when the requirement is written as:

flowchart LR
    E["Event Trigger (e.g. OrderPlaced)"] -->|"Synchronous or Async"| N["Notify User (Push / SMS / Email)"]

In production, that one arrow hides a surprising amount of distributed-systems complexity:

• durable acceptance
• asynchronous processing
• fan-out to multiple channels
• user preferences
• priorities
• retries
• backoff and jitter
• dead-letter handling
• idempotency
• provider rate limits
• circuit breakers
• expiry
• observability
• the database/queue dual-write problem
• and the fact that end-to-end exactly-once delivery is usually not something we can honestly promise

This article designs a notification platform that supports:

• Push notifications
• Email
• SMS
• In-app notifications
• Transactional and marketing traffic
• User preferences
• High burst traffic
• Provider failures
• At-least-once message processing with duplicate protection

The design is intentionally production-oriented rather than interview-oriented. The goal is not to name as many technologies as possible. The goal is to make the failure modes explicit and choose mechanisms that address them.

1. Start with the requirements

Before drawing architecture boxes, define what the system is expected to guarantee.

Functional requirements

The platform should:

1. Accept notification requests from internal services.
2. Support push, email, SMS, and in-app channels.
3. Respect user notification preferences where business rules allow.
4. Support both transactional and marketing notifications.
5. Track the lifecycle of each channel delivery independently.
6. Support notification expiry.
7. Expose enough state for operations and support teams to investigate failures.

Non-functional requirements

Different notification types have different expectations.

A single global SLA is therefore not enough.

A production notification system should make priority and expiry explicit data, not hidden assumptions in code.

2. The naive synchronous design

A first implementation often looks like this:

flowchart LR
    A[Order Service] --> B[Notification API]
    B --> C[Email Provider]
    B --> D[SMS Provider]
    B --> E[Push Provider]
    C --> B
    D --> B
    E --> B
    B --> A

The application receives a request and immediately calls every external provider.

A simplified controller might look like:

[HttpPost("notifications")]
public async Task<IActionResult> Send(
    SendNotificationRequest request,
    CancellationToken cancellationToken)
{
    await emailSender.SendAsync(request, cancellationToken);
    await smsSender.SendAsync(request, cancellationToken);
    await pushSender.SendAsync(request, cancellationToken);
 
    return Ok();
}

This is easy to understand, but it couples request latency and availability to third-party systems.

If email takes two seconds, your endpoint waits two seconds.

If the SMS provider is degraded, your endpoint is degraded.

If one million campaign notifications arrive, your API tries to turn one million incoming requests into immediate outbound provider calls.

The notification API has now become a traffic amplifier.

The first architectural decision is therefore:

Accept the notification durably, then perform delivery asynchronously.

The caller should not wait for the actual email, SMS, or push request to complete.

3. Model the data correctly

One of the easiest mistakes is to put a single Status column on the notification record.

That stops being meaningful as soon as a notification uses multiple channels.

For example:

flowchart LR
    N["Notification #42"] --> P["Push: Delivered ✅"]
    N --> E["Email: Failed ❌"]
    N --> S["SMS: Retrying ⏳"]
    N --> I["In-App: Seen 👁️"]

Is notification 42 Sent, Failed, or Pending?

The better model separates the logical notification from the individual channel deliveries.

Notification

public sealed class Notification
{
    public Guid Id { get; init; }
    public Guid UserId { get; init; }
    public required string Type { get; init; }
    public required string TemplateKey { get; init; }
    public required string PayloadJson { get; init; }
    public NotificationPriority Priority { get; init; }
    public DateTimeOffset CreatedAt { get; init; }
    public DateTimeOffset? ExpiresAt { get; init; }
}

Notification delivery

public sealed class NotificationDelivery
{
    public Guid Id { get; init; }
    public Guid NotificationId { get; init; }
    public NotificationChannel Channel { get; init; }
 
    public required string Destination { get; init; }
    public DeliveryStatus Status { get; set; }
 
    public int AttemptCount { get; set; }
    public DateTimeOffset? NextAttemptAt { get; set; }
    public DateTimeOffset? SentAt { get; set; }
    public DateTimeOffset? DeliveredAt { get; set; }
 
    public string? ProviderMessageId { get; set; }
    public string? LastErrorCode { get; set; }
    public string? LastErrorMessage { get; set; }
 
    public required string IdempotencyKey { get; init; }
}

This gives us independent lifecycle tracking per channel.

A good state model is something like:

stateDiagram-v2
    direction TB
    [*] --> Pending: Request Accepted (202)
    Pending --> Queued: Outbox Published to Broker
    Queued --> Processing: Worker Claims Delivery
    Processing --> ProviderAccepted: External Provider Confirms Receipt
    Processing --> Expired: TTL Passed Before Send
    Processing --> Suppressed: User Preferences Disallow
    Processing --> Failed: Permanent Failure (4xx / Bad Config)
    Processing --> RetryScheduled: Transient Failure (5xx / 429)
    RetryScheduled --> Queued: Exponential Backoff + Jitter
    ProviderAccepted --> Delivered: Carrier Delivery Webhook
    ProviderAccepted --> Failed: Provider Bounce / Rejection
    ProviderAccepted --> Expired: Terminal Expiry Window

Not every provider exposes every state. For example, ProviderAccepted is not always equivalent to Delivered, and email open tracking should not be treated as a guaranteed proof that a person read a message.

The model should represent only states the system can justify.

4. Durable acceptance and the dual-write trap

We want the API to:

1. Store the notification.
2. Publish work to a message broker.
3. Return 202 Accepted.

The dangerous implementation is:

flowchart LR
    A["1. INSERT Notification into PostgreSQL"] -->|"Uncoordinated Writes"| B["2. Publish Message to Broker"]

Those are two independent writes to two independent systems.

Consider this sequence:

flowchart TD
    subgraph ScenarioA ["Failure Mode 1: DB Succeeds, Broker Publish Fails"]
        A1["1. COMMIT Notification in DB (Success)"] --> B1["💥 Process Crash / Network Outage"]
        B1 --> C1["❌ Queue Publish Never Happens<br/>(Notification is orphaned in DB)"]
    end
    subgraph ScenarioB ["Failure Mode 2: Broker Succeeds, DB Write Fails"]
        A2["1. Publish Notification to Broker (Success)"] --> B2["💥 DB Transaction Fails / Rolls Back"]
        B2 --> C2["❌ Worker receives message for<br/>non-existent DB entity"]
    end

Now a worker can receive work for a notification that does not exist.

This is the classic dual-write problem.

Use the Transactional Outbox pattern

Write both the notification and an outbox message to the same database transaction.

flowchart LR
    A[Producer] --> B[Notification API]
    B --> C[(PostgreSQL)]
    C --> D[Notification row]
    C --> E[Outbox row]
    E --> F[Outbox Publisher]
    F --> G[Message Broker]

EF Core example

public async Task<Guid> CreateNotificationAsync(
    CreateNotification command,
    CancellationToken cancellationToken)
{
    var notificationId = Guid.NewGuid();
 
    var notification = new Notification
    {
        Id = notificationId,
        UserId = command.UserId,
        Type = command.Type,
        TemplateKey = command.TemplateKey,
        PayloadJson = JsonSerializer.Serialize(command.Payload),
        Priority = command.Priority,
        CreatedAt = DateTimeOffset.UtcNow,
        ExpiresAt = command.ExpiresAt
    };
 
    var outboxMessage = new OutboxMessage
    {
        Id = Guid.NewGuid(),
        Type = "NotificationCreated",
        AggregateId = notificationId,
        PayloadJson = JsonSerializer.Serialize(
            new NotificationCreated(notificationId)),
        OccurredAt = DateTimeOffset.UtcNow
    };
 
    await using var transaction =
        await db.Database.BeginTransactionAsync(cancellationToken);
 
    db.Notifications.Add(notification);
    db.OutboxMessages.Add(outboxMessage);
 
    await db.SaveChangesAsync(cancellationToken);
    await transaction.CommitAsync(cancellationToken);
 
    return notificationId;
}

The important property is not EF Core itself.

The important property is:

The business record and the outbox record commit atomically in the same database transaction.

Either both exist or neither exists.

5. Publishing the outbox safely with multiple instances

Once an outbox table exists, a background publisher polls unprocessed rows and publishes them to the broker.

A naive query is:

SELECT *
FROM outbox_messages
WHERE processed_at IS NULL
ORDER BY occurred_at
LIMIT 100;

That is not enough when multiple publisher instances run concurrently.

Two publishers can read the same rows.

PostgreSQL provides a useful pattern for queue-like tables:

SELECT id, type, payload_json
FROM outbox_messages
WHERE processed_at IS NULL
ORDER BY occurred_at
FOR UPDATE SKIP LOCKED
LIMIT 100;

SKIP LOCKED allows concurrent consumers to skip rows already locked by another transaction instead of waiting on them. PostgreSQL's documentation explicitly notes that this can be useful for multiple consumers accessing a queue-like table.

A simple illustrative pattern is:

flowchart TD
    A["1. Begin DB Transaction"] --> B["2. SELECT ... FOR UPDATE SKIP LOCKED<br/>(Claim batch without contention)"]
    B --> C["3. Publish Batch to Message Broker"]
    C --> D["4. UPDATE outbox SET processed_at = NOW()"]
    D --> E["5. Commit DB Transaction"]

However, do not confuse row claiming with exactly-once publishing.

A publisher can still crash after the broker accepts a message but before processed_at is committed.

The same outbox event can therefore be published again.

That is expected.

The downstream consumer must be designed for redelivery.

Holding database locks while making remote broker calls also couples transaction duration to network latency. A production relay can instead claim a bounded batch with leases, commit the claim, publish outside the transaction, and record confirmed results. Lease expiry, ordering and duplicate publication then need their own design; SKIP LOCKED alone is not a complete relay protocol.

6. Fan out by channel

One notification may need multiple delivery channels.

A useful logical architecture is:

flowchart TB
    A[NotificationCreated] --> B[Notification Router]
    B --> C[Push Queue]
    B --> D[Email Queue]
    B --> E[SMS Queue]
    B --> F[In-App Queue]
 
    C --> C1[Push Workers]
    D --> D1[Email Workers]
    E --> E1[SMS Workers]
    F --> F1[In-App Workers]

Why not one giant queue?

Because the channels have independent failure characteristics.

If the email provider becomes slow and all channels share one worker pool, email work can consume worker capacity while SMS and push messages wait behind it.

Separate queues and worker pools create failure isolation:

flowchart LR
    subgraph DegradedChannel ["Degraded Channel (Email)"]
        E1["Email Queue"] --> E2["Email Workers"]
        E2 -->|"High Latency / Rate Limits"| E3["Email Provider (Degraded)"]
    end
    subgraph HealthyChannels ["Healthy Channels (Isolated Failure Domains)"]
        H1["Push Queue"] --> H2["Push Workers"] --> H3["APNs / FCM (Real-time)"]
        S1["SMS Queue"] --> S2["SMS Workers"] --> S3["Twilio (Real-time)"]
        I1["In-App Queue"] --> I2["In-App Workers"] --> I3["PostgreSQL (Instant)"]
    end

This is not just a scaling technique. It is a resilience boundary.

7. Priority must be first-class

Channel isolation is not enough.

Imagine a marketing campaign publishes one million emails at 09:00.

At 09:00:01, a user requests a password reset.

If both share the same FIFO backlog, the password-reset notification may wait behind campaign traffic.

That is a business failure even if the queue is working perfectly.

A useful priority model is:

Logically:

flowchart LR
    A[Notification Router] --> B[Critical Queue]
    A --> C[Transactional Queue]
    A --> D[Bulk / Marketing Queue]
 
    B --> B1[Dedicated Workers]
    C --> C1[Dedicated Workers]
    D --> D1[Elastic Bulk Workers]

Whether these become physically separate queues depends on scale and operational needs, but the isolation boundary should be explicit.

A useful operational rule is:

Backpressure should degrade low-priority traffic before critical traffic.

8. The consumer contract: acknowledge only after responsibility is transferred

Message brokers rely on acknowledgment semantics.

With manual acknowledgments, a message is not considered complete just because a worker received it. The worker acknowledges after it has safely processed the message according to the application contract.

That distinction matters.

Suppose a worker receives a delivery ticket, then crashes before acknowledging it.

A reliable broker can make that work available again.

This produces at-least-once processing.

At-least-once is valuable because it protects against silent loss.

But it also means:

The same logical message can be processed more than once.

Therefore every side-effecting consumer must have a duplicate strategy.

9. Idempotency is per delivery, not merely per notification

Suppose notification 123 should be sent through both SMS and email.

This key is too broad:

notification:123

If SMS marks that key complete, the email worker could mistakenly conclude the whole notification has already been handled.

Use a key scoped to the external side effect:

notification:123:sms
notification:123:email

or simply use an immutable delivery ID:

delivery:31fa89d8-...

That key should be unique for one logical delivery operation.

10. Why a Redis lock is helpful but not an exactly-once guarantee

A common duplicate-control technique is an atomic Redis key:

SET delivery:{id} processing NX EX 60

Redis supports SET with conditional NX semantics and expiry options such as EX or PX.

This can coordinate contenders during the lease window; it cannot guarantee exclusivity after expiry or failover. Use an ownership token for safe release/renewal and consider fencing where the downstream resource supports it. A sender may continue running after its lease expires.

Conceptually:

sequenceDiagram
    autonumber
    actor WA as Worker A
    participant Redis as Redis Cache
    actor WB as Worker B
 
    WA->>Redis: SET delivery:{id} processing NX EX 60
    Redis-->>WA: OK (Lock Acquired)
    WB->>Redis: SET delivery:{id} processing NX EX 60
    Redis-->>WB: (nil) (Contention Detected)
    Note over WA: Dispatches to Provider
    Note over WB: Skips / Awaits Completion

But this is not enough to claim exactly-once external delivery.

Consider the failure window:

sequenceDiagram
    autonumber
    actor W as Worker
    participant P as SMS / Email Provider
 
    W->>P: 1. POST /v1/messages (Payload, IdempotencyKey)
    activate P
    Note over P: Provider accepts and dispatches<br/>message to carrier network!
    P--xW: 2. HTTP 200 OK (TCP Disconnect / Lost Ack / 504 Timeout)
    deactivate P
    Note over W: Worker catches TimeoutException!<br/>Side effect occurred, but outcome is ambiguous.

Did the SMS send?

Our system cannot know with certainty.

If we retry, the user may receive a duplicate.

If we never retry, the user may receive nothing.

This is the fundamental ambiguity of performing side effects across a network boundary.

The strongest option is provider-supported idempotency, where the same idempotency key is sent with retries and the provider promises to deduplicate repeated requests according to its documented contract.

If a provider does not offer such a guarantee, our application can still reduce duplicate probability substantially, but should not claim mathematical end-to-end exactly-once delivery.

This is why the accurate language is:

At-least-once transport + idempotent processing + provider-side deduplication where supported.

Not:

“The notification is guaranteed to be delivered exactly once.”

11. Persisted idempotency is stronger than an ephemeral lock for completed work

For business-critical delivery history, I prefer the durable database to remain the source of truth for completed delivery state.

One useful pattern is a unique constraint on the logical delivery key.

For example:

CREATE UNIQUE INDEX ux_delivery_idempotency_key
ON notification_deliveries (idempotency_key);

The worker reads and transitions a known delivery row rather than inventing send state entirely in Redis.

Redis can still be used for short-lived coordination or load shedding, but durable status belongs in durable storage.

A state transition can be guarded with optimistic concurrency or a conditional update:

UPDATE notification_deliveries
SET status = 'Processing',
    attempt_count = attempt_count + 1
WHERE id = @delivery_id
  AND status IN ('Pending', 'RetryScheduled');

If zero rows are updated, another worker or a previous attempt may already have changed the state.

The correct implementation depends on the broker and database isolation model, but the principle is stable:

Coordinate transient work cheaply; record durable truth durably.

12. Retry only failures that are likely to recover

“Retry on failure” is too broad.

Some failures are transient:

• HTTP 408
• HTTP 429
• HTTP 5xx
• temporary DNS/network errors
• provider timeout
• provider unavailable

Some are usually permanent until data or configuration changes:

• malformed destination
• invalid/unregistered device token
• invalid template
• blocked recipient
• authentication/authorization configuration failure

A permanent failure should not be retried 20 times simply because a retry policy exists.

The worker should classify provider responses.

public static FailureKind Classify(HttpStatusCode statusCode)
{
    return statusCode switch
    {
        HttpStatusCode.RequestTimeout => FailureKind.Transient,
        HttpStatusCode.TooManyRequests => FailureKind.Transient,
        >= HttpStatusCode.InternalServerError => FailureKind.Transient,
        HttpStatusCode.BadRequest => FailureKind.Permanent,
        HttpStatusCode.Unauthorized => FailureKind.Configuration,
        HttpStatusCode.Forbidden => FailureKind.Configuration,
        _ => FailureKind.Unknown
    };
}

Real providers have provider-specific error codes, so production adapters should map those codes explicitly rather than relying only on HTTP status.

For example, FCM documents retry behavior for server-unavailable and internal errors, including honoring Retry-After, using exponential backoff, and adding randomness to avoid synchronized retry traffic.

13. Exponential backoff needs jitter

Without delay:

flowchart TD
    A["Provider Degraded (503 / 429 Throttle)"] --> B["10,000 Workers Fail Simultaneously"]
    B --> C{"Retry Strategy"}
    C -->|"Naive Immediate Retries"| D["Massive Synchronized Retry Spike (10,000 req/s)<br/>💥 Destroys Provider Recovery"]
    C -->|"Exponential Backoff + Jitter"| E["Randomized Uniform Request Spacing<br/>✅ Allows Provider to Drain & Recover"]

Basic exponential backoff:

1s → 2s → 4s → 8s → 16s

is better, but thousands of workers can still synchronize around the same delays.

Add jitter:

1.2s → 2.7s → 3.6s → 8.9s → 14.3s

The exact policy should respect provider guidance and Retry-After where available.

.NET resilience example

Modern .NET applications can use Microsoft.Extensions.Http.Resilience, which provides HTTP resilience strategies built on Polly.

builder.Services
    .AddHttpClient<EmailProviderClient>()
    .AddStandardResilienceHandler(options =>
    {
        options.Retry.MaxRetryAttempts = 3;
        options.Retry.UseJitter = true;
        options.Retry.BackoffType = DelayBackoffType.Exponential;
 
        options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(20);
        options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(5);
    });

One critical warning: blindly retrying POST requests can duplicate side effects.

Microsoft's resilience documentation explicitly warns that retries for unsafe HTTP methods can be harmful for operations that create data. A notification provider call should only be retried automatically when your provider contract and idempotency strategy make that retry safe enough.

14. Add a circuit breaker for sustained provider outages

Retries are useful for transient faults.

They are harmful when a dependency is clearly down for an extended period.

If an email provider is unavailable for 30 minutes, repeatedly sending hundreds of thousands of failing requests wastes resources and may slow recovery.

A circuit breaker changes the behavior:

stateDiagram-v2
    direction LR
    [*] --> Closed: Normal Operation
    Closed --> Open: Consecutive Failures > Threshold
    Open --> HalfOpen: Cooldown Timer Expires
    HalfOpen --> Closed: Probe Request Succeeds
    HalfOpen --> Open: Probe Request Fails

Architecture:

flowchart LR
    A[Email Queue] --> B[Email Worker]
    B --> C[Circuit Breaker]
    C --> D[Email Provider]
    C -->|open| E[Retry / Delay path]

A circuit breaker does not replace a durable queue.

The queue keeps work safe.

The circuit breaker prevents workers from continuously hammering a dependency that is already failing.

For high-value channels, a provider-routing layer can optionally support a secondary provider, but failover must consider provider-specific templates, sender identities, compliance requirements, rate limits, and idempotency behavior.

“Just switch providers” is not operationally free.

15. Provider rate limits are part of your architecture

Worker autoscaling is not unlimited throughput.

Suppose the queue has 500,000 messages.

You scale from 20 workers to 500 workers.

If the downstream provider allows only 1,000 requests per second, the worker fleet can now create a provider-throttling storm.

The outbound path needs its own controls:

flowchart LR
    A[Queue] --> B[Workers]
    B --> C[Concurrency / Rate Limiter]
    C --> D[Circuit Breaker]
    D --> E[Provider]

This is a different problem from inbound API rate limiting.

Here we are protecting both:

• the third-party provider
• our own connection pools, sockets, CPU, and worker capacity

A queue lets us absorb a burst.

It does not give us permission to drain the backlog faster than downstream dependencies can safely handle.

16. Expiry is a correctness rule, not an optimization

Some notifications become useless after a deadline.

Examples:

OTP                → 2 minutes
Password reset     → 15 minutes
Delivery update    → 1 hour
Flash sale         → campaign-defined deadline

If an OTP is delivered 45 minutes late, the system may technically have “delivered” it, but functionally it failed.

Workers should check expiry before an external side effect:

if (notification.ExpiresAt is { } expiresAt &&
    expiresAt <= DateTimeOffset.UtcNow)
{
    delivery.Status = DeliveryStatus.Expired;
    await db.SaveChangesAsync(cancellationToken);
    return DeliveryResult.Expired;
}

Expiry also prevents a recovered provider from causing a massive burst of obsolete traffic after a long outage.

17. User preferences should usually be checked close to send time

Suppose a marketing notification is created at 09:00 but sits behind a backlog.

At 09:05 the user opts out of marketing email.

At 09:10 the worker finally processes the notification.

If preferences were only checked when the notification was created, the user may still receive a message they just opted out of.

For preference-controlled traffic, checking again close to the side effect is generally safer.

A preference model might look like:

You may also need:

• locale
• timezone
• quiet hours
• frequency caps
• legal/compliance consent state

Not every notification should be suppressible. Security, legal, or transactional requirements may follow separate product and regulatory rules.

18. In-app notifications are different

In-app delivery is the one channel fully controlled by your product.

You do not need APNs, FCM, an SMS carrier, or an email service to display an item inside your own application.

A simple model is:

sequenceDiagram
    autonumber
    actor User as Mobile / Web Client
    participant API as In-App Notification API
    participant DB as PostgreSQL
 
    Note over DB: Notification delivery row written (Status: Pending)
    User->>API: 1. GET /api/notifications/unread
    API->>DB: 2. Query unread deliveries (SeenAt IS NULL)
    DB-->>API: 3. Return unread notifications
    API-->>User: 4. Display badge counter & list
    User->>API: 5. POST /api/notifications/{id}/seen
    API->>DB: 6. UPDATE delivery SET SeenAt = NOW()

Instead of a single boolean Seen, prefer timestamps when they matter operationally:

public DateTimeOffset? SeenAt { get; set; }
public DateTimeOffset? ReadAt { get; set; }

This preserves information rather than collapsing it into true/false.

19. Dead-letter queues are not garbage bins

After a bounded number of retry attempts, some messages will still fail.

These messages should not disappear.

A dead-letter queue (DLQ) creates an explicit operational state for work that needs investigation or a controlled replay.

Common reasons include:

• malformed payloads
• unexpected provider responses
• repeated transient failures
• broken templates
• application bugs
• bad configuration

A production DLQ needs more than existence.

You also need:

• alerting on DLQ growth
• tooling to inspect messages safely
• a remediation workflow
• controlled replay
• protection against infinite poison-message loops

RabbitMQ supports dead-letter exchanges, and Amazon SQS supports dead-letter queues/redrive policies.

The operational principle is:

A failed notification must end in an observable state, not disappear into logs.

20. Observability: monitor age, not just queue depth

Queue depth is useful but incomplete.

Consider two queues:

Queue A
Depth: 1,000,000
Oldest message: 8 seconds
 
Queue B
Depth: 10,000
Oldest message: 23 minutes

For transactional notifications, Queue B may be much more serious.

Important metrics include:

notifications_requested_total
notification_deliveries_total
notification_delivery_failures_total
notification_retries_total
notification_expired_total
 
queue_depth
queue_oldest_message_age_seconds
worker_processing_latency_seconds
end_to_end_delivery_latency_seconds
provider_request_duration_seconds
provider_error_rate
provider_throttle_rate
circuit_breaker_state
dlq_depth

Useful dimensions include:

channel
priority
notification_type
provider
result
error_code
region

Be careful with high-cardinality dimensions. User IDs, notification IDs, phone numbers, and email addresses generally do not belong as metric labels.

They are better suited to structured logs or traces, with appropriate privacy controls.

Trace the entire path

A notification can cross many boundaries:

flowchart LR
    A["1. API Endpoint"] -->|"traceparent Context"| B["2. PostgreSQL Outbox"]
    B -->|"Propagated W3C ID"| C["3. Outbox Publisher"]
    C -->|"Message Metadata"| D["4. Broker Queue"]
    D -->|"Envelope Headers"| E["5. Background Worker"]
    E -->|"traceparent HTTP Header"| F["6. Downstream Provider"]

Propagating correlation identifiers across these steps makes it possible to answer:

“Why did notification 7f8d... take 38 seconds?”

This is where observability becomes an architectural feature, not an afterthought.

21. A production worker flow

A useful delivery pipeline is:

flowchart TD
    A[Receive broker message] --> B[Load delivery record]
    B --> C{Already terminal?}
    C -->|Yes| Z[Acknowledge / finish]
    C -->|No| D{Expired?}
    D -->|Yes| E[Mark Expired]
    E --> Z
    D -->|No| F{Preference allows send?}
    F -->|No| G[Mark Suppressed]
    G --> Z
    F -->|Yes| H[Acquire/verify processing ownership]
    H --> I[Apply provider rate/concurrency limit]
    I --> J[Call provider through timeout + circuit breaker]
    J --> K{Result}
    K -->|Success| L[Persist provider accepted/sent state]
    L --> Z
    K -->|Transient| M[Schedule retry with backoff + jitter]
    M --> Z
    K -->|Permanent| N[Mark Failed]
    N --> Z
    K -->|Poison / max attempts| O[DLQ]
    O --> Z

The ordering of these checks is deliberate.

Do cheap local validation before consuming external provider capacity.

22. Example delivery service in .NET

The following snippet is intentionally simplified to show boundaries rather than hide them behind framework code.

public sealed class DeliveryProcessor(
    NotificationDbContext db,
    IProviderRouter providerRouter,
    IPreferenceService preferences,
    ILogger<DeliveryProcessor> logger)
{
    public async Task ProcessAsync(
        Guid deliveryId,
        CancellationToken cancellationToken)
    {
        var delivery = await db.NotificationDeliveries
            .SingleOrDefaultAsync(x => x.Id == deliveryId, cancellationToken);
 
        if (delivery is null)
        {
            logger.LogWarning("Delivery {DeliveryId} not found", deliveryId);
            return;
        }
 
        if (delivery.Status is DeliveryStatus.ProviderAccepted
            or DeliveryStatus.Delivered
            or DeliveryStatus.Failed
            or DeliveryStatus.Expired
            or DeliveryStatus.Suppressed)
        {
            return;
        }
 
        var notification = await db.Notifications
            .SingleAsync(x => x.Id == delivery.NotificationId, cancellationToken);
 
        if (notification.ExpiresAt is { } expiresAt &&
            expiresAt <= DateTimeOffset.UtcNow)
        {
            delivery.Status = DeliveryStatus.Expired;
            await db.SaveChangesAsync(cancellationToken);
            return;
        }
 
        var allowed = await preferences.IsAllowedAsync(
            notification.UserId,
            notification.Type,
            delivery.Channel,
            cancellationToken);
 
        if (!allowed)
        {
            delivery.Status = DeliveryStatus.Suppressed;
            await db.SaveChangesAsync(cancellationToken);
            return;
        }
 
        delivery.Status = DeliveryStatus.Processing;
        delivery.AttemptCount++;
        await db.SaveChangesAsync(cancellationToken);
 
        var provider = providerRouter.Resolve(delivery.Channel);
 
        ProviderResult result;
        try
        {
            result = await provider.SendAsync(
                new ProviderMessage(
                    delivery.Destination,
                    notification.TemplateKey,
                    notification.PayloadJson,
                    delivery.IdempotencyKey),
                cancellationToken);
        }
        catch (Exception ex) when (ProviderFailureClassifier.IsTransient(ex))
        {
            delivery.Status = DeliveryStatus.RetryScheduled;
            delivery.NextAttemptAt = RetrySchedule.Next(delivery.AttemptCount);
            delivery.LastErrorMessage = ex.Message;
            await db.SaveChangesAsync(cancellationToken);
            return;
        }
 
        if (result.Success)
        {
            delivery.Status = DeliveryStatus.ProviderAccepted;
            delivery.ProviderMessageId = result.ProviderMessageId;
            delivery.SentAt = DateTimeOffset.UtcNow;
        }
        else if (result.IsTransient)
        {
            delivery.Status = DeliveryStatus.RetryScheduled;
            delivery.NextAttemptAt = RetrySchedule.Next(delivery.AttemptCount);
            delivery.LastErrorCode = result.ErrorCode;
        }
        else
        {
            delivery.Status = DeliveryStatus.Failed;
            delivery.LastErrorCode = result.ErrorCode;
        }
 
        await db.SaveChangesAsync(cancellationToken);
    }
}

This excerpt is illustrative and has not been compiled or integration-tested in this revision. A persisted Processing value is not itself an exclusive claim. Production code needs an atomic ownership transition, a recoverable lease, an Unknown/ReconciliationRequired outcome for ambiguous sends, and a durable retry handoff before acknowledging the original message. ProviderAccepted should not be resent while awaiting delivery evidence. Cancellation, acknowledgement semantics and permanent failures also need explicit mapping. Do not copy this fragment as a production worker.

The point of the snippet is the architecture of the decision pipeline.

23. What happens when a provider times out after accepting the request?

This is the most important edge case in the whole design.

Timeline:

sequenceDiagram
    autonumber
    actor W as Worker
    participant P as SMS / Email Provider
 
    W->>P: 1. POST /v1/messages (Payload, IdempotencyKey)
    activate P
    Note over P: Provider accepts and dispatches<br/>message to carrier network!
    P--xW: 2. HTTP 200 OK (TCP Disconnect / Lost Ack / 504 Timeout)
    deactivate P
    Note over W: Worker catches TimeoutException!<br/>Side effect occurred, but outcome is ambiguous.

Our worker sees a timeout.

The external side effect may already have happened.

There are only three broad possibilities:

1. Provider supports idempotent retries. Reuse the same provider-recognized idempotency key.
2. Provider exposes a reliable lookup/reconciliation API. Query by provider request/message identifier where the contract allows it.
3. Provider offers neither. Retrying is a business trade-off between possible duplicate delivery and possible loss.

No amount of local Redis locking can prove what happened inside a remote provider after an ambiguous network failure.

This is exactly why architecture discussions should distinguish:

• message processing guarantees inside our platform
• API request guarantees between our platform and a provider
• actual human/device delivery beyond the provider

Those are different boundaries.

24. Final architecture

flowchart TB
    subgraph Producers
        A1[Order Service]
        A2[Payment Service]
        A3[Auth Service]
        A4[Marketing Platform]
    end
 
    A1 --> B[Notification API]
    A2 --> B
    A3 --> B
    A4 --> B
 
    B --> C[(PostgreSQL)]
 
    subgraph Database
        C1[Notifications]
        C2[Notification Deliveries]
        C3[User Preferences]
        C4[Outbox]
    end
 
    C --> C1
    C --> C2
    C --> C3
    C --> C4
 
    C4 --> D[Outbox Publisher]
    D --> E[Message Broker]
 
    E --> P0[Critical]
    E --> P1[Transactional]
    E --> P3[Marketing]
 
    P0 --> R[Channel Router]
    P1 --> R
    P3 --> R
 
    R --> Q1[Push Queue]
    R --> Q2[Email Queue]
    R --> Q3[SMS Queue]
    R --> Q4[In-App Queue]
 
    Q1 --> W1[Push Workers]
    Q2 --> W2[Email Workers]
    Q3 --> W3[SMS Workers]
    Q4 --> W4[In-App Workers]
 
    W1 --> X1[Rate Limit / Timeout / Circuit Breaker]
    W2 --> X2[Rate Limit / Timeout / Circuit Breaker]
    W3 --> X3[Rate Limit / Timeout / Circuit Breaker]
 
    X1 --> FCM[FCM / APNs]
    X2 --> MAIL[Email Provider]
    X3 --> SMS[SMS Provider]
 
    W4 --> C
 
    W1 -. failure .-> DLQ[Dead-Letter Queue]
    W2 -. failure .-> DLQ
    W3 -. failure .-> DLQ

The key design properties are:

• PostgreSQL is the durable source of truth for notification intent and delivery state.
• The Transactional Outbox removes the unsafe database-plus-broker dual write from the request path.
• The broker decouples acceptance from delivery and absorbs bursts.
• Channel and priority isolation prevent one backlog from blocking unrelated critical traffic.
• Workers assume redelivery can happen.
• Retries are bounded and limited to retryable failures.
• Backoff includes jitter and respects provider guidance.
• Expired work is dropped intentionally.
• External calls use timeouts, rate/concurrency limits, and circuit breakers.
• DLQs turn terminal failures into an operational workflow.
• Observability tracks latency, backlog age, retries, failures, and provider behavior.

25. What the system guarantees — and what it does not

A production architecture should state its guarantees explicitly.

We can reasonably design for

• Durable recording of notification intent in our database.
• At-least-once publication/processing across the asynchronous pipeline.
• Strong duplicate suppression inside our own system.
• Independent delivery state for each channel.
• Bounded retry behavior.
• Priority isolation.
• Recovery from worker crashes without silently losing queued work.
• Operational visibility into failed and delayed notifications.

We should not casually promise

• End-to-end exactly-once delivery to a human.
• Guaranteed provider delivery after a provider has merely accepted a request.
• Zero duplicates across every carrier/device/network failure mode.
• Global ordering across independent notifications unless the product explicitly requires and pays for it.

The most accurate production mindset is:

Prefer durable at-least-once processing, then make repeated processing safe wherever the boundary allows it.

26. Scaling strategy

When load grows, scale the bottleneck that is actually saturated.

Notification API

Scale stateless API instances horizontally.

The API should do little work beyond validation, authorization, durable persistence, and returning 202 Accepted.

Outbox publisher

Run multiple publishers and coordinate row claiming safely.

PostgreSQL FOR UPDATE SKIP LOCKED is one practical mechanism for queue-like outbox polling.

Broker

Partition/shard where the chosen broker requires it. Avoid introducing strict global ordering if the product does not require it.

Workers

Scale independently by channel and priority.

Do not scale worker concurrency past safe downstream provider throughput.

PostgreSQL

Index for actual access paths, especially:

CREATE INDEX ix_outbox_unprocessed
ON outbox_messages (occurred_at)
WHERE processed_at IS NULL;
 
CREATE INDEX ix_delivery_status_next_attempt
ON notification_deliveries (status, next_attempt_at);
 
CREATE INDEX ix_notification_user_created
ON notifications (user_id, created_at DESC);

Indexes should be validated against real query plans and write volume rather than copied blindly.

Redis

Use Redis only for problems that benefit from low-latency ephemeral coordination, deduplication windows, rate limiting, or caching.

Do not turn it into the only durable record of a financially or operationally important delivery outcome.

27. Security and privacy considerations

Notification systems process sensitive user contact information and sometimes sensitive message content.

Production design should include:

• encryption in transit
• encryption at rest where appropriate
• secrets in a secret manager rather than source/config files
• least-privilege access to provider credentials
• redaction of phone numbers, email addresses, tokens, and message bodies from logs
• strict access controls around DLQ inspection
• retention policies for notification payloads
• auditability for preference changes where required
• secure storage and rotation of push-device tokens

Avoid placing full notification bodies into message metadata, metric labels, or logs by default.

The broker event can carry identifiers and routing metadata while workers load sensitive details from an authorized data store.

28. Key lessons

A production notification platform is not mainly an email/SMS integration problem.

It is a distributed-systems problem involving unreliable networks and independently failing components.

The most important lessons are:

1. Accept first, deliver asynchronously

Do not make the caller depend on provider latency.

2. Avoid database + broker dual writes

Use a Transactional Outbox or another mechanism with equivalent durability semantics.

3. Design consumers for redelivery

At-least-once processing means duplicates are part of normal failure recovery.

4. Scope idempotency to the actual side effect

Email and SMS deliveries from one notification are different idempotency boundaries.

5. Do not claim exactly-once across a remote provider boundary

A timeout can leave the result ambiguous unless the provider contract gives you a way to deduplicate or reconcile it.

6. Retry selectively

Transient failures deserve retries. Permanent failures usually do not.

7. Use backoff, jitter, timeouts, rate limits, and circuit breakers together

Resilience is a pipeline, not one retry loop.

8. Isolate channels and priorities

A marketing backlog should never trap an OTP behind it.

9. Expiry is part of correctness

Late can be equivalent to wrong.

10. Observe the whole pipeline

Queue age, delivery latency, provider errors, retry rate, and DLQ depth should be measurable before production incidents happen.

Closing

The difficult part of notification delivery is not calling an API.

The difficult part is deciding what should happen when every boundary can fail independently:

Database succeeds, broker publish fails.
 
Broker redelivers after a worker crash.
 
Provider accepts the message, but the response disappears.
 
Provider becomes unavailable for 30 minutes.
 
Marketing creates a million-message burst.
 
The user opts out while work is still queued.
 
An OTP expires before the backlog clears.

A production-grade notification system is built by designing those failure paths deliberately.

The goal is not a magical exactly-once promise.

The goal is a system where work is durable, retries are controlled, duplicates are minimized, critical traffic stays responsive, failures are observable, and every important edge case ends in a known operational state.

That is what makes the system reliable.

References and further reading

The claims below were checked against primary and vendor documentation available at the time of writing.

1. PostgreSQL 18 — SELECT, FOR UPDATE, SKIP LOCKED
   PostgreSQL documents that SKIP LOCKED skips rows that cannot immediately be locked and notes that this is useful for multiple consumers accessing a queue-like table.
   https://www.postgresql.org/docs/18/sql-select.html

2. RabbitMQ — Consumer Acknowledgements and Publisher Confirms
   Covers manual acknowledgements, requeue behavior, publisher confirms, and why distributed messaging needs explicit delivery/processing confirmation.
   https://www.rabbitmq.com/docs/confirms

3. RabbitMQ — Reliability Guide
   Explains the semantics of acknowledgements and notes that acknowledgements enable at-least-once delivery, while applications must handle retransmission/redelivery.
   https://www.rabbitmq.com/docs/reliability

4. RabbitMQ — Dead Letter Exchanges
   Official documentation for dead-lettering rejected, expired, over-limit, or over-delivery-limit messages.
   https://www.rabbitmq.com/docs/dlx

5. Amazon SQS — At-least-once delivery
   AWS explicitly documents that standard SQS messages can be delivered more than once and applications should be idempotent.
   https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html

6. Amazon SQS — Queue / Dead-letter configuration
   Documents visibility timeout and redrive/dead-letter queue configuration.
   https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_CreateQueue.html

7. Redis — SET command
   Official syntax for conditional NX writes and expiry options such as EX/PX, useful for short-lived coordination/deduplication windows.
   https://redis.io/docs/latest/commands/set/

8. Microsoft Learn — Build resilient HTTP apps
   Documents the .NET HTTP resilience pipeline, including retry, timeout, circuit breaker, concurrency/rate limiting and jitter. It also warns that retrying unsafe HTTP methods can cause harmful duplicated side effects.
   https://learn.microsoft.com/en-us/dotnet/core/resilience/http-resilience

9. Microsoft Learn — Introduction to resilient app development
   Overview of retry, circuit breaker, timeout, rate limiting, concurrency limiting, fallback, and hedging in modern .NET resilience packages.
   https://learn.microsoft.com/en-us/dotnet/core/resilience/

10. Firebase Cloud Messaging — Error codes
    FCM documents retrying selected transient failures, honoring Retry-After, using exponential backoff, and adding randomness to avoid synchronized retries.
    https://firebase.google.com/docs/cloud-messaging/error-codes

Editorial note

This article is inspired by the classic “Design a Notification System” architectural problem, but the production model, failure analysis, code structure, outbox discussion, resilience pipeline, delivery-state model, and operational guidance have been expanded and cross-checked against primary technical documentation.

Additional architecture references

• Microsoft Learn — Queue-Based Load Leveling
• Microsoft Learn — Bulkhead
• Microsoft Learn — Priority Queue
• Chris Richardson — Transactional Outbox
• AWS Builders’ Library — Idempotent APIs
• Microsoft Learn — Service Bus dead-letter queues