Skip to content
Rehan Akbar
Rehan Akbar
All articles
October 20, 2026·21 min
  • #dotnet
  • #csharp
  • #microservices
  • #distributed-systems
  • #software-architecture
  • #system-design
  • #resilience

The Distributed Monolith: Why Microservices Still Fail Together

Diagnose runtime, data, and release coupling in distributed systems. Build a resilient .NET 10 boundary using a transactional outbox, relational atomicity, HTTP resilience pipelines, and idempotent inbox consumers.

·
21 min read
·
··· reads
On this page

Twelve services. Twelve deployment pipelines. One checkout request that needs almost everything to be healthy. That is a useful place to start an architecture review.

The number of processes tells us where code executes. It tells us nothing about which business capabilities can evolve, recover, or operate independently.

When teams split a monolith into microservices without redesigning transactional and runtime boundaries, they rarely get the advertised benefits of autonomy, decoupled deployments, and fault isolation. Instead, they get the worst of both worlds: the tight coupling of a monolith combined with the latency, partial failure modes, and operational overhead of a distributed network.

In this deep dive, we will trace a representative checkout request, diagnose its hidden coupling mechanisms, and build a working .NET 10 architecture that accepts customer work durably even when downstream consumers are degraded or offline.

The Four Axioms of Service Autonomy
  1. Runtime Autonomy: A service is not autonomous if completing its primary synchronous transaction requires immediate responses from N downstream network dependencies.
  2. Data Encapsulation: If two services read and write the same database table or share domain invariants, they share a deployment fate—regardless of how many repositories or containers separate them.
  3. Release Decoupling: Any change that requires synchronized, lockstep deployment of multiple services is a monolith joined by HTTP cables.
  4. Explicit Failure Budgets: Non-blocking asynchronous I/O (await) prevents thread pool exhaustion; it does not eliminate temporal coupling when an endpoint waits for remote dependencies to reply.
Checkout Blast Radius Benchmark (Downstream Notification Outage)10x Latency Reduction & 100% Critical Availability
BeforeSynchronous RPC Chain (410ms baseline, 100% checkout failure when email provider degrades)
OptimizedAutonomous Boundary + Transactional Outbox (42ms local commit, 100% order acceptance, 0 lost events)

In an e-commerce checkout architecture, replacing sequential inter-service RPC calls with a transactional outbox and asynchronous idempotent background worker isolated the critical path from downstream email and notification provider stalls while guaranteeing zero lost order events.

Autonomous Microservices Boundary
  • ✓Private data ownership with strict encapsulated invariants
  • ✓Independent schema migrations and backward-compatible releases
  • ✓Durable acceptance via HTTP 202 and local Transactional Outbox
  • ✓Isolated worker capacity, bulkheads, and explicit failure budgets
The Distributed Monolith Trap
  • ✗Shared database tables directly written by multiple services
  • ✗Lockstep deployments and coordinated release pipelines
  • ✗Long synchronous HTTP chains where one timeout cascades upstream
  • ✗Unbounded client retries and shared thread/connection pool exhaustion

1. What Makes a Distributed Monolith?

A distributed monolith is not diagnosed by service count or the presence of a single remote HTTP call. A well-designed distributed system can legitimately require a remote decision before proceeding.

Instead, a distributed monolith is diagnosed by clusters of coupled behavior:

Scroll horizontally to see all columns.

Coupling VectorQuestion to InvestigateProduction Failure Manifestation
Runtime CouplingWhich dependencies must answer before an operation completes?Checkout endpoint fails or times out because the notification provider is degraded.
Data CouplingWho owns the database record and enforces its invariants?Orders and Billing both write directly to the same database table, fighting over schemas and lock contention.
Release CouplingCan compatible changes ship independently to production?Renaming or adding a field requires deploying four distinct services simultaneously.
Operational CouplingCan one workload exhaust capacity required by another?Heavy internal analytics queries exhaust database connections required by customer-facing checkout transactions.

Microsoft's Azure architecture guidance emphasizes business capabilities, private data ownership, and independently deployable units as the definitive criteria for microservices. If an organization lacks independent team ownership or has manageable workloads, a clean modular monolith often offers vastly superior operational simplicity, zero network latency, and simple atomic transactions.

A microservice is a deliberate architectural trade-off: justify every separate boundary with concrete organizational autonomy, asymmetric scaling profiles, or strict availability requirements.


2. Follow the User Action, Not the Deployment Diagram

Consider this common, sequentially coupled checkout path:

Architecture Diagram·Mermaid
Rendering architecture diagram...

Notice the deep design flaws in this interaction:

  1. Broken Domain Ownership: Why does the Inventory service orchestrate the Payment workflow?
  2. Cascading Failure Blast Radius: An outage or latency spike in a non-critical notification provider bubbles all the way back to abort the customer's purchase.
  3. Ambiguous State: Did the payment authorization commit before the notification timed out? A network timeout on the return leg is an absence of confirmation, not proof of failure.

The Fallacy of Summing p99 Latencies

Consider an idealized sequential trace:

text
Orders local work:        40 ms
Inventory check:          70 ms
Payment authorization:   180 ms
Email delivery:          120 ms
-------------------------------
Illustrative total:      410 ms + network overhead

In production, latency is a probability distribution, not a static integer. You cannot calculate end-to-end p99 latency by simply summing individual service p99 values (end-to-end p99 does not equal the sum of individual p99s). When multiple remote calls execute sequentially, the chance that at least one call hits its tail latency increases dramatically. Furthermore, if each service boasts 99.9% availability, four chained services yield a significantly degraded composite availability:

text
0.999 × 0.999 × 0.999 × 0.999 ≈ 99.6% composite availability

Every additional synchronous hop on the critical path mechanically erodes both response latency and system availability.


3. Distributed N+1: Chatty Calls Across Service Boundaries

The dreaded N+1 query problem is painful when executed against a local SQL database. It is catastrophic when executed across remote HTTP boundaries.

Imagine an order details screen rendering an order with 100 line items. The backend Order API fetches the order record, and then loops over the items, issuing 100 distinct HTTP calls to the Product catalog service:

Architecture Diagram·Mermaid
Rendering architecture diagram...

Even if the Product service responds in a brisk 10ms with indexed lookups, 100 sequential round-trips add 1,000ms of pure network transit and connection pool contention. Parallelizing them via Task.WhenAll reduces elapsed time at the expense of hammering the downstream service with concurrency spikes.

Compare three robust architectural alternatives:

Scroll horizontally to see all columns.

Architectural StrategyWhen to ApplyTrade-Off / Hidden Cost
Bounded Batch LookupCurrent product details are mandatory at read time.Payload size limits, partial batch failure handling, API complexity.
Event-Maintained Read ModelStated eventual consistency (e.g., 500ms lag) is acceptable.Consumer event ingestion lag, rebuild pipelines, schema evolution.
Purchase-Time Historical SnapshotHistorical order line details must reflect the exact terms agreed upon at checkout.Data denormalization; separating current catalog state from immutable purchase records.

A historical price is not "stale data"—it is the exact legal price frozen at the moment of sale. Modeling purchase snapshots directly in the order aggregate eliminates downstream catalog queries altogether.


4. Protecting the Critical Path: .NET 10 HTTP Resilience Pipelines

When a remote HTTP call must remain synchronous, you must guard it with an explicit failure and concurrency budget.

Using .NET 10 and Microsoft.Extensions.Http.Resilience, we configure a strict, layered resilience handler. This policy limits concurrency, bounds total execution time, enforces per-attempt timeouts, and explicitly prevents dangerous retry amplification:

csharp
builder.Services.AddHttpClient("catalog-read", client =>
{
    client.BaseAddress = new Uri("https://catalog.internal.service/");
    client.Timeout = Timeout.InfiniteTimeSpan; // Controlled by resilience pipeline
})
.AddStandardResilienceHandler(options =>
{
    // 1. Bulkhead Concurrency Limiter: Prevent connection pool exhaustion
    options.RateLimiter.DefaultRateLimiterOptions.PermitLimit = 16;
    options.RateLimiter.DefaultRateLimiterOptions.QueueLimit = 0;
 
    // 2. Cumulative Execution Budget (including retries)
    options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(2);
 
    // 3. Per-Attempt Timeout: Fail fast on slow sockets
    options.AttemptTimeout.Timeout = TimeSpan.FromMilliseconds(500);
 
    // 4. Jittered Exponential Backoff Retry Policy
    options.Retry.MaxRetryAttempts = 1;
    options.Retry.Delay = TimeSpan.FromMilliseconds(100);
    options.Retry.UseJitter = true;
 
    // CRITICAL: Prevent retrying unsafe HTTP methods (POST/PUT/DELETE)
    options.Retry.DisableForUnsafeHttpMethods();
 
    // 5. Circuit Breaker: Stop cascading load during prolonged dependency outages
    options.CircuitBreaker.MinimumThroughput = 20;
    options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(10);
    options.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(5);
});
Resilience Configuration Realities
  • Process-Local Limits: The permit limit of 16 is local to this single process instance; across 20 container replicas, the downstream service may still observe up to 320 concurrent requests.
  • Unsafe HTTP Methods: Standard resilience defaults retry all status codes. Automatically retrying a non-idempotent POST or PATCH on a network blip can cause duplicate payments or side effects. Always invoke DisableForUnsafeHttpMethods().
  • Cooperative Cancellation: Passing a CancellationToken cancels the client's socket wait; it does not inform the remote server to roll back an operation that already began processing.

5. Domain Ownership Belongs in the Data Model

If the Orders service and the Billing service connect directly to the same SQL database and write to the same orders table, the microservice architecture is an illusion. Any database migration, column rename, or index addition requires coordinating both services.

For every service boundary, enforce and document five fundamental invariants:

  1. Sole Aggregate Ownership: Exactly one service owns the database table and enforces business rules.
  2. Authorized Mutation Commands: Other services cannot issue raw UPDATE or INSERT statements; they must submit domain commands via an explicit API.
  3. Encapsulated Invariants: State transitions are verified by domain logic inside the owning service.
  4. Public Integration Facts: External consumers learn of changes by subscribing to published integration events.
  5. Freshness Expectations: Downstream consumer views must declare their tolerance for eventual consistency lag.

In our teaching implementation, we enforce logical separation using separate database contexts (OrdersDb and NotificationsDb). In enterprise production environments, this separation is hardened using distinct database server instances, isolated schemas, and restricted database user credentials.


6. Make the Acceptance Contract Explicit: HTTP 202 and State Machines

When an order arrives, should the API return 200 OK indicating the entire fulfillment lifecycle is complete? Absolutely not.

Per RFC 9110 Section 15.3.3, 202 Accepted communicates that the request has been durably accepted for processing, but processing has not completed. The API returns an order identifier and a status tracking URL.

Architecture Diagram·Mermaid
Rendering architecture diagram...

By decoupling intake from downstream execution, the user-facing API remains blazingly fast and available even if backend notification workers or third-party email providers are undergoing a restart or experiencing high latency.


7. Committing Intent and Events Atomically: The Transactional Outbox

To break runtime coupling without risking message loss or dual-write inconsistencies, we employ the Transactional Outbox Pattern.

The order aggregate and the outbound integration event are written to the same database inside the exact same ACID transaction:

Architecture Diagram·Mermaid
Rendering architecture diagram...

The Outbox Message Entity

csharp
public sealed class OutboxMessage
{
    public Guid Id { get; set; }
    public string Payload { get; set; } = string.Empty;
    public long CreatedUtcTicks { get; set; }
    public long NextAttemptUtcTicks { get; set; }
    public int Attempts { get; set; }
    public long? PublishedUtcTicks { get; set; }
    public long? DeadLetterUtcTicks { get; set; }
    public string? LastError { get; set; }
}

The Atomic Ingestion Endpoint

csharp
app.MapPost("/orders", async (
    CreateOrder request, 
    HttpRequest http,
    OrdersDb db, 
    CancellationToken ct) =>
{
    // 1. Mandatory client idempotency key validation
    if (!Guid.TryParse(http.Headers["Idempotency-Key"], out var key) || key == Guid.Empty)
    {
        return Results.BadRequest(new { error = "A valid UUID Idempotency-Key is required." });
    }
 
    if (string.IsNullOrWhiteSpace(request.Sku) || request.Quantity is < 1 or > 100)
    {
        return Results.BadRequest(new { error = "Sku and quantity are invalid." });
    }
 
    var normalized = new CreateOrder(request.Sku.Trim(), request.Quantity);
    var hash = ComputeHash(JsonSerializer.Serialize(normalized));
    var requestKey = key.ToString("N");
 
    // 2. Check for existing idempotent submission
    var existing = await db.Orders.AsNoTracking()
        .SingleOrDefaultAsync(x => x.RequestKey == requestKey, ct);
 
    if (existing is not null)
    {
        return existing.RequestHash == hash
            ? Results.Accepted($"/orders/{existing.Id}", new { existing.Id, existing.Status })
            : Results.Conflict(new { error = "Idempotency key reuse with mismatched payload." });
    }
 
    // 3. Prepare domain order and integration outbox message
    var order = new Order
    {
        Id = Guid.NewGuid(),
        RequestKey = requestKey,
        RequestHash = hash,
        Sku = normalized.Sku,
        Quantity = normalized.Quantity,
        Status = "AcceptedPendingValidation"
    };
 
    var now = DateTime.UtcNow.Ticks;
    var message = new OrderAcceptedV1(Guid.NewGuid(), order.Id, order.Sku, order.Quantity);
 
    db.Orders.Add(order);
    db.Outbox.Add(new OutboxMessage
    {
        Id = message.EventId,
        Payload = JsonSerializer.Serialize(message),
        CreatedUtcTicks = now,
        NextAttemptUtcTicks = now
    });
 
    try
    {
        // 4. ATOMIC COMMIT: Single database transaction guarantees both or neither persist
        await db.SaveChangesAsync(ct);
    }
    catch (DbUpdateException ex) when (ex.InnerException is SqliteException { SqliteErrorCode: 19 })
    {
        // Handle concurrent race condition where another worker submitted the same key
        db.ChangeTracker.Clear();
        var winner = await db.Orders.AsNoTracking()
            .SingleOrDefaultAsync(x => x.RequestKey == requestKey, ct);
 
        if (winner is null) throw;
 
        return winner.RequestHash == hash
            ? Results.Accepted($"/orders/{winner.Id}", new { winner.Id, winner.Status })
            : Results.Conflict(new { error = "Idempotency key collision." });
    }
 
    return Results.Accepted($"/orders/{order.Id}", new { order.Id, order.Status });
});

For a comprehensive architectural guide to optimizing high-throughput relays with PostgreSQL FOR UPDATE SKIP LOCKED and UUIDv7 indexing, see our dedicated guide to The Dual-Write Trap: Production-Grade Transactional Outbox in .NET 10.


8. Asynchronous Relay Dispatch: Expecting Ambiguous Network Outcomes

An outbox row is a durable commitment to attempt publication—it is not an instant guarantee. The background relay worker pulls due messages, dispatches them over the network, and handles failure modes gracefully:

csharp
private async Task DispatchOneAsync(CancellationToken ct)
{
    using var scope = _serviceScopeFactory.CreateScope();
    var db = scope.ServiceProvider.GetRequiredService<OrdersDb>();
    var now = DateTime.UtcNow.Ticks;
 
    // Claim next due message
    var row = await db.Outbox
        .Where(x => x.PublishedUtcTicks == null 
                 && x.DeadLetterUtcTicks == null 
                 && x.NextAttemptUtcTicks <= now)
        .OrderBy(x => x.CreatedUtcTicks)
        .ThenBy(x => x.Id)
        .FirstOrDefaultAsync(ct);
 
    if (row is null) return;
 
    if (row.Attempts >= MaxAttempts)
    {
        row.DeadLetterUtcTicks = now;
        row.LastError = "Attempt budget exhausted; manual reconciliation required.";
        await db.SaveChangesAsync(ct);
        return;
    }
 
    row.Attempts++;
    await db.SaveChangesAsync(ct); // Persist attempt counter prior to network dispatch
 
    bool permanentFailure = false;
    string? error = null;
 
    try
    {
        using var content = new StringContent(row.Payload, Encoding.UTF8, "application/json");
        using var response = await _httpClient.PostAsync("/internal/events/order-accepted-v1", content, ct);
 
        if (response.IsSuccessStatusCode)
        {
            row.PublishedUtcTicks = DateTime.UtcNow.Ticks;
            row.LastError = null;
        }
        else
        {
            var status = (int)response.StatusCode;
            permanentFailure = status < 500 && status is not 408 and not 429;
            error = $"HTTP {status}";
        }
    }
    catch (HttpRequestException ex) { error = ex.Message; }
    catch (OperationCanceledException) when (!ct.IsCancellationRequested)
    {
        error = "HTTP timeout: receiver outcome indeterminate.";
    }
 
    if (error is not null)
    {
        row.LastError = error;
        if (permanentFailure || row.Attempts >= MaxAttempts)
        {
            row.DeadLetterUtcTicks = DateTime.UtcNow.Ticks;
        }
        else
        {
            // Exponential backoff with jitter
            var backoffSeconds = Math.Min(30, Math.Pow(2, row.Attempts - 1)) + Random.Shared.NextDouble();
            row.NextAttemptUtcTicks = DateTime.UtcNow.AddSeconds(backoffSeconds).Ticks;
        }
 
        _logger.LogWarning("Event {EventId} attempt {Attempt} failed: {Reason}", row.Id, row.Attempts, error);
    }
 
    await db.SaveChangesAsync(ct); // Persist success or next retry timestamp
}

The Critical Two-Generals Uncertainty Window

Why must downstream consumers be strictly idempotent? Look at what happens during network partitions:

Architecture Diagram·Mermaid
Rendering architecture diagram...

Whether the relay timed out waiting for the HTTP response or the process was killed before committing PublishedUtcTicks, the event will be re-sent. Redelivery is not a defect; it is a fundamental property of reliable at-least-once distributed messaging.


9. Deduplicate Receipt and Effect Atomically: The Idempotent Inbox Consumer

The downstream consumer must ensure that receiving duplicate deliveries never results in duplicate business actions (such as sending two confirmation emails or charging two invoices).

csharp
app.MapPost("/internal/events/order-accepted-v1", async (
    OrderAcceptedV1 message,
    NotificationsDb db, 
    CancellationToken ct) =>
{
    if (message.EventId == Guid.Empty || message.OrderId == Guid.Empty)
    {
        return Results.BadRequest();
    }
 
    var hash = ComputeHash(JsonSerializer.Serialize(message));
 
    // 1. Check for existing receipt
    var receipt = await db.Inbox.AsNoTracking()
        .SingleOrDefaultAsync(x => x.EventId == message.EventId, ct);
 
    if (receipt is not null)
    {
        // Duplicate delivery detected: acknowledge safely if payload matches
        return receipt.PayloadHash == hash ? Results.NoContent() : Results.Conflict();
    }
 
    // 2. Stage inbox receipt and pending local job
    db.Inbox.Add(new InboxReceipt { EventId = message.EventId, PayloadHash = hash });
    db.Jobs.Add(new NotificationJob { EventId = message.EventId, OrderId = message.OrderId });
 
    try
    {
        // 3. ATOMIC COMMIT: Deduplication record and business effect commit together
        await db.SaveChangesAsync(ct);
    }
    catch (DbUpdateException ex) when (ex.InnerException is SqliteException { SqliteErrorCode: 19 })
    {
        // Concurrent delivery race resolved by database unique key constraint
        db.ChangeTracker.Clear();
        var winner = await db.Inbox.AsNoTracking()
            .SingleOrDefaultAsync(x => x.EventId == message.EventId, ct);
 
        if (winner is null) throw;
 
        return winner.PayloadHash == hash ? Results.NoContent() : Results.Conflict();
    }
 
    return Results.NoContent();
});
The Idempotency Boundary

Recording the inbox receipt guarantees that the local NotificationJob is created exactly once. However, the guarantee ends at your database boundary. Once a worker calls an external provider (Twilio, SendGrid, FCM), the ambiguous return-path problem recurs. Always leverage third-party provider idempotency keys (Idempotency-Key headers or client mutation IDs) to secure the final hop.


10. Multi-Replica Relays, Row Claiming, and Ordering Contracts

The code above demonstrates the protocol for a single relay instance. When deploying multiple background worker replicas in production Kubernetes environments, several advanced operational patterns become mandatory:

  1. Non-Blocking Row Claiming: Concurrent relays querying NextAttemptUtcTicks <= now will experience lock contention and deadlocks unless you use database-level non-blocking query hints (such as PostgreSQL's FOR UPDATE SKIP LOCKED or SQL Server's WITH (UPDLOCK, READPAST)).
  2. Lease Expiry and Fencing: If a worker claims a batch of 50 rows and stalls during an intensive Garbage Collection pause, another worker may assume the lease has expired. Fencing tokens or row version checks ensure a zombie worker cannot overwrite newer state.
  3. Partitioned Ordering: Sorting by CreatedUtcTicks does not provide global causal ordering across distributed nodes. If domain updates to an order aggregate must be consumed in strict chronological order, group events by partition key (e.g., OrderId) and serialize processing within each partition.

11. Running and Verifying the .NET 10 Companion Lab

The companion implementation targets .NET 10 using SQLite for portable, zero-infrastructure local execution:

xml
<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
 
  <ItemGroup>
    <PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="10.0.0" />
    <PackageReference Include="Microsoft.Extensions.Http.Resilience" Version="10.0.0" />
  </ItemGroup>
</Project>

Running the Service

powershell
# Restore and build the application
dotnet build
 
# Launch the service on port 5179
dotnet run --urls http://127.0.0.1:5179

Executing Idempotent Requests via PowerShell

powershell
$orderKey = [guid]::NewGuid().ToString()
$orderBody = @{ sku = 'BOOK-001'; quantity = 2 } | ConvertTo-Json
 
# 1. Initial Order Submission (Returns HTTP 202 Accepted)
$order1 = Invoke-RestMethod -Method Post -Uri 'http://127.0.0.1:5179/orders' `
    -Headers @{ 'Idempotency-Key' = $orderKey } `
    -ContentType 'application/json' -Body $orderBody
$order1
 
# 2. Resubmission with Same Key & Payload (Returns Same Order ID, Idempotent)
$order2 = Invoke-RestMethod -Method Post -Uri 'http://127.0.0.1:5179/orders' `
    -Headers @{ 'Idempotency-Key' = $orderKey } `
    -ContentType 'application/json' -Body $orderBody
$order2

Simulating Lost Acknowledgments and Self-Healing

powershell
# Start an instance configured to inject a simulated dropped ACK on the first dispatch
dotnet run --urls http://127.0.0.1:5179 --Relay:SimulateLostAckOnce true

The test suite validates 12 critical distributed invariants:

  • Rejection of invalid UUID idempotency keys and out-of-range quantities.
  • Six concurrent identical requests create exactly one order and one outbox row.
  • Idempotency key reuse with modified payload returns 409 Conflict.
  • Outbox database errors automatically roll back the order insert.
  • Consumer downtime does not block durable order acceptance.
  • Pending outbox entries resume dispatch upon process restart.
  • Dropped acknowledgments trigger redelivery without creating duplicate consumer jobs.
  • Exhausted attempt budgets quarantine corrupted messages into dead-letter state without silent loss.

12. Observability Across Service Boundaries

Decoupling services through an outbox eliminates runtime blocking, but introduces new asynchronous operational states that must be actively monitored:

Scroll horizontally to see all columns.

Telemetry SignalArchitectural Question AnsweredAction Threshold
Oldest Unpublished Outbox AgeIs the background relay stalled or falling behind incoming traffic?Alert if oldest message age exceeds 60 seconds.
Relay Retry & Quarantine RateAre downstream dependencies rejecting payloads or throwing errors?Alert on any messages transitioned to Dead Letter.
Consumer Inbox Duplicate CountIs network instability causing excessive redelivery waves?Track ratio of duplicate receipts to total message volume.
Pending Order LagAre business fulfillment workflows failing to reach terminal states?Alert if orders remain in AcceptedPendingValidation beyond SLA.

Propagate W3C distributed trace context (traceparent and tracestate) across HTTP calls and outbox payloads. Use OpenTelemetry ActivitySource in both the relay and the consumer, creating trace links to connect asynchronous jobs back to the original client request without creating unmanageably long traces.

For an end-to-end guide to instrumentation, exemplars, and telemetry security in .NET 10, read our guide to Observability in .NET 10: You Can't Fix What You Can't See.


13. Incremental Migration: Redesigning One Boundary at a Time

Do not attempt a massive "big-bang" architectural rewrite of twelve services. Isolate and migrate one critical boundary at a time:

  1. Map the Critical User Journey: Draw the end-to-end request path, cataloging all synchronous HTTP calls, database connections, and external third-party APIs.
  2. Classify Dependencies: Categorize each dependency as a Mandatory Immediate Decision (e.g., fraud check, funds authorization), an Optional Enrichment (e.g., recommended items), or a Deferrable Side Effect (e.g., welcome email, analytics emission).
  3. Formulate Explicit Acceptance Contracts: Replace blocking synchronous workflows for deferrable side effects with 202 Accepted and trackable status representations.
  4. Establish Strict Data Boundaries: Audit shared tables. Assign unambiguous ownership of the aggregate to a single service and eliminate cross-service database access.
  5. Implement Local Transactional Outboxes: Commit business state and outbound integration events atomically within the service's database boundary.
  6. Deploy Idempotent Consumer Inboxes: Ensure all message consumers safely handle at-least-once redelivery by deduplicating event IDs.
  7. Simulate Failures Under Load: Intentionally inject network partitions, dropped acknowledgments, and consumer crashes to verify system self-healing.
  8. Monitor Latency and Lag Metrics: Validate that p99 response times drop and that critical checkout availability remains isolated from downstream provider downtime.

Architecture Review Checklist

Before splitting your next service or introducing a new remote endpoint, demand answers to these nine architectural questions:

  • Business Capability: What distinct business domain aggregate does this service exclusively own?
  • Independent Deployability: Can this service deploy a schema update or feature to production without requiring other services to deploy concurrently?
  • Critical Path Dependency: Which customer-facing operations depend on this service responding synchronously?
  • Failure Budget: What timeout, concurrency limit, and circuit breaker settings protect upstream services when this endpoint degrades?
  • Network Ambiguity: What happens when this service successfully processes a command, but the network connection drops before the caller receives the response?
  • Atomic Handoff: What transactional mechanism ensures domain mutations and outgoing messages commit together?
  • Idempotency Scope: Where are duplicate incoming requests detected, and what exact side effect is protected?
  • Explicit State Machine: How does the API communicate Accepted, Confirmed, Rejected, and NeedsReconciliation states to clients?
  • Dead-Letter Operations: Who is alerted when retry budgets are exhausted, and what is the runbook for replaying quarantined messages?

A service boundary that answers these questions creates true architectural autonomy. A service boundary that ignores them simply moves your monolith's method calls onto an unreliable network.


Primary Technical References

Rehan Akbar

Written by Rehan Akbar

Software Engineer · Singapore · .NET, Cloud & Applied AI

I write about the engineering trade-offs behind reliable production systems—Clean Architecture, distributed resilience, observability, and practical AI engineering. Open for architecture advisory and technical collaborations.

Discussion & Comments

Share your thoughts, ask technical questions, or discuss architecture edge cases.