Observability in .NET 10: You Can't Fix What You Can't See

A production API can return 200 OK on every health check and still deliver a disastrous user experience.

Requests gradually slow down. Downstream payment gateways begin timing out under peak load. Unindexed database queries quietly consume connection pool slots. Error rates spike on a single tenant endpoint while aggregate dashboards stay soothingly green.

The hard part of engineering distributed systems is rarely writing the bugfix.

The hard part is locating exactly where the failure or latency originated.

The Four Observability Axioms

Instrument the dependencies, not just the entry point. Basic ASP.NET Core and HttpClient instrumentation leaves a massive blind spot right where database and caching bottlenecks occur.
Instrument the failure and cancellation paths. Most sample code records durations only on happy paths, discarding telemetry on the exact failed or aborted requests you need to investigate.
Enable exemplars. They are the automated link that jumps directly from a p99 latency spike on a metric chart to the exact distributed trace that caused it.
Treat telemetry as production data. Unfiltered distributed traces and structured logs frequently become an unencrypted, unredacted shadow copy of your production database with weaker access controls.

Investigation Flow (8.7s API Stall)Same data, fewer tools

BeforeGrep raw logs, cross-reference timestamps by hand

OptimizedMetric → Exemplar → Trace → Span Logs, one click each step

Correlating metrics to distributed traces via Exemplars lets engineers click a p99 histogram spike, inspect the 4.8s database span, and view isolated query logs without cross-referencing unsynchronized log stores. The exact time saved depends on your incident, your tooling, and how good your dashboards already are — the point is the number of manual steps, not a fixed multiplier.

A Production Problem: The 8.7-Second "200 OK"

Consider a routine e-commerce endpoint:

http

GET /api/orders/12345

A customer support escalation lands in your team's Slack channel:

"Checkout and order status pages are taking 8 to 10 seconds to load."

You check the application logs in your log aggregation tool:

text

[14:22:01 INF] Request started: GET /api/orders/12345
[14:22:09 INF] Request completed: 200 OK (8721 ms)

Nothing threw an unhandled exception. The database is reachable, the Redis cache is healthy, external shipping APIs are responsive, and container health probes are returning 200 OK.

What actually happened during those 8.7 seconds?

flowchart LR
    Client["React / Browser Client"] --> API["ASP.NET Core API<br/>(8.7s total)"]
    API --> DB["PostgreSQL / SQL Server<br/>(4.8s unindexed query)"]
    API --> External["External Shipping API<br/>(2.7s network latency)"]
    API --> Cache["Redis Cache<br/>(12ms lookup)"]
    API --> App["App Serialization<br/>(0.9s processing)"]

Architecture Diagram·Mermaid

Rendering architecture diagram...

The real time breakdown across the distributed boundary reveals the problem immediately:

text

GET /api/orders/12345                 8.7s  [200 OK]
├── Database (Orders Query)           4.8s  (Missing composite index)
├── External Shipping Gateway API     2.7s  (HTTP retry backoff)
├── In-Memory Serialization           0.9s  (Large JSON payload)
└── Middleware & Framework Overhead   0.3s

Without distributed tracing, diagnosing this involves guesswork, adding speculative log lines, and deploying ad-hoc patches. With an integrated observability pipeline, the bottleneck is self-evident in seconds.

Observability Is More Than Logging

A common misconception in backend engineering is:

"We write detailed log statements to Elasticsearch or Seq, so our system is fully observable."

Logs are necessary, but they represent only one corner of the telemetry triangle.

Scroll horizontally to see all columns.

Signal	Primary Question Answered	Data Characteristics
Logs	What happened at a specific instant?	Discrete textual/structured events with rich contextual metadata.
Metrics	How is the system behaving over time?	Aggregated numerical timeseries counters, gauges, and histograms.
Traces	Where did a single request spend its execution time?	Directed acyclic graphs of causal spans spanning process boundaries.

The fundamental breakdown in engineering teams during high-severity incidents is having three disconnected tools: a Grafana dashboard for metrics, a separate portal for logs, and a standalone APM for traces. Correlating across disparate timestamps and unaligned clocks burns precious MTTR (Mean Time to Resolution).

What you need is a unified correlation pipeline:

Unified Telemetry Correlation Path

1. Metric AlertDetection

Histogram detects p99 latency surge on /api/orders.

›

2. Trace ExemplarCorrelation

Exemplar attaches TraceId directly to the metric spike.

›

3. Trace WaterfallIsolation

Identifies the exact 4.8s database span and query parameters.

What .NET 10 Brings to Observability

The .NET runtime provides native, zero-dependency diagnostics APIs built into the Base Class Library (BCL):

ILogger<T> for structured application logs.
System.Diagnostics.Metrics (Meter, Counter, Histogram) for high-throughput metrics.
System.Diagnostics.ActivitySource and Activity for distributed tracing spans.

OpenTelemetry .NET bridges these native BCL primitives to the vendor-neutral OTLP (OpenTelemetry Protocol) wire format, eliminating vendor SDK lock-in from your application codebase.

What's New in .NET 10 Diagnostics:

Native Framework Security Metrics: .NET 10 introduces built-in framework meters for Microsoft.AspNetCore.Authentication, Microsoft.AspNetCore.Authorization, Microsoft.AspNetCore.Identity, and Microsoft.AspNetCore.MemoryPool. Authorization attempts, sign-in outcomes, and password-check failures are now counters you can graph and alert on directly, instead of something you had to instrument by hand. They're a useful signal for anomalies like a spike in failed sign-ins — not a credential-stuffing detector on their own, and they belong alongside a real detection system, not instead of one.
Telemetry Schema URLs: ActivitySource and Meter constructors support telemetry schema URLs and the ActivitySourceOptions configuration model, aligning .NET diagnostics with OpenTelemetry semantic convention versioning.
Enhanced Out-of-Process Serialization: ActivityLink and ActivityEvent instances are fully serialized through the Microsoft-Diagnostics-DiagnosticSource EventSource provider for external diagnostic agents.
Standardized W3C Trace Context: The default propagator strictly adheres to the W3C traceparent and tracestate standards across all distributed service boundaries.

The End-to-End Telemetry Architecture

flowchart LR
    subgraph AppProcess ["ASP.NET Core (.NET 10 Service)"]
        Logs["ILogger<T>"]
        Metrics["System.Diagnostics.Metrics"]
        Traces["ActivitySource / Activity"]
        OTel["OpenTelemetry .NET SDK"]
 
        Logs --> OTel
        Metrics --> OTel
        Traces --> OTel
    end
 
    OTel -->|OTLP gRPC / Protobuf| Collector["OpenTelemetry Collector<br/>(Tail Sampling + Redaction)"]
 
    subgraph Storage ["Observability Backends"]
        Collector --> Prometheus["Prometheus / Mimir (Metrics)"]
        Collector --> Tempo["Grafana Tempo / Jaeger (Traces)"]
        Collector --> Loki["Grafana Loki / Seq (Logs)"]
    end
 
    Prometheus --> Dashboard["Unified Grafana Dashboards & Alerts"]
    Tempo --> Dashboard
    Loki --> Dashboard

Architecture Diagram·Mermaid

Rendering architecture diagram...

This decoupled architecture guarantees that your .NET code remains vendor-agnostic. The OpenTelemetry Collector acts as a security boundary, policy enforcement layer, and redaction proxy.

1. High-Performance Structured Logging

Never use string interpolation inside logging invocations. Interpolated strings allocate memory on the heap and flatten rich data into unstructured text:

csharp

// ❌ ANTI-PATTERN: Allocates memory and generates an unindexable string
logger.LogInformation($"Order {orderId} processed in {durationMs}ms for user {userId}");

Use semantic message templates with named placeholders:

csharp

// ✅ CLEAN PATTERN: Zero string allocation if LogLevel disabled; queryable properties
logger.LogInformation(
    "Order {OrderId} processed in {DurationMs}ms for user {UserId}",
    orderId,
    durationMs,
    userId);

High-Throughput Source Generators

For performance-critical code paths (such as hot API pipelines processing thousands of requests per second), use the compile-time logging source generator ([LoggerMessage]):

csharp

public static partial class OrderLogMessages
{
    [LoggerMessage(
        EventId = 1001,
        Level = LogLevel.Information,
        Message = "Order {OrderId} processed in {DurationMs}ms with outcome {Outcome}")]
    public static partial void OrderProcessed(
        this ILogger logger,
        long orderId,
        double durationMs,
        string outcome);
 
    [LoggerMessage(
        EventId = 1002,
        Level = LogLevel.Warning,
        Message = "Order {OrderId} lookup failed: record not found")]
    public static partial void OrderNotFound(
        this ILogger logger,
        long orderId);
}

The C# compiler emits a zero-allocation, strongly typed delegate that caches log templates and skips argument boxing entirely.

What to Never Log

Never record raw authentication tokens, passwords, credit card numbers, national identifiers, or complete Authorization header values. Before adding any log statement, ask: "Does this log provide operational insight without creating an unencrypted copy of sensitive customer data?"

2. Metrics: Measuring Population Behavior Over Time

While logs capture individual transactions, metrics aggregate behavior across the entire population.

Prefer `IMeterFactory` over Static `Meter`

Static meter singletons create cross-test pollution during parallel unit/integration test runs. Since .NET 8, the recommended enterprise pattern is injecting IMeterFactory:

csharp

using System.Diagnostics.Metrics;
 
public sealed class OrderMetrics
{
    public const string MeterName = "Enterprise.Orders.Api";
 
    private readonly Counter<long> _ordersProcessedCounter;
    private readonly Histogram<double> _orderProcessingDurationHistogram;
 
    public OrderMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create(MeterName);
 
        _ordersProcessedCounter = meter.CreateCounter<long>(
            name: "orders.processed.count",
            unit: "{order}",
            description: "Total number of orders processed.");
 
        // OpenTelemetry semantic conventions specify seconds for durations.
        _orderProcessingDurationHistogram = meter.CreateHistogram<double>(
            name: "orders.processing.duration",
            unit: "s",
            description: "Duration of order processing operations in seconds.");
    }
 
    public void OrderProcessed(string outcome) =>
        _ordersProcessedCounter.Add(1, new KeyValuePair<string, object?>("outcome", outcome));
 
    public void RecordDuration(TimeSpan elapsed, string outcome) =>
        _orderProcessingDurationHistogram.Record(
            elapsed.TotalSeconds,
            new KeyValuePair<string, object?>("outcome", outcome));
}

csharp

builder.Services.AddMetrics();
builder.Services.AddSingleton<OrderMetrics>();

Why Histograms Beat Averages Every Time

Look at this telemetry summary:

text

Average Response Time: 115 ms

It looks exceptional. Now inspect the percentile distribution:

text

p50 (Median):    65 ms
p90:            120 ms
p95:            240 ms
p99:          4,850 ms  <-- 1 out of 100 users experiences a 5-second hang!

Averages hide tail-latency anomalies. Histograms preserve the full mathematical distribution, allowing you to configure alerts on p95 and p99 SLOs (Service Level Objectives).

3. Distributed Tracing with `ActivitySource`

In .NET, distributed traces consist of spans represented by System.Diagnostics.Activity, created via ActivitySource.

csharp

using System.Diagnostics;
 
public static class Diagnostics
{
    public const string ActivitySourceName = "Enterprise.Orders.Api";
 
    public static readonly ActivitySource ActivitySource = new(
        new ActivitySourceOptions(ActivitySourceName)
        {
            Version = "1.0.0",
            // .NET 10 OpenTelemetry Semantic Conventions Schema
            TelemetrySchemaUrl = "https://opentelemetry.io/schemas/1.30.0"
        });
}

Semantic Tagging and Activity Kinds

csharp

using var activity = Diagnostics.ActivitySource.StartActivity(
    name: "OrderService.ProcessOrder",
    kind: ActivityKind.Internal);
 
activity?.SetTag("order.id", orderId);
activity?.SetTag("order.item_count", items.Count);

Always Guard Activity Calls with Optional Chaining (?.)

When no trace listener or exporter is actively collecting from your ActivitySource, StartActivity() returns null. Always invoke methods using activity?.SetTag() to prevent NullReferenceException.

4. Production-Grade Service Instrumentation

Here is a complete, production-hardened service implementation that handles cancellations, exceptions, durations, and exemplars correctly:

csharp

using System.Diagnostics;
using Microsoft.EntityFrameworkCore;
 
public sealed class OrderService
{
    private readonly OrdersDbContext _db;
    private readonly HttpClient _httpClient;
    private readonly OrderMetrics _metrics;
    private readonly ILogger<OrderService> _logger;
 
    public OrderService(
        OrdersDbContext db,
        HttpClient httpClient,
        OrderMetrics metrics,
        ILogger<OrderService> logger)
    {
        _db = db;
        _httpClient = httpClient;
        _metrics = metrics;
        _logger = logger;
    }
 
    public async Task<OrderDetails?> GetOrderAsync(
        long orderId,
        CancellationToken cancellationToken)
    {
        using var activity = Diagnostics.ActivitySource.StartActivity(
            "OrderService.GetOrder",
            ActivityKind.Internal);
 
        activity?.SetTag("order.id", orderId);
 
        var start = Stopwatch.GetTimestamp();
        var outcome = "success";
 
        try
        {
            var order = await _db.Orders
                .AsNoTracking()
                .Where(x => x.Id == orderId)
                .Select(x => new OrderDetails
                {
                    Id = x.Id,
                    Status = x.Status,
                    TotalAmount = x.TotalAmount
                })
                .SingleOrDefaultAsync(cancellationToken);
 
            if (order is null)
            {
                activity?.SetTag("order.found", false);
                outcome = "not_found";
                _logger.OrderNotFound(orderId);
                return null;
            }
 
            activity?.SetTag("order.found", true);
 
            // Dispose HttpResponseMessage promptly to return socket to pool
            using var response = await _httpClient.GetAsync(
                $"/shipping/{orderId}",
                cancellationToken);
 
            response.EnsureSuccessStatusCode();
 
            return order;
        }
        catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
        {
            // Client disconnect or timeout is NOT a server 500 error
            outcome = "cancelled";
            activity?.SetTag("cancellation.reason", "client_aborted");
            throw;
        }
        catch (Exception ex)
        {
            outcome = "error";
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
 
            // Native .NET 9+ BCL method (supersedes legacy OTel RecordException)
            activity?.AddException(ex);
            throw;
        }
        finally
        {
            // Guaranteed measurement across all happy, failed, and aborted execution paths
            var elapsed = Stopwatch.GetElapsedTime(start);
            _metrics.RecordDuration(elapsed, outcome);
            _metrics.OrderProcessed(outcome);
        }
    }
}

Production-Hardened Pattern

✓Records duration in finally block, capturing timeouts and failed calls
✓Uses Stopwatch.GetElapsedTime() without allocating Stopwatch instances
✓Marks Activity status as Error and calls Activity.AddException(ex)
✓Categorizes OperationCanceledException separately from server crashes
✓Restricts metric tag dimensions to bounded string sets ('success', 'error')

Naive Anti-Pattern

✗Measures duration only at end of try block, losing all failure metrics
✗Instantiates new Stopwatch() on every request, adding GC pressure
✗Leaves Activity status as Unset (200 OK) when downstream calls throw
✗Increments 500 Internal Error counters when mobile clients disconnect
✗Tags metrics with customerId, causing catastrophic cardinality explosion

5. Complete OpenTelemetry Setup in ASP.NET Core

Add the required NuGet packages to your project:

bash

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Instrumentation.Runtime
dotnet add package OpenTelemetry.Instrumentation.EntityFrameworkCore

Configure OpenTelemetry logging, tracing, metrics, and exemplars in Program.cs:

csharp

using System.Diagnostics;
using OpenTelemetry;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
 
var builder = WebApplication.CreateBuilder(args);
 
const string ServiceName = "Enterprise.Orders.Api";
 
builder.Services.AddMetrics();
builder.Services.AddSingleton<OrderMetrics>();
 
// 1. Configure OpenTelemetry Core
builder.Services
    .AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(
            serviceName: ServiceName,
            serviceVersion: typeof(Program).Assembly.GetName().Version?.ToString() ?? "1.0.0")
        .AddAttributes(new Dictionary<string, object>
        {
            ["deployment.environment.name"] = builder.Environment.EnvironmentName,
            ["host.name"] = Environment.MachineName
        }))
    .WithTracing(tracing =>
    {
        tracing
            .AddAspNetCoreInstrumentation(options =>
            {
                // Filter out synthetic health checks to reduce trace ingestion noise
                options.Filter = httpContext =>
                    !httpContext.Request.Path.StartsWithSegments("/health");
            })
            .AddHttpClientInstrumentation(options =>
            {
                // Strip sensitive query strings containing tokens from client spans
                options.EnrichWithHttpRequestMessage = (activity, httpRequest) =>
                {
                    activity.SetTag(
                        "url.full",
                        httpRequest.RequestUri?.GetLeftPart(UriPartial.Path));
                };
            })
            .AddEntityFrameworkCoreInstrumentation(options =>
            {
                // Keep raw SQL parameter text disabled to prevent PII leakage
                options.SetDbStatementForText = false;
            })
            .AddSource(Diagnostics.ActivitySourceName)
            // Head-based parent sampling: samples 10% of new root traces
            .SetSampler(new ParentBasedSampler(new TraceIdRatioBasedSampler(0.10)))
            .AddOtlpExporter();
    })
    .WithMetrics(metrics =>
    {
        metrics
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddRuntimeInstrumentation()
            // .NET 10 Built-in Security & Infrastructure Meters
            .AddMeter("Microsoft.AspNetCore.Authentication")
            .AddMeter("Microsoft.AspNetCore.Authorization")
            .AddMeter("Microsoft.AspNetCore.Identity")
            .AddMeter("Microsoft.AspNetCore.MemoryPool")
            .AddMeter(OrderMetrics.MeterName)
            // Enable Exemplars to bridge metric histogram buckets to active TraceIds
            .SetExemplarFilter(ExemplarFilterType.TraceBased)
            .AddOtlpExporter();
    });
 
// 2. OpenTelemetry Structured Logging Provider
builder.Logging.AddOpenTelemetry(logging =>
{
    logging.IncludeScopes = true;
    logging.IncludeFormattedMessage = true;
    logging.AddOtlpExporter();
});
 
var app = builder.Build();
 
app.MapGet("/health", () => Results.Ok(new { status = "healthy", timestamp = DateTime.UtcNow }));
 
app.Run();

6. Exemplars: Bridging Metrics Directly to Traces

Exemplars represent the most critical link in modern observability.

Without exemplars, when your dashboard shows a sudden spike in p99 latency on /api/orders, you have to guess which trace in your APM backend corresponds to that spike.

When you enable .SetExemplarFilter(ExemplarFilterType.TraceBased), the OpenTelemetry SDK captures the active sampled TraceId and SpanId alongside the recorded latency value inside the histogram bucket.

text

Grafana Metric Dashboard: orders.processing.duration [p99 = 4.8s]
      │
      └── [Click Exemplar Dot in Histogram Bucket]
            │
            ▼
Distributed Trace Waterfall (TraceId: 4bf92f3577b34da6a3ce929d0e0e4736)
      ├── ASP.NET Core GET /api/orders/12345        [8.7s]
      │     ├── Npgsql ExecuteReader (Orders)       [4.8s]  <-- Root Bottleneck!
      │     └── HttpClient GET /shipping/12345      [2.7s]
      └── Correlated Logs for TraceId (6 log lines with exact SQL query)

7. Head Sampling vs. Tail Sampling

Collecting 100% of distributed traces in high-volume production environments is cost-prohibitive.

Head Sampling (In-Process)

Evaluated at span creation inside your .NET process.
Lightweight, zero memory overhead on the application.
Limitation: Outcome-blind. At the start of a request, you do not know whether the request will succeed in 10ms or fail after 10 seconds.

Tail Sampling (Collector-Level)

Evaluated in the OpenTelemetry Collector after the entire trace graph completes.
Keeps 100% of errors, 100% of traces exceeding latency thresholds (e.g. > 2000ms), and 2% of routine healthy traffic.

yaml

# otel-collector-config.yaml
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 10000
    expected_new_traces_per_sec: 2000
    policies:
      - name: capture-all-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
 
      - name: capture-slow-requests
        type: latency
        latency: { threshold_ms: 2000 }
 
      - name: probabilistic-healthy-sample
        type: probabilistic
        probabilistic: { sampling_percentage: 2.0 }

8. Telemetry Security & Cardinality Defense

Telemetry is sensitive production data. In many enterprise environments, telemetry pipelines inadvertently create unencrypted replicas of sensitive customer databases.

Telemetry Security Checklist

Disable Raw SQL Text: Ensure SetDbStatementForText = false in EF Core and SqlClient instrumentation so customer PII and query literals are never exported to trace stores.
Sanitize Query Strings: Strip access tokens, session tokens, and passwords from outbound HttpClient URLs.
Collector-Side Redaction: Enforce attribute deletion at the OpenTelemetry Collector layer as defense-in-depth against accidental application-side leaks.
Strict Cardinality Controls: Metric dimensions must strictly come from closed enum-like sets. Never use user IDs, email addresses, or unvalidated URLs as metric labels.

Defense-in-Depth Collector Redaction

yaml

# Redact sensitive attributes at the collector before storage
processors:
  attributes/redact:
    actions:
      - key: customer.email
        action: delete
      - key: http.request.header.authorization
        action: delete
      - key: db.statement
        action: hash

Summary Checklist for Production Observability

flowchart TD
    A["Production Observability in .NET 10"] --> B["Structured Logs"]
    A --> C["Metrics & Exemplars"]
    A --> D["Distributed Tracing"]
    A --> E["Security & Governance"]
 
    B --> B1["Compile-time [LoggerMessage]"]
    B --> B2["Automatic TraceId correlation"]
 
    C --> C1["IMeterFactory with closed dimensions"]
    C --> C2["TraceBased Exemplar filtering"]
 
    D --> D1["ActivitySource with schema URL"]
    D --> D2["EF Core + HttpClient spans"]
 
    E --> E1["Tail sampling in OTel Collector"]
    E --> E2["PII / Token sanitization"]

Architecture Diagram·Mermaid

Rendering architecture diagram...

True observability is not about generating voluminous data—it is about transforming telemetry into immediate, actionable answers. By pairing .NET 10's native diagnostic primitives with OpenTelemetry and Exemplars, your engineering team can isolate and remediate production incidents in seconds rather than hours.

Observability in .NET 10: You Can't Fix What You Can't See

A Production Problem: The 8.7-Second "200 OK"

Observability Is More Than Logging

What .NET 10 Brings to Observability

What's New in .NET 10 Diagnostics:

The End-to-End Telemetry Architecture

1. High-Performance Structured Logging

High-Throughput Source Generators

2. Metrics: Measuring Population Behavior Over Time

Prefer `IMeterFactory` over Static `Meter`

Why Histograms Beat Averages Every Time

3. Distributed Tracing with `ActivitySource`

Semantic Tagging and Activity Kinds

4. Production-Grade Service Instrumentation

5. Complete OpenTelemetry Setup in ASP.NET Core

6. Exemplars: Bridging Metrics Directly to Traces

7. Head Sampling vs. Tail Sampling

Head Sampling (In-Process)

Tail Sampling (Collector-Level)

8. Telemetry Security & Cardinality Defense

Defense-in-Depth Collector Redaction

Summary Checklist for Production Observability

References & Official Documentation

Written by Rehan Akbar

Discussion & Comments