Production-Grade Java Metrics: Histograms, Percentiles, and SLOs with Micrometer and OpenTelemetry

Most Java backends ship metrics that confidently tell the wrong story. The average latency looks fine, dashboards look calm, and yet customers still hit timeouts and retries. The root cause is usually not "Java is slow." It is that the service measured the wrong thing, or measured the right thing in a way that cannot survive production traffic.

As teams migrate toward OpenTelemetry while still using Micrometer in Spring Boot, a few defaults can quietly make numbers misleading: missing histograms, ad-hoc percentiles, high-cardinality tags, counters that are hard to aggregate, and dashboards that hide tail latency.

This guide is a practical playbook to instrument Java services so latency, throughput, and error metrics reflect what users actually feel. We will use Micrometer and OpenTelemetry together, enable histogram-based percentiles, pick safe tag boundaries, and wire exemplars so a p99 spike can lead to a representative trace.

The Java community keeps moving toward better measurement patterns, including more discussion around metrics in Quarkus and OpenTelemetry. That is useful momentum, but the work still has to happen inside each service.

TL;DR

• Use histograms for latency. Averages hide tail behavior, while histogram-based p95 and p99 values better reflect user pain.
• Publish percentile histograms from Micrometer and compute percentiles in the backend. Avoid relying on client-side percentiles for service-level aggregation.
• Set explicit SLO buckets and expected min/max values for timers and distribution summaries.
• Keep tags low-cardinality and stable. Ban user IDs, session IDs, UUIDs, raw URLs, exception messages, and free-text labels.
• Export Micrometer metrics to OTLP when you want vendor-neutral pipelines.
• Wire exemplars so latency spikes can connect to traces during incident triage.
• Review every new metric like production code. A bad metric can create cost, alert fatigue, and false confidence.

Why It Matters

Good metrics drive real decisions: autoscaling thresholds, SLO alerts, release confidence, incident triage, and capacity planning.

If your service averages away tail latency, the dashboard can look healthy while users experience timeouts. If one feature adds unbounded labels, your observability bill and query latency can explode. If percentiles are calculated in the wrong place, they may look precise while being impossible to aggregate across pods or regions.

Histograms, percentiles, and safe tags give teams stable, queryable, and explainable signals. Micrometer and OpenTelemetry provide the building blocks; the hard part is choosing defaults that survive production.

This fits naturally with the production checklist mindset in Java and Spring Boot Production Checklist and the operational habits in Building Calm Backend Systems.

The Practical Problem Engineers Need To Solve

Java teams need latency metrics that:

• Reflect user impact with p95 and p99, not only averages.
• Aggregate across instances, pods, nodes, regions, and releases.
• Support SLO burn-rate alerts.
• Connect spikes to traces for root cause analysis.
• Stay affordable and queryable as traffic grows.

At the same time, teams must avoid:

• High-cardinality tags that explode time series.
• Client-side percentiles that cannot be meaningfully aggregated.
• Ambiguous units.
• Missing histogram buckets near the actual SLO.
• Raw paths, user IDs, or exception messages in labels.

The goal is not "add more metrics." The goal is to add metrics that help engineers make better decisions under pressure.

Decision Framework: Which Metric, Which Aggregation, Which Tags

Pick the metric shape intentionally, then keep it consistent.

Throughput

Use counters for events that only go up: requests handled, messages consumed, orders shipped, retries attempted.

Counters should use stable tags such as service, route template, outcome, method, or region. Do not add user-specific or request-specific values.

Latency

Use timers with percentile histograms and SLO buckets.

Compute p95 and p99 in the metrics backend from histogram buckets. Avoid relying on client-side percentiles when you need aggregation across pods.

Payload Sizes and Queue Lag

Use distribution summaries with expected minimum and maximum values. This keeps buckets reasonable and prevents pathological values from distorting your data.

Gauges

Use gauges for sampled state such as thread pool size, queue depth, connection pool usage, and memory. Do not model events as gauges.

A Safe Workflow Teams Can Adopt This Week

Start with a small standard, then apply it everywhere.

1. Use Spring Boot and Micrometer as the local facade.
2. Export metrics through OTLP for a vendor-neutral pipeline.
3. Enable histogram-based latency for primary HTTP server and client metrics.
4. Define SLO buckets near real user-facing thresholds.
5. Adopt a tag budget.
6. Use route templates instead of raw URLs.
7. Group exceptions into coarse outcomes.
8. Configure exemplars so traces can explain p99 spikes.

A useful tag budget might be:

• Maximum 8 tags per metric.
• Maximum 50 distinct values per tag per service and region.
• No raw identifiers.
• No raw paths or query strings.
• No exception messages.
• No free-form labels from request input.

This does not have to be perfect on day one. The important move is making the rule explicit and reviewable.

Step-by-Step Implementation

There are two common paths: Micrometer-first with Spring Boot defaults, and OpenTelemetry-first with manual instrumentation or the Java agent. You can use both safely if you keep metric names and tags disciplined.

1. Configure Spring Boot and Micrometer

Start with histogram-based HTTP metrics and an OTLP exporter.

management:
  endpoints:
    web:
      exposure:
        include: health,metrics,prometheus
  metrics:
    tags:
      service: orders-api
      env: prod
    distribution:
      percentiles-histogram:
        http.server.requests: true
      slo:
        http.server.requests: 50ms,100ms,200ms,400ms,800ms,1500ms,3000ms
    enable:
      jvm: true
      logback: true
      http.client.requests: true
      http.server.requests: true
 
management.otlp.metrics.export:
  url: http://otel-collector:4318/v1/metrics
  step: 10s

This publishes histogram buckets for http.server.requests so the backend can calculate p95 and p99 per route template.

2. Instrument Hot Paths with Timers and Distribution Summaries

Use explicit SLO buckets for timers and min/max values for distribution summaries.

import io.micrometer.core.instrument.DistributionSummary;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
 
import java.time.Duration;
 
public class PricingService {
  private final Timer priceCalc;
  private final DistributionSummary payloadSize;
 
  public PricingService(MeterRegistry registry) {
    this.priceCalc = Timer.builder("pricing.calc.duration")
        .description("Time to compute price for a cart")
        .publishPercentileHistogram()
        .serviceLevelObjectives(
            Duration.ofMillis(25),
            Duration.ofMillis(50),
            Duration.ofMillis(100),
            Duration.ofMillis(250),
            Duration.ofMillis(500),
            Duration.ofMillis(1000))
        .tag("component", "pricing")
        .register(registry);
 
    this.payloadSize = DistributionSummary.builder("pricing.cart.payload.bytes")
        .minimumExpectedValue(100.0)
        .maximumExpectedValue(200_000.0)
        .serviceLevelObjectives(500.0, 1_000.0, 5_000.0, 10_000.0, 100_000.0)
        .register(registry);
  }
 
  public PriceResult compute(Cart cart) {
    payloadSize.record(cart.approxBytes());
    return priceCalc.record(() -> expensiveComputation(cart));
  }
 
  private PriceResult expensiveComputation(Cart cart) {
    return new PriceResult();
  }
}

The important details are not only the metric names. The important details are the histogram, SLO buckets, and bounded distribution summary.

3. Add OpenTelemetry Manual Metrics When Needed

If you rely on the OpenTelemetry Java agent or mix manual instrumentation, use GlobalOpenTelemetry so manual metrics flow into the same pipeline.

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.metrics.LongCounter;
import io.opentelemetry.api.metrics.Meter;
 
public final class OTelMetrics {
  private static final OpenTelemetry OTEL = GlobalOpenTelemetry.get();
  private static final Meter METER = OTEL.getMeter("orders-api", "1.0.0");
 
  private static final LongCounter shipped = METER.counterBuilder("orders.shipped.total")
      .setDescription("Total shipped orders")
      .build();
 
  public static void markShipped(String region) {
    shipped.add(1, Attributes.of(
        AttributeKey.stringKey("region"), region
    ));
  }
}

Manual OTel metrics are useful for domain events, background workers, and places where framework instrumentation does not see enough context.

4. Drive Load and Verify Metrics Locally

Generate traffic and inspect metrics before calling the service production-ready.

curl http://localhost:8080/actuator/prometheus
 
for i in {1..2000}; do
  curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/api/orders
done

In Prometheus-style backends, query histogram buckets like this:

histogram_quantile(
  0.95,
  sum(rate(http_server_requests_seconds_bucket[5m])) by (le, route, status)
)

For error rate:

sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
/
sum(rate(http_server_requests_seconds_count[5m]))

The exact metric names vary by framework and exporter version. The pattern matters: aggregate rates from histogram buckets, then calculate quantiles centrally.

Concrete Example: Turning Noisy HTTP Latency Into SLO Signals

Problem: the average latency for /api/search looks good, but users report frequent timeouts at peak.

When to care: immediately. If you alert on averages or max values, you are probably blind to the actual tail.

What to do:

• Switch to histogram-based percentiles with explicit SLO buckets.
• Use route templates like /api/search, not /api/search?q=....
• Attach outcome tags such as success, timeout, and error.
• Avoid exception text and request IDs as tags.
• Add dashboards for p50, p90, p95, p99, error rate, RPS, and saturation.

What to avoid: using client-side percentiles and histograms simultaneously without understanding which one your backend uses. In most production setups, histogram exports are more useful because they can be aggregated.

Review Checklist for Code and Configuration

Use this in pull requests. If any answer is no, request changes.

• Are timers publishing histograms, or only averages?
• Are SLO buckets explicitly set for primary latency metrics?
• Are percentiles computed in the backend from histograms?
• Are tags low-cardinality and documented?
• Are raw URLs, user IDs, session IDs, or request IDs banned?
• Are distribution summaries clamped with min/max values?
• Are error classes bucketed into coarse categories?
• Is OTLP export configured with a sensible step such as 10 to 30 seconds?
• Are JVM and runtime metrics enabled for context?
• Are exemplars enabled so engineers can pivot from metrics to traces?

Security Considerations

Labels can leak personally identifiable information, secrets, and operational details. Metrics endpoints can also become an attack surface.

Do this:

• Prohibit user IDs, emails, tokens, raw query strings, and raw paths in tags.
• Validate tags at instrumentation boundaries.
• Expose metrics endpoints only inside the cluster or behind authentication.
• Restrict scrape access with network policy.
• Cap label cardinality to prevent denial-of-wallet or denial-of-observability failures.
• Avoid labels that serialize exception messages or stack traces.

This is close to the principle in Practical Error Handling in Spring Boot for Reliable Production APIs: operational signals should help engineers debug without leaking sensitive implementation details.

Performance and Operations Considerations

Metrics can become expensive. The usual cause is not "too much observability"; it is undisciplined labels and poor bucket choices.

Bucket Design

Keep buckets denser near your SLO and coarser elsewhere.

For example, if an API SLO is 300ms, useful buckets might include:

• 50ms
• 100ms
• 200ms
• 300ms
• 400ms
• 800ms
• 1500ms
• 3000ms

Do not blindly generate many fine-grained buckets across a wide range.

Tag Cardinality

Track time series count per metric. A single bad tag can multiply cost and slow queries.

Prefer:

• route="/api/orders/{id}"
• outcome="success"
• exception="timeout"

Avoid:

• url="/api/orders/893742?token=..."
• userId="..."
• exceptionMessage="Connection timed out for customer ..."

Export Step and Backpressure

Choose an export step that matches operational needs. Ten to thirty seconds is a common range for service metrics.

If the collector is unavailable, your application should degrade safely. Metrics export should not block request processing.

Decision Boundaries: Metric Type vs Use Case

Use this table during design reviews.

Production Readiness Checklist

Before a service sees real traffic, verify:

Common Mistakes and How to Fix Them

"We Alert on Average Latency"

Replace averages with p95 and p99 alerts from histogram buckets. Keep averages as a supporting signal, not the primary user-impact signal.

"Percentiles Do Not Match Across Pods"

Client-side percentiles usually cannot be aggregated correctly. Export histograms and compute percentiles centrally.

"The Observability Bill Spiked After a Feature"

You probably added high-cardinality tags. Revert the label, add a tag budget, and introduce static checks for banned tag keys.

"We Cannot Find the Slow Requests Behind the Spike"

Enable exemplars and trace correlation. A p99 spike should lead to sample traces, not a guessing session.

"Raw URLs Created Millions of Time Series"

Templatize paths and drop query strings at instrumentation time.

How To Apply This In a Real Team

Make metrics quality part of delivery, not an optional cleanup task.

1. Create a one-pager per service.

   Define primary SLOs, chosen metrics, tags, dashboard queries, and owners.

2. Add PR template checks.

   Ask: does this PR add metrics? If yes, require histogram status, SLO buckets, and tag list.

3. Add CI guardrails.

   Forbid tag keys such as userId, sessionId, url, error, message, token, and email.

4. Review one service per week.

   Fix the worst offenders first: averages, unbounded tags, missing histograms, and dashboards without p95/p99.

5. Tie metrics to release habits.

   Deploy markers, rollout windows, and rollback behavior matter. The patterns in DevOps Release Habits for Small Teams make metrics far more useful during change.

Example: Spring Boot Controller with Safe Route Tags

Use framework instrumentation where possible. If you need custom timers, keep tags stable and bounded.

import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
 
import java.util.List;
import java.util.concurrent.TimeUnit;
 
@RestController
@RequestMapping("/api")
class SearchController {
 
  private final MeterRegistry registry;
 
  SearchController(MeterRegistry registry) {
    this.registry = registry;
  }
 
  @GetMapping("/search")
  public ResponseEntity<List<Result>> search(@RequestParam("q") String query) {
    if (query == null || query.isBlank()) {
      return ResponseEntity.badRequest().build();
    }
 
    long start = System.nanoTime();
 
    try {
      List<Result> results = performSearch(query);
      return ResponseEntity.ok(results);
    } finally {
      registry.timer("search.query.duration", "component", "search")
          .record(System.nanoTime() - start, TimeUnit.NANOSECONDS);
    }
  }
 
  private List<Result> performSearch(String query) {
    return List.of();
  }
}

With the earlier application.yml, latency for /api/search can be captured as a histogram with SLO buckets and aggregated across pods and regions.

Why the Current Java Metrics Conversation Matters

The ecosystem keeps converging on histogram-first measurement, clean tags, and trace links. That matters because teams increasingly run distributed Java systems where averages and raw logs are not enough.

The lesson is not to chase hype around a specific tool. The lesson is to standardize instrumentation so your production data is trustworthy.

Teams that align with these patterns get:

• More stable alerts.
• Lower time series cost.
• Faster incident triage.
• Clearer SLO conversations.
• Better confidence during releases.

FAQ

Should we enable both client-side percentiles and histograms in Micrometer?

Prefer publishing percentile histograms and computing percentiles server-side. Client-side percentiles are often non-aggregable and redundant in distributed systems.

How many tags are too many?

It depends on your backend, but a practical starting budget is 8 or fewer tags per metric and 50 or fewer values per tag per service and region. Ban unbounded tags such as IDs and raw URLs.

Do we need the OpenTelemetry Java agent if we already use Micrometer?

You can use Micrometer as the facade and export to OTLP. The OpenTelemetry Java agent can still add automatic runtime, HTTP, database, and messaging telemetry. Manual OTel metrics can coexist through GlobalOpenTelemetry.

How do exemplars work in Java?

When enabled, the metrics pipeline can attach sampled trace context to histogram buckets or measurements. Your backend can then show a link from a metric point, such as a p99 spike, to a representative trace.

What SLO buckets should we start with?

Start around your external SLOs and timeouts. For example: 50ms, 100ms, 200ms, 400ms, 800ms, 1500ms, and 3000ms. Keep buckets denser near the SLO.

Why not just use averages?

Averages hide tail behavior. Users experience the tail. Histogram-based p95 and p99 values align better with SLOs and incident detection.

Conclusion

Stop shipping metrics that lie.

Move Java services toward histogram-first latency, explicit SLO buckets, and safe tags. Export to OTLP, compute percentiles server-side, and wire exemplars so p99 points to a trace instead of a debate.

Your immediate next step is simple: enable percentile histograms for primary HTTP timers, define SLO buckets around real user-facing thresholds, and add a tag budget check to CI. The rest of the observability practice becomes easier once the measurements are trustworthy.

FAQ Schema

{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "Should we enable both client-side percentiles and histograms in Micrometer?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Prefer publishing percentile histograms and computing percentiles server-side. Client-side percentiles are often non-aggregable and redundant in distributed systems."
      }
    },
    {
      "@type": "Question",
      "name": "How many tags are too many for Java metrics?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "A practical starting budget is eight or fewer tags per metric and 50 or fewer values per tag per service and region. Ban unbounded tags such as IDs, raw URLs, and free text."
      }
    },
    {
      "@type": "Question",
      "name": "Do we need the OpenTelemetry Java agent if we already use Micrometer?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "You can use Micrometer and export to OTLP. The OpenTelemetry Java agent can add automatic runtime, HTTP, database, and messaging telemetry, while manual OTel metrics can coexist through GlobalOpenTelemetry."
      }
    },
    {
      "@type": "Question",
      "name": "How do exemplars connect Java metrics to traces?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "When enabled, the metrics pipeline can attach sampled trace context to histogram buckets or measurements. Your backend can link a metric point, such as a p99 spike, to a representative trace."
      }
    },
    {
      "@type": "Question",
      "name": "What latency SLO buckets should we start with?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Begin around external SLOs and timeouts, such as 50ms, 100ms, 200ms, 400ms, 800ms, 1500ms, and 3000ms. Keep buckets denser near the SLO and refine with traffic data."
      }
    },
    {
      "@type": "Question",
      "name": "Why are averages a bad latency signal?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Averages mask tail behavior. Users experience tail latency, so histogram-based p95 and p99 values align better with SLOs and incident detection."
      }
    }
  ]
}