Monitoring AI Microservices

Build comprehensive observability for distributed AI systems using the three pillars: metrics, logs, and traces - plus AI-specific monitoring.

The Three Pillars of Observability

AI-Specific Metrics

Beyond standard service metrics, AI microservices need additional monitoring:

from prometheus_client import Histogram, Counter, Gauge

# Standard service metrics
request_latency = Histogram(
    "model_request_duration_seconds",
    "Time spent processing inference request",
    ["model_name", "model_version"],
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0, 30.0]
)

# AI-specific metrics
prediction_confidence = Histogram(
    "model_prediction_confidence",
    "Distribution of prediction confidence scores",
    ["model_name"],
    buckets=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99]
)

gpu_memory_used = Gauge(
    "gpu_memory_used_bytes",
    "GPU memory currently in use",
    ["gpu_id"]
)

tokens_processed = Counter(
    "model_tokens_processed_total",
    "Total tokens processed",
    ["model_name", "direction"]  # direction: input/output
)

data_drift_score = Gauge(
    "model_data_drift_score",
    "Current data drift score vs training distribution",
    ["model_name", "feature_name"]
)

Distributed Tracing for AI Pipelines

When a prediction request flows through 4-5 microservices, tracing shows exactly where time is spent:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

tracer = trace.get_tracer("ai-pipeline")

@app.post("/predict")
async def predict(request: PredictRequest):
    with tracer.start_as_current_span("prediction_pipeline") as span:
        span.set_attribute("model.name", request.model)

        # Trace each stage
        with tracer.start_as_current_span("feature_extraction"):
            features = await extract_features(request)
            span.set_attribute("features.count", len(features))

        with tracer.start_as_current_span("model_inference"):
            result = await run_inference(features)
            span.set_attribute("inference.latency_ms", result.latency)
            span.set_attribute("inference.confidence", result.confidence)

        with tracer.start_as_current_span("postprocessing"):
            response = format_response(result)

        return response

Alerting Strategy

Grafana Dashboard Essentials

Every AI microservice dashboard should include:

• Request rate: Requests per second by model and endpoint.
• Latency percentiles: P50, P95, P99 latency with TTFT for streaming endpoints.
• Error rate: 4xx and 5xx rates by error type.
• GPU utilization: GPU compute, memory, and power usage per pod.
• Model metrics: Prediction confidence distribution, throughput in tokens/second.
• Queue depth: Number of pending requests waiting for GPU capacity.