ML Pipeline Logging Intermediate

Structured logging transforms unreadable training output into queryable, correlated records. This lesson covers designing a logging strategy for ML pipelines including structured JSON logs, log levels for ML events, correlation with trace IDs, and centralized aggregation using Grafana Loki or Elasticsearch.

Structured Log Format for ML

import structlog

logger = structlog.get_logger()

logger.info("training_epoch_complete",
    epoch=15,
    train_loss=0.342,
    val_loss=0.418,
    learning_rate=0.001,
    gpu_utilization=94.2,
    samples_per_second=1250,
    experiment_id="exp-2024-001",
    trace_id="abc123def456"
)

ML-Specific Log Levels

• INFO - Epoch completions, checkpoint saves, pipeline stage transitions
• WARNING - Loss plateaus, learning rate adjustments, data quality anomalies, GPU thermal warnings
• ERROR - Training failures, OOM events, data loading errors, model validation failures
• DEBUG - Batch-level metrics, gradient statistics, individual feature computation times

Log Aggregation with Grafana Loki

Loki is the lightweight log aggregation system that integrates with Grafana. For ML pipelines, configure Loki to label logs by pipeline stage, experiment ID, and model name for efficient querying:

• Label design - Use labels for pipeline_name, stage, experiment_id, and environment
• Retention - Keep ML training logs for at least 90 days for experiment comparison
• LogQL queries - Use Loki's query language to filter and aggregate ML events

Common ML Error Patterns