AI Cost Dashboards Intermediate

Cost dashboards transform raw billing data into actionable insights. This lesson covers building dashboards that show real-time AI spending, per-team breakdowns, cost trends, anomaly detection, and cost-per-inference metrics that help teams understand the financial impact of their AI workloads.

Executive Cost Dashboard

• Total monthly AI spend - Stat panel with month-over-month comparison
• Cost by team - Pie chart showing proportional spend per team
• Cost trend - Line chart showing daily spend over the last 90 days with forecast
• Budget vs actual - Bar chart comparing budgeted amount with actual spend per project
• Cost per model inference - Table showing unit economics for each production model

Team-Level Cost Dashboard

Each ML team needs visibility into their own spending:

# GPU-hours consumed by team namespace
sum by (namespace) (
  count_over_time(DCGM_FI_DEV_GPU_UTIL{namespace=~"$team"}[1h])
) / 3600 * $gpu_hourly_rate

# Cost per training run
sum by (job_name) (
  gpu_seconds_used{namespace="$team"} * $gpu_per_second_rate
)

Cost Anomaly Detection

Detect unexpected cost spikes before they blow your budget:

• Daily spend threshold - Alert when daily spend exceeds 150% of the 7-day rolling average
• Idle GPU detection - Alert on GPUs allocated but unused for more than 30 minutes
• Runaway jobs - Alert on training jobs running longer than expected duration with no checkpoints
• New expensive resources - Alert when a new high-cost instance type is launched without approval

Unit Economics Dashboard

Track the cost efficiency of your ML operations: