Advanced

Advanced Sweeps

Go beyond basic hyperparameter search with multi-objective optimization, early termination, distributed sweeps, and custom agents.

Early Termination

Stop underperforming runs early to save compute. W&B supports the Hyperband early termination strategy.

YAML - Sweep config with early termination

program: train.py
method: bayes
metric:
  name: val_loss
  goal: minimize
early_terminate:
  type: hyperband
  min_iter: 5        # Minimum epochs before stopping
  eta: 3             # Aggressiveness (higher = more aggressive)
  s: 2               # Number of brackets
parameters:
  learning_rate:
    distribution: log_uniform_values
    min: 0.0001
    max: 0.1
  batch_size:
    values: [16, 32, 64, 128]
  dropout:
    distribution: uniform
    min: 0.0
    max: 0.5
  optimizer:
    values: ["adam", "sgd", "adamw"]

Distributed Sweeps

Terminal - Run sweep agents on multiple machines

# Machine 1: Create the sweep
wandb sweep sweep_config.yaml
# Returns: sweep_id = "entity/project/sweep_id"

# Machine 1: Start an agent
wandb agent entity/project/sweep_id

# Machine 2: Start another agent (same sweep)
wandb agent entity/project/sweep_id

# Machine 3: And another
wandb agent entity/project/sweep_id

# All agents coordinate through the W&B server
# Each gets different hyperparameter combinations

Advanced Search Strategies

Strategy	When to Use	Pros	Cons
Grid	Small discrete spaces	Exhaustive coverage	Exponential cost
Random	Large spaces, initial exploration	Good coverage, parallelizable	No learning
Bayesian	Continuous parameters, expensive runs	Learns from history	Sequential bottleneck

Programmatic Sweep Control

Python - Create and manage sweeps via API

import wandb

sweep_config = {
    "method": "bayes",
    "metric": {"name": "val_accuracy", "goal": "maximize"},
    "early_terminate": {"type": "hyperband", "min_iter": 3},
    "parameters": {
        "learning_rate": {"distribution": "log_uniform_values",
                          "min": 1e-5, "max": 1e-2},
        "hidden_size": {"values": [128, 256, 512, 1024]},
        "num_layers": {"min": 1, "max": 5},
        "activation": {"values": ["relu", "gelu", "silu"]},
    },
    "run_cap": 50,  # Maximum number of runs
}

sweep_id = wandb.sweep(sweep_config, project="advanced-sweeps")

def train():
    run = wandb.init()
    config = wandb.config
    model = build_model(config.hidden_size, config.num_layers,
                        config.activation)
    # ... training loop ...
    wandb.log({"val_accuracy": val_acc})

wandb.agent(sweep_id, function=train, count=50)

✅

Cost-saving strategy: Start with 20-30 random search runs to understand the parameter landscape, then switch to Bayesian optimization with early termination. This typically finds good configurations in 50-100 total runs instead of hundreds.

← Previous Tables Next → Launch

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka

📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp

🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX