Advanced

Hyperparameter Tuning

Master Bayesian optimization with Optuna, systematic tuning strategies, early stopping, learning rate scheduling, and competition-winning techniques for gradient boosting.

Bayesian Optimization with Optuna

Python

import optuna
from sklearn.model_selection import cross_val_score
import lightgbm as lgb

def objective(trial):
    params = {
        "n_estimators": 1000,
        "max_depth": trial.suggest_int("max_depth", 3, 12),
        "num_leaves": trial.suggest_int("num_leaves", 20, 150),
        "learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3, log=True),
        "subsample": trial.suggest_float("subsample", 0.5, 1.0),
        "colsample_bytree": trial.suggest_float("colsample_bytree", 0.5, 1.0),
        "reg_alpha": trial.suggest_float("reg_alpha", 1e-8, 10.0, log=True),
        "reg_lambda": trial.suggest_float("reg_lambda", 1e-8, 10.0, log=True),
        "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
    }
    model = lgb.LGBMClassifier(**params, random_state=42, verbose=-1)
    scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
    return scores.mean()

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=100)
print(f"Best AUC: {study.best_value:.4f}")
print(f"Best params: {study.best_params}")

Systematic Tuning Strategy

Fix learning_rate=0.1, find optimal n_estimators
Use early stopping with a large n_estimators (5000). The best iteration tells you how many rounds you need.
Tune tree structure
Optimize max_depth, num_leaves, and min_child_samples together. These control model complexity.
Tune sampling
Adjust subsample and colsample_bytree. Values between 0.6-0.9 usually work best.
Tune regularization
Optimize reg_alpha and reg_lambda to prevent overfitting.
Lower learning_rate, increase n_estimators
Drop learning_rate to 0.01-0.05 and let early stopping find the right number of rounds.

Early Stopping

Python

# XGBoost early stopping
model = XGBClassifier(
    n_estimators=5000,
    learning_rate=0.01,
    early_stopping_rounds=100
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)])
print(f"Stopped at iteration {model.best_iteration}")

# LightGBM early stopping
model = lgb.LGBMClassifier(n_estimators=5000, learning_rate=0.01)
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    callbacks=[lgb.early_stopping(100)]
)

Competition Strategies

Strategy	Description
Multi-seed averaging	Train with different random seeds and average predictions
Framework blending	Blend XGBoost + LightGBM + CatBoost predictions
K-fold training	Train on each fold and average out-of-fold predictions
Feature engineering	Often more impactful than hyperparameter tuning
Target encoding	Encode categorical features using target statistics (with CV)