Derivatives Beginner

A derivative measures how a function's output changes when its input changes by a tiny amount. In ML, derivatives tell us how the loss changes when we adjust a model parameter - the key information needed for training.

The Derivative Defined

The derivative of f(x) at point x is the slope of the tangent line at that point. It tells you the instantaneous rate of change:

Python 
import numpy as np

# Numerical derivative approximation
def numerical_derivative(f, x, h=1e-7):
    return (f(x + h) - f(x - h)) / (2 * h)

# Example: f(x) = x^3
f = lambda x: x ** 3
# Analytical derivative: f'(x) = 3x^2

x = 2.0
print("Numerical:", numerical_derivative(f, x))  # ~12.0
print("Analytical:", 3 * x**2)                   # 12.0

Common Derivatives in ML

Function Derivative ML Context
xn n · xn-1 Polynomial features
ex ex Softmax, exponential distributions
ln(x) 1/x Cross-entropy loss
sigmoid(x) σ(x)(1 - σ(x)) Binary classification activation
ReLU(x) 0 if x < 0, 1 if x > 0 Most common activation function

Partial Derivatives

When a function has multiple inputs (like a loss function with many weights), a partial derivative measures the sensitivity to just one input while holding others constant:

Python 
# f(w1, w2) = w1^2 + 3*w1*w2 + w2^2
# Partial derivative w.r.t. w1: df/dw1 = 2*w1 + 3*w2
# Partial derivative w.r.t. w2: df/dw2 = 3*w1 + 2*w2

def f(w1, w2):
    return w1**2 + 3*w1*w2 + w2**2

def df_dw1(w1, w2):
    return 2*w1 + 3*w2

def df_dw2(w1, w2):
    return 3*w1 + 2*w2

# At w1=1, w2=2:
print("df/dw1 =", df_dw1(1, 2))  # 8
print("df/dw2 =", df_dw2(1, 2))  # 7
ML Interpretation: If df/dw1 = 8, it means increasing w1 by a tiny amount will increase the loss by approximately 8 times that amount. To decrease the loss, we should decrease w1. This is the core logic of gradient descent.

Derivatives of Loss Functions

Python 
# Mean Squared Error loss and its derivative
def mse_loss(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

def mse_gradient(y_true, y_pred):
    return -2 * np.mean(y_true - y_pred)

# Binary Cross-Entropy loss derivative
def bce_gradient(y_true, y_pred):
    return -(y_true / y_pred) + (1 - y_true) / (1 - y_pred)

Next Up: Gradients

Now that you understand individual derivatives, let's combine them into gradient vectors that guide multi-parameter optimization.

Next: Gradients →
← Introduction Gradients →

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka
📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp
🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX