Intermediate

Hypothesis Testing

Hypothesis testing is a structured framework for making data-driven decisions. It lets you determine whether observed patterns are statistically significant or likely due to random chance.

The Hypothesis Testing Framework

  1. State the Hypotheses

    Define the null hypothesis (H₀) - the default assumption of no effect or no difference - and the alternative hypothesis (H₁) - what you are trying to prove.

  2. Choose a Significance Level (α)

    Set the threshold for rejecting H₀. Common choices are 0.05 (5%) or 0.01 (1%). This is your tolerance for a false positive.

  3. Select the Appropriate Test

    Choose a test statistic based on your data type, sample size, and question (t-test, chi-square, ANOVA, etc.).

  4. Calculate the Test Statistic and p-value

    Compute how far your observed result is from what H₀ predicts. The p-value is the probability of seeing a result this extreme if H₀ is true.

  5. Make a Decision

    If p-value ≤ α, reject H₀ (statistically significant). If p-value > α, fail to reject H₀ (not enough evidence).

Understanding p-values

💡
What a p-value IS: The probability of observing data as extreme as yours (or more extreme) assuming the null hypothesis is true.

What a p-value is NOT: The probability that the null hypothesis is true, or the probability that your results are due to chance.

Type I and Type II Errors

H₀ is True H₀ is False
Reject H₀ Type I Error (False Positive, α) Correct (Power, 1-β)
Fail to Reject H₀ Correct Type II Error (False Negative, β)
  • Type I Error (α): Concluding there is an effect when there is not. Like a fire alarm going off with no fire.
  • Type II Error (β): Failing to detect a real effect. Like a fire alarm not going off during a fire.
  • Power (1-β): The probability of correctly detecting a real effect. Increase power with larger sample sizes.

Common Statistical Tests

t-test: Comparing Means

Use a t-test to determine whether two groups have significantly different means.

Python 
from scipy import stats
import numpy as np

# One-sample t-test: Is the mean different from a known value?
sample = [23, 25, 28, 22, 26, 24, 27, 25, 29, 26]
t_stat, p_value = stats.ttest_1samp(sample, popmean=24)
print(f"t-statistic: {t_stat:.4f}, p-value: {p_value:.4f}")

# Two-sample t-test: Are two groups different?
group_a = [85, 90, 88, 92, 87, 91, 86, 89]
group_b = [78, 82, 80, 85, 79, 83, 81, 77]
t_stat, p_value = stats.ttest_ind(group_a, group_b)
print(f"t-statistic: {t_stat:.4f}, p-value: {p_value:.4f}")

if p_value < 0.05:
    print("Statistically significant difference")
else:
    print("No statistically significant difference")

Chi-Square Test: Categorical Data

Use chi-square to test whether categorical variables are independent or related.

Python 
import pandas as pd
from scipy.stats import chi2_contingency

# Is there a relationship between department and attrition?
contingency_table = pd.crosstab(df['department'], df['attrition'])
print(contingency_table)

chi2, p_value, dof, expected = chi2_contingency(contingency_table)
print(f"Chi-square: {chi2:.4f}")
print(f"p-value: {p_value:.4f}")
print(f"Degrees of freedom: {dof}")

ANOVA: Comparing Multiple Groups

Analysis of Variance (ANOVA) tests whether the means of three or more groups are significantly different.

Python 
# One-way ANOVA: Compare salary across departments
engineering = df[df['dept'] == 'Engineering']['salary']
marketing = df[df['dept'] == 'Marketing']['salary']
sales = df[df['dept'] == 'Sales']['salary']

f_stat, p_value = stats.f_oneway(engineering, marketing, sales)
print(f"F-statistic: {f_stat:.4f}")
print(f"p-value: {p_value:.4f}")

# If significant, use Tukey's HSD for pairwise comparisons
from statsmodels.stats.multicomp import pairwise_tukeyhsd
tukey = pairwise_tukeyhsd(df['salary'], df['dept'], alpha=0.05)
print(tukey)

A/B Testing

A/B testing applies hypothesis testing to compare two versions of something (a webpage, email, feature) to determine which performs better.

Python 
from scipy import stats

# A/B test: Does the new button color increase click-through rate?
# Control (A): 500 visitors, 45 clicked
# Treatment (B): 500 visitors, 63 clicked

n_a, clicks_a = 500, 45
n_b, clicks_b = 500, 63

rate_a = clicks_a / n_a  # 9.0%
rate_b = clicks_b / n_b  # 12.6%

# Two-proportion z-test
count = np.array([clicks_a, clicks_b])
nobs = np.array([n_a, n_b])

from statsmodels.stats.proportion import proportions_ztest
z_stat, p_value = proportions_ztest(count, nobs, alternative='two-sided')
print(f"Control rate: {rate_a:.1%}")
print(f"Treatment rate: {rate_b:.1%}")
print(f"p-value: {p_value:.4f}")
print(f"Lift: {(rate_b - rate_a) / rate_a:.1%}")
✅
A/B testing checklist: Define your metric before starting. Calculate the required sample size for desired power. Run the test long enough. Do not peek at results and stop early. Account for multiple comparisons if testing many variants.
⚠
Statistical significance is not practical significance. A p-value of 0.01 means the effect is unlikely due to chance, but it does not tell you if the effect is large enough to matter. Always report effect sizes alongside p-values.

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka
📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp
🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX