Intermediate

Data Cleaning with tidyr

Reshape and clean data using pivot operations, string splitting, and missing value handling to get your data into tidy format.

Tidy Data Principles Recap

Tidy data follows three rules: each variable is a column, each observation is a row, each value is a cell. Most real-world data is not tidy - tidyr helps you get it there.

pivot_longer() - Wide to Long

Convert columns into rows (the most common tidying operation):

R
library(tidyr)
library(dplyr)

# Wide data: year columns contain values
wide <- tibble(
  country = c("USA", "UK", "Japan"),
  `2020` = c(100, 80, 90),
  `2021` = c(110, 85, 95),
  `2022` = c(120, 90, 100)
)

# Pivot to tidy (long) format
long <- wide |>
  pivot_longer(
    cols = `2020`:`2022`,
    names_to = "year",
    values_to = "value"
  )
# country  year   value
# USA      2020   100
# USA      2021   110
# ...

pivot_wider() - Long to Wide

R
# Convert back to wide format
long |>
  pivot_wider(
    names_from = year,
    values_from = value
  )

separate() and unite()

R
# Split one column into two
df <- tibble(date = c("2024-01-15", "2024-02-20", "2024-03-25"))
df |> separate(date, into = c("year", "month", "day"), sep = "-")

# Combine multiple columns into one
df2 <- tibble(first = c("Alice", "Bob"), last = c("Smith", "Jones"))
df2 |> unite("full_name", first, last, sep = " ")

Handling Missing Values

R
df <- tibble(
  x = c(1, NA, 3, NA, 5),
  y = c(NA, 2, NA, 4, 5)
)

# Drop rows with any NA
df |> drop_na()

# Drop rows where specific column is NA
df |> drop_na(x)

# Replace NA with a value
df |> replace_na(list(x = 0, y = 0))

# Fill NA with previous/next value
df |> fill(x, .direction = "down")
df |> fill(x, .direction = "up")

# Complete cases (no NAs in any column)
df |> filter(complete.cases(df))

nest() and unnest()

R
# Nest data by group
nested <- mtcars |>
  group_by(cyl) |>
  nest()
# Each group's data stored as a tibble in a list-column

# Fit a model to each group
library(purrr)
models <- nested |>
  mutate(model = map(data, \(df) lm(mpg ~ wt, data = df)))

# Unnest back to flat data
nested |> unnest(data)
Rule of thumb: If your data has values in column names (like year columns "2020", "2021"), you need pivot_longer(). If you have repeated measurements in rows that should be columns, you need pivot_wider().

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.