Data Cleaning with tidyr
Reshape and clean data using pivot operations, string splitting, and missing value handling to get your data into tidy format.
Tidy Data Principles Recap
Tidy data follows three rules: each variable is a column, each observation is a row, each value is a cell. Most real-world data is not tidy - tidyr helps you get it there.
pivot_longer() - Wide to Long
Convert columns into rows (the most common tidying operation):
library(tidyr) library(dplyr) # Wide data: year columns contain values wide <- tibble( country = c("USA", "UK", "Japan"), `2020` = c(100, 80, 90), `2021` = c(110, 85, 95), `2022` = c(120, 90, 100) ) # Pivot to tidy (long) format long <- wide |> pivot_longer( cols = `2020`:`2022`, names_to = "year", values_to = "value" ) # country year value # USA 2020 100 # USA 2021 110 # ...
pivot_wider() - Long to Wide
# Convert back to wide format long |> pivot_wider( names_from = year, values_from = value )
separate() and unite()
# Split one column into two df <- tibble(date = c("2024-01-15", "2024-02-20", "2024-03-25")) df |> separate(date, into = c("year", "month", "day"), sep = "-") # Combine multiple columns into one df2 <- tibble(first = c("Alice", "Bob"), last = c("Smith", "Jones")) df2 |> unite("full_name", first, last, sep = " ")
Handling Missing Values
df <- tibble( x = c(1, NA, 3, NA, 5), y = c(NA, 2, NA, 4, 5) ) # Drop rows with any NA df |> drop_na() # Drop rows where specific column is NA df |> drop_na(x) # Replace NA with a value df |> replace_na(list(x = 0, y = 0)) # Fill NA with previous/next value df |> fill(x, .direction = "down") df |> fill(x, .direction = "up") # Complete cases (no NAs in any column) df |> filter(complete.cases(df))
nest() and unnest()
# Nest data by group nested <- mtcars |> group_by(cyl) |> nest() # Each group's data stored as a tibble in a list-column # Fit a model to each group library(purrr) models <- nested |> mutate(model = map(data, \(df) lm(mpg ~ wt, data = df))) # Unnest back to flat data nested |> unnest(data)
pivot_longer(). If you have repeated measurements in rows that should be columns, you need pivot_wider().Ready to Go Deeper?
Live instructor-led courses from our partners. Affiliate disclosure.
AI & ML Courses - 30% Off
Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.
EdurekaDataCamp - AI & Data Science
Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.
DataCampedX - Top AI Courses
University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.
edX