Advanced

DVC Best Practices

Integrate DVC into CI/CD pipelines, optimize storage, collaborate with teams, and deploy ML models to production.

CI/CD Integration

YAML - GitHub Actions with DVC

name: ML Pipeline

on:
  push:
    branches: [main]

jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install "dvc[s3]"

      - name: Configure DVC remote
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: dvc pull

      - name: Run pipeline
        run: dvc repro

      - name: Check metrics
        run: |
          dvc metrics show
          python scripts/check_metrics.py --min-accuracy 0.90

      - name: Push results
        if: success()
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          dvc push
          git add dvc.lock metrics/
          git commit -m "Update model [skip ci]"
          git push

CML Integration

CML (Continuous Machine Learning) is a companion tool from the DVC team that adds ML reporting to pull requests.

YAML - CML in GitHub Actions

name: ML Report

on: pull_request

jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: iterative/setup-cml@v2

      - name: Run pipeline and generate report
        env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          pip install -r requirements.txt
          dvc pull
          dvc repro

          # Generate report as PR comment
          echo "## Model Performance" >> report.md
          echo "" >> report.md
          dvc metrics diff --md >> report.md
          echo "" >> report.md
          echo "### Plots" >> report.md
          dvc plots diff --target plots/ --out plots_diff
          echo '![](./plots_diff/index.html)' >> report.md

          cml comment create report.md

Team Workflows

Shared remote storage
Configure a shared DVC remote (S3 bucket, GCS) that all team members can access.
Branch-based development
Each team member works on a Git branch. DVC data is shared via the remote.
Pull request reviews
Use CML to automatically post metric comparisons on pull requests.
Merge and deploy
Merge the best branch, CI/CD runs the pipeline, pushes results to production.

Storage Optimization

Garbage collection: Run dvc gc --workspace to clean up unused cache files.
Shared cache: On shared machines, use dvc cache dir to set a common cache directory.
Symlinks: Configure dvc config cache.type symlink to avoid duplicating large files.
Remote cleanup: Periodically remove old data versions from remote storage that are no longer referenced.

Bash - Storage management

# Clean unused cache (keep only current workspace data)
dvc gc --workspace

# Clean cache for cloud remote too
dvc gc --workspace --cloud

# Show cache size
du -sh .dvc/cache/

# Configure symlinks to save disk space
dvc config cache.type symlink
dvc config cache.shared group

# Use hardlinks (faster than copies)
dvc config cache.type hardlink

Common Pitfalls

⚠

Forgetting to push: Always run dvc push after git push. Your teammates need both the Git pointer files and the actual data.
Not committing .dvc files: The .dvc pointer files and dvc.lock must be committed to Git. Without them, data versions are lost.
Editing tracked files directly: After dvc add, editing a file without re-running dvc add creates inconsistencies.
Missing remote credentials: Ensure CI/CD environments have access to your DVC remote storage.
Large .dvc/cache: The local cache grows over time. Schedule regular dvc gc runs.

Frequently Asked Questions

Yes. DVC works with a local cache only. You can use dvc add and dvc repro without configuring a remote. However, without a remote, you cannot share data with teammates or use dvc push/pull. A local directory remote works for testing.

Git LFS is simpler but stores data on your Git server, which can be expensive for large datasets. DVC stores data on cheap cloud storage (S3, GCS). DVC also adds pipelines and experiment tracking. Use Git LFS for a few large files; use DVC for ML-specific workflows.

Absolutely, and this is a popular combination. Use DVC for data versioning and pipeline management (what data and code produced each model). Use W&B for experiment visualization, real-time dashboards, and team collaboration. They complement each other well.