Versioning & Regression Testing

A prompt change that improves one case and silently breaks five others is worse than no change at all. The only way to know is to test before you ship.

Prompts Are Code

The framing shift that makes this lesson possible: prompts are software. They have inputs, outputs, and a behavior contract. They can regress. They need tests. Teams that treat prompt changes as “just updating some text” discover this the hard way, usually after a prompt update that seemed like an improvement started silently breaking cases that were working.

The practices in this lesson are not exotic engineering - they are the same discipline applied to software since before LLMs existed: version control, changelog, regression suite, and a gate that blocks regressions from shipping.

Version Control for Prompts

Store prompts as files in your version-control system, not as strings inside application code or hardcoded in a database. The reasons are identical to why you version-control code: history, blame, rollback, diff, and code review.

prompts/
  support-classifier/
    v1.0.0.txt          # original release
    v1.1.0.txt          # added "feature_request" category
    v1.2.0.txt          # tightened output format instructions
    current.txt         # symlink or copy of the active version
    CHANGELOG.md        # human-readable change log
  refund-decision/
    v1.0.0.txt
    current.txt
    CHANGELOG.md

## v1.2.0 - 2026-05-15
- Tightened output format: added "no preamble" instruction after observing
  "Here is the classification:" prefix in 2.3% of outputs (broke parser)
- Added negative constraint for "request_for_refund" subcategory routing

## v1.1.0 - 2026-04-22
- Added "feature_request" category after PM request
- Expanded few-shot examples from 3 to 5 (added feature_request example
  and edge-case multi-issue example)

## v1.0.0 - 2026-03-10
- Initial release

Building a Regression Test Set

A regression test set is a collection of (input, expected_output) pairs that captures the behavior you care about preserving. When you change a prompt, you run the new version against the test set and compare results to the baseline.

How to build one:

1. Start with the happy path. 5-10 inputs representing normal expected usage, with the correct output labeled.
2. Add edge cases. Every edge case you handled deliberately in the prompt design should have a corresponding test case.
3. Add real failure cases. When you find a bug in production, add a test case for it before fixing it. The test case proves the fix worked and prevents regression.
4. Add adversarial inputs. Inputs that try to jailbreak scope, override format, or trigger edge-case behavior. The test verifies your constraints hold.
5. Target 30-50 test cases as the working size for most production prompts. Below 20 you have poor coverage; above 100 the evaluation time and cost starts to hurt iteration speed.

Manual vs. Automated Evaluation

For prompts with deterministic outputs (classification, extraction, structured generation), exact-match or schema-validation checks are sufficient. Write a script that runs the test set, compares outputs to expected values, and reports a pass/fail rate.

For prompts with quality-rated outputs (summaries, explanations, responses), exact-match is meaningless - there are many correct outputs. Two approaches:

• Rubric-based human evaluation: A human reviewer rates each output 1-5 on specific criteria (accuracy, tone, completeness). Reliable but slow and doesn’t scale to CI.
• LLM-as-judge: Use another LLM call to evaluate the output against a rubric. Fast and scalable enough for CI, with the caveat that the judge LLM has its own biases and should be validated against human judgment periodically.

You are evaluating the quality of an AI assistant's response.

Task: [description of the task the prompt performs]
Input: {user_input}
Response to evaluate: {llm_output}

Rate this response on the following criteria (1 = poor, 5 = excellent):
- Accuracy: Does it correctly address the request?
- Scope adherence: Does it stay within the defined scope?
- Format: Does it match the required output format?
- Completeness: Does it address all parts of the request?

Return ONLY a JSON object:
{
  "accuracy": [1-5],
  "scope_adherence": [1-5],
  "format": [1-5],
  "completeness": [1-5],
  "overall": [1-5],
  "issues": [list of specific issues, or empty list]
}

CI/CD Integration

The goal is to make prompt regressions as catchable as code regressions. A minimal CI integration:

1. On any change to a prompt file, the CI pipeline runs the regression test set against the new version.
2. Results are compared to a baseline (the previous version’s scores).
3. If overall accuracy drops by more than a threshold (commonly 5%), or if any critical test case fails, the CI step fails and the change requires manual review.
4. If tests pass, the change can be deployed. The baseline is updated to the new version’s scores.