Eval Rubric Builder

Define the criteria that matter for your LLM outputs, weight them, and get two ready-to-use artifacts: a clean markdown rubric for humans and an LLM-as-judge prompt with a strict JSON scoring schema.

Criteria

Weights are auto-normalized to 100%. Scale anchors (1-5) are auto-suggested from the criterion name; edit them freely.

Markdown rubric (for humans)

LLM-as-judge prompt (strict JSON output)

How to use the judge prompt: replace the {{OUTPUT_TO_EVALUATE}} placeholder (and optionally {{TASK_CONTEXT}}) with your data at eval time, send it to a strong model at temperature 0, and parse the JSON response. Weighted total is computed from normalized weights, so scores are comparable across rubric versions. Tips: keep criteria independent of each other, use 3-6 criteria (more dilutes signal), and spot-check the judge against a handful of human-graded examples before trusting it at scale.