Evaluating Translation Quality Intermediate

How do you know if a translation is good? This lesson covers automatic metrics (BLEU, chrF, COMET), their strengths and limitations, and how to set up human evaluation for production translation systems.

Automatic Metrics Overview

Metric What It Measures Range Correlation with Humans
BLEU N-gram overlap with reference 0-100 Moderate
chrF Character-level F-score 0-100 Good
COMET Neural quality estimation 0-1 High
TER Edit distance from reference 0-inf (lower better) Moderate

BLEU Score

BLEU (Bilingual Evaluation Understudy) is the most widely used MT metric. It measures how many n-grams in the machine translation appear in the reference translation:

Python 
import sacrebleu

# Reference translations (can have multiple references)
refs = [["The cat sat on the mat."]]

# System output (hypothesis)
hyps = ["The cat is sitting on the mat."]

# Calculate BLEU
bleu = sacrebleu.corpus_bleu(hyps, refs)
print(f"BLEU: {bleu.score:.1f}")
print(bleu)  # Shows detailed breakdown

# Interpreting BLEU scores:
# < 10: Almost useless
# 10-19: Hard to understand
# 20-29: Clear gist, significant errors
# 30-39: Understandable, some errors
# 40-49: High quality
# 50+: Very high quality / near-human

chrF Score

chrF measures character-level overlap and works better for morphologically rich languages:

Python 
# chrF is available in sacrebleu
chrf = sacrebleu.corpus_chrf(hyps, refs)
print(f"chrF: {chrf.score:.1f}")

COMET Score

COMET uses a trained neural model to estimate translation quality and correlates much better with human judgments:

Python 
from comet import download_model, load_from_checkpoint

# Download and load COMET model
model_path = download_model("Unbabel/wmt22-comet-da")
model = load_from_checkpoint(model_path)

# COMET uses source, hypothesis, and reference
data = [{
    "src": "Die Katze sitzt auf der Matte.",
    "mt": "The cat is sitting on the mat.",
    "ref": "The cat sat on the mat."
}]

output = model.predict(data, batch_size=8)
print(f"COMET: {output.system_score:.4f}")
Which metric should you use? Use BLEU for quick comparisons and reporting (it is the industry standard). Use COMET for the most reliable quality assessment. Always combine automatic metrics with periodic human evaluation for critical applications.

Human Evaluation

For production systems, set up a human evaluation process:

  • Adequacy - Does the translation convey the same meaning? (1-5 scale)
  • Fluency - Does it read naturally in the target language? (1-5 scale)
  • Ranking - Given two translations, which is better?
  • Post-edit distance - How much does a translator need to change?

Try It Yourself

Translate 50 sentences with two different models, compute BLEU and COMET for both, and compare the rankings. Do the metrics agree?

Next: Best Practices →
← Fine-Tuning Best Practices →

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka
📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp
🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX