Fine-Tuning Translation Models Advanced

Generic translation models often struggle with domain-specific terminology (medical, legal, technical). Fine-tuning a pretrained model on your own parallel data can dramatically improve translation quality for your specific use case. This lesson walks through fine-tuning MarianMT and NLLB-200 using Hugging Face.

When to Fine-Tune

Fine-tune when:
  • You have domain-specific terminology (medical, legal, financial)
  • Generic APIs produce poor quality for your language pair
  • You need consistent terminology across translations
  • You have at least 5,000-10,000 parallel sentence pairs

Preparing Parallel Data

Python 
import pandas as pd
from datasets import Dataset

# Load your parallel data
data = pd.read_csv("parallel_data.csv")
# Columns: 'source' (English), 'target' (German)

# Split into train/validation
train_data = data.sample(frac=0.9, random_state=42)
val_data = data.drop(train_data.index)

# Convert to Hugging Face Dataset
train_dataset = Dataset.from_pandas(train_data)
val_dataset = Dataset.from_pandas(val_data)

print(f"Train: {len(train_dataset)}, Val: {len(val_dataset)}")

Fine-Tuning MarianMT

Python 
from transformers import (
    MarianMTModel, MarianTokenizer,
    Seq2SeqTrainer, Seq2SeqTrainingArguments,
    DataCollatorForSeq2Seq
)

model_name = "Helsinki-NLP/opus-mt-en-de"
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = MarianMTModel.from_pretrained(model_name)

# Tokenize the data
def preprocess(examples):
    inputs = tokenizer(
        examples["source"], max_length=128,
        truncation=True, padding="max_length"
    )
    targets = tokenizer(
        text_target=examples["target"], max_length=128,
        truncation=True, padding="max_length"
    )
    inputs["labels"] = targets["input_ids"]
    return inputs

train_tokenized = train_dataset.map(preprocess, batched=True)
val_tokenized = val_dataset.map(preprocess, batched=True)

# Training configuration
training_args = Seq2SeqTrainingArguments(
    output_dir="./mt-finetuned",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    learning_rate=2e-5,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    predict_with_generate=True,
    fp16=True,
)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=train_tokenized,
    eval_dataset=val_tokenized,
    tokenizer=tokenizer,
    data_collator=DataCollatorForSeq2Seq(tokenizer, model=model),
)

trainer.train()

Fine-Tuning NLLB-200

NLLB-200 (No Language Left Behind) supports 200+ languages, making it ideal for low-resource language pairs:

Python 
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

# Load NLLB-200 (distilled version for faster training)
model_name = "facebook/nllb-200-distilled-600M"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Set source and target language codes
tokenizer.src_lang = "eng_Latn"  # English
target_lang = "swh_Latn"       # Swahili

# Translate (before fine-tuning)
inputs = tokenizer("The patient has a fever.", return_tensors="pt")
translated = model.generate(
    **inputs,
    forced_bos_token_id=tokenizer.convert_tokens_to_ids(target_lang)
)
print(tokenizer.decode(translated[0], skip_special_tokens=True))

# Fine-tuning follows the same Seq2SeqTrainer pattern
# as MarianMT above, with NLLB-specific language codes

Data Augmentation with Back-Translation

When you have limited parallel data, back-translation can generate synthetic training pairs:

Python 
# 1. Collect monolingual target language text
target_sentences = ["Der Patient hat Fieber.", ...]

# 2. Translate back to source language using a reverse model
reverse_model = MarianMTModel.from_pretrained("Helsinki-NLP/opus-mt-de-en")
reverse_tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-de-en")

synthetic_pairs = []
for tgt in target_sentences:
    inputs = reverse_tokenizer(tgt, return_tensors="pt")
    translated = reverse_model.generate(**inputs)
    src = reverse_tokenizer.decode(translated[0], skip_special_tokens=True)
    synthetic_pairs.append({"source": src, "target": tgt})

# 3. Combine with real parallel data and fine-tune
Training Tips:
  • Start with a low learning rate (1e-5 to 3e-5) to avoid catastrophic forgetting
  • Use early stopping based on validation BLEU score
  • Mix domain-specific data with general data (80/20) to maintain general translation ability
  • Fine-tune for 3-5 epochs; more can cause overfitting on small datasets

Try It Yourself

Download a small parallel corpus from OPUS, fine-tune a MarianMT model for 3 epochs, and compare the output with the base model on domain-specific sentences.

Next: Evaluation →
← Translation APIs Evaluation →

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka
📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp
🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX