Microsoft Phi Models Intermediate

Microsoft's Phi series has redefined what small language models can achieve. Starting with Phi-1 in 2023, each generation has pushed the boundaries of what is possible with fewer parameters, proving that data quality matters more than model size for many tasks.

The Phi Family

Data-Centric Training

1. Textbook-quality filtering

   Rather than training on all available web data, Phi models are trained on carefully filtered data that resembles high-quality textbooks - clear explanations, logical structure, and educational content.

2. Synthetic data generation

   GPT-4 and other large models generate training examples that demonstrate step-by-step reasoning, correct coding patterns, and structured problem-solving. This "distillation" transfers capabilities from large to small models.

3. Curriculum learning

   Training data is ordered from simpler to more complex examples, helping the model build foundational understanding before tackling harder problems.

Using Phi Models

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "microsoft/Phi-3-mini-4k-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

messages = [
    {"role": "user", "content": "Explain quicksort in simple terms."}
]
inputs = tokenizer.apply_chat_template(
    messages, return_tensors="pt"
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))