Google Gemma Models Intermediate

Gemma is Google's family of lightweight, open models built from the same research and technology used to create the Gemini models. Available in 2B, 7B, 9B, and 27B parameter variants, Gemma models bring Google-level AI capabilities to developers who need efficient, deployable models.

Gemma Model Variants

Architecture Highlights

1. Sliding Window Attention

   Gemma 2 alternates between local sliding window attention and full global attention layers. This reduces memory usage while maintaining long-range understanding.

2. Knowledge Distillation

   Smaller Gemma models are trained using outputs from larger Gemini models as "soft targets." This transfers nuanced understanding that would be impossible to learn from text data alone at this scale.

3. RoPE Embeddings

   Rotary Position Embeddings enable better handling of positional information and support context length extension through fine-tuning.

Fine-Tuning Gemma

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-9b-it",
    torch_dtype="auto",
    device_map="auto"
)

# LoRA configuration for efficient fine-tuning
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
print(f"Trainable params: {model.print_trainable_parameters()}")

Gemma vs. Phi: When to Choose Which

• Choose Gemma when you need strong multilingual support, want Google ecosystem integration (Vertex AI, Colab), or need the 27B size point.
• Choose Phi when you need the smallest possible model (1.3-3.8B), want MIT licensing flexibility, or prioritize coding and reasoning tasks.
• Benchmark both on your specific task - performance varies significantly across tasks and domains, and the "best" model depends on your use case.