Choosing the Right Model

With dozens of model types and hundreds of specific models available, selecting the right one for your project can feel overwhelming. This lesson provides a practical decision framework to help you navigate the landscape and make confident choices.

The Model Selection Decision Tree

Start by answering these questions in order. Each answer narrows your options significantly:

What is your PRIMARY data type?
|
+-- Text
|   +-- Generate text / chat / summarize  -->  LLM (Lesson 2)
|   +-- Search / find similar documents   -->  Embedding Model (Lesson 3)
|   +-- Classify / sentiment / spam       -->  Classification Model (Lesson 6)
|   +-- Extract structured data           -->  LLM or Fine-tuned Model (Lesson 9)
|
+-- Images
|   +-- Generate images from text         -->  Generative Model (Lesson 11)
|   +-- Classify / detect / segment       -->  Vision Model (Lesson 4)
|   +-- Describe images in text           -->  Multimodal Model (Lesson 10)
|
+-- Audio
|   +-- Transcribe speech to text         -->  Speech Model (Lesson 5)
|   +-- Generate speech from text         -->  Speech Model (Lesson 5)
|   +-- Classify audio / music            -->  Classification Model (Lesson 6)
|
+-- Tabular / Structured Data
|   +-- Predict a number (regression)     -->  Traditional ML (Lesson 8)
|   +-- Predict a category                -->  Traditional ML (Lesson 8)
|   +-- Find patterns / clusters          -->  Traditional ML (Lesson 8)
|   +-- Recommend items to users          -->  Recommendation Model (Lesson 7)
|
+-- Multiple Data Types
|   +-- Process text + images + audio     -->  Multimodal Model (Lesson 10)
|
+-- Sequential Decisions / Actions
    +-- Agent interacting with environment --> Reinforcement Learning (Lesson 12)

Factors to Consider

1. Data Type

The type of data you are working with is the single strongest signal for which model family to choose. Text data points you toward LLMs and embeddings. Image data points to vision models. Tabular data points to traditional ML. If you have multiple data types, multimodal models or multi-model pipelines are the answer.

2. Task Type

Within each data type, the specific task further narrows your options:

• Generation (creating new content): LLMs, image generators, speech synthesis, generative models
• Classification (categorizing inputs): Classification models, traditional ML, fine-tuned LLMs
• Search / Retrieval (finding similar items): Embedding models + vector databases
• Prediction (forecasting values): Traditional ML models (XGBoost, random forests)
• Recommendation (suggesting items): Recommendation models, collaborative filtering
• Decision-making (sequential actions): Reinforcement learning

3. Data Availability

• No training data: Use pre-trained APIs (GPT-4, Claude, Whisper). Zero-shot or few-shot prompting with LLMs.
• Small dataset (100-10K examples): Fine-tune a pre-trained model, or use few-shot prompting. Traditional ML models like random forests work well with small tabular datasets.
• Large dataset (10K+ examples): Train custom models, fine-tune extensively, or use traditional ML approaches that thrive on data volume.

4. Latency Requirements

• Real-time (<100ms): Small models, edge-deployed models, cached embeddings. Avoid large LLMs for real-time paths. Consider distilled or quantized models.
• Near real-time (100ms-2s): Most API-based models work here. Use smaller LLMs (GPT-4o-mini, Claude Haiku) for speed.
• Batch processing (seconds to minutes): Any model works. Optimize for accuracy over speed. Use the largest models for best quality.

5. Cost Constraints

• API costs: LLM APIs charge per token. At high volume, costs add up quickly. A single GPT-4 call costs ~$0.03-0.10, but millions of calls per day can cost thousands.
• Compute costs: Training or self-hosting large models requires expensive GPUs. A single A100 GPU costs ~$1-3/hour on cloud.
• Team expertise: Custom ML models require data scientists. API-based solutions can be built by any developer. Factor in the human cost.
• Free tier strategy: Many providers offer free tiers (Google Gemini, Hugging Face, Colab). Start there and scale when needed.

6. Accuracy Requirements

• High stakes (medical, legal, finance): Use the best models regardless of cost. Add human review. Consider ensemble approaches. Fine-tune for your domain.
• Medium stakes (business applications): Balance cost and accuracy. Mid-tier models with good prompting often suffice.
• Low stakes (internal tools, prototypes): Use the cheapest option that works. Small models and free tiers are often enough.

7. Interpretability Needs

• Must explain decisions: Traditional ML (decision trees, logistic regression, SHAP explanations). Avoid black-box neural networks in regulated industries.
• Nice to have: Use attention visualizations, chain-of-thought prompting, or LIME/SHAP for neural models.
• Not needed: Use whatever model gives the best performance.

8. Deployment Environment

• Cloud: Any model works. Use managed APIs for simplicity or self-host for control and cost optimization at scale.
• Edge / Mobile: Small, quantized models only. TensorFlow Lite, ONNX Runtime, Core ML. Models must be under ~100MB for comfortable mobile deployment.
• On-premises: Self-hosted open-source models (LLaMA, Mistral, Whisper). No data leaves your infrastructure.
• Browser: ONNX.js, TensorFlow.js, or WebGPU-accelerated models. Very limited model size.

Quick Reference: "I Want To..." Guide

Combining Models: Multi-Model Architectures

Real-world AI systems rarely use a single model. Instead, they combine multiple model types in a pipeline where each model handles what it does best:

RAG Pipeline (Retrieval-Augmented Generation)

• Embedding model converts documents and queries to vectors
• Vector database retrieves relevant documents
• LLM generates an answer grounded in the retrieved context
• Optionally: reranker model improves retrieval precision

Multimodal AI Assistant

• Speech model transcribes user audio input
• Vision model processes uploaded images
• LLM reasons about the combined text + image context
• Speech model converts the text response back to audio

Intelligent Document Processing

• Vision model (OCR) extracts text from scanned documents
• Classification model categorizes the document type
• LLM extracts structured fields (names, dates, amounts)
• Embedding model indexes the document for future search

Build vs. Buy Decision

Recommended Strategy

1. Prototype with APIs. Start with the best commercial APIs (GPT-4o, Claude, etc.) to validate your use case quickly.
2. Optimize with smaller models. Once validated, try smaller/cheaper models (GPT-4o-mini, Haiku, open-source) to reduce costs.
3. Self-host if needed. Only invest in self-hosting when you hit cost ceilings, need data privacy, or require deep customization.

Cost Comparison Across Model Types

The Model Selection Checklist

Follow this step-by-step checklist before committing to a model for your project:

1. Define the problem clearly. What input goes in? What output comes out? What does "good" look like? Write this down before looking at any models.
2. Identify your data type. Text, image, audio, tabular, or multi-modal? This immediately narrows your model family.
3. Determine your task type. Generation, classification, search, prediction, recommendation, or decision-making?
4. Assess your data. How much labeled data do you have? None means you need pre-trained models or zero-shot approaches.
5. Set your constraints. Maximum latency, budget per query, deployment environment, privacy requirements.
6. Start with the simplest approach. Try a prompt-based LLM API call before training a custom model. Try logistic regression before deep learning on tabular data.
7. Benchmark with your actual data. Run a quick evaluation on a sample of your real data, not just published benchmarks.
8. Consider the full pipeline. Do you need one model or several working together? Map out the complete data flow.
9. Plan for iteration. Your first model choice is rarely your last. Design your system to make it easy to swap models.
10. Monitor in production. Track accuracy, latency, and cost over time. Models degrade, data drifts, and requirements change.

Future Trends: Where Model Types Are Heading

• Convergence toward unified models: The boundary between model types is blurring. Models like GPT-4o handle text, images, and audio in a single architecture. Expect more "one model to rule them all" approaches.
• Smaller, faster, cheaper: Distillation, quantization, and architectural innovations are making powerful models accessible on consumer hardware. What needed a data center in 2023 runs on a laptop in 2026.
• Agentic systems: Individual models are becoming components in larger agent architectures that can plan, use tools, and collaborate. The model is no longer the product; the system is.
• Domain-specific models: While general models get better, specialized models for medicine, law, finance, and science continue to outperform generalists in their domains.
• On-device AI: Apple Intelligence, Google Gemini Nano, and Qualcomm's NPUs are pushing AI to run locally on phones and laptops. This trend will accelerate.
• Open source closing the gap: Open models (LLaMA, Mistral, Stable Diffusion) are rapidly approaching proprietary model quality, giving more control and reducing costs for everyone.

Course Completion