Few-Shot Design That Scales

Few-shot examples are expensive to write and easy to get wrong. Here is how to design a set that improves output quality without brittleness or token waste.

When Few-Shot Is Worth the Token Cost

Few-shot examples add tokens to every request. Before building a few-shot set, decide if the task actually needs one. The test: can you describe the output format and quality bar in explicit instructions, or does showing an example communicate something that words can’t?

How Many Examples

The practical range for production few-shot sets is 2-8 examples in the prompt at any time. Beyond 8, marginal returns fall rapidly and token cost accumulates. Below 2, a single bad example can anchor the model’s behavior in the wrong direction.

The heuristic that holds in practice: start with 3. Evaluate accuracy. Add examples for specific failure cases you observe. Stop adding when new examples don’t measurably improve the test set. Most tasks plateau at 3-5 examples in the prompt.

Coverage: How to Choose Representative Examples

The most common mistake in few-shot design is choosing examples that look good rather than examples that cover the failure space. Good coverage means your examples collectively represent the range of inputs your system will see - not just the easy ones.

A practical selection process:

1. Identify 3-4 input archetypes that represent the real distribution. For a customer support classifier, these might be: straightforward billing question, angry tone with technical issue, vague unclear request, multi-issue message.
2. Write one example per archetype. Each example covers a different region of the input space.
3. Include at least one edge case that is semantically ambiguous or atypical. The model learns from how you handled the hard case more than from how you handled the easy ones.
4. Add examples for specific failure modes you observe after deployment. When you find a class of inputs the model handles poorly, add an example that demonstrates the correct behavior.

Static vs. Dynamic Few-Shot Injection

Static few-shot means the same examples are included in every prompt. Dynamic few-shot means the examples are selected at runtime based on the input query.

Static is simpler and usually sufficient for tasks with a relatively uniform input distribution. Dynamic is worth implementing when:

• The task has many distinct input types that each benefit from different examples
• You have accumulated many examples and can’t include all of them (token budget)
• You want to improve accuracy by always showing examples similar to the current input

# Build an embedding-indexed library of (input, expected_output) pairs
example_library = EmbeddingIndex(labeled_examples)

def build_prompt(user_input):
    # Find the 3 examples most similar to this input
    relevant_examples = example_library.search(user_input, top_k=3)

    prompt = system_instructions + "\n\n"
    for ex in relevant_examples:
        prompt += f"Input: {ex.input}\nOutput: {ex.output}\n\n"
    prompt += f"Input: {user_input}\nOutput:"
    return prompt

The embedding step adds latency (typically 10-50ms for a fast embedding model) but can significantly improve accuracy on tasks where the input distribution is wide. The tradeoff is worth measuring concretely for your task before committing to the infrastructure cost.

The Edge-Case Coverage Problem

No finite set of examples covers all possible inputs. When the model receives an input that doesn’t match any of its examples, it generalizes - and that generalization may be wrong in surprising ways. Two strategies for the gaps:

• Add a catch-all example that demonstrates how to handle inputs that don’t fit the primary categories (e.g., a “this doesn’t fit our support categories - route to general inquiry” example).
• Add a confidence field to your output schema. When the model is uncertain, it should say so. A confidence: 0.4 output you route to a human fallback is better than a confident wrong classification.

Maintaining Few-Shot Libraries

A few-shot library that is not maintained is a reliability liability. Examples that were correct when written may become incorrect as the task definition evolves, the product changes, or the input distribution shifts. Treat your few-shot library as a versioned asset:

• Store examples in a version-controlled file (JSON or YAML), not hardcoded in the prompt string.
• Label each example with the date it was added and the reason (initial coverage, edge case, failure fix).
• Review the library quarterly for examples that no longer reflect current correct behavior.
• When you change what “correct” means for a task, update the examples in the same change - stale examples silently teach the wrong behavior.