Ensemble & Voting Pattern

The ensemble pattern queries multiple AI models and combines their answers to produce a more accurate, reliable result. Just as a panel of experts gives better advice than any single expert, multiple models voting on an answer consistently outperform individual models on complex tasks.

What Is the Ensemble Pattern?

The ensemble pattern is borrowed from classical machine learning, where combining predictions from multiple models (random forests, boosting, bagging) almost always outperforms any single model. The same principle applies to LLMs: different models have different strengths, different failure modes, and different biases. By combining their outputs, you get answers that are more accurate and more robust than any individual model.

The core idea is simple: send the same question to multiple models, collect their answers, and combine them. The combination method depends on the task: for classification, take the majority vote; for generation, have a judge pick the best response; for factual questions, look for consensus across models.

Ensembles trade cost and latency for accuracy and reliability. You are paying for multiple model calls instead of one. This tradeoff makes sense for high-stakes decisions (medical diagnosis, legal analysis, financial recommendations) where the cost of an error far exceeds the cost of extra model calls, and less sense for casual conversations where good enough is good enough.

Majority Voting

The simplest ensemble method: ask N models the same question and take the answer that most models agree on. This works best for tasks with discrete answers - classification, yes/no questions, multiple choice, or factual queries with definitive answers.

For majority voting to work well, you need:

• Diverse models: Models from different providers or families (Claude, GPT, Gemini) tend to have different biases. Models from the same family (different sizes of the same model) often make correlated errors, reducing the benefit of voting.
• Independent errors: The power of voting comes from the assumption that models fail independently. If all models make the same mistake on the same input, voting won't help. Diversity of training data and architecture is key.
• Odd number of voters: Use 3, 5, or 7 models to avoid ties. If you must use an even number, have a tiebreaking strategy (e.g., prefer the answer from the most capable model).

Weighted Voting

Not all models are equally good at all tasks. Weighted voting assigns higher weight to models that historically perform better on the task at hand. A model that's been right 90% of the time on similar questions gets more vote weight than one that's been right 70% of the time.

How to determine weights:

• Historical accuracy: Track each model's accuracy on a labeled evaluation set and use accuracy as the weight.
• Task-specific benchmarks: Weight based on performance on benchmarks relevant to your use case (e.g., coding benchmarks for code tasks, reasoning benchmarks for analytical tasks).
• Dynamic weights: Adjust weights based on recent performance. A model that's been degrading gets its weight reduced automatically.
• Confidence-weighted: Let each model report its own confidence, and weight votes by confidence. Caveat: models are often poorly calibrated.

LLM-as-Judge

Instead of mechanical voting, use a powerful LLM as a judge to evaluate the responses from multiple models and select the best one. The judge sees all candidate responses and picks the winner based on quality criteria you specify (accuracy, completeness, clarity, tone).

The LLM-as-judge pattern is particularly effective for open-ended generation tasks where there's no single "correct" answer - creative writing, explanation quality, code solutions with different approaches, and nuanced analysis where judgment is needed.

Key design choices for LLM-as-judge:

• Judge model selection: The judge should be at least as capable as the candidate models. Using a weaker model as judge degrades quality. Typically, you use the strongest available model (e.g., Claude Opus) as judge.
• Evaluation criteria: Provide specific, measurable criteria. "Pick the best response" is too vague. "Pick the response that is most factually accurate, complete, and clearly written" gives the judge clear guidance.
• Position bias mitigation: LLMs tend to prefer the first or last response they see. Randomize the order of candidate responses to reduce this bias.
• Explanation requirement: Ask the judge to explain its reasoning before giving the verdict. This improves decision quality (chain-of-thought effect) and provides useful debugging information.

The Debate Pattern

In the debate pattern, models don't just answer independently - they argue with each other. One model makes a claim, another critiques it, the first defends or revises its position, and a judge evaluates the debate to arrive at the best answer.

This is inspired by the adversarial collaboration concept: when intelligent agents with different perspectives argue constructively, the truth tends to emerge. It is especially powerful for complex reasoning tasks where a single model might miss important considerations.

A typical debate flow:

1. Initial positions: Two or more models independently answer the question
2. Cross-examination: Each model sees the other's answer and writes a critique, pointing out errors or weaknesses
3. Defense: Each model responds to the critiques, either defending its position with evidence or conceding and revising
4. Judgment: A judge model reads the full debate transcript and renders a final verdict, synthesizing the strongest arguments from both sides

The debate pattern is expensive (many model calls) but produces remarkably high-quality answers for complex questions. It is most valuable for tasks where correctness is critical and the cost of getting it wrong is high.

Mixture of Agents

The mixture of agents approach uses multiple specialized models, each expert in a different aspect of the task, and combines their outputs. Unlike voting where every model answers the same question, mixture of agents splits the work so each model handles what it's best at.

• Factual agent: A model fine-tuned for factual accuracy handles the information retrieval and verification
• Reasoning agent: A model strong in logical reasoning handles the analysis and inference
• Writing agent: A model optimized for clear communication handles the final writing and formatting
• Aggregator: A final model combines the outputs of all specialists into a coherent response

This pattern is particularly effective when you have access to models with clearly different strengths. For example, a small model that's been fine-tuned on your domain data for facts, combined with a large general model for reasoning and writing.

Self-Consistency

Self-consistency is an ensemble technique that uses a single model but samples multiple outputs with temperature > 0, then takes the majority answer. It is the cheapest form of ensembling because you only need one model API, but you still get the benefits of aggregating multiple answers.

Self-consistency works because temperature sampling explores different reasoning paths. Even though the model is the same, different samples may follow different chains of thought. When multiple reasoning paths arrive at the same answer, that answer is more likely to be correct.

Self-consistency is most effective for:

• Math and logic problems: Different reasoning paths to the same numerical answer strongly indicate correctness
• Multiple-choice questions: Discrete answers make majority voting straightforward
• Factual questions: When the model "knows" the answer, most samples will agree; when it's guessing, samples will disagree (useful as a confidence signal)

Code Example: Multi-Model Voting System

This system sends the same question to Claude, GPT, and Gemini in parallel, collects their answers, and uses a judge to select the best one:

import anthropic
import openai
import google.generativeai as genai
import asyncio
import json
import random

# Initialize clients
claude_client = anthropic.Anthropic()
openai_client = openai.OpenAI()
genai.configure(api_key="your-gemini-key")

async def query_claude(question: str) -> dict:
    """Query Claude and return the response."""
    response = claude_client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=2000,
        messages=[{"role": "user", "content": question}]
    )
    return {
        "model": "Claude Sonnet",
        "provider": "Anthropic",
        "response": response.content[0].text
    }

async def query_gpt(question: str) -> dict:
    """Query GPT and return the response."""
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        max_tokens=2000,
        messages=[{"role": "user", "content": question}]
    )
    return {
        "model": "GPT-4o",
        "provider": "OpenAI",
        "response": response.choices[0].message.content
    }

async def query_gemini(question: str) -> dict:
    """Query Gemini and return the response."""
    model = genai.GenerativeModel("gemini-1.5-pro")
    response = model.generate_content(question)
    return {
        "model": "Gemini 1.5 Pro",
        "provider": "Google",
        "response": response.text
    }

async def multi_model_vote(question: str) -> dict:
    """Query multiple models and have a judge select the best answer."""

    # Step 1: Query all models in parallel
    print("[Ensemble] Querying 3 models in parallel...")
    results = await asyncio.gather(
        query_claude(question),
        query_gpt(question),
        query_gemini(question),
        return_exceptions=True
    )

    # Filter out failures
    valid_results = [r for r in results if isinstance(r, dict)]
    if not valid_results:
        raise Exception("All models failed")

    print(f"[Ensemble] Got {len(valid_results)} responses")

    # Step 2: Randomize order to avoid position bias
    random.shuffle(valid_results)

    # Step 3: Have Claude Opus judge the responses
    candidate_text = "\n\n".join(
        f"--- Response from {r['model']} ({r['provider']}) ---\n{r['response']}"
        for r in valid_results
    )

    judge_response = claude_client.messages.create(
        model="claude-opus-4-20250514",
        max_tokens=2000,
        messages=[{
            "role": "user",
            "content": (
                "You are a judge evaluating multiple AI responses to a question.\n\n"
                f"QUESTION: {question}\n\n"
                f"CANDIDATE RESPONSES:\n{candidate_text}\n\n"
                "Evaluate each response on:\n"
                "1. Factual accuracy (most important)\n"
                "2. Completeness - does it fully answer the question?\n"
                "3. Clarity - is it well-organized and easy to understand?\n\n"
                "Return JSON with:\n"
                '- "winner": the model name of the best response\n'
                '- "reasoning": your evaluation of each response (2-3 sentences each)\n'
                '- "scores": {model_name: {accuracy: 1-5, completeness: 1-5, clarity: 1-5}}\n'
                '- "final_answer": the best response text (you may improve it slightly)'
            )
        }]
    )

    judgment = json.loads(judge_response.content[0].text)

    return {
        "question": question,
        "candidates": valid_results,
        "judgment": judgment,
        "winner": judgment["winner"],
        "final_answer": judgment["final_answer"]
    }

# Usage
result = asyncio.run(multi_model_vote(
    "What are the key differences between TCP and UDP, "
    "and when should you use each?"
))
print(f"Winner: {result['winner']}")
print(f"Answer: {result['final_answer']}")

Code Example: Self-Consistency with Temperature Sampling

This example samples multiple answers from a single model using temperature and takes the majority answer. It is the most cost-effective ensemble technique:

import anthropic
import json
from collections import Counter

client = anthropic.Anthropic()

def self_consistency(
    question: str,
    n_samples: int = 5,
    temperature: float = 0.7
) -> dict:
    """Sample multiple answers and take the majority vote."""

    # Step 1: Generate N samples with temperature
    samples = []
    for i in range(n_samples):
        response = client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=1000,
            temperature=temperature,
            messages=[{
                "role": "user",
                "content": (
                    f"{question}\n\n"
                    "Think step by step, then give your final answer on "
                    "the last line in the format: ANSWER: [your answer]"
                )
            }]
        )
        text = response.content[0].text
        samples.append(text)

    # Step 2: Extract final answers from each sample
    answers = []
    for sample in samples:
        lines = sample.strip().split("\n")
        for line in reversed(lines):
            if "ANSWER:" in line.upper():
                answer = line.split(":", 1)[1].strip()
                answers.append(answer)
                break

    # Step 3: Find majority answer
    if not answers:
        return {"error": "Could not extract answers from samples"}

    counter = Counter(answers)
    majority_answer, majority_count = counter.most_common(1)[0]
    confidence = majority_count / len(answers)

    # Step 4: Find the best reasoning for the majority answer
    best_reasoning = None
    for sample, answer in zip(samples, answers):
        if answer == majority_answer:
            best_reasoning = sample
            break

    return {
        "question": question,
        "answer": majority_answer,
        "confidence": confidence,
        "agreement": f"{majority_count}/{len(answers)}",
        "all_answers": dict(counter),
        "reasoning": best_reasoning,
        "n_samples": n_samples,
        "temperature": temperature
    }

# Usage - works great for math and logic
result = self_consistency(
    "A store offers 20% off, then an additional 15% off the discounted price. "
    "What is the total percentage discount from the original price?",
    n_samples=7,
    temperature=0.8
)

print(f"Answer: {result['answer']}")
print(f"Confidence: {result['confidence']:.0%}")
print(f"Agreement: {result['agreement']}")
print(f"All answers: {result['all_answers']}")

When Ensembles Are Worth the Cost

Ensembles are not always the right choice. They multiply your costs and add complexity. Use them when:

• High stakes: Medical diagnosis, legal analysis, financial decisions - where the cost of a wrong answer is orders of magnitude more than the cost of extra model calls
• No ground truth: When you can't easily verify the model's output, agreement among multiple models serves as a proxy for correctness
• Adversarial inputs: Different models are vulnerable to different adversarial attacks. An ensemble is harder to fool than any single model
• Benchmark evaluation: When evaluating model performance on benchmarks, ensembles provide more reliable scores than single model runs
• Compliance requirements: Some regulated industries require multiple independent assessments before a decision

Don't use ensembles when:

• The task is simple enough that a single model handles it reliably (>95% accuracy)
• Latency is critical and you can't parallelize the model calls
• Cost sensitivity is extreme and accuracy requirements are moderate
• The task is purely creative with no "correct" answer to converge on

Accuracy vs Latency vs Cost Tradeoffs

Comparison with Single Model Performance

Research consistently shows that ensembles outperform individual models, but the margin depends on task difficulty and model diversity:

• Easy tasks (>90% single-model accuracy): Ensembles add 1-3% accuracy. Rarely worth the cost unless error rates must be near zero.
• Medium tasks (70-90% single-model accuracy): Ensembles add 5-15% accuracy. This is the sweet spot where ensembles provide the most value for the cost.
• Hard tasks (<70% single-model accuracy): Ensembles add 8-20% accuracy. If individual models struggle, diverse ensembles can find the right answer through complementary strengths. However, if the task is fundamentally beyond current model capabilities, ensembles won't save you.

The key insight is that ensemble benefits follow diminishing returns. Going from 1 to 3 models provides most of the improvement. Going from 3 to 5 adds a little more. Going from 5 to 7 adds very little. Beyond 7, the improvement is usually negligible and not worth the cost.