Sampling and voting

If a model gives a different answer each time you ask, asking once is a coin flip you did not know you were making. Sampling three times and taking the majority is the cheapest reliability improvement available, and the disagreement rate is a free confidence signal.

How it works

Run the same decision three or five times, with temperature above zero.
Take the majority answer.
Record how many votes it won by.
Treat a narrow margin as low confidence, which usually means escalate.
Never vote on anything with a side effect: vote on the decision, then act once.

Optional: load a real model

See it work

When it pays, and when it does not

Use it when	Skip it when
The step is a decision, not an action	The step has a side effect, where voting would mean doing it three times
Wrong answers are expensive	The task is cheap to redo, so a single attempt plus a retry is fine
You want a confidence signal for free	Latency matters more than accuracy, since voting multiplies it
The model disagrees with itself in testing	Temperature is zero and the output is already stable

How it fails

Voting on side effects

Three samples that each send an email sends three emails. Separate deciding from acting, always.

Majority of wrong

If the model is systematically wrong, three samples agree confidently. Voting fixes variance, never bias.

Ignoring the margin

The margin is the most useful output, and most implementations throw it away.

What it costs

Tokens	Three or five times the decision step. Only worth it on the steps that matter.
Latency	Parallel sampling keeps it near single-call latency, sequential does not.
Complexity	Small. This is one of the cheapest reliability patterns to add.

The margin is the product. Everyone remembers the majority answer and forgets that the vote count is a calibrated confidence number you got for nothing.

Related: Testing agents · Reflection · Routing · all patterns · agent jobs

Free from AI School - no signup, everything runs in your browser.