The problem with small samples
You try a new strategy and win four of your next five games. It is tempting to conclude that it works. But a player who wins half their games will win four or more out of five fairly often by chance alone. A short run says very little.
The central idea is that results are a mix of skill and luck, and luck averages out only over many games. The more games you have, the closer the observed win rate tends to be to your true underlying rate. How many is “enough” depends on how big a difference you are trying to detect.
A rough measure of the noise
Treat each game as an independent win or loss with the same chance of winning. This is a simplification, since opponents, teammates and your own form vary, but it gives a useful ballpark. If your true win chance is p, then over n games the win rate typically wanders from p by about the square root of p times (1 minus p) divided by n. This quantity is called the standard error.
For a true win rate of 50%, p times (1 minus p) is 0.25, so:
| Games played | Standard error | Rough 95% range around 50% |
|---|---|---|
| 25 | 10 points | about 30% to 70% |
| 100 | 5 points | about 40% to 60% |
| 400 | 2.5 points | about 45% to 55% |
The rough range is about two standard errors either side (1.96, rounded to 2 here). So over 100 games, a player whose real win chance is 50% will usually finish somewhere from about 40% to 60%. Over 25 games, a result anywhere from 30% to 70% would not be surprising.
Notice that quadrupling the number of games only halves the noise. That is why confirming small improvements takes so long.
Worked example
Suppose your win rate has historically been around 50%. After changing your strategy, you win 56 of your next 100 games.
A 56% win rate is 6 points above 50%. The noise over 100 games is about 5 points (one standard error), and the rough range is plus or minus 10. A 6 point difference falls well inside that range, so this result is consistent with no real improvement at all. It does not show that the change failed either. It only shows that 100 games cannot settle the question.
Now imagine the same 6 point gap sustained over 400 games, say 224 wins. Here the standard error is 2.5 points, and 6 points is a bit more than two standard errors, so it starts to look like a real difference, though it is still not proof. A larger sample is what lets a modest improvement stand out from the noise.
How many games do you need?
Detecting a small edge needs many games. Very roughly, to notice a 5 point improvement from a 50% baseline with reasonable confidence, you may need several hundred games for each condition, since you are comparing two noisy numbers (before and after). Detecting a 20 point gain needs far fewer. The smaller the real effect, the larger the sample you need, so be realistic about what you can test. A casual player with ten games a week would take months to judge a subtle change by win rate alone.
Practical consequences
- Do not judge by streaks. Winning or losing runs occur naturally in random sequences, as described under random clumping. They tempt you to see patterns that are not there, including the hot hand belief.
- Use measures with less noise. If you can track a specific behaviour that you control, such as how often you completed an opening correctly, a few dozen games may reveal whether the habit changed, even if wins are too noisy to show it. Win rate is the noisiest and slowest signal.
- Change one thing at a time. If you alter several things together, even a large sample cannot tell you which one mattered.
- Fix the test in advance. Decide how many games you will play before looking at the result. Stopping the moment the numbers look good makes false positives much more likely.
- Keep conditions comparable. Different ranks, teammates, maps, patches and times of day all change the baseline. A balance patch in the middle of a test can invalidate it, as can playing a different opponent pool.
- Expect regression. An unusually good run tends to be followed by something closer to average, and this applies to your rank as well. Rating volatility is partly the same effect.
Ratings and sample size
Competitive ratings face the same issue. A rating that has moved after a few games carries a lot of uncertainty, which is why systems that track rating uncertainty widen their estimates for new or inactive players. When you read someone’s win rate, ask how many games it comes from. A 70% record over 20 games is much weaker evidence than 58% over 500. The wider question of how much of a result is skill is covered under separating luck from skill.
A reasonable approach
Test changes with a plan, not a feeling. Pick a behaviour measure and a win rate measure, play a fixed batch of games, and compare with your earlier numbers while remembering how wide the noise is. If the evidence is unclear, which it often will be, you can keep the change if it makes sense for other reasons, continue testing, or go back. Be wary of strong conclusions in either direction from small batches, and accept that some changes will stay uncertain.