The problem with a good-looking result
Most players have had a stretch where everything worked: a run of ranked wins, a tournament placement, a new character that seemed to click. The natural conclusion is that something improved. Sometimes that is correct. Often part of the result was luck, and the difficulty is that from the inside, the two feel identical.
The methods below are not about proving anything with certainty. They are habits that make it harder to fool yourself, and they work for individual players, coaches and anyone reading results.
Decide the test before you look at the result
The most useful single habit is to say in advance what would convince you. If you change a playstyle, decide on a number of games to play before judging it, and judge only after that many. Checking after each game and stopping when the numbers look good greatly raises the chance of a false conclusion, because random fluctuations will eventually look impressive at some point.
How many games is enough depends on the size of the effect you are hoping to detect. A large improvement can show in a few dozen games. A small one may need hundreds. The general principle is that a smaller effect requires a larger sample size before it can be told apart from noise.
Ask how often chance alone would do this
Before crediting skill, estimate how common a result would be if the player were merely average. Take someone who wins 8 of their first 10 games with a new character. For a player with a true 50% win rate, the chance of 8 or more wins in 10 is (45 + 10 + 1) / 1024 = 56 / 1024, or about 5.5%. That is unusual for any single player.
But the number of players matters too. If 20 people each try a new character and each has a true 50% rate, the chance that at least one of them wins 8 or more of 10 is about two in three. So an impressive streak from someone is expected, and the person who has it is simply the one the story gets told about. This is the point at which the question changes from “how unlikely is this result” to “how unlikely is it that anyone did this.”
It also connects to how random clumping works. Streaks appear in random sequences more often than intuition suggests, so a streak by itself should not be treated as a signal.
Expect regression toward the average
When a result is extreme, the next set of results is usually less extreme, even if nothing about the player changed. This is regression to the mean. A player who tops a tournament partly because of skill and partly because of favourable draws will, on average, do slightly worse next time, since the favourable draws will not repeat.
This has a practical use. If you are choosing between two players, or trying to predict next month’s performance, shrink extreme results toward the average. How much to shrink depends on how noisy the measure is. A very noisy measure deserves heavy shrinkage, and a stable one needs little.
The same effect distorts how people judge their own changes. A player who changes settings after a bad week often sees improvement, and credits the change, when part of the improvement was just bad luck fading.
Measure the process, not only the outcome
Outcomes are noisy, but some measures are less so. These are often called process measures: things you control and can observe directly.
- Accuracy on a clearly defined drill rather than match results.
- How often you took a decision you had previously identified as correct.
- Resource management stats such as how consistently you hit a timing target.
- Mistakes logged during replay review, counted per game.
A rise in a process measure alongside a rise in wins is more convincing than a rise in wins alone. A rise in wins with no change in the process is a reason for caution. Stat tracking in a spreadsheet makes these comparisons possible, because memory tends to flatter recent success.
This also supports results thinking in reverse: judging whether a decision was good given the information available, not by how the game ended.
Look for repeated, comparable conditions
Results are more informative when conditions are similar. Comparing 30 games against comparable opponents on one version of the game says more than comparing a mixed set of 30 across patches, modes and queues. If the conditions differ, ask whether the difference could explain the gap before attributing it to ability.
Repeated play against the same opponents, with sides swapped, is the cleanest version of this, and it is the least noisy way to compare two players directly.
Beware of choosing your best slice
A common error is cherry-picking: finding a subset of games where the numbers look good and presenting it as the story. “I am 70% on this map as the second side with this character” sounds precise, but if you searched through many such combinations, one of them was bound to look good by chance. A safer approach is to name the comparison before looking, and treat any pattern found afterwards as a hypothesis to test on new games.
Rating movements need the same care. Rating volatility can make a player look as if they are rising or falling over short windows when the underlying level has not moved much.
A short checklist
- Did I decide how many games to judge before looking?
- How often would an average player produce this by chance, and how many players were in the pool that produced the story?
- Would I expect the next run to be more ordinary?
- Did a process measure move as well, or only the outcomes?
- Were conditions comparable across the games I am counting?
- Did I pick this slice before or after seeing the data?
None of these questions will tell you for certain whether a result was earned. Their value is in lowering the chance of drawing a firm conclusion from a result that could easily have happened by luck.