Minecraft anticheat guide
Measuring Anticheat False Positives Through Appeals
Use appeal outcomes to estimate false-positive pressure: which fields to log, how to calculate overturn rates per check, and what the numbers cannot prove.
How appeals reveal false positives
An appeal that is reviewed and overturned is one of the few direct records that a punishment was wrong. Count those outcomes per check and per window. The result is a lower bound on false positives among punished players, because many players never appeal.
Log the fields an appeal analysis needs
Appeal data is only useful if the punishment record holds the check name and the violation level at the moment of punishment. Log these fields for every punishment and every appeal:
Calculate overturn rate per check
For each check, divide the number of overturned appeals by the number of punishments issued from that check in the same window. Keep punishment types in separate series. Mixing mitigation events with bans makes the rate hard to read.
Set a minimum sample size before you read any rate. A check with three punishments and one overturn is not a trend. The table shows the layout with placeholder values.
| Check | Punishments in window | Overturned appeals | Overturn rate | Sample OK? |
|---|---|---|---|---|
| [PLACEHOLDER: check A] | [PLACEHOLDER] | [PLACEHOLDER] | [PLACEHOLDER] | [PLACEHOLDER: yes/no] |
| [PLACEHOLDER: check B] | [PLACEHOLDER] | [PLACEHOLDER] | [PLACEHOLDER] | [PLACEHOLDER: yes/no] |
| [PLACEHOLDER: check C] | [PLACEHOLDER] | [PLACEHOLDER] | [PLACEHOLDER] | [PLACEHOLDER: yes/no] |
Handle appeals that arrive after a reversal
Some appeals arrive after a punishment was already lifted, for example after a staff spot check. Record these as a separate outcome, such as already reversed, instead of forcing them into upheld or overturned. Otherwise the overturn rate rises for reasons that have nothing to do with the check.
Also record whether the appeal included new evidence. An appeal with a screen recording or a replay is more informative than one that only says the player did nothing wrong. Keep the two groups separate when you read the numbers.
Why appeals undercount false positives
Appeals miss a lot. Some players never notice a punishment, some assume an appeal will fail, and some leave the server. Mitigation events often have no appeal path at all, so they never enter the appeal data. Treat the overturn rate as one signal among several, such as player reports and staff spot checks.
Use reason codes, not free text
Free-text reviewer notes cannot be counted. Define a short reason code list, such as lag or latency, legitimate client feature, evidence error, or confirmed cheating. Revisit the list each quarter, and keep an 'other' code with a required note so new patterns surface.
Read the result without overclaiming
An overturn caused by a reviewer error is still a false positive from the player's view, but it points to a process fix, not a detection fix. Separate those two causes in your reason codes. Before you change a threshold, check whether the overturns cluster around a version update, a hosting change, or a single day with lag.
Worked example of an overturn rate
Take a check with PLACEHOLDER inputs of [PLACEHOLDER: 120] punishments in the window, of which [PLACEHOLDER: 9] appeals were overturned. The overturn rate is [PLACEHOLDER: 9] / [PLACEHOLDER: 120], or [PLACEHOLDER: 7.5] percent. The figures are arithmetic inputs, not measurements. Before you act on a rate like this, check that the sample meets the minimum you set in advance, and that the punishment type matches the other rows you are comparing.
Step-by-step overturn analysis
Use this order each time you calculate the rate, so the numbers stay comparable from one review to the next:
Pitfalls in overturn analysis
The most frequent mistake is treating a low count as a clean record. A handful of punishments with no overturns says little, because the sample may be too small to show an error. Another mistake is comparing a month with a mass cleanup against a normal month. Mark any review batch that was cleared together, since one reviewer's decision moves the rate for every row at once.
Watch the appeal window too. An appeal filed late may land in the next window, which lowers this window's rate and raises the next. Use the filing date for the window, not the decision date, and say which one you used.
What we have not verified
This page does not give an acceptable overturn rate. We have not measured overturn rates across servers, so no published rate should be used as a target. The method above is a measurement plan for your own data, not evidence about any anticheat's accuracy.
FAQ
Does a low overturn rate prove the anticheat has no false positives?
No. It shows only the punishments that were appealed and reviewed. Players who never appeal, and mitigation events without an appeal path, are missing from the rate.
Should mitigation events count as punishments?
Track them in their own series. A hit cancellation or damage reduction has a different impact on the player than a ban, so combining them hides the pattern in both.
How long should we wait before reading the overturn rate?
Use a window long enough to reach your minimum sample size for the checks you care about. Decide the window and the sample minimum before you look at the data, so you do not pick the window that makes the rate look better.
Can we compare overturn rates between two checks?
Only when the sample sizes and punishment types are similar. Checks that issue different severities of punishment will show different appeal rates for reasons unrelated to accuracy, so compare like with like.