Where KICK agrees with the world, where it doesn't

What this page is

Any rating system can claim to be good. We score ours, publish the results, and commit to not shipping a new version unless the scores improve on binding pass criteria. This page is the permanent record.

Five internal metrics (computed from our own data) and three external metrics (external authoritative signals) feed into every validation run. The code is in validate_kick.py in our repo and runs in under a minute.

v1.1 results (current)

External signals

• Brownlow top-10 overlap: 4.36/10 average across 2000–2025 seasons.
• All-Australian squad overlap: 9.09/22 (42.7%) average across 1991–2025. KICK top-22 and AA 22 agree on fewer than half their selections — the strongest signal that our rating needs position-aware work.
• AFLCA Champion Player winner in KICK top-10: 82.6% hit rate (19/23 years since 2003). The four misses: Barry Hall (2005), Robbie Gray (2014), Dan Hannebery (2015), Zak Butters (2023). Two of the four are forwards — consistent with our internal position-gap finding.

Internal signals

• Position-coherence gaps (mean career KICK across players with 50+ games):
  • Midfielder: 59.5 baseline
  • Defender: 36.8 — 22.6 pts below mid
  • Ruck: 39.3 — 20.2 pts below mid
  • Forward: 27.5 — 31.9 pts below mid (the biggest visible gap)
• Era gap (1890s → 2020s top-100 mean): 45.8 points. Expected, given pre-2000 data sparsity. To be addressed by v1.3's era normaliser.
• Single-game volatility: forwards 30% of top-100 games — within the healthy range; goals are not over-weighted in aggregate.

v1.2 — what we tried, why we held

v1.2's target was to narrow the three position gaps without regressing Brownlow alignment. Binding pass criteria: D ≤ 19.0, R ≤ 16.0, F ≤ 27.0, Brownlow ≥ 4.1 (no more than −0.3 from v1.1).

We ran a 48-configuration weight sweep across five weights. Result: top configs narrowed the defender gap to ~18.9, but the forward gap barely moved (best 31.4 vs target 27.0). Brownlow alignment dropped 0.4–0.6 points across every top config. No config met all four criteria simultaneously.

Decision: held. Full sweep in data/validation/sweep/results.md.

v1.3 — same verdict, deeper explanation

v1.3 moved past weight tuning to three new mechanisms, with an expanded six-criterion bar (added AA squad overlap ≥ 50% and AFLCA winner hit rate ≥ 87%).

• Role-aware multipliers. Auto-classify each player's position from their stat profile; apply position-specific multipliers (defender: one_percenters ×2.5; forward: marks_inside_50 ×2.5; ruck: hit_outs ×1.4).
• Scoring-involvement composite. Forward-specific raw-score addition combining goals + goal_assists + 0.5×marks_inside_50.
• Era normaliser. Separate divisors for <1965 / 1965–1999 / 2000+ to address the 45.8-point era gap.

A 32-configuration mechanism sweep. Zero configs cleared all six criteria.

Pattern across all 32:

• Position gaps responded strongly to role_aware + era_on (best config: D12.5, R10.5, F12.5 — crushing all three gap targets).
• Every such config tanked Brownlow and AFLCA alignment — Brownlow dropped as low as 2.67, AFLCA to 56.5%.
• AA squad overlap never cleared 45.23% (max). The 50% target was unreachable with available mechanisms.
• AFLCA came within 0.04 of the 87% target at rank 8 (role_mild configs), but paid for it in Brownlow and D-gap.

The tension is structural. Every mechanism that lifts defenders/rucks/forwards relative to midfielders pushes mids out of year-by-year top-N rankings, which drags Brownlow and AFLCA down (both signals reward midfielders). You can't close the position gap AND hold top-10 alignment with the same lever.

Decision: held. Same process as v1.2 — criteria published in advance, criteria not met, no ship.

What v1.4 has to do differently

v1.2 taught us weights aren't enough. v1.3 taught us mechanisms layered on box-score data aren't enough either. The six-way tradeoff between position gaps, Brownlow alignment, AA overlap, and AFLCA hit rate is fundamental to what's in and what's missing from a public AFL box score.

v1.4 needs signals the box score doesn't carry:

• Hit-out-to-advantage. Ruckmen's actual value, not just total tap count. Champion Data has it; public box scores don't.
• Intercept marks. Distinct from uncontested-mark-in-defensive-50. Identifies the Harris Andrews / Sam Taylor impact directly.
• Score involvements. Goals + goal-assists + disposals-in-the-scoring-chain. Captures forward creation beyond what goal_assists alone records.
• Contested-mark-in-traffic flag. Distinguishes forward pack marks from rebound-50 uncontested grabs.

None of these are in AFL Tables. Options we'll work in order:

1. Scrape afl.com.au match centre pages directly for the modern era (2018+) — may expose some of these fields.
2. Reach out to community data projects (HPN Footy, Wheelo Ratings, Footy Industries) about access to their derived datasets.
3. Commercial option: Champion Data licence — five-figure annual cost, not yet justified by our revenue.

Until v1.4 has at least hit-out-to-advantage and intercept marks plumbed in, shipping new KICK versions against the same six criteria is an exercise in polishing the wrong surface. v1.1 stays live.

Full v1.4 plan: V1.4_PLAN.md in the repo.

Can you help?

If you know of a data source for hit-out-to-advantage, intercept marks, contested-possession-differential, or other box-score-adjacent metrics that would make KICK fairer to specific positions, tell us. hello@kicker.au.

The validation code and full reports are in the data/validation/ directory of our repo. No Kicker+ paywall, no subscription — this is the work we show alongside the rating.