The tail is the product
Why we throw away aggregate F1 on tagging benchmarks, and what we report instead. A short argument with a long table attached.
Every tagging model we have ever benchmarked looks good on average and bad where it matters. This is not a coincidence, it is arithmetic: the head of a tag distribution contains most of the mass, so a model that ignores the tail can still post a respectable aggregate score.
The specific failure
Take a vocabulary of twelve thousand tags. Two hundred of them account for roughly seventy per cent of all annotations. A model that predicts only those two hundred perfectly, and nothing else at all, lands at an aggregate F1 that would have been publishable three years ago.
It is also useless. Nobody searches a corpus for girl.
What we report
Three bands, three sets of numbers, always together:
| Band | Definition | Why it exists |
|---|---|---|
| Head | top 200 tags | sanity check; should be near-saturated |
| Body | next 2,300 | where most real improvement happens |
| Tail | remaining 9,500 | where models quietly give up |
Reporting all three makes one class of paper impossible to write, which we consider the point.
The uncomfortable part
When we applied this retroactively to three published taggers, the rank order reversed. The model with the best headline number had the worst tail. We had cited it. That is roughly when this became a policy rather than an opinion.
A number you cannot defend in the follow-up thread is a number you should not have published.
Keep reading
Regions, not repairs
Inpainting treats a mask as damage. PrismStream treats it as composition. Notes from six weeks of getting that distinction to survive training.
Filters are opinions, so ship the score
On building Aurora: why every quality filter in the pipeline emits a column instead of deleting a row, and what that costs in storage.
Scheduling for four GPUs
Kubernetes answers a question small labs are not asking. Notes on writing a scheduler whose main feature is explaining itself.