LumenEval
Benchmarks with their failure cases attached
An evaluation harness that refuses to emit a single number without the error analysis that explains it.
- Category
- Evaluation
- Status
- Active
- Language
- Python
- Licence
- Apache-2.0
- Stars
- 640
- Since
- 2026
A leaderboard without error analysis is marketing. LumenEval is the harness we built after publishing one number we could not defend in the follow-up thread.
What a run emits
- Aggregate metrics, stratified by whatever axes the task declares.
- The worst hundred cases, rendered.
- The exact config, dataset revision and model revision, hashed together.
Status
Active, and used for every number this site publishes.
Writing
The tail is the product
Why we throw away aggregate F1 on tagging benchmarks, and what we report instead. A short argument with a long table attached.
Related projects
Aurora Corpus
The dataset everything else is trained on
An 18M-image illustration corpus with tags, captions and quality bands, plus the whole filtering pipeline that produced it.
- Language
- Python
- Stars
- 980
- Licence
- CC-BY-4.0
- Category
- Data
BluzenTagger
Multi-label tagging for illustration
A tagger over a 12k-tag vocabulary that holds up on the long tail, where general-purpose vision models quietly collapse into the fifty most common labels.
- Language
- Python
- Stars
- 1.4k
- Licence
- Apache-2.0
- Category
- Research
MochiCore
Adapter training that fits on one GPU
The trainer the rest of the lab runs on. Adapter families, quantised paths and a config format you can read out loud.
- Language
- Python
- Stars
- 3.6k
- Licence
- MIT
- Category
- Training