Filters are opinions, so ship the score
On building Aurora: why every quality filter in the pipeline emits a column instead of deleting a row, and what that costs in storage.
Every large dataset is a set of deletions wearing a filename. The images that did not make it in are usually more informative about the dataset's biases than the ones that did, and they are almost never recoverable.
The rule
No filter in the Aurora pipeline deletes anything. Each one writes a column. The curated slice is then a query rather than a separate artefact:
SELECT * FROM aurora
WHERE aesthetic > 0.62
AND dedup_cluster_rank = 1
AND resolution_min >= 768
Anyone who disagrees with our thresholds writes a different query. Nobody has to re-crawl the internet to argue with us.
What it costs
About eleven per cent more storage and a genuinely annoying amount of pipeline complexity, because a filter that cannot delete has to be idempotent and versioned. We think it is worth it. Ask again after the next petabyte.
Keep reading
The tail is the product
Why we throw away aggregate F1 on tagging benchmarks, and what we report instead. A short argument with a long table attached.
Regions, not repairs
Inpainting treats a mask as damage. PrismStream treats it as composition. Notes from six weeks of getting that distinction to survive training.
Scheduling for four GPUs
Kubernetes answers a question small labs are not asking. Notes on writing a scheduler whose main feature is explaining itself.