Scheduling for four GPUs
Kubernetes answers a question small labs are not asking. Notes on writing a scheduler whose main feature is explaining itself.
The lab has four GPUs. For a while we allocated them the way every small lab does: by asking in chat.
Why that stops working
It does not stop working because of contention. It stops working because of opacity. A run is queued, and nobody can tell whether it is queued because the hardware is busy, because it asked for more memory than exists, or because somebody's dead session is still holding a lock.
The feature that mattered
river why <job-id> prints, in plain sentences, the placement decision and
every constraint that failed. It took two days to write and eliminated most of
the chat traffic the scheduler was supposed to replace.
Keep reading
The tail is the product
Why we throw away aggregate F1 on tagging benchmarks, and what we report instead. A short argument with a long table attached.
Regions, not repairs
Inpainting treats a mask as damage. PrismStream treats it as composition. Notes from six weeks of getting that distinction to survive training.
Filters are opinions, so ship the score
On building Aurora: why every quality filter in the pipeline emits a column instead of deleting a row, and what that costs in storage.