The GPU fleet
Rendering and the episode agents run on NVIDIA L4 GPUs (24 GB, Ada generation) in g6 machines on AWS. Each is a software twin made from a golden image — the same driver, Chrome, Blender, Node, Python packages, speech recogniser and agent CLI on every box — so a scene renders identically anywhere. The original programme's fleet has grown from 12 boxes on 3 November to 30, in 13 pools: one each for Projects 1 and 2, four Studio lanes, the films, and six for Blender episodes.
Each box is started only while work waits for it and stopped two to five minutes after the last job ('silence means off').
Reconcilers: GPUs on only while working#
Work declares intent; a reconciler per pool turns intent into machines. A batch or job keeps touching an intent file while it needs a GPU; every minute the pool's reconciler makes the provider state match: a fresh intent starts a box (or keeps one running), a stale one stops it. Nothing has to remember to clean up — silence means off — and absolute caps stop any box that runs too long. A box serving a batch is never switched away from mid-run, and a box's SSH host keys are pinned by name so a recycled address can never reach the wrong machine.
Capacity and quotas#
Cloud GPUs are scarce: on 3 November alone, L4 starts were refused in three us-east-1 zones at different times of the day. The fleet is built for it: boxes sit in different zones, regions and accounts, a refusal moves on to the pool's next box, and a refusal of the account itself is treated the same way. The work is fitted to the quotas there are rather than the other way round: the Studio's newest projects use lanes of their own, kept off the live projects' boxes, and give way to them whenever a live batch has waited eight minutes. Every new box is checked — GPU, driver, Vulkan, software versions — and proven by rendering reference stills byte-identical to the fleet's references before it joins a pool.