Autonomous Video Studios
HomePART IV · Infrastructure, operations and economics
CHAPTER 20 · INFRASTRUCTURE

The GPU fleet

Rendering and the episode agents run on NVIDIA L4 GPUs (24 GB, Ada generation) in g6 machines on AWS. Each is a software twin made from a golden image — the same driver, Chrome, Blender, Node, Python packages, speech recogniser and agent CLI on every box — so a scene renders identically anywhere. The original programme's fleet has grown from 12 boxes on 3 November to 30, in 13 pools: one each for Projects 1 and 2, four Studio lanes, the films, and six for Blender episodes.

Pool
The boxes it tries, in order
Project 1
1c
1a
Ohio b
1c
Project 2
1d
1b
Ohio b
1d
Studio lane 1
1d
1c
Ohio b
1c
Studio lane 2
1d
1c
Ohio c
1c
Studio lane 3
1c
1c
Studio lane 4
1b
1d
Long-form films
1c
1d
Ohio a
Blender · Project 1
1d
Blender · Project 2
1c
Blender · Studio 1
1d
Blender · Studio 2
1c
Blender · Studio 3
1c
Blender · Studio 4
1c
1a
ChannelForge
1a
1c
AWS account oneAWS account twoChannelForge's own fleet

Each box is started only while work waits for it and stopped two to five minutes after the last job ('silence means off').

Figure 36. The original programme's 13 GPU pools and ChannelForge's fleet: the boxes each pool tries, in order, by zone (1a–1d: us-east-1's zones; Ohio: us-east-2) and by account. A zone without L4 capacity, or an account that refuses the start, sends it to the pool's next box.

Reconcilers: GPUs on only while working#

Work declares intent; a reconciler per pool turns intent into machines. A batch or job keeps touching an intent file while it needs a GPU; every minute the pool's reconciler makes the provider state match: a fresh intent starts a box (or keeps one running), a stale one stops it. Nothing has to remember to clean up — silence means off — and absolute caps stop any box that runs too long. A box serving a batch is never switched away from mid-run, and a box's SSH host keys are pinned by name so a recycled address can never reach the wrong machine.

Capacity and quotas#

Cloud GPUs are scarce: on 3 November alone, L4 starts were refused in three us-east-1 zones at different times of the day. The fleet is built for it: boxes sit in different zones, regions and accounts, a refusal moves on to the pool's next box, and a refusal of the account itself is treated the same way. The work is fitted to the quotas there are rather than the other way round: the Studio's newest projects use lanes of their own, kept off the live projects' boxes, and give way to them whenever a live batch has waited eight minutes. Every new box is checked — GPU, driver, Vulkan, software versions — and proven by rendering reference stills byte-identical to the fleet's references before it joins a pool.

Project 1442 hProject 2293 hStudio lane 1522 hStudio lane 2413 hStudio lane 341 hStudio lane 452 hLong-form films230 hBlender · Project 1103 hBlender · Project 2100 hBlender · Studio 1101 hBlender · Studio 272 hBlender · Studio 336 hBlender · Studio 446 h
Figure 37. Billed L4 hours by pool, 12 October – 14 November 2026: 2,451 hours in all. The Studio's first lane has run the most; the Blender pools began on 10 November.