Autonomous Video Studios
HomePART II · The original programme
CHAPTER 10 · OPERATIONS

Self-healing operations

The owner's rule is simple: no slot is ever missed and no episode fails for ever. A sentinel enforces it. Every five minutes it observes every part of the pipeline, acts on what is wrong and records why.

CheckWhat it does when something is wrong
Slotsa slot n8n never fired is published on the spot; a slot that found nothing ready becomes an owed post
Stock (pace)if a slot would go empty before the next scheduled generation, it starts a top-up batch
Owed postsmade up from surplus stock only: never two posts on a channel within 110 minutes, never within 55 minutes of a slot
In stepa platform behind its siblings is caught up, so every platform posts the same episodes
Repaira post given up after three failures is retried at 2, 4, 8… hours, up to ten times
Catch-up generationmissed videos are made in gaps before the next scheduled run, charged to a budget
GPUsa box running with nothing asking for it, a reconciler that stopped, a start refused for hours
Cloud accountsan account that refuses to start a box is passed over like a full zone, and the pool's next box is started — in the other account
Yieldpre-launch work gives its GPUs to a live project whose batch has waited eight minutes for a box
Batchesa batch silent for ten minutes is marked failed, so the next one can start
YouTubea video YouTube leaves stuck in processing is uploaded again by itself
Archivea video kept on the host because the archive lacks it, a stale archive listing
Posting guarda platform that could not be checked before posting; a host whose records are behind
Hostdisk space, load, the portal's pulls, n8n's reachability and errors

Every incident is written to a log the portal shows as Eagle Eye's Observer, and errors are emailed. A failure the sentinel cannot fix by itself — a platform token the owner must renew, credits to top up at a provider — is said in plain words with what to do, and since 13 November with the lever that does it (the previous chapter).

Lessons, written down#

Every incident that cost something — a slot, a batch, a day of films — ends as a numbered lesson in the operations handbook, with what happened, why, and the rule that now prevents it: fourteen so far, from 'capacity before volume' and 'a retry must never erase the record of a failure' to 'one blocked cloud account must never stall a pool that has other boxes'. The same lessons are turned into the code's own checks.

Eagle Eye#

Eagle Eye is the read-only automation console: the pipeline as stages (schedules, generation, render, review, publish), every channel's state today, what is going wrong grouped by cause, and everything in time order. It reads the control host through a forced command that can only report — it cannot change anything.

Figure 17. Eagle Eye, the automation console (development build, demonstration data).