Self-healing operations
The owner's rule is simple: no slot is ever missed and no episode fails for ever. A sentinel enforces it. Every five minutes it observes every part of the pipeline, acts on what is wrong and records why.
| Check | What it does when something is wrong |
|---|---|
| Slots | a slot n8n never fired is published on the spot; a slot that found nothing ready becomes an owed post |
| Stock (pace) | if a slot would go empty before the next scheduled generation, it starts a top-up batch |
| Owed posts | made up from surplus stock only: never two posts on a channel within 110 minutes, never within 55 minutes of a slot |
| In step | a platform behind its siblings is caught up, so every platform posts the same episodes |
| Repair | a post given up after three failures is retried at 2, 4, 8… hours, up to ten times |
| Catch-up generation | missed videos are made in gaps before the next scheduled run, charged to a budget |
| GPUs | a box running with nothing asking for it, a reconciler that stopped, a start refused for hours |
| Cloud accounts | an account that refuses to start a box is passed over like a full zone, and the pool's next box is started — in the other account |
| Yield | pre-launch work gives its GPUs to a live project whose batch has waited eight minutes for a box |
| Batches | a batch silent for ten minutes is marked failed, so the next one can start |
| YouTube | a video YouTube leaves stuck in processing is uploaded again by itself |
| Archive | a video kept on the host because the archive lacks it, a stale archive listing |
| Posting guard | a platform that could not be checked before posting; a host whose records are behind |
| Host | disk space, load, the portal's pulls, n8n's reachability and errors |
Every incident is written to a log the portal shows as Eagle Eye's Observer, and errors are emailed. A failure the sentinel cannot fix by itself — a platform token the owner must renew, credits to top up at a provider — is said in plain words with what to do, and since 13 November with the lever that does it (the previous chapter).
Lessons, written down#
Every incident that cost something — a slot, a batch, a day of films — ends as a numbered lesson in the operations handbook, with what happened, why, and the rule that now prevents it: fourteen so far, from 'capacity before volume' and 'a retry must never erase the record of a failure' to 'one blocked cloud account must never stall a pool that has other boxes'. The same lessons are turned into the code's own checks.
Eagle Eye#
Eagle Eye is the read-only automation console: the pipeline as stages (schedules, generation, render, review, publish), every channel's state today, what is going wrong grouped by cause, and everything in time order. It reads the control host through a forced command that can only report — it cannot change anything.