Autonomous Video Studios
HomePART IV · Infrastructure, operations and economics
CHAPTER 21 · INFRASTRUCTURE

Media, backups and disaster recovery

The owner's rule: everything made is kept, in more than one place, and the whole system can be rebuilt from its backups on new infrastructure — turn-key.

GitHub · source

channels-workspace: all code, kits, docs pushed every four hours

GitHub · what runs

channels-deployed: every host's trees, units, cron, DB settings, DRIFT.md

Media archive

Google Cloud Storage, versioned, 90-day soft delete; copied hourly

Databases

Cloud SQL nightly + 7-day point-in-time; the portal's Postgres in host backups

DR.md — the turn-key runbook

what lives where, three ways back, steps per disaster, drills

1provision
2restore
3hold posting
4verify
5release
Figure 38. What the systems are recovered from, and the order a recovery follows.

The media archive#

Every video since the first day is in Google Cloud Storage: all 2,015 episodes of the original programme with their Facebook edits and thumbnails, every film, every kit, Blender asset and brand source, every record — 730 GB in all — and all 944 of ChannelForge's episodes in its own bucket. The original's archive keeps every overwritten version, and anything deleted from it can be restored for 90 days. Every hour a transfer job copies both buckets, whole, into a second pair of buckets kept in the archive storage class, so every video exists twice. The control host deletes a local video only when the archive holds it at its exact size.

Code and configuration#

Two private GitHub repositories are refreshed every four hours and after every deploy (every night until 8 November): the workspace (every line of code, the kits, the brand art, the documentation) and a snapshot of what actually runs on every host — code trees, services, timers, cron, database settings, GPU inventory, the workflows — with a drift report against the workspace. A secret scanner, which knows the estate's real secrets and the shapes of every kind of key, stands in front of both pushes; nothing secret is ever committed. Since 14 November both repositories are also kept as git bundles in a cloud bucket, so even a lost GitHub account would lose nothing.

Golden images in both clouds#

Every GPU box is made from a golden image, and the images are kept on both clouds: on AWS in two accounts, and on Google Cloud, where the image was rebuilt on 14 November with the whole current stack — the driver, Chrome, Node, the agent CLI, Blender, the word aligner and its weights, the 2.9 GB library of CC0 assets — and checked against a live AWS box: the same versions, and seven test renders identical to the pixel. What a new box would otherwise fetch from third parties (the Blender release, the asset library, the aligner's weights) is stored with its checksum in a bucket on each cloud. If one provider refused every start, both systems' boxes could be made on the other from the same recipe, in minutes.

The runbook#

DR.md describes where everything lives, the three ways back (standby, backups, rebuild from GitHub), the steps for each kind of disaster — a lost GPU box, a lost host, a lost database, a lost cloud account — and the drills that prove them. A restored host starts with posting held; the posting guard reads every platform before the first post; only then is posting released.