Storing 90 days of video from 10,000 cameras, on site

Video is the workload that punishes every default. It never stops, it is large, and the person who needs a clip needs it now, at the site, while the uplink is down. This is a worked example at a scale where the arithmetic decides the architecture: ten thousand cameras across two hundred sites, ninety days of retention, analytics running on the same footage.

cameras (example)

10,000

sites

200

retention

90 days

per stream

1080p / 15 fps

The arithmetic

Start from the bitrate, not from the drive catalogue. A 1080p stream at 15 frames per second in H.265, on a mostly static scene, sits around 1 Mbit/s; H.264 roughly doubles it. Everything else follows:

1 Mbit/s  = 0.125 MB/s
per camera per day   = 0.125 × 86,400 s  ≈ 10.8 GB
per camera, 90 days  ≈ 0.97 TB
per site, 50 cameras ≈ 48.6 TB of video to keep at all times

That is the payload. Storage that survives a dead drive costs more than the payload, and that overhead is the first design decision.

Why not the cloud

Push the same site to an object store in a region and you are uploading 50 Mbit/s per site, continuously, forever, over links that were sized for a web browser. Two hundred sites is 10 Gbit/s of sustained upload before you have watched a single clip. Retrieval bills egress. A public-sector operator may not be allowed to move the footage off national territory at all. And the analytics has to run next to the video anyway, because moving frames to a GPU somewhere else costs more than the GPU.

So the storage lives at the site, and the site has to be its own small cloud.

One cluster per site

Four commodity nodes per site, each with the same drives, running Kubernetes with the video management system, the analytics containers and MinIO on the same hardware. MinIO turns the drives of all four nodes into one erasure-coded pool:

# each node runs the same command; MinIO discovers the set from the expansion
minio server http://node{1...4}.site42.local/mnt/disk{1...4} \
  --console-address ":9001"

With 16 drives and the default parity of 4 (EC:4), any four drives, or one entire node, can fail with no data loss, and the usable capacity is 75% of raw. Sizing the site becomes a division:

payload to keep            48.6 TB
usable ratio at EC:4      × 1/0.75  →  64.8 TB raw
plus 20% for growth/burst           →  ~78 TB raw
16 drives → 5 TB each (use 6 TB, the next catalogue size)

The VMS writes one-minute chunks as objects, named by camera and timestamp, so retention is not a cron job that deletes files. It is a lifecycle rule the store enforces itself:

mc alias set site42 https://minio.site42.local $ACCESS $SECRET
mc mb site42/video
mc ilm rule add site42/video --expire-days 90
mc ilm rule ls site42/video

Analytics reads the same objects through the same S3 API, and clustered failover on the VMS side means a dead node is an event in the dashboard, not a phone call at two in the morning, four hundred kilometres away.

MinIO on site versus S3 in a region

The interesting part is that the application does not know which one it is talking to. MinIO speaks the S3 API, so the VMS, the analytics and the lifecycle rules are written once. The decision is about physics, money and law, not about code.

	MinIO at the site	S3 in a region
Where the bytes live	On the site’s own drives	In the cloud provider’s region
Bandwidth	LAN; uplink carries metadata	Sustained upload per site, forever
Playback latency	Milliseconds, works during uplink outages	Depends on the link; none without it
Cost model	Hardware up front, no per-GB or egress fees	Per GB stored, per request, per GB retrieved
Durability	Yours to engineer: EC:4, monitoring, drive swaps	Eleven nines, someone else’s problem
Capacity	What you racked; growth is a site visit	Elastic
Sovereignty	Footage never leaves the building	Region choice, provider’s jurisdiction
Operations	Firmware, drives, remote hands at 200 sites	An API

Swapping one for the other

Because the contract is the S3 API, moving a tier to the cloud, or the whole thing, is configuration:

# the only code that knows which store this is
s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["VIDEO_S3_ENDPOINT"],   # https://minio.site42.local or https://s3.eu-west-1.amazonaws.com
    aws_access_key_id=os.environ["VIDEO_S3_KEY"],
    aws_secret_access_key=os.environ["VIDEO_S3_SECRET"],
    config=Config(s3={"addressing_style": "path"}), # MinIO default; virtual-hosted works too
)

Lifecycle rules have a one-to-one equivalent on the S3 side, and MinIO can replicate a bucket to S3 for the footage you do want off site, such as flagged events:

# keep events off site too: replicate only objects tagged by the analytics
mc replicate add site42/video --remote-bucket arn:aws:s3:::central-events \
  --tags "event=true" --replicate "delete,delete-marker"

That gives a hybrid that a lot of teams end up wanting: ninety days of everything at the site, the interesting minutes in the cloud, and one API in the code.

What I would do differently

Buy fewer, larger drives than the first sizing suggests and leave two bays empty; growth always arrives as “just two more cameras”. Automate the site rollout on day one, because two hundred hand-installed sites are two hundred different sites. And put the lifecycle rule in Terraform next to the bucket, so the retention policy is reviewed like code.