Problem

Companies buying GPUs usually buy in bulk, and that fixed pool ends up shared across every user and every workload type on the box, from inference to training, submitted at any time by anyone. Without coordination, two things go wrong: too many jobs land on the same card and blow past its real VRAM and crash, or the system plays it safe and queues everything conservatively, wasting idle capacity. The goal is to run as many jobs concurrently as the hardware can actually hold, and make everyone else wait cleanly instead of erroring.

Solution

Two cooperating layers, and both are required.

The obvious idea is to solve this with scheduling alone: track how much capacity each workload needs, keep a live ledger of what's already committed on each card, and only admit a job once there's room. That idea is necessary, but on its own it isn't sufficient. A scheduling decision only means something if it's actually enforced once the job is running, and admission control by itself has no way to stop an already-admitted job from reaching past the GPU it was assigned.

Layer 1, admission and capacity accounting, decides who gets to run. A manifest declares how much capacity each workload type needs per GPU model. A live ledger tracks what's currently held on which physical card. A queue holds every submitted job with a released flag. The scheduling loop takes a Postgres advisory lock so multiple worker processes can't race and double-book the same GPU, reclaims capacity from jobs that finished, then walks the waiting queue in order and admits whatever fits, dispatching it immediately and leaving the rest queued.

Layer 2, hardware isolation, enforces the decision. Layer 1 only decides which GPU a job is assigned to. It doesn't stop the job's process from seeing every GPU on the machine once it starts. That enforcement happens at container launch: passing the specific assigned device to the container runtime restricts its CUDA and driver visibility to just that one physical card.

Architecture

A job is submitted. The admission layer checks the capacity ledger under the advisory lock and either admits it, dispatching a container fenced to its assigned GPU, or leaves it queued until capacity frees up, at which point the next scheduling pass picks it up.

Tech stack

• Django + PostgreSQL, for the capacity and queue models. An admission decision is a database read and write, so the ORM plus a Postgres advisory lock gives safe cross-worker coordination without standing up a separate coordination service.
• A Postgres advisory lock, because multiple worker processes field submissions concurrently, and the scheduling loop has to be the only one making an admission decision at any instant.
• Docker, with the container runtime's per-device GPU flag instead of a privileged flag. That flag is the layer that actually fences what a process can see. A scheduling decision by itself enforces nothing.

What I learned

I assumed getting the admission math right in layer 1 would be the whole problem. It wasn't. Under a concurrent stress test, jobs that had each been correctly assigned their own GPU were still colliding and crashing. Layer 1 was making the right call every time, three jobs in, three GPUs, each assigned a distinct card, but layer 2 was launching containers with a privileged runtime flag that grants raw access to every device node on the host, which completely bypasses GPU visibility restriction. The containers could see and grab memory on all three cards regardless of what they'd been assigned, and stepped on each other.

Switching to a flag that fences each container to its one assigned device, and dropping the privileged flag, fixed it. An A/B test made the causality clean: identical admission decisions either way, only the isolation flag changed, and only the fenced version stopped the collisions. The lesson: a correct scheduling decision is worthless without enforcement, and enforcement without a scheduling decision controls nothing about who gets admitted in the first place. Both layers matter.