Ask for an empty street and you’ll get a street with a man walking a dog, two teenagers on a bench, and someone in a red hat staring directly into the camera like they know what you did. Background people are the most disobedient part of AI video generation. Your lead does what the prompt says; the crowd behind them does whatever it wants — wandering into focus, multiplying between shots, changing clothes mid-scene, and occasionally upstaging the performance you actually generated the shot for.
Extras matter more than creators expect. A short film lives or dies on whether the world around the lead feels continuous, and crowds are where continuity goes to die: the same café has four patrons in shot one, eleven in shot two, and a completely different set of faces in the reverse.
The good news? Background behaviour responds to strategy. You can prompt crowds into behaving, shoot empty and populate later, or salvage the shots where one extra goes rogue. This guide covers all three, plus the honest trade-offs of each.
Why Backgrounds Get Busy Behind Your Back
Video models are trained on real footage, and real footage of a café, station or street is full of people. To the model, people in the background are part of what makes the scene type look right — an empty café reads as wrong, even closed. So when you prompt a location, you’re also implicitly prompting its typical population. The busier the location type, the harder the model works to fill it.
That alone would be manageable. The real problems are behavioural:
- Extras have no continuity between generations. Each clip re-casts the background from scratch. Your lead gets a reference image; the woman at the next table gets reinvented every time, in a new outfit, sometimes as a different person entirely.
- Background faces are low-resolution inventions. Faces far from camera are rendered loosely, and loose faces morph — features slide, expressions change mid-shot, heads subtly reshape. Up close this is the well-known face drift problem; in the background it happens faster and uglier.
- Movement draws the eye. A sharply defined extra walking through frame pulls more attention than your softly lit lead standing still. The model’s background pedestrians often move with odd purpose — crossing frame for no reason, stopping, staring — and odd movement is catnip to a viewer’s eye.
- Crowds interact with each other badly. Two generated extras passing each other can merge at the shoulders, swap a bag, or briefly share an arm. In a busy crowd shot, some corner of the frame is usually doing something impossible.
None of this is a reason to avoid populated scenes. It’s a reason to direct the background with the same intent as the foreground — which starts, as most good things in this workflow do, at the storyboard stage, where you decide how populated each shot actually needs to be before generating anything.
Prompting Crowds: Numbers, Distance And Blur
When the crowd is genuinely part of the shot — a market, a platform, a party — the prompt has to direct it like a crowd coordinator would: how many, where, how clearly seen, and what they’re doing. Vague crowd prompts (“busy street, people walking”) give the model total freedom, and it spends that freedom badly.
The levers that work:
- Give a number range, not a vibe. “A few people” becomes anything from two to thirty. “Six to eight people in the background” lands closer to six to eight. Counts also stabilise across shots: if shot one and shot two both say “about five patrons seated at tables,” the café’s population at least lives in the same neighbourhood.
- Fix them in space. Seated beats standing, standing beats walking, walking beats crossing frame. “Patrons seated at tables, talking quietly” generates far calmer backgrounds than “people moving around the café.” Motion is where extras go wrong, so assign motion only where the story needs it.
- Put distance and blur in the prompt explicitly. “Background people far from camera, softly out of focus, shallow depth of field” isn’t just an aesthetic choice — blur is a consistency tool. A defocused extra can’t morph visibly, can’t stare into the lens, and can’t steal focus. Most of my populated scenes are shot, in prompt terms, like portraits: lead sharp, world soft.
- Dress them down. Specify muted, ordinary clothing for extras — “neutral colours, plain clothes, no hats, no bright logos.” Bright and unusual items attract both the model’s attention and the viewer’s. Your lead should own the only red coat in the film: distinctiveness is a budget, and you spend it on the lead, not the crowd. The same logic governs how you keep the lead identifiable across a whole run of episodes — the series consistency approach of fixed descriptions and fixed references.
- Ban the stare. Add “no one looking at the camera” to crowd prompts. One extra locking eyes with the lens can hijack an otherwise perfect shot, and models produce it constantly because real footage is full of people glancing at cameras.
Keep the crowd description in a fixed block, the same wording across the scene’s shots, next to your lighting line. “Café background: five patrons seated at tables, softly out of focus, muted clothing, no one looking at camera” pasted into every shot of the scene won’t make the extras identical — nothing will — but it keeps their number, energy and prominence in the same range, which is most of what viewers actually track.
The Empty-Set Strategy
The most reliable crowd is the one that isn’t there. The empty-set strategy is exactly what it sounds like: generate the scene with nobody in the background at all, then decide later whether it needs people and add them on your terms. Prompt it directly: “empty café, no other people, tables and chairs unoccupied.” Models resist empty public spaces, so say it twice — once positively (“deserted, quiet, empty”) and once in the negative prompt (“no crowd, no pedestrians, no extra people”).
When empty is right:
- Dialogue and performance shots. If the audience should be watching faces, background motion is competition. Empty or near-empty backgrounds make close dialogue feel intimate and keep every generation failure away from your lead’s big moment.
- Story beats where emptiness is the point. Early morning, closed hours, an off-season town — the story wants quiet anyway.
- Any shot you plan to reuse or extend. An empty plate is a flexible asset. A populated one is a frozen casting decision.
One more habit ties the prompted-crowd shots together across a scene: keep a population note in the shot list. Not art — bookkeeping. “Shot 1: wide, ~6 patrons. Shot 2: medium, 2 patrons behind lead, blurred. Shot 3: reverse, same corner table pair.” When a generation comes back with a visibly different crowd density than its neighbours, you catch it at the review stage by comparing numbers, not by staring at faces. Counting is fast and objective in a way face-checking never is.
It also stops a subtle failure I see often: the crowd that grows across a scene, two patrons becoming nine by the final shot, because each generation independently decided the café was getting busier. Scenes should only get busier if the story says so.
The trade-off is real, and worth stating plainly: empty public spaces can look staged, even eerie. A rush-hour station with nobody in it reads as a dream sequence whether you wanted one or not. The judgement call is shot-by-shot. My usual pattern in a scene is mixed: wides get the prompted crowd (distance and blur make them safe), close and medium shots get the empty set (the background is mostly wall and bokeh anyway). The scene feels populated where the audience looks for population, and controlled where they look for performance.
When To Add People In The Edit Instead
Between “generate the crowd” and “leave it empty” sits the option creators underuse: generate empty, then add background people in the edit. In Premiere Pro this ranges from simple to fiddly, but for the right shots it gives you something generation never will — the exact same extras in every shot.
The practical versions:
- The blurred passerby layer. A separate generated or stock clip of someone walking, masked, blurred and dropped behind the lead for a few frames. Because it’s a layer, you control its timing, position and softness, and it can’t morph mid-shot — it’s finished footage.
- Reused background plates. Generate one good populated wide shot of the location, then use it (or crops of it) as the background element behind tighter generated shots where the location art allows. The crowd is literally identical across cuts, because it’s the same footage.
- Do nothing, but warm the set. Sometimes a scene reads “empty” not because it lacks people but because it lacks life: no steam from cups, no movement in curtains, no distant sound. Ambient sound design in the edit sells population you never have to render. A murmur of café noise behind an “empty” café shot does more for believability than three morphing patrons.
The honest cost: edit-added people need masking and grading care, and a badly tracked layer looks worse than a generated extra. This route earns its keep on hero scenes and repeat locations, not on every shot of a quick project. It’s the same judgement as compositing real products — do it where the audience’s attention actually is.
Salvaging A Shot With One Bad Extra
Sooner or later you get the near-perfect take with one disaster in the corner: an extra whose face melts on the walk-by, a bystander staring down the lens, a gentleman whose arm belongs to someone else. Regenerating throws away a lead performance you may not get back. Before you do, run the salvage ladder, cheapest first:
- Crop or reframe. If the offender is near an edge, a modest punch-in or reposition in Premiere Pro removes them. On vertical deliverables especially, you often have horizontal room to spare from the generated frame.
- Cut earlier or later. Many extras only go wrong for a second or two — the morph happens mid-crossing. Trimming the shot’s head or tail, or moving the cut point, often exits the shot before the meltdown.
- Blur them out. A tracked blur or a subtle defocus patch over one background figure is invisible when the rest of the background is already soft. This is the single most-used salvage in my own edits.
- Cover with a cutaway. An insert — hands, an object, a reaction — over the bad seconds lets the lead’s shot survive underneath. Classic editing, still undefeated.
- Regenerate the background only, if your tool allows masked or regional re-generation; otherwise regenerate the shot with the crowd block tightened (“background people distant and heavily blurred”) and hope the lead lands again. With a decent seed and reference setup you can usually get back to comparable quickly — the seed and iteration methodology is built for exactly this kind of targeted re-roll.
Run the finished sequence once through your normal review with an eye only on the background — a dedicated extras pass sounds excessive until it catches the one staring gentleman who would have become the most-commented thing on the video.
Backgrounds will never be as obedient as leads. But a crowd that is numbered, seated, blurred and dressed down behaves most of the time; an empty set behaves all of the time; and the edit can fix or furnish the rest. Direct the people behind your people, and they’ll stop auditioning for roles they were never cast in.