One video is worth a thousand pictures

Turning entire novels into narrated, lip-synced, motion video — locally, on a single 16GB GPU, with FLUX, Wan2.2, PuLID, MuseTalk, and ComfyUI.

There’s an old line: a picture is worth a thousand words. I’d extend it — a video is worth a thousand pictures. So I spent the last few months testing that idea the hard way: I built a pipeline that turns an entire novel into a narrated, lip-synced, motion video. Feed it Pride and Prejudice, or a 400-year-old tale from Strange Tales from a Chinese Studio (聊斋志异), and out comes a finished film with consistent character faces across a hundred-plus scenes.

One person. One RTX 4060Ti with 16GB of VRAM. Everything local, no cloud inference.

I called it iTube. And honestly — the videos came out better than I expected. Watching a novel you know become a moving, voiced, paced thing is a genuinely different experience from reading a summary of it. That’s the bet: that video is a better medium for conveying a story than text is, and the finished clips have mostly convinced me the bet was right.

See for yourself before reading another word about how it works:

• 🎬 Pride and Prejudice — a Western public-domain classic
• 🎬 聊斋志异 / Strange Tales from a Chinese Studio — a playlist of classical Chinese tales

But this post isn’t a demo reel. It’s about the part nobody warns you about — the part between the models — because that’s where the real work turned out to be.

The models are instruments, not magic boxes

Here’s the single most important thing I learned, and the thing I’d want any collaborator to understand before touching a pipeline like this:

Each AI package can do exactly one thing well, and your entire job is knowing precisely where each one’s ability ends — then writing your script logic to route around the gaps.

The stack is not exotic. Most people in generative media know these pieces:

• FLUX generates the still image for a scene. That’s all it does — a single frame. Beautiful, controllable with the right prompt, but frozen.
• Wan2.2 takes that still and adds motion — image-to-video. Crucially, it animates movement, not speech. Wind, water, a turning head, a walk. It does not do lip-sync, and if you let it, it will invent motion you never asked for.
• MuseTalk is the lip-sync specialist. Give it a face and an audio clip, and it drives the mouth to match the speech. It does one job, and it’s the only thing in the stack that does it.
• edge-tts synthesizes the narration and dialogue audio.
• PuLID locks a character’s face so the same person looks like the same person across every scene.

The insight isn’t the list. It’s that the audio comes first and everything else bends around it. MuseTalk needs a face and an audio clip of a known length — so the length of the spoken line determines the length of the scene. You don’t pick a scene duration and fit the speech into it; the speech is fixed, hard, immovable, and the visuals are soft, stretched or trimmed to match. Once you internalize that inversion — audio is the skeleton, video is the flesh — the whole scripting logic falls into place. Before you internalize it, nothing lines up and you can’t figure out why.

So writing the “screenplay” that drives this pipeline isn’t creative writing. It’s writing with the exact capabilities and failure modes of five models held in your head simultaneously. A few examples of what that means in practice — each one a bug I hit before I understood the instrument:

Wan will animate a mouth on a silent scene. A moonlit courtyard, no dialogue — and Wan helpfully makes a bystander silently mouth nothing, which is deeply unsettling. Fix: scenes with a face but no speech never go to Wan at all; they route to a static frame. The script has to know the difference and mark it.

Wan will conjure a person into an empty landscape and animate them, because image-to-video models fill vacuums — they find plausible motion whether or not you wanted any. Fix: faceless environment scenes get an explicit negative prompt (person, people, human, face, figure) and a positive redirect toward environmental motion — leaves, water, light. Tell it exactly what should move, or it decides for you.

FLUX renders a Chinese ghost-story demon with Western facial structure, because it defaults to the aesthetic it’s seen most. Fix: anchor every creature scene with explicit East Asian bone structure and Chinese folktale design cues, or the whole thing drifts into a culturally wrong uncanniness. (For the grotesque figures I also chose a Chinese-opera / woodblock stylization over photoreal horror — partly taste, partly to avoid rendering something that reads as caricature.) You can see the results of this in the 聊斋志异 playlist.

None of these are bugs in the models. They’re the models doing exactly what they do. The bug, every time, was mine — asking an instrument to play a note outside its range. It’s the combination of packages plus the scripting logic. Neither alone is the product.

The hardest part: learning to wire the graph

Everything above assumes a working ComfyUI workflow. Getting there was its own education, and I want to be honest about how it actually felt rather than how it reads in hindsight.

ComfyUI is a node graph in the browser. You double-click the canvas, search for a node, drop it in, and drag a line from one node’s output socket to the next node’s input. Sometimes the node you need isn’t there at all and you have to install the package that provides it first. That’s the whole interface. It sounds simple written down.

In practice, for the first stretch I had no idea what I was doing. I was following AI-generated instructions step by step, and every step felt arbitrary — do this, click that, connect here — with no sense of why. It was only later that I realized a whole run of those baffling steps had just been installing packages. That’s the part nobody tells you about learning a new tool: you can execute a procedure perfectly for an hour without having any mental model of what the procedure is. The steps aren’t hard. Not knowing what category of thing you’re doing is hard.

The model showed up eventually. Wires connect data — a model loader feeds an encoder, which feeds a sampler, which feeds a decoder. Once that clicked, the graph stopped being a ritual and became something I could reason about and debug.

And then the payoff. I ran it, and a still image moved. After all that fumbling, seeing a frame I’d generated actually come to life was genuinely thrilling — one of those moments that makes the preceding confusion feel worth it. I’d built the thing badly, by accident, without understanding it, and it worked.

That working graph is now frozen in the repo as wan_i2v_480.json, so the pipeline can replay it programmatically instead of me rebuilding it by hand in a browser every time. Turning a hand-wired graph into a reusable artifact is, in a sense, the whole transition from “I got it to work once” to “this is a pipeline.”

Two ComfyUI backends, one GPU, taking turns

I’ve built a fair amount of software, and the single most painful stretch of this whole project was getting the ComfyUI backends wired correctly. It’s an install unlike any I’d seen documented anywhere.

Here’s the shape of it. Wan2.2’s model needs fp8 support. The fp8-capable build lives in one Python environment; MuseTalk and its mmpose dependency live in another. They can’t cleanly coexist in a single environment. So the pipeline runs two separate ComfyUI installations on one machine:

• a “cons” stack on 127.0.0.1:8188 under the system Python, and
• a “move” stack on 127.0.0.1:8189 in a dedicated venv that can load the fp8 Wan models.

Each pipeline stage is routed to the backend whose dependencies actually live there. And because a 16GB card cannot hold both servers’ models resident at once, the orchestration has to tear one backend down — kill it by port to free the VRAM — before bringing the other up. Two servers, one GPU, taking turns.

I want to be clear that this split is a workaround, not a design choice — it exists only because upgrading the shared system Python to the fp8 build risked breaking the MuseTalk stack that already worked, so two isolated environments was the safe path, not the necessary one. It’s very likely mergeable into a single server once one environment can hold both the fp8 Wan build and mmpose without conflict, and I’m already partway there. But merging the servers wouldn’t touch the deeper constraint: on 16GB, the models still can’t all sit in memory at once. Today I free VRAM by killing a server; after a merge I’d free it by swapping models within one. The mechanism gets cleaner — the memory ceiling doesn’t move.

Getting that routing right, though, took a while — because for a long time it failed in the worst possible way: silently. The motion stage was dispatching to the wrong backend, the one that couldn’t load the fp8 model, the load failed, and the pipeline fell back to a static frame — no crash, no traceback, just… no motion. Every scene quietly rendered as a still while reporting success. A crash would have been a mercy; it points at a line number. This pointed at nothing, because I’d built the graceful fallback so well that it hid the very thing it was falling back from.

The lesson has stuck with me harder than any other: a graceful fallback is a place bugs go to hide. If your system silently degrades, it has to be loud about the fact that it degraded — otherwise “success” and “total failure” look identical in the logs.

Trust the bytes, not the label

One more debugging lesson worth passing on, because it generalizes far beyond this project.

My audio stage occasionally produced files that passed every check — the metadata said valid MP3, correct duration, right bitrate — but were corrupt the moment you tried to actually decode them. Downstream, MuseTalk would choke, and I’d waste retry cycles blaming out-of-memory, the usual suspect.

The fix was to stop trusting metadata and validate by real decoding: run the file through ffmpeg and confirm it produces actual samples, not just that the header claims it should. A metadata check is a claim; a decode is a fact. In any pipeline, validate the thing itself at each hand-off, never the label on the thing. That one principle has saved me more hours than any other.

What it can’t do yet — and why the limits are mostly memory

I want to be honest about the seams, because the gaps are as informative as the wins.

No non-verbal sound. Right now the pipeline handles speech, but not the sounds that actually make a scene feel alive — a shout, a horse’s whinny, hoofbeats, the howl of wind and snow. Adding a sound-effects layer is next.

Background music is still a work in progress. I have a per-scene BGM-planning scheme, but getting music to sit right under narration — ducking, mood-matching, transitions — is unfinished.

Every scene is generated independently of the others. This is the limitation I find most interesting, because I don’t think it’s theoretical — I think it’s a memory problem. In principle the pipeline could carry context between scenes so a character’s coat, or a room’s layout, stays truly consistent shot to shot. In practice, holding that much video context in memory — character consistency, object consistency, across a hundred-plus scenes — is enormously expensive, and I’m already pinned against 16GB. So each scene is rendered in isolation and I enforce consistency through PuLID face-locking and disciplined prompting rather than true shared context. With more VRAM, this changes.

Resolution is capped by VRAM too. My base renders are small — small enough that the quality ceiling is visibly set by how much I can fit on the card. I’m confident the output quality would jump substantially with more memory and a larger base resolution. Right now, memory is the wall almost everything runs into.

If there’s a theme to the limitations, it’s that: on consumer hardware, memory is the binding constraint on almost everything that matters — consistency, resolution, richness. The models are capable of more than my card can hold.

Why “vibe coding” finally makes sense to me

Building this changed how I understand a phrase people throw around: that AI has turned programming into “vibe coding.” I used to find it glib. Now I get it.

Because the prompt — to FLUX, to Wan, to the whole routing logic — is doing the work that code used to do. Getting a Chinese demon to render correctly, stopping Wan from animating a phantom person, matching a scene to its narration: these aren’t solved by writing more functions. They’re solved by understanding each model deeply enough to ask it the right thing, in the right words, with the right constraints. The skill has shifted from “can you implement it” to “do you understand the tool well enough to direct it.” That’s real, and it connects straight back to the instrument point: you can’t prompt your way out of a capability the model doesn’t have. You have to know where each one’s range ends.

We’re at the very beginning

I’m aware that everything I built will look primitive in a year. The two-server dance, the memory ceilings, the missing sound — these are the friction of an early era, and the field is moving fast enough that some of them will be gone before I finish writing about them.

That’s exactly why it’s the most exciting thing to be working on. We’re at the start of something genuinely new, and it’s compounding quickly. I built a system that turns real books into watchable films on hardware that fits under a desk — and that was hard-won today and will be trivial soon. The distance between those two states is where the whole opportunity lives.

The repo is on GitHub. If any of this resonated — the silent-fallback flavor of bug, the two-backend juggling, or just the conviction that video is where storytelling is heading — I’d love to trade notes.

Built with FLUX, PuLID, Wan2.2, MuseTalk, edge-tts, and ComfyUI, orchestrated in Python, on a single 16GB GPU. Currently adapting public-domain literature — Austen, the Brontës, Shelley, Stoker, and classical Chinese tales — into video. Open to interesting problems in generative media.