Skip to main content

​
Previz-to-video — 3D-blocked reference scenes for motion-controlled AI video (vision doc)

Status: PARKED — preserved from RFC PR #187 (branch spec/previz-to-video, closed to clear the queue; the branch also carried a ROADMAP.md “Theme H” entry, reproduced in the addendum at the bottom) · Field research: previz-research-notes.md
Prior art: Doug Hogan’s Seedance Playblast2Render (playblast → ByteDance2ReferenceNode, fps read from the playblast itself), Mickmumpitz’s AI Renderer 2.0 (local control-pass school: depth/outline/color-ID passes → Wan VACE), and the Awesome Blender + Seedance use-case collection (28 cases, several agent-built via Blender MCP).
Living doc. Casual “run it by a smart friend” framing at the top; concrete architecture + phases below. Draft — shaping it before building.

​
The pitch

Prompting alone can’t direct a shot. You can get a beautiful video out of Wan or Seedance, but you can’t get the shot — this character crosses frame left while the camera dollies past that prop, beat lands on frame 40. The most reliable control signal we have today isn’t words, it’s a pre-animated 3D scene: block the shot in Blender with rigged characters and a keyframed camera, render a cheap untextured preview, and hand that to a video model as a motion/camera reference. The model keeps the body movement, timing, camera motion and pacing — and repaints everything else from reference images. Swap the reference images, keep the animation: infinite restyles of one shot. ComfyUI’s own demo of exactly this recipe: Seedance 2.0 restyling a Blender/Mixamo previz — Mixamo characters + a generated bus + keyframed camera, rendered rough, then “follow the exact same body movement, timing, camera motion and pacing from the reference video; change the appearance to match the reference images.” The reason this belongs in comfyui-mcp: every step of that pipeline is agent-drivable now. Blender ships an official MCP (5.1+). Meshy 6 partner nodes live in ComfyUI core for text/image→3D assets. Wan 2.2 Animate runs the free local path; Seedance 2.0 Reference-to-Video is the paid API path. Our agent already does story→scenes→frames→clips (the director skill). What’s missing is the cinematography department: a skill + pack that teaches the agent to previz a scene in Blender and cash it out through ComfyUI. You’d tell the panel “two characters argue on a rooftop, camera circles them, then one walks off a ledge — make it look like a 90s anime” and the agent blocks it, renders the reference, and restyles it — asking before it spends credits. The panel’s job, first and foremost, is natural motion between characters you created inside ComfyUI (Discord feedback, verbatim intent): ComfyUI is the character shop, Mixamo auto-rigs what it produces, and Claude — with full visibility into the Blender scene — directs the performance from the user’s natural language. Why this is the panel’s home turf (community articulation, from a live session): the panel’s whole value is that the agent acts on live, current state instead of frozen training data — live model cards correct guessed settings, the daily-changing workflow catalog can’t live in weights, and the agent doesn’t know how anything on disk works until it reads it. A Blender stage extends that same superpower to 3D: the agent sees your actual rig, scene, and pose rather than hallucinating plausible bpy calls. And a posed scene solves multi-character composition by construction — who’s touching whom, from what angle — which text prompting has never done reliably.

​
Evidence this works today

  • ComfyUI (official): style-swapping one Blender animation — the core recipe end-to-end: Mixamo + Tripo asset + camera keyframes → rough render → Seedance 2.0 R2V with character/environment ref images. Confirms: no textures/lighting needed in the previz; camera motion survives.
  • PixelArtistry: Claude took over ComfyUI + Blender — Claude driving Blender MCP + a ComfyUI MCP together: batch asset kits (image model → img2mesh → import to Blender), retopo/bake cleanup, and text-to-animation (Kinetix) retargeted onto rigged characters. Confirms: the dual-MCP orchestration pattern holds up for real multi-step 3D work.
  • Stefan 3D AI: official Blender MCP setup — the official 5.1 add-on is drag-drop + auto-start, Claude connector is one-click; agent-built materials, geometry nodes, baking, LODs all in single prompts. Confirms: setup friction is low enough to put in a skill.
  • PixelArtistry: 3D AI news #13 — the surrounding ecosystem (retopo, splats, markerless mocap) is compounding fast; spatial reasoning in current Claude models is measurably better.
  • Community, on Discord: a member (seanmcmagic) built a fully-local character shop with the Agent Panel’s help and shared the workflow — SDXL (WAI-Illustrious) hero image → Qwen-Image-Edit 2511 + multiple-angles LoRA batch-generating consistent views → Hunyuan3D v2 multi-view → GLB. Preserved at previz-assets/master-3d-model-maker.community.json (Discord CDN links expire). Confirms: the free asset path is real today, and people are already using the panel to build this pipeline’s front half. The same thread asked for 3D-modeling and audio skill files — H1 answers the 3D half.

​
The pieces on the board

​
Architecture — who talks to whom

comfyui-mcp does not wrap Blender. The agent (Claude) holds two MCP connections — comfyui-mcp for generation/canvas, Blender MCP for scene work — and a skill teaches it the choreography. That keeps us out of the business of proxying bpy, and it’s exactly the pattern proven in the videos. Handoff points (all already representable):
  1. ComfyUI → Blender: Meshy/Trellis writes GLB/FBX into ComfyUI output/; the agent imports it via Blender MCP execute_python (bpy.ops.import_scene.gltf) — directly from the output folder, the same file awareness the panel already has on the drive (get_image (action:"list_outputs"), workspace paths). No copy step, no asking the user where the file went.
  2. ComfyUI → Mixamo (auto-rig): a ComfyUI-authored character mesh from output/ goes to mixamo.com for auto-rigging (or a Mixamo-family Blender add-on) and comes back as a skeleton-bound FBX. See the character loop below.
  3. Mixamo → Blender: user-curated local FBX clip library (see below); import + retarget onto the character rig via add-on or bpy.
  4. Blender → ComfyUI: viewport/OpenGL render (solid shading, no lights, no materials — it’s a motion reference) to mp4 or frame sequence → upload_image (action:"video") → reference input of the Wan Animate graph or Seedance node.
  5. Reference stills: character/environment look-images from any local image pack (Z-Image, Krea2, …) or provided by the user.
The character loop (Discord feedback, revised by round-3 research): characters are authored in ComfyUI first — image model → img2mesh → FBX/GLB in output/ — then auto-rigged, and only then animated and directed in Blender. Rigging now has a fully-automatic local path: ComfyUI-UniRig (UniRig + Make-It-Animatable) turns a mesh into a Mixamo-compatible rigged FBX locally, free, inside the same ComfyUI the agent already drives — no website hop at all. Mixamo’s site remains the fallback rigger and the clip library; when used, the skill runs it as a guided handoff: stage the exact file, tell the user what to upload and click, pick the rigged FBX back up from the download folder. (Cloud alternates with real APIs: Meshy rigging + 600-clip retarget, Tripo one-shot rig+retarget.) Direction stays verifiable: the agent inspects the scene graph and screenshots the viewport, so it sees every scene it’s directing — the same trust model as the canvas, where it reads the graph it mutates. Mixamo reality check: there is no official API and scraping is a ToS minefield, so the skill treats Mixamo’s clip library as a one-time shopping trip — a documented starter list (idle/walk/run/turn/sit/fight/dance/death…) the user downloads once into a conventional folder (e.g. ~/.comfyui-mcp/previz/clips/), which the agent then reuses forever. Auto-rigging is per-character but follows the same stage → guide → pick-up pattern. Text-to-animation add-ons (Kinetix-style) and markerless mocap slot into the same folder convention later. Model-agnostic exports (adopted from community feedback): everything that leaves Blender is a generic control artifact — clay/skeleton video, depth pass, camera-keyframe JSON — never tied to one backend. The downstream node decides: Seedance/Kling for hosted cinematic work, Wan Animate / VACE / ControlNet for local work. Same Blender tooling either way. Corollary from the same feedback: hosted backends filter content server-side, so the local leg isn’t just the free leg — it’s the only leg for content the APIs won’t render. One more scope rule adopted: pose export and camera export ship together (H0/H1), not camera-first — multi-character shots need pose matching from day one, camera motion alone doesn’t hold characters consistent. Orchestrator gap + the architecture question: the panel agent currently speaks only to comfyui-mcp. Claude Desktop / Claude Code users can add the Blender MCP alongside today, but the panel needs companion MCP server support — a config that passes extra MCP servers into the spawned agent session (H3). A community counter-proposal (research notes §6.6) goes further: fork a bpy socket bridge and ship our own two-package Blender panel (blender-agent-panel add-on + blender-mcp-orchestrator) exposing ~15 curated named tools instead of raw execute_python — reusing the panel’s consent file. The concern behind it is real: raw bpy is hostile to small local models (the exact lesson compact tool mode taught us on the ComfyUI side) and hard to consent-gate. Current lean: don’t own a Blender add-on yet — official MCP underneath, curated recipes in the H1 skill on top, and revisit a thin curated tool layer (possibly proxying the official add-on’s socket rather than forking) if H0/H1 show local models can’t drive recipes reliably. Prior art either way: alexisrolland/ComfyUI-Blender (Blender→ComfyUI trigger, one-directional, UI-pattern reference).

​
Deliverables

  • previz-director skill (plugin/skills/previz-director/SKILL.md) — the whole recipe as knowledge, with the field-research numbers baked in (research notes): Blender MCP setup + port sanity check; scene blocking conventions (real-world scale — Seedance needs character proportions to match, not just height; camera rig patterns: dolly/orbit/crane as reusable bpy snippets); Mixamo library convention + auto-rig handoff + retarget procedure (free Retarget extension / ARP); viewport render recipe (bpy.ops.render.opengl, Workbench MatCap/solid, outlines on, fps read from the playblast and held identical through the workflow, silhouette-readable for DWPose); the routing rule — camera language and staging go to Seedance R2V, character-performance transfer with a calm camera goes to Wan Animate; role-tagged prompt templates (“Keep the motion of @Video1 … replace subject with @Image1”, “Follow the camera path from @Video1”), shot length guidance (6–8s per shot, stitch longer); known-weak-spots watchlist (foot contact, cloth, multi-cut references); style-sweep pattern (one previz, N reference-image sets).
  • wan-animate pack (packs/wan-animate/) — the free path, installable: Wan 2.2 Animate 14B fp8 + controlnet_aux (DWPose) + SAM2 nodes, template workflow wired for reference-video input. Follows the existing manifest conventions; pack.yaml links the skill.
  • character-shop pack (community seed) — productize the shared Master 3D Model Maker graph (SDXL hero → Qwen-Edit 2511 multi-angle LoRA → Hunyuan3D v2 MV → GLB) as the free character-authoring pack, with credit to its author, extended with a ComfyUI-UniRig stage so the pack’s output is a rigged, animation-ready FBX — the full character loop with zero manual hops. It’s the ComfyUI half of the character loop, already panel-built.
  • Meshy + Seedance guidance — no pack needed (they’re core API nodes); the skill documents the built-in templates and wraps both in the list_packs action:“check_runtime” / ask-before-spending convention. Free alternates (Trellis/Hunyuan3D for assets, Wan Animate for video) are always named first.
  • 3D-asset file awareness — extend the existing output-folder awareness (get_image (action:"list_outputs") already tags kind: "video") to 3D assets (kind: "model" for GLB/FBX/OBJ in output/), so “import the character I just generated into Blender” and “stage this mesh for Mixamo auto-rigging” resolve to exact paths without the user hunting for files. Small, and it’s the glue the whole character loop stands on.
  • Companion MCP servers for the panel orchestrator — config + session-spawn plumbing so the panel agent can reach Blender MCP. Specced separately when we get there (it’s generic, not Blender-specific).
  • director integration — a previz mode for the existing story pipeline: scene list → one Blender previz per shot → batch restyle. Character consistency comes free (same 3D characters every shot); the director skill’s hero-frame/ref-image machinery supplies the style side.

​
Phases

H0/H1 ship value with no code at all — that’s the point of leading with a skill. H2 is manifest-writing. Only H3 touches the orchestrator.

​
Costs & safety

Two of the pieces are paid (Meshy, Seedance — Comfy credits billed to the user’s Comfy account). Existing convention applies unchanged: the agent calls list_packs with action: "check_runtime", treats api/mixed as possibly-paid, and asks before spending credits — never silently. Every paid step has a named free alternative (local img2mesh; Wan Animate), and the skill presents the free path as the default. Blender MCP is local and free; its execute_python is arbitrary code execution inside the user’s Blender — the skill should say so plainly and keep generated bpy scripts inspectable in chat, same spirit as our graph mutations being undoable.

​
Where I actually want a gut-check

  • How early does the panel need Blender? Discord feedback says the side panel controlling motion is the headline feature — which pulls H3 (companion MCP servers) forward. Counter-pressure: H0–H2 already work from Claude Desktop/Code and prove the recipe before we touch the orchestrator. Current lean: keep H3 third, but spec it in parallel with H1 so panel parity lands fast once the skill is validated.
  • Curated Blender tools vs raw bpy over the official MCP? The community proposal (notes §6.6) wants ~15 named tools (blender_export_pose_map, blender_set_bone_rotation, …) instead of execute_python. Real tension: named tools are what small local backends can actually drive, and what a consent gate can reason about — but 15 tools can’t cover retargeting, physics, and the long tail that makes Blender worth having. H0/H1 decide: if skill recipes over the official MCP work for Claude but fail for qwen/gemma-class backends, a thin curated layer (proxying the official add-on’s socket, not forking a new one) becomes an H3-adjacent item.
  • Bundle a tiny CC0 previz kit? A mannequin character + 3–5 CC0 motion clips shipped in the pack would make H0-style demos work without the Mixamo shopping trip (and without leaning on Adobe’s ToS). Leaning yes — and round-3 research softens the need further: UniRig rigs locally, and text-to-animation APIs (DeepMotion SayMotion, Text2Motion’s Blender add-on, Move AI phone mocap) can generate clips instead of shopping for them. Mixamo becomes the curated-quality option, not the bottleneck.
  • Yedp Action Director as the on-ramp? A 3D previz viewport inside ComfyUI (Mixamo rigs, cameras, 7 baked control passes) covers simple shots with zero Blender setup, and the full Blender MCP stage takes over when shots need real set dressing, physics, or multi-take direction. Should H1 teach both rungs, or does two previz stages confuse the story?
  • Headless Blender? The official MCP can execute in a background Blender process — a render-farm-ish mode for batch shots. Interactive-first feels right (the user watches the blocking happen, same as the canvas), headless later.
  • DWPose readability as a skill rule — resolved by field research: the split is real and stable. Camera moves + staging → Seedance (follows the reference video’s trajectory faithfully); character performance with a calm camera → Wan Animate (DWPose needs a readable humanoid; aggressive camera moves mutate identity). One open sub-question for H0: does DWPose track a flat-grey mannequin reliably, or does the previz need a MatCap/toon-shaded humanoid with a visible face?
  • A third leg worth a later pack? The Mickmumpitz school renders explicit control passes (depth via compositor Map Range, Freestyle/Workbench outlines, per-object color-ID masks for regional prompting) into a local Wan-VACE restyle. More setup, more control, fully free — and color-ID masks are the best multi-character answer anyone has. Candidate previz-restyle pack after H2, not in scope now.
  • Where does the clip library live — ~/.comfyui-mcp/previz/ vs the ComfyUI workspace vs a pack templates/ dir. Orchestrator config already has a home dir; leaning ~/.comfyui-mcp/previz/.
  • Seedance “5.0” — resolved: it doesn’t exist (“Seedream 5.0” is the image model). The current video line is 2.0 / 2.0 Fast / 2.0 Mini in ComfyUI, with Seedance 2.5 (native 30s, ≤50 refs) released but not yet in ComfyUI partner nodes. The skill names capabilities, not node class names, so 2.5 slots in when it lands.

​
Addendum — ROADMAP.md Theme H entry (from the PR branch, preserved verbatim)

​
Theme H — 3D previz → motion-controlled video (Blender MCP + Mixamo + Meshy) (#187)

The most reliable way to direct an AI video shot is a pre-animated 3D scene: block it in Blender (agent-driven via the official Blender MCP, 5.1+), populate with Mixamo motion clips and Meshy 6 / local img2mesh assets, render a rough viewport reference, then restyle it with Wan 2.2 Animate (free, local) or Seedance 2.0 Reference-to-Video (paid API node) — motion, timing and camera survive; appearance comes from reference images. Full vision: design/previz-to-video.md. comfyui-mcp does not wrap Blender — the agent holds both MCPs and a skill teaches the choreography. Headline (community feedback): the panel directs natural motion between characters authored in ComfyUI — ComfyUI is the character shop, Mixamo auto-rigs its output, Blender is the stage Claude can fully see and direct.
  • H0 — Manual spike. Run the recipe once end-to-end on this rig (asset → Mixamo auto-rig → blocking → viewport render → Wan Animate AND Seedance R2V); capture every snag.
  • H1 — previz-director skill. The whole recipe as knowledge (blocking conventions, character loop incl. the guided Mixamo auto-rig handoff, retarget procedure, render settings, handoff commands, R2V prompt templates). Zero code; works today from Claude Desktop/Code with both MCPs configured.
  • H2 — wan-animate + character-shop packs + 3D-asset file awareness. The free video path installable via apply_manifest (Wan 2.2 Animate 14B fp8 + DWPose + SAM2 + wired template); the community-seeded character-shop pack (SDXL hero → Qwen-Edit 2511 multi-angle → Hunyuan3D v2 MV → GLB, design/previz-assets/master-3d-model-maker.community.json); plus kind: "model" (GLB/FBX/OBJ) in the output listing so import-to-Blender / stage-for-Mixamo resolve exact paths.
  • H3 — Companion MCP servers in the orchestrator. Config + session-spawn plumbing so the panel agent can reach Blender MCP (generic mechanism, Blender is just the first customer). Open question tracked in the RFC: whether small local backends additionally need a thin curated tool layer (blender_export_pose_map-style named tools) over the official MCP’s raw execute_python — community proposal, decided by H0/H1 evidence.
  • H4 — Director × previz. Shot lists drive previz scenes; multi-shot cast/set continuity; style sweeps (one previz, N looks) as a first-class op.
Paid pieces (Meshy, Seedance) stay behind the existing list_packs action: "check_runtime" / ask-before-spending convention; every paid step has a named free alternative.