/v1/chat/completions) through the custom-provider backend with your own key.
Same provider chips, same chat, same live canvas.
The problem: ~200 tool schemas vs. a 4B model
The headlesscomfyui MCP server exposes roughly 200 tools — queue,
models, custom nodes, workflows, generation, the lot — and the panel adds
around 40 panel_* live-canvas tools on top. A frontier model swallows
that catalog without blinking. A 4B model drowns in it: the schemas alone eat
the context window, and tool selection accuracy collapses long before the
model gets to your request.
The fix is the compact tool router. Run the MCP server with
COMFYUI_MCP_TOOL_MODE=compact (or --compact) and the ~200-tool surface
collapses into three meta-tools:
list_tools— search the catalog by keyworddescribe_tool— fetch one tool’s full schema, on demandcall_tool— invoke it
panel_list_tools / panel_describe_tool / panel_call_tool over its
loopback panel MCP — so the model sees exactly six tools, total, and can
still reach every one of the ~240 underneath.
The other structural difference from the Claude/Codex paths: those providers
bring their own agentic loop. Ollama doesn’t — it’s a plain HTTP server with
OpenAI-style tool calling and no harness. So the Ollama backend
(src/orchestrator/ollama-backend.ts) owns the whole loop itself: it
streams /api/chat NDJSON (or OpenAI-dialect SSE for LM Studio, llama.cpp,
OpenRouter, DeepSeek, and friends), dispatches tool calls against a headless
comfyui MCP subprocess spawned in compact mode, feeds results back, and repeats
until the model produces a final answer. The two dialects normalize into one
history — the OpenAI wire wants tool-call arguments as JSON strings paired by
tool_call_id; the native Ollama wire pairs by name — and the backend
translates so the rest of the orchestrator never knows which endpoint it’s
talking to.
What small models actually needed
Wiring up the loop was the easy half. Live end-to-end runs against real 4B–12B models surfaced failure modes a frontier harness never shows you, and each one got a specific guard. The exact-repeat loop-breaker. Small models wedge into calling the same tool with the same arguments forever — same search, same result, again. The backend fingerprints every call asname + JSON(args) per turn. The second
identical call isn’t dispatched; it gets a corrective tool result instead
(“REPEAT CALL BLOCKED: … the result has not changed. Use the earlier result,
or try DIFFERENT arguments”). Because every emitted tool call still needs a
paired result or the wire format breaks, the block is a tool result — the
conversation stays well-formed. At four repeats the turn is cut with an honest
stop message rather than burning rounds.
The discovery-spam breaker. The subtler wedge is a model that never repeats
exactly — it calls list_tools with a different search string every
round, hunting for a tool that doesn’t exist, so the exact-repeat breaker never
fires. Discovery tools get their own per-name counter: at four searches with no
hit, the corrective result says the quiet part out loud (“it is very likely NOT
in this catalog — STOP searching”) and names the common traps: canvas actions
live behind panel_call_tool, not the headless catalog; model families like
wan or ltxv are installer packs, found via list_packs, not tools. At eight,
the turn ends with a breaker-specific explanation. A live E2E run caught the
original stop copy recommending our fine-tuned model to the fine-tuned model
— the current copy checks first.
Empty-final recovery. At low temperature, after a run of tool rounds, a
small model sometimes emits a final message with no content — the turn would
“complete” in total silence. The backend nudges it exactly once (“your reply
was EMPTY. In 1–3 sentences, tell the user what you found”) and never loops on
it: a second empty reply falls through to a visible fallback instead.
Never-silent turns. The rule underneath all of these: a tool-using turn
can never end in silence. A real panel session hit a Civitai 503 → empty
final → empty retry, and the user stared at a raw tool error with no
explanation. Now every failure path paints something visible — a request
failure yields a ⚠️ The model request failed: … chat line alongside the
error event, an exhausted round budget says so, a tripped breaker explains
which breaker and why. An error event alone leaves the panel looking wedged;
a sentence doesn’t.
One GPU, two models: the VRAM pause
The pitch of local-first is that a single 4–8 GB GPU runs the whole stack — the diffusion model and the LLM driving it. Our gemma4 fine-tune ladder is sized for exactly that: at q4,:e2b is ~2 GB, :e4b ~3.5 GB, :12b ~8 GB. But
naively, the two models fight: a chat model resident in VRAM can OOM a render,
and a chat sent mid-render reloads the LLM on top of the running generation.
So the orchestrator has a VRAM pause (in src/orchestrator/index.ts).
While a render is in flight on a local provider, it does two things: it
unloads the local model to hand its VRAM to ComfyUI, and it holds any
chat you send instead of forwarding it. When the render finishes, held
messages flush — the panel posts ”✅ Render finished — the local agent is back.
Answering your queued message now.” — and if nothing was queued, the model is
proactively warmed so your next message is instant. It’s local-only by design:
LM Studio joins the same handoff when its server is on this machine, hosted
OpenAI-dialect endpoints don’t touch your VRAM and are never paused. Default
on; COMFYUI_MCP_OLLAMA_PAUSE_ON_GEN=0 opts out.
This is the piece that makes the one-GPU story sane rather than aspirational.
The LLM and the diffusion model take turns; neither ever OOMs the other.
Same panel, same tools, same chips
None of this forked the product. Local providers are just more chips in the same backend picker, next to Claude and ChatGPT — one port, one panel. They get the samepanel_* live-canvas surface every provider gets: add and wire nodes,
set widgets, load packs, run the graph, all undoable with Ctrl+Z. The knowledge
tools are there too — list_packs, list_skills, read_skill — reached
through the router like everything else.
Vision rides along where the model supports it: images are delivered to every
Ollama-family backend, and capability is judged per-model, not
per-provider — delivery is always attempted, and an endpoint that rejects image
input triggers one graceful strip-and-retry with an honest 📎 note. (A sibling
post covers that machinery.)
Which model should you actually run? The Arena page exists for
exactly that question — a repeatable ten-scenario, 20-point benchmark across
local and hosted models, where our artokun/gemma4-comfyui-mcp fine-tunes currently lead the
local field. (Also its own post.)
Why bother, when Claude exists?
Because the failure mode of “the AI tool needs an account” is that most people never start. A local model on the compact router is genuinely capable of the day-to-day: find a pack, load a workflow, wire a node, kick off a render, read the result. It’s free, it’s private — prompts and images never leave the machine — and it works on a train. And when a task outgrows the small model, switching is one chip: the conversation surface, the tools, and the canvas are identical across every provider. You’re choosing a brain, not a product. The engineering lesson we’d underline: making small models work wasn’t about prompting harder. It was six tools instead of two hundred, breakers for the two distinct ways they wedge, one nudge for empty finals, a hard rule against silent turns, and a VRAM handoff so the GPU is never contested. Every one of those came out of a live failure, and every one is code you can read.Drive ComfyUI with a free local model — no account, no API key, fully offline: install comfyui-mcp and add the Panel, then pick Ollama, LM Studio, or llama.cpp in the backend picker — see Local & self-hosted LLMs for setup. Star the repo or file an idea at artokun/comfyui-mcp.