> ## Documentation Index > Fetch the complete documentation index at: https://comfyui-mcp.artokun.io/docs/llms.txt > Use this file to discover all available pages before exploring further. # Bring any LLM: free local models now drive ComfyUI > The ComfyUI agent panel now runs on Ollama, LM Studio, llama.cpp, or any OpenAI-compatible endpoint — no account, no API key, fully offline. How a 6-tool compact router makes ~200 tool schemas survivable for a 4B model, the reliability work small models actually needed (loop-breakers, empty-final recovery, never-silent turns), and the VRAM pause that lets one GPU run both the diffusion model and the LLM driving it. *by [artokun](https://github.com/artokun) · July 14, 2026 · local llms · tool router · reliability* The [Panel](../panel) agent — the autonomous assistant in ComfyUI's sidebar that adds nodes, wires graphs, installs packs, and runs renders — started life on Claude and ChatGPT subscriptions. That's still the strongest way to run it. But a lot of people running ComfyUI locally have exactly one question: *can I do this without an account?* Now you can. Pick **Ollama**, **LM Studio**, or **llama.cpp** in the panel's backend picker and the agent runs on a model on **your** disk — no account, no API key, no per-token bill, and it keeps working with the network cable unplugged. If you'd rather bring a hosted key, the same driver speaks any OpenAI-compatible endpoint: **OpenRouter** as a first-class backend, or **DeepSeek** (and vLLM, Together, Azure, anything with a `/v1/chat/completions`) through the custom-provider backend with your own key. Same provider chips, same chat, same live canvas. ``` you ⇄ panel (pick a provider) ⇄ orchestrator (local agentic loop) ⇄ your model ⇄ your graph ``` This post is about what it actually took. Frontier models come with an agent harness that does the hard parts. A local model is a bare HTTP daemon that streams tokens — everything else, we had to build. ## The problem: \~200 tool schemas vs. a 4B model The headless `comfyui` MCP server exposes roughly **200 tools** — queue, models, custom nodes, workflows, generation, the lot — and the panel adds around **40 `panel_*` live-canvas tools** on top. A frontier model swallows that catalog without blinking. A 4B model drowns in it: the schemas alone eat the context window, and tool selection accuracy collapses long before the model gets to *your* request. The fix is the **compact tool router**. Run the MCP server with `COMFYUI_MCP_TOOL_MODE=compact` (or `--compact`) and the \~200-tool surface collapses into **three meta-tools**: * `list_tools` — search the catalog by keyword * `describe_tool` — fetch one tool's full schema, on demand * `call_tool` — invoke it The model discovers tools the way you'd browse documentation: search, read one page, act. Only the schemas it actually needs ever enter the context. The panel orchestrator mirrors the same pattern for the live canvas — `panel_list_tools` / `panel_describe_tool` / `panel_call_tool` over its loopback panel MCP — so the model sees exactly **six tools**, total, and can still reach every one of the \~240 underneath. The other structural difference from the Claude/Codex paths: those providers bring their own agentic loop. Ollama doesn't — it's a plain HTTP server with OpenAI-style tool calling and no harness. So the Ollama backend (`src/orchestrator/ollama-backend.ts`) **owns the whole loop itself**: it streams `/api/chat` NDJSON (or OpenAI-dialect SSE for LM Studio, llama.cpp, OpenRouter, DeepSeek, and friends), dispatches tool calls against a headless comfyui MCP subprocess spawned in compact mode, feeds results back, and repeats until the model produces a final answer. The two dialects normalize into one history — the OpenAI wire wants tool-call arguments as JSON strings paired by `tool_call_id`; the native Ollama wire pairs by name — and the backend translates so the rest of the orchestrator never knows which endpoint it's talking to. ## What small models actually needed Wiring up the loop was the easy half. Live end-to-end runs against real 4B–12B models surfaced failure modes a frontier harness never shows you, and each one got a specific guard. **The exact-repeat loop-breaker.** Small models wedge into calling the *same tool with the same arguments* forever — same search, same result, again. The backend fingerprints every call as `name + JSON(args)` per turn. The second identical call isn't dispatched; it gets a corrective tool result instead ("REPEAT CALL BLOCKED: … the result has not changed. Use the earlier result, or try DIFFERENT arguments"). Because every emitted tool call still needs a paired result or the wire format breaks, the block *is* a tool result — the conversation stays well-formed. At four repeats the turn is cut with an honest stop message rather than burning rounds. **The discovery-spam breaker.** The subtler wedge is a model that never repeats *exactly* — it calls `list_tools` with a **different** search string every round, hunting for a tool that doesn't exist, so the exact-repeat breaker never fires. Discovery tools get their own per-name counter: at four searches with no hit, the corrective result says the quiet part out loud ("it is very likely NOT in this catalog — STOP searching") and names the common traps: canvas actions live behind `panel_call_tool`, not the headless catalog; model families like wan or ltxv are installer *packs*, found via `list_packs`, not tools. At eight, the turn ends with a breaker-specific explanation. A live E2E run caught the original stop copy recommending our fine-tuned model *to the fine-tuned model* — the current copy checks first. **Empty-final recovery.** At low temperature, after a run of tool rounds, a small model sometimes emits a final message with *no content* — the turn would "complete" in total silence. The backend nudges it exactly once ("your reply was EMPTY. In 1–3 sentences, tell the user what you found") and never loops on it: a second empty reply falls through to a visible fallback instead. **Never-silent turns.** The rule underneath all of these: **a tool-using turn can never end in silence.** A real panel session hit a Civitai 503 → empty final → empty retry, and the user stared at a raw tool error with no explanation. Now every failure path paints something visible — a request failure yields a `⚠️ The model request failed: …` chat line alongside the error event, an exhausted round budget says so, a tripped breaker explains which breaker and why. An error event alone leaves the panel looking wedged; a sentence doesn't. ## One GPU, two models: the VRAM pause The pitch of local-first is that a single 4–8 GB GPU runs the whole stack — the diffusion model *and* the LLM driving it. Our gemma4 fine-tune ladder is sized for exactly that: at q4, `:e2b` is \~2 GB, `:e4b` \~3.5 GB, `:12b` \~8 GB. But naively, the two models fight: a chat model resident in VRAM can OOM a render, and a chat sent mid-render reloads the LLM *on top of* the running generation. So the orchestrator has a **VRAM pause** (in `src/orchestrator/index.ts`). While a render is in flight on a local provider, it does two things: it **unloads the local model** to hand its VRAM to ComfyUI, and it **holds any chat you send** instead of forwarding it. When the render finishes, held messages flush — the panel posts "✅ Render finished — the local agent is back. Answering your queued message now." — and if nothing was queued, the model is proactively warmed so your next message is instant. It's local-only by design: LM Studio joins the same handoff when its server is on this machine, hosted OpenAI-dialect endpoints don't touch your VRAM and are never paused. Default on; `COMFYUI_MCP_OLLAMA_PAUSE_ON_GEN=0` opts out. This is the piece that makes the one-GPU story sane rather than aspirational. The LLM and the diffusion model take turns; neither ever OOMs the other. ## Same panel, same tools, same chips None of this forked the product. Local providers are just more chips in the same backend picker, next to Claude and ChatGPT — one port, one panel. They get the same `panel_*` live-canvas surface every provider gets: add and wire nodes, set widgets, load packs, run the graph, all undoable with Ctrl+Z. The knowledge tools are there too — `list_packs`, `list_skills`, `read_skill` — reached through the router like everything else. Vision rides along where the model supports it: images are delivered to every Ollama-family backend, and capability is judged **per-model**, not per-provider — delivery is always attempted, and an endpoint that rejects image input triggers one graceful strip-and-retry with an honest 📎 note. (A sibling post covers that machinery.) Which model should you actually run? The [Arena](../arena) page exists for exactly that question — a repeatable ten-scenario, 20-point benchmark across local and hosted models, where our `artokun/gemma4-comfyui-mcp` fine-tunes currently lead the local field. (Also its own post.) ## Why bother, when Claude exists? Because the failure mode of "the AI tool needs an account" is that most people never start. A local model on the compact router is genuinely capable of the day-to-day: find a pack, load a workflow, wire a node, kick off a render, read the result. It's free, it's private — prompts and images never leave the machine — and it works on a train. And when a task outgrows the small model, switching is one chip: the conversation surface, the tools, and the canvas are identical across every provider. You're choosing a brain, not a product. The engineering lesson we'd underline: making small models *work* wasn't about prompting harder. It was six tools instead of two hundred, breakers for the two distinct ways they wedge, one nudge for empty finals, a hard rule against silent turns, and a VRAM handoff so the GPU is never contested. Every one of those came out of a live failure, and every one is code you can read. *** Drive ComfyUI with a free local model — no account, no API key, fully offline: install [comfyui-mcp](https://github.com/artokun/comfyui-mcp) and add the [Panel](../panel), then pick Ollama, LM Studio, or llama.cpp in the backend picker — see [Local & self-hosted LLMs](../local-llms) for setup. Star the repo or file an idea at [artokun/comfyui-mcp](https://github.com/artokun/comfyui-mcp/issues).