Skip to main content
by artokun · June 25, 2026 · AgentBackend · provider parity · architecture The Agent Panel is an autonomous AI agent embedded in ComfyUI’s sidebar. You type a request — “add a KSampler and wire it to my checkpoint”, “build a Flux txt2img graph and run it” — and the agent edits the graph you’re looking at, runs it, and replies right there next to your canvas. It runs in the background on your own subscription, with no API keys, and every edit it makes is undoable with Ctrl+Z. The new part: that agent now runs on either Claude or ChatGPT. The panel’s settings have a backend picker — Claude / ChatGPT chips — and whichever you pick runs the same canvas-driving agent on your subscription for that provider. You pick a provider, not a port.
Both providers reach full feature parity: the same live-canvas tools, the same model knowledge, the same one-shot workflow loads, the same local-GPU-vs-paid-API cost guardrail. This post is about how that parity actually works — and two debugging stories from getting Codex to behave exactly like the Claude path.

​
One picker, no keys

Sign in once for the provider you want — claude (or claude setup-token) for Claude, codex login for ChatGPT — open the Agent tab, pick a provider, and click Connect. The panel starts that provider’s background orchestrator (npx -y comfyui-mcp@latest connect) on your subscription and links to its loopback bridge. Nothing is ever started without your click; Disconnect stops it. Each backend runs its own orchestrator on its own loopback bridge port (Claude defaults to ws://127.0.0.1:9180), which is why the panel can present a provider chooser instead of asking you to think about ports. Switching providers starts a fresh chat — conversations aren’t shared across Claude and ChatGPT — and the panel posts a system note so you’re never surprised. The bridge is loopback-only; nothing is reachable from your LAN.

​
How full parity actually works

The interesting engineering is that two very different providers expose one identical surface. It comes down to a single seam.

​
One port, two adapters

The orchestrator doesn’t know what an LLM is. It depends on a provider-neutral AgentBackend port — classic dependency injection — and each provider is an adapter behind it: Both adapters normalize their provider’s native messages into one canonical event stream (session, assistant_delta, tool_call, result, rate_limit, error), so the orchestrator’s queue, turn-gate, bridge push, and rewind tracking are written once and work for both. The Codex side is driven over the structured codex app-server JSON-RPC protocol (thread/start, turn/start, turn/interrupt, streaming notifications, config/read) — not string-scraping codex exec output — which is what makes interrupt-mid-turn and model enumeration real on the ChatGPT backend too.

​
Giving Codex the live canvas

The agent drives your graph through a fixed allowlist of panel_* commands — add/wire/move/retitle/color/collapse/group/lay-out nodes, plus one-shot panel_load_workflow — and there is no arbitrary JavaScript. Every mutation goes through LiteGraph’s change tracking, so Ctrl+Z reverts an agent edit exactly like your own. Those panel_* definitions live in one shared list. On Claude they register onto an in-process SDK MCP server. Codex can’t host in-process tools — it only talks to MCP servers declared in its config — so the orchestrator stands the same tool list up as a loopback streamable-HTTP MCP server and points Codex at it. The headless comfyui MCP (queue, models, custom nodes, workflows, generation) is injected into both backends the same way: in-process for Claude, declared via codex app-server -c mcp_servers for ChatGPT. Because neither path reimplements a tool — they both consume the shared definitions — parity is automatic, including the destructive-confirm gating on panel_clear and panel_restart_comfyui. That same loopback-MCP trick is how knowledge parity works. Claude loads comfyui-mcp’s bundled model-family skills natively (IDEOGRAM, WAN, LTX, Qwen, and more). Codex can’t, so that expertise is published as MCP tools any backend can call — list_skills, read_skill, list_packs, read_pack_workflow, list_workflow_templates — alongside the check_workflow_runtime cost guardrail. So the ChatGPT agent isn’t a blank slate either; it knows the models you run.

​
Vision and effort, on both

Both backends take image input — attach, drag, or paste images (also video, workflow .json, and text) into the composer, and the agent can see them. And both expose a reasoning-effort picker, even though the two providers use different scales:
  • Claude: low · medium · high · max
  • ChatGPT (Codex): none · minimal · low · medium · high · xhigh
The picker is per-provider, and a chosen effort survives a provider switch by mapping to the nearest valid level on the target backend — the panel and the orchestrator do the same mapping, so the picker stays honest about what’s actually selectable. There are a few honest, capability-gated differences. Code/graph rollback (/revert, double-Esc, per-turn snapshots) works on both backends because it lives in the orchestrator, not the provider. Conversation rollback (forking the chat back to a past turn) is currently Claude-only — the Codex app-server resumes whole threads, so the panel gates that scope off for ChatGPT rather than faking it. Provider slash commands (/compact, /loop, …) are Claude-only too. A capability descriptor per backend is what lets the panel degrade gracefully here instead of showing a button that wouldn’t work.

​
Two war stories

Parity sounds clean in a table. Getting there involved two bugs worth keeping. The tool that silently vanished. Early on, panel_add_node and several other edit tools simply weren’t in Codex’s tool list — no error, no warning, just absent. The cause was a schema detail. We’d typed fixed-length coordinate vectors (positions, sizes) with zod’s z.tuple([...]). Zod emits that as JSON-Schema draft-04 tuple validation — items as an array of schemas — and Codex’s strict function-schema validator rejects any tool whose schema uses array-form items, dropping it from the list entirely. The Claude SDK didn’t care, so it worked on one provider and the tool “disappeared” on the other. The fix was to switch those params to a plain z.array(z.number()) (single-object items with minItems/maxItems), which both validators accept. A tool that vanishes silently is worse than one that errors loudly. The watchdog that interrupted healthy renders. Each turn has an idle watchdog: if a turn in flight emits no events at all for a few minutes, treat it as stalled, surface a clear error, and let the next queued batch run — a safety net for a genuinely wedged backend. But a long video render is exactly the case that looks dead from the outside. A Codex panel_run that kicks off a multi-minute ComfyUI generation emits raw app-server notifications throughout, yet those can translate to zero canonical AgentEvents during the wait — so the watchdog would false-trip and interrupt a perfectly healthy generation. The fix was a liveness signal: re-arm the idle timer on any sign the backend is alive, not just on translated events. A true zero-event freeze still trips it; a slow-but- working render no longer does.

​
What it enables

The headline is simple: the same canvas-driving, node-installing, workflow-building agent now runs on whichever subscription you already have. Pick Claude or pick ChatGPT — you get the same panel_* live-canvas surface, the same one-shot pack loads, the same model knowledge, the same “ask before spending paid API credits” guardrail, the same install → restart → continue autonomy. No API keys, no per-token billing, no juggling ports. Just a provider chip and Connect.
Drive ComfyUI from an autonomous agent on your own Claude or ChatGPT subscription: install comfyui-mcp and add the Agent Panel — see Backends / providers for the full capability matrix. Star the repo or file an idea at artokun/comfyui-mcp.