​

Fine-tune 2.0 — a better in-house panel model, trained on the community’s own successful sessions (plan / RFC)

Living doc. Casual “run it by a smart friend” framing up top; concrete architecture, pipeline, and phases below. Opinionated on purpose — react to it.

​

⚠️ Freshness check — re-verified 2026-08-14 (written 2026-08-01)

This RFC names external models and libraries, which go stale fast. Everything below was re-checked against live Ollama / GitHub listings on 2026-08-14. Read this box before acting on §1 or §3. Still true. #642 is still open; the shipped model is still artokun/gemma4-comfyui-mcp (:e4b default — see docs/local-llms.mdx); every in-repo anchor in the grounding table below still exists. No part of this plan has been built — Phases 0–5 are all unstarted (details in §7). Corrected since writing (all marked inline):

• One candidate does not exist as named. richardyoung/qwen3.6-14b-abliterated:agent is not a real model — see §1.
• huihui_ai/Qwen3.6-abliterated is vision-capable, not text-only as the original table claimed — which changes the Qwen shortlist.
• huihui_ai/qwen3-coder-next-abliterated is ~52 GB, so it cannot serve as the local “tool-only sibling” this doc proposes.
• The default lean is no longer safe to assert on freshness grounds — the proposed pick is the oldest build on the shortlist. See §1.
• distilabel and Argilla both lost their original maintainers. Neither is archived; neither is getting new features. See §3.

Claims that could not be checked against a primary source are labelled [unverified] rather than guessed at.

​

The pitch

1. It’s the wrong base for the workload. The panel loop is (a) tool-calling and (b) visual verification. The current fine-tune is built on a Heretic-abliterated Gemma 4 merge that drops the vision + audio projector (#642) — so the “panel specialist” is the one model that structurally cannot look at its own renders, while the generic huihui_ai/gemma-4-abliterated build of the same family, same size, same quant, keeps vision. And independent of vision: across 2026 write-ups, Gemma-family tool-calling is consistently rated weaker than Qwen / Llama / Mistral, while Qwen 3 is repeatedly called the best local agent model of 2026 for clean, argument-complete tool calls. We may be optimizing the wrong horse.
2. It’s trained on synthetic self-play, not real successful work. Today’s dataset is TOUCAN-style synthesis against a mock/real MCP server plus arena trajectories. That’s a fine bootstrap, but the best training signal in the world for “drive this panel well” is real sessions where a strong model already drove this panel well — and we have a fleet of community members running exactly that, with Kimi K3 (our moonshot backend, the current frontier agentic coder) as the teacher. We should ask them, opt-in, for their scrubbed successful transcripts, distill those into the small local model, and close the loop.

• Re-pick the base from the current uncensored/abliterated field with tool-calling and vision as first-class criteria — Gemma 4 huihui vs Qwen 3, decided by our own arena ladder, not vibes.
• Open a community transcript pipeline — opt-in, consent-first, PII-scrubbed (reusing the scrubber we already ship), so “I chose Kimi K3 and it worked” becomes a training pair.
• Make cleaning agentic with a popular open-source curation framework (distilabel), and gate every candidate through an automated, extended version of the arena eval ladder before anything is published to Ollama.

​

Where we are today (grounding — read before proposing changes)

Reviewer warning — half this table is not in the repo. /finetune-v2/ is gitignored (.gitignore L56, alongside /finetune/ L55: “multi-GB, never committed”). It exists on the maintainer’s machine — verified 2026-08-14, all files dated 2026-07-09, and every path cited below was confirmed present there — but it is not on GitHub, so a reviewer reading this PR in a browser cannot open train_qlora.py, panel-arena.mjs, datagen/lib.mjs or the Modelfile.template to check any claim made about them. Rows below are marked (local-only) where this applies. The tracked scripts/arena-*.mjs files are a different, smaller surface — do not confuse the two. Making the pipeline reviewable at all may be a prerequisite for the community phases (§2–§3), since contributors cannot match a format spec they cannot read.

​

1. Base-model research (Aug 2026) — Gemma 4 huihui vs Qwen 3 vs the new field

​

The field

​

Reading of the evidence

• Tool-calling quality — the panel’s #1 job — favors Qwen 3. Multiple independent 2026 evaluations say Gemma tool-calling is comparatively weak and Qwen 3 emits clean, argument-complete calls. This directly matches the complaints we get. ⚠️ 2026-08-14: those “evaluations” are blog round-ups, [unverified] and not re-checked this pass — and one of them was shown wrong on a checkable detail (see Appendix). Note also that huihui_ai/gemma-4-abliterated does advertise a tools capability chip. This bullet is the main argument for the Qwen pivot and it currently rests on the weakest sources in the doc.
• Vision — the #642 lesson — is satisfiable in both families if we pick the vision-retaining build: huihui_ai/gemma-4-abliterated (retains proj.+audio) or huihui_ai/qwen3-vl-abliterated / Qwen3.6-14B-:agent.
• Template fragility — where past complaints actually bit (§4) — is a format problem, not a base problem, and is solvable for either family by never hand-authoring the chat/tool template (derive it from the base’s stock Modelfile).
• Abliteration doesn’t cost us tool-calling — both the current bake-off (“abliteration didn’t hurt tool calling”) and #642 corroborate. So “uncensored” and “good agent” are not in tension here.

​

Recommended shortlist + default pick

Framed as start the conversation — artokun is explicitly undecided Gemma-4-huihui vs Qwen 3 vs new. This is the lean, not the verdict; the arena ladder (§5) casts the deciding vote in a re-run bake-off.

⚠️ 2026-08-14 — the default lean no longer follows from the evidence. The draft leaned Qwen partly on recency, but the freshness column added above inverts that: qwen3-vl-abliterated (~9 mo) is the oldest build on the shortlist, while the Gemma co-primary (~2 mo) is the newest, and Qwen3.6-abliterated (~3 mo) — which this doc wrongly filed as text-only — is a vision+tools Qwen that was never actually considered. Treat “default lean: Qwen 3” as unsettled until the bake-off runs; the three viable entrants are listed below. No verdict is asserted here.

• Lean, now qualified → Qwen 3 (vision-capable), abliterated by huihui-ai. Concretely: huihui_ai/qwen3-vl-abliterated at :8b (6.1 GB) for the 4090 and :4b (3.3 GB) as the wide-adoption sibling — tiers verified 2026-08-14. Rationale as written: it wins on the panel’s dominant axis (tool-calling) and clears #642 (vision), with a well-supported Ollama tool template. Caveat: the build is ~9 months old, so “from a maintained abliteration publisher” is the claim to re-check before committing — the publisher is active, but this particular model has not moved.
• New entrant the draft missed → huihui_ai/Qwen3.6-abliterated :27b. Vision + tools, 256K context, ~3 months old. The strongest current Qwen option. Blocker: at 17 GB it fits the 4090 but has no small sibling, so shipping it means either dropping the wide-adoption tier or pairing it with a different family for the small rungs.
• Co-primary → huihui_ai/gemma-4-abliterated (vision-retaining). If we stay Gemma, this is the build — it directly fixes #642 by keeping the projector, reuses everything the current pipeline already knows about Gemma templates, and keeps the size ladder we’ve validated. The single change from today is swap the Heretic base for the vision-retaining huihui base.
• Tool-only sibling (optional) → huihui_ai/qwen3-coder-next-abliterated ❌ Withdrawn 2026-08-14 — it is ~52 GB quantised and cannot run on the hardware this doc targets. The use case (users who opt out of sending pixels via the panel’s “Blind” toggle and just want the strongest tool driver) is still legitimate; it needs a different, smaller model. No verified replacement is proposed here — picking one is now an open question, not a recommendation.

1. huihui_ai/qwen3-vl-abliterated:8b — the draft’s lean (oldest build)
2. huihui_ai/gemma-4-abliterated:12b — least pipeline churn, fixes #642 directly (newest build)
3. huihui_ai/Qwen3.6-abliterated:27b — newest vision+tools Qwen (no small sibling)

​

2. Community transcript-collection pipeline (opt-in, consent-first)

​

What counts as a “successful session”

• Atomic render success: CompletionNotification.status === "success" (src/services/job-watcher.ts) — the render actually executed and produced outputs (derived from ComfyUI execution_success, not the model’s say-so).
• Session success: the arena-style aggregate — the session reached a completed render (or the user’s stated goal) with no unrecovered execution_error, and the agent used a teacher backend on the allowlist (moonshot/kimi). Sessions where the teacher hit a wall and gave up are not eligible.
• Optional human confirm: a one-tap “this session was good — contribute it?” in the panel, so the contributor, not a heuristic, has the final say.

​

Consent / opt-in (never auto-harvest)

• Off by default. A panel setting “Contribute successful sessions to the fine-tune dataset” (three states: off / ask-each-time / on) — mirrors the existing image-feed “Blind” opt-out pattern, inverted to opt-in.
• Per-session review before anything leaves the machine. On an eligible session, the panel shows the already-scrubbed transcript for review with an explicit “Send” — same “scrub first, human confirms, then submit” shape as report_issue.
• Teacher ToS respected by construction. Only allowlisted teachers (moonshotai/, kimi) are eligible; Anthropic/OpenAI/Google/xAI transcripts are refused before the offer is even shown — reusing ALLOWED_TEACHER_PREFIXES / isAllowedTeacher() from finetune-v2/finetune/datagen/lib.mjs.
• Provenance + revocability. Each submission carries an opaque contributor token so a contributor can later request deletion of their contributions.

​

The scrub step — reuse what we ship

1. Client-side, before display/submit: redactTokens() from src/services/oauth-flow.ts over the whole transcript (tool args, tool results, assistant text), plus the report-bug/SKILL.md Step-5 policy (home paths → ~/…, strip .env/Authorization:/?token=/?key=). Extend redactTokens coverage if transcript-specific leaks show up (e.g. absolute output paths, civitai URLs) — but extend that function, don’t fork it.
2. Server-side backstop: the issue-worker already runs a second secret-scrub pass; the transcript endpoint runs the same backstop before persistence. A submission that still trips the secret detector server-side is dropped, not stored.

​

Submission path — extend the worker we already have

​

Dataset storage & format

• Storage: an R2/object bucket behind the worker (private), one JSONL shard per accepted submission, keyed by contributor token + timestamp.
• Format: normalized to the training schema the existing pipeline already reads — the tool-schema-aware trajectory JSONL used by finetune-v2/finetune/train/prepare_dataset.py (system prompt + per-turn messages + tool calls/results, teacher-labeled). Capture the complete assistant message including reasoning_content for Kimi K3 (Moonshot’s API requires the full message be replayed across tool turns — dropping it destabilizes multi-turn tool calls, so we must store it, not strip it).

​

Bootstrap (artokun-first)

​

3. Agentic data-cleaning — recommend distilabel (+ Argilla)

​

Options considered

​

Recommendation & justification

⚠️ 2026-08-14 — both halves of this recommendation lost their original maintainers. Neither project is archived and neither is broken, but the “most widely adopted, actively developed” framing below is now too strong:

• distilabel — README: “The original authors have moved on to other projects. A group of community members have recently joined the GitHub project as collaborators to maintain the project and are actively working towards the next release. Check out the develop branch for access to the latest fixes.” Community-maintained; ~3.4k stars.
• Argilla — README: “The original authors have moved on to exciting new projects! … While we won’t be adding new features going forward, we’re committed to solve bug fixes and publish patches as needed.” Explicit maintenance mode, no new features; ~5.1k stars.

This does not invalidate the choice — a mature, stable, feature-frozen curation library is a defensible dependency for a boutique pipeline, and the primitives we need (LLM-as-judge, MinHash dedup, Argilla export) already exist and work. But “most widely adopted OSS framework, actively developed” is no longer the reason to pick it, and a reviewer should weigh the handover risk before this becomes load-bearing. The alternatives in the table above were not re-verified on 2026-08-14 [unverified] — if distilabel’s community maintenance is judged too thin, that comparison needs redoing, not guessing.

• distilabel’s core primitive is exactly what we need — an LLM-as-judge step that scores each candidate trajectory and keeps only top-K, plus MinHash dedup out of the box. It is widely adopted and HF-integrated. (The draft called it “the most widely adopted … actively developed” — see the maintenance note above; the primitives are what justify it now, not the momentum.)
• Argilla gives us the review surface the opt-in pipeline wants anyway — a place to spot-check contributed transcripts before they train anything.
• It composes cleanly with our reality: contributions arrive as JSONL shards; distilabel Steps map over them.

​

The agentic cleaning flow

raw scrubbed shards (from §2)
  └─(optional) data-juicer heuristic pre-pass: dedupe exact/near dupes,
     drop empty/degenerate turns, language/format filters
  └─ distilabel pipeline (LLM-curated):
       1. SCRUB-VERIFY judge  — LLM confirms no residual PII/secret survived
                                 the code scrubber; quarantine on any hit
       2. SUCCESS judge       — LLM confirms the trajectory truly solved the
                                 stated task (guards against false "success"
                                 that slipped the run-completion filter)
       3. TOOL-CALL-FORMAT normalizer — reshape every tool call/result into the
                                 chosen base model's exact tool schema/template;
                                 drop malformed calls (this is where format bugs die)
       4. QUALITY judge (LLM-as-judge, reward score) — dedupe via MinHash,
                                 rank, keep top-K, discard bottom percentile
       5. Argilla export      — human spot-review of a sampled slice
  └─ curated train/val JSONL → prepare_dataset.py (existing)

​

4. Model generation with model-appropriate Ollama syntax

​

Training

• Method: LoRA/QLoRA via Unsloth, reusing finetune-v2/finetune/train/ (train_qlora.py, config.yaml). No full fine-tune — LoRA keeps the size ladder cheap and preserves the base’s (now vision-capable) behavior.
• Preserve the projector. The #642 fix is a training discipline: fine-tune on top of the vision-retaining base and do not merge/drop the multimodal projector. Publish capabilities including vision so hosts route correctly.
• Hardware. The in-repo config targets rented A100/H200-class GPUs for the full ladder; the 24 GB 4090 covers the small/e-tier LoRA runs locally. (There’s a WSL2-4090 recipe referenced in an external plan file, not in-tree — open question: pull that recipe into the repo so local runs are reproducible.)

​

Correct Ollama Modelfile + tool TEMPLATE (the part that must not be fumbled)

• Capture, don’t compose. Get the base model’s exact template from its stock Modelfile (ollama show <base> --modelfile) and use the model-family RENDERER/PARSER, never a hand-written TEMPLATE. For Gemma: RENDERER gemma4 / PARSER gemma4. For Qwen 3: the Hermes/qwen3 renderer+parser that handle <tool_call> XML — Ollama ships native Qwen 3 tool parsing; use it, don’t reinvent the <tool_call> grammar by hand.
• Per-family Modelfile templates checked into finetune-v2/finetune/package/ (Modelfile.gemma4, Modelfile.qwen3), each pinning num_ctx (currently 65536), temperature 0, and the captured renderer/parser for that family.
• A gate that proves tool calls parse before publish — see §5’s “100% parseable” gate. The template is only “done” when the eval harness emits and re-parses tool calls with zero malformed calls.

​

5. Automated eval suite — extend the arena ladder into a release gate

​

Scenario tiers (existing + new)

• Base / Gauntlet / Crucible — keep as-is (health, models, registry, queue, generate; precision, breakfix, provenance; multiout, pipeline).
• New — VISION tier (the #642-driven addition): scenarios that are unpassable without looking: “generate, then confirm the image matches the ask before declaring success”; debug-render (tap a wire with PreviewImage, run-to-node, report what’s actually in the intermediate); reject-and-retry when the render doesn’t match. A vision-blind model must fail these — which is exactly the signal #642 says we’re missing today.
• New — FORMAT tier: adversarial tool-call scenarios (nested args, optional params, multi-call turns) whose only pass condition is every emitted tool call parses and round-trips under the model’s Ollama parser.

​

Metrics + pass thresholds (release gate)

​

6. Phasing + open questions

​

Build status — re-checked 2026-08-14

​

Phasing

• Phase 0 — Bootstrap & contract (artokun). Hand-produce a few Kimi-K3 panel sessions; lock capture → redactTokens → review → /contribute → worker backstop → JSONL; land the golden reference shards + format spec. No base decision needed yet.
• Phase 1 — Base bake-off. Extend the arena ladder with the VISION + FORMAT tiers; re-run the head-to-head (qwen3-vl-abliterated vs vision-retaining gemma-4-abliterated) weighting vision. Output: the base decision.
• Phase 2 — Community pipeline. Ship the opt-in setting + review UI + worker /contribute route + dataset bucket. Announce; start collecting.
• Phase 3 — Agentic cleaning. Stand up the distilabel (+Argilla, +optional data-juicer) pipeline; produce the first curated train/val from contributed + synthetic data.
• Phase 4 — Train & gate. LoRA the chosen base per tier; build per-family Modelfiles; run arena:gate; publish only on a green ladder.
• Phase 5 — Iterate. Contributions keep flowing; periodic re-train; watch for drift; re-open the base question when the field moves again.

​

Open questions for the community (react here)

1. Base: now a three-way question (updated 2026-08-14) — qwen3-vl-abliterated:8b (the original lean, but the oldest build), vision-retaining gemma-4-abliterated:12b (fixes #642 with least pipeline churn, newest build), or Qwen3.6-abliterated:27b (newest vision+tools Qwen, but no small sibling)? And should Xiaomi MiMo-V2.5 — which our own docs/local-llms.mdx already recommends — be entered too? The ladder decides; which do you want to win?
2. Teacher parity threshold: how close to Kimi K3 must a candidate be to ship (proposed within 15%)? And do we allow other allowlisted teachers (glm, minimax, xiaomi) to contribute too, or Kimi-only for consistency? Note: xiaomi/ is on the datagen allowlist but there is no xiaomi backend in OPENAI_KEY_PROVIDERS (glm | kimi | moonshot | minimax), so allowing it as a teacher means shipping a backend first.
3. Contribution incentive & trust: what makes contributors comfortable sending transcripts — the double-scrub + review UI + revocable token, or do we need more (e.g. fully local pre-review, published dataset transparency, contributor credit)? How do we keep low-quality/poisoned contributions out beyond the LLM judges (§3)?
4. Vision cost: do we ship vision on every tier, or a vision tier + a tool-only sibling for the “Blind” crowd? (Updated 2026-08-14: the proposed sibling qwen3-coder-next-abliterated is ~52 GB and is withdrawn — if we want this tier, which small text-only model? Unanswered.)
5. WSL2-4090 recipe: pull the external local-training recipe into the repo so community members can reproduce small-tier LoRA runs on consumer GPUs?
6. New (2026-08-14) — does /finetune-v2/ need to become reviewable? It is gitignored, so the training code, arena harness, teacher allowlist and Modelfile template that this plan builds on are invisible to everyone but the maintainer. Phases 2–3 ask outsiders to match a format spec they cannot read. Do we commit the code (excluding the multi-GB artifacts that motivated the ignore), publish it separately, or keep the community role limited to submitting transcripts and not inspecting the pipeline?

​

Appendix — sources (Aug 2026)

Verification note (2026-08-14). The Ollama model pages below were re-fetched and their capability chips / tag lists copied verbatim into §1 — those are primary sources and are reliable. The blog-style “best model of 2026” round-ups (locallyuncensored, dev.to, localaimaster, insiderllm, lushbinary) were not re-verified, and at least one secondary source was contradicted by a primary page this pass: write-ups describing an :agent tag on richardyoung/qwen3-14b-abliterated are not borne out by the model page. Treat the ranking claims sourced from those round-ups — including “Gemma tool-calling is weaker than Qwen” and “Qwen 3 8B is the best local agent model of 2026”, which are load-bearing for §1’s argument — as [unverified]. The bake-off (§5) exists precisely so we don’t have to trust them.

• Issue #642 — vision/audio dropped by the current fine-tune; huihui base retains it.
• huihui-ai abliterated models (Ollama): gemma-4-abliterated, qwen3-abliterated, qwen3.5-abliterated, qwen3-vl-abliterated, qwen3-coder-next-abliterated, Qwen3.6-abliterated.
• Abliterated model guides: Abliterated Models 2026 by VRAM (locallyuncensored), Abliterated Models Guide (dev.to).
• Tool-calling comparisons: Best Ollama Models for AI Agents 2026 (Local AI Master), Best Local LLMs for Tool & Function Calling 2026 (Local AI Master), Function Calling Local LLMs: Qwen 3.6, Gemma 4 (InsiderLLM), Hermes Agent + Gemma 4 & Qwen 3.5 (Lushbinary).
• Kimi K3 (teacher): Simon Willison, Northflank self-hosting, CometAPI benchmarks.
• Data cleaning: distilabel (GitHub), distilabel docs, Clean a dataset with LLMs-as-judges (HF cookbook), Data-Juicer 2.0, NeMo Curator, Synthetic data pipelines: distilabel/Augmentoolkit/Nemotron (Spheron).