Skip to content

A Qwen load sets the trunk kernel for the whole process, so a Kimi context already open switches to i8mm #68

Description

@marcobambini

trunk_kern is a file-static in src/model.c, read by matvec_t_inner for every 4-bit trunk tensor in the process. Until now nothing but WASTE_TRUNK_KERNEL and tests/sweep.c ever wrote it, so it was one setting chosen once from the environment. #63 makes a container load write it:

if (m->cfg.arch_qwen && !trunk_kern_env)
    waste_model_set_sdot4(TK_I8MM, sdot4_sg);

That is the right default for Qwen — LEARNED has the measurement, 29% for a difference the text cannot see — but the setting is process-wide and the load is per-context. On a machine with i8mm:

  • a host that opens a Kimi or GLM container and then a Qwen one moves the first container's 4-bit trunk onto i8mm as well, including mid-generation, because matvec_t_inner reads the static on every call and does not care which model it is serving;
  • the same prompt through the same K3 container therefore gives different logits depending on whether a Qwen container happened to be opened earlier in the same process;
  • nothing reports it. i8mm is explicitly not the exact arithmetic — that is what the WASTE_TRUNK_KERNEL=0 escape hatch is for — so this is a numerical change, not a scheduling one.

The comment at the call site and docs/QWEN.md both say the kernel is process-wide, so the behaviour is known. What makes it worth an issue rather than a note is that it is the one thing in this engine that an embedder cannot see coming: waste.h is an opaque waste_ctx with no global state, and waste_open on model B is not supposed to be able to change what model A computes. It is also the exact property tests/run.sh asserts everywhere else — WASTE_XPAR, the residency decision, the scalar and NEON VQ4P paths are all bit-identical by contract, precisely so that results never depend on how the process got where it is.

x86 is unaffected: waste_model_set_sdot4 falls back to TK_F32 when the CPU has no DOTPROD, so the write lands but selects nothing.

What it would take

Move the choice from the static onto waste_model and have matvec_t_inner read m->. WASTE_TRUNK_KERNEL stays what it is — a process-wide pin that overrides the per-model default — and waste_model_set_sdot4 stays for tests/sweep.c, which wants exactly the global it has today.

A check that would have caught it

Two contexts in one process, a 4-bit-trunk Kimi container opened first, logits taken before and after a Qwen waste_open, compared byte for byte. There is no check in tests/run.sh today that opens two containers at once, which is also why #31 (two auto-budget opens each sizing against the whole machine) went unnoticed for as long as it did.

Found reviewing #63; the code is on feature/qwen38-flash-next and not yet on main.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions