Support for Kimi-K2.7 Code

[kalifvaughn]

Greetings,

I have used your gmlx with great success in streaming Kimi-K3 GGUF (q2) on my Mac Studio. I am curious if Kimi-Code (K2.7) has support?

Thank you!

[asher]

Hi! Kimi-Code wasn't on my radar but I'm downloading Kimi-K2.7-Code-UD-Q2_K_XL right now. Kimi 2.5/2.7 appears to be derived from the deepseek2 arch which is supported, so it might just work, but I'll work out any issues I find and report back.

[kalifvaughn]

Hello! I can confirm it is working - I downloaded the Q8_K_XL quant from Unsloth (~595gb). It streams responses on the server! Puzzlingly, it seems a little slower compared to Kimi-K3 (Q2 ~ 861gb). That's a bit strange to me, but I have only just started testing it.

Thanks for creating this! I hope you will continue with this project as I find it very useful.

Edit: It is actually faster. I got a prefill rate of 19t/s (which seems great) and then a decode of 0.2 t/s. It took < 3 hours for a response. The response was overall very helpful, but parts of it were odd. It would say that the code had variable mismatches in naming, but then would paste the same variables (e.g., "varA" vs. "varA"). It referenced my code during the response, but then later on said, "If you post your full code, I can add these fixes." So, it had the full code, but then acted as if it didn't have the full code. Strange.

[asher]

I found a couple issues with attention and tools parsing vs. the deepseek2 default it's getting. A fix for proper kimi k2.5/2.7 support should land this morning!

[asher]

Proper kimi-k2 support is now on main, let me know if you have any issues with it. I'll cut a new pypi release in the PDT evening.

[asher]

0.3.1 is now on pypi with Kimi2 support. Vision now works in streaming mode too, tested against Kimi-K2.7 Code.