quantizations

Run a <400ms latency Voice Agent on just 4GB VRAM. Fully offline, no API keys required. Optimized for GTX 1650 and edge robotics with zero-copy inference. (Apache 2.0)

High-performance late-interaction retrieval engine for on-prem AI. ColBERT/ColPali multi-vector search with Rust fused MaxSim, Triton GPU kernels, ROQ quantization, LEMUR routing, WAL-backed CRUD, and a FastAPI server — single machine, CPU or GPU.

NestQuant: Post-Training Integer-Nesting Quantization for On-Device DNN [IEEE TMC 2025]

Implementation of "Low-Cost and Effective Fault-Tolerance Enhancement Techniques for Emerging Memories-Based Deep Neural Networks." 2021 58th ACM/IEEE Design Automation Conference (DAC).

SCYTH-J is a neural network-based web application for classifying Japanese texts into user-defined dynamic topics. The system uses a quantized supervised trained deep learning model and natural language processing (NLP) techniques to analyze and categorize texts.

Can I run this LLM? Open-source deployment intelligence for local AI — VRAM estimation, quantization selection, hardware compatibility, speed prediction. Built with FastAPI + Next.js.

Batched QLoRA fine-tuning of FLAN-T5-Large for three-way stance classification, with systematic evaluation of clustering, embedding probes, and full model inference

RAG with Binary Quantization for enhanced performance