ShiftQuant

Analyzing the Limits of Shift-Based Post-Training Quantization for LLMs

Shift-based quantization restricts weights to exact powers of two {−4, −2, −1, 0, +1, +2, +4}, enabling multiply-free inference via bit-shifts. This repo contains the full research pipeline: quantization code, AWQ adaptation, WikiText-103 benchmark, diagnostic analysis, and paper draft.

Model: Qwen2-1.5B · Dataset: WikiText-103 · Hardware: RTX 5080 16 GB

---

Key results

Method	PPL	Δ vs FP16	Recovery
FP16 baseline	9.58	—	—
Shift PTQ (Grid A, bs=32)	12.53	+2.95	0%
+ 9-value uniform grid	11.61	+2.03	31%
+ AWQ (isocompression)	11.88	+2.30	22%
+ AWQ × 9v grid	11.06	+1.48	50%

The grid improvement (−0.92 PPL) and AWQ improvement (−0.65 PPL) are 93.6% orthogonal — they can be optimised independently without interaction loss.

---

Findings

1. +30.8% PPL baseline cost at block size 32. Dominant cause: a structural gap at ±3 in the log-uniform grid leaves 25.6% of normalized weights in a region with 2× higher max quantization error than a uniform 4-bit quantizer (4.48× MSE ratio).

2. No 7-value grid escapes the gap. Uniform Grid B {−3..+3} and asymmetric Grid C {−4..+3} are both worse than the log-uniform baseline. Outlier coverage and gap coverage are incompatible objectives at 3-bit precision.

3. Weight-MSE scale optimisation backfires (+13.6 PPL at bs=128). Minimising ‖W − Q(W)‖² clips high-magnitude weights that dominate model output — an empirical rediscovery of the GPTQ/AWQ motivation.

4. AWQ recovers 22% at isocompression, and calibration saturates at 30 windows (15k tokens, ~1 second). No benefit from additional calibration data.

---

Repository structure

PTQ/
├── ptq/
│   ├── quantize.py          # Grid A/B/C/9v quantization + MSE calibration
│   ├── quantized_linear.py  # Drop-in nn.Linear replacement
│   ├── shift_matmul.py      # Dequantize-and-multiply + pure shift reference
│   ├── awq.py               # AWQ diagonal-Hessian adaptation
│   ├── calibrate.py         # Activation scale collection (forward hooks)
│   ├── model_wrapper.py     # Model-level layer replacement
│   └── utils.py             # Memory footprint accounting
├── bench/
│   ├── perplexity.py        # WikiText-103 PPL (non-overlapping 2048-token windows)
│   └── run_benchmark.py     # CLI: all grids, block sizes, AWQ, calibration
├── analysis/
│   └── diagnose.py          # H1/H2/H3 diagnostic scripts
├── paper/
│   ├── abstract.md
│   ├── introduction.md
│   ├── related_work.md
│   ├── method.md
│   ├── results.md
│   ├── conclusion.md
│   ├── build_pdf.py         # Assembles sections → PDF via WeasyPrint
│   └── shiftquant.pdf       # Compiled paper
└── tests/                   # 66 unit tests

---

Usage

# Install dependencies
uv sync   # or: pip install -r requirements.txt

# FP16 baseline + Grid A across block sizes
python -m bench.run_benchmark --model Qwen/Qwen2-1.5B

# Full experiment: all grids + AWQ
python -m bench.run_benchmark \
    --model Qwen/Qwen2-1.5B \
    --block-sizes 32 \
    --grids A B C 9v \
    --awq --awq-grids A 9v \
    --calib-samples 30

# MSE calibration ablation
python -m bench.run_benchmark --model Qwen/Qwen2-1.5B --calibrated

# Run tests
pytest tests/

---

Paper

The full paper draft is in paper/. To rebuild the PDF:

uv tool install weasyprint markdown
python paper/build_pdf.py

---

Citation

@misc{shiftquant2026,
  title   = {ShiftQuant: Analyzing the Limits of Shift-Based Post-Training
             Quantization for LLMs},
  author  = {Zorko},
  year    = {2026},
  url     = {https://github.com/Kyworn/ShiftQuant}
}