Add optional int8 weight-only quantization

[Talpik]

Optional int8 weight-only quantization, for when memory is the binding constraint.

Stacked PR — contains #5 and #6 and should be merged last. Only the final two commits are new here; everything rebases cleanly if the earlier ones land first.

Usage

pip install optimum-quanto
FIRERED_QUANT=int8 FIRERED_WEIGHT_DTYPE=bfloat16 python your_script.py

optimum-quanto is imported lazily and stays an optional dependency: the default FIRERED_QUANT=none does not touch the model and requires nothing installed. A missing package raises a message naming the install command rather than an ImportError traceback.

Results

M4 Max (macOS 26, torch 2.8), Base zero-shot cloning of a 3-sentence paragraph over two reference voices, RTF as the range across 3 interleaved runs, speaker similarity via the bundled CAM++:

Weights	Resident weights	RTF	Speaker sim
float32, no quant (default)	11.42 GiB	0.87–0.96	0.947 / 0.935
bfloat16, no quant	5.71 GiB	0.73–0.76	0.941 / 0.933
bfloat16 + int8	3.90 GiB	1.03–1.05	0.937 / 0.934

int8 is 2.9x smaller than the default at ~40% higher latency than plain bf16, with speaker similarity intact. It is a memory play, not a speed play — worth stating plainly, since where the RAM is available bf16 without quantization is both faster and smaller than the fp32 default.

Why this scope

Quantization is applied to the transformer backbones (tts_core.backbone_llm and redae) and not to the DiT flow head. Chosen by measurement, not assumption:

Scope	Resident weights	RTF	Speaker sim
backbone only	4.40 GiB	1.12	0.926
backbone + redae	3.90 GiB	1.16	0.934
every nn.Linear	3.59 GiB	2.41	0.921

Including the DiT doubles latency for 8% more savings: it runs once per flow timestep with a CFG-doubled batch, so per-call dequantization dominates.

int4 is rejected with an explanatory error rather than silently accepted — measured 3x slower than bf16 with a clear similarity drop (0.828 vs 0.913), and quanto's int4 path conflicts with the @torch.inference_mode() decorators (RuntimeError: Cannot set version_counter for inference tensor).

Also here

One commit corrects the dtype benchmark added in #6. That table timed a single short sentence, where fixed per-call overhead dominates, and wrongly concluded half weights are slower on MPS. On a realistic multi-sentence workload, measured 3 times interleaved, bf16 is consistently ~20% faster than fp32 as well as half the size. #6's own description has been updated accordingly.

Testing

All five inference paths under FIRERED_QUANT=int8: Base and Instruct cloning, voice design, acoustic edit, semantic edit. Worth noting that quantizing the backbone also quantizes lm_head, so the voice-design CoT was the thing to watch — it comes out identical to the unquantized run:

[gender] male [age] middle_age [pitch] low [texture] warm [volume] normal
[accent] american [emotion] calm [fluency] fluent [speed] slow [clarity] clear
[tone] narrative [personality] gentle

Acoustic edit stays numerically correct under int8 (16.5 s input, speed 0.8x → 20.80 s = 16.5/0.8). Invalid values (FIRERED_QUANT=int7) and the default off-path were both checked.

Not tested on CUDA — no CUDA machine here. The default is off, so the CUDA path is untouched unless the variable is set.

Carried from #6

Two commits here belong to #6's subject rather than quantization, and are listed for review clarity:

• FIRERED_WEIGHT_DTYPE now defaults to the checkpoint's own dtype instead of hard-coding float32, which was overriding what transformers does by itself and would silently upcast a half-precision checkpoint back to fp32.
• scripts/convert_to_bf16.py halves the checkpoints on disk (19 GB → 9.7 GB), verifying every tensor against the fp32 original for exact bfloat16 rounding before replacing it. It makes the two routes to half precision explicit: cast on load via FIRERED_WEIGHT_DTYPE, or convert once and load with no variable set. int8 stacks on either.

🤖 Generated with Claude Code