Quantization: compare size, speed and prediction changes

Quantization represents model values with lower-precision numbers. It can reduce storage and, with supported kernels and hardware, improve latency or energy use. Smaller weights alone do not prove faster inference. Measure size, output changes, task metrics and target-runtime speed separately.

A symmetric int8 weight experiment

The local lab maps each convolutional and dense weight tensor to signed int8 using a scale based on its maximum absolute value, then dequantizes back to float for the existing NumPy forward pass.

from deep_learning_cases import quantization_case

result = quantization_case()
assert result["compression_ratio"] == 8.0
assert result["weight_bytes"]["float64"] == 320
assert result["weight_bytes"]["int8"] == 40
print(result["maximum_probability_change"])

Raw weight storage falls from 320 to 40 bytes, an 8× ratio expected when float64 values become int8. The largest probability change across 60 images is about 0.000562; float and dequantized-int8 accuracy both remain 100% on the easy stripe fixture.

This implementation still computes float operations after dequantization. It supplies no integer-kernel speed evidence. Real artifacts also store scales, zero-points, metadata and graph structure, so complete file compression will differ from the raw-weight ratio.

Choose the quantization scheme

Per-tensor and per-channel scales trade metadata and fidelity. Symmetric and asymmetric mappings handle ranges differently. Weight-only, dynamic and static activation quantization require different calibration and runtime support. Document which operators fall back to float.

Use a representative calibration set that is separate from test when activation ranges are estimated. Compare output differences, task metrics, class and slice regressions, artifact size, memory and warmed latency on the deployment target.

Treat conversion as a model change

Version the quantized artifact separately. Run reload-equivalence within an appropriate tolerance. Define maximum acceptable metric loss and probability drift before conversion. Verify thresholds and calibration because a small score change can flip decisions near a cutoff.

The Data Science course places compression inside the same measured model-selection and release process as architecture changes.

Exercise

Compare per-tensor and per-channel int8 weight quantization. Report full artifact bytes, output drift by confidence band, validation metric and target-hardware median/p95 latency. Reject any candidate that misses the predeclared quality gate.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Measure inference latency with warmup and repeated trials.
• Continue with Write a deep-learning experiment report with negative results.

Reference: current torchao quantization documentation. PyTorch's main documentation now directs active quantization development to torchao; the local NumPy lab does not claim torchao execution.