Why can phones run large models? What are INT4/INT8?
Computers use different levels of "precision" to represent numbers. Higher precision means more possible values and greater accuracy, but also more storage space. Every parameter in a large model is a number — precision determines how much memory is needed to store them.
FP32 (32-bit floating point): Each number takes 4 bytes and can represent about 4 billion distinct values. This is the standard precision used in traditional training.
FP16 (16-bit floating point): Each number takes 2 bytes and can represent 65,536 distinct values. Sufficient precision at half the size.
INT8 (8-bit integer): Each number takes 1 byte with only 256 possible values. A common choice for quantization.
INT4 (4-bit integer): Each number is just half a byte with only 16 possible values. Maximum compression.
Click different precision levels to see how the gradient bar's "resolution" changes. Lower precision means coarser color blocks.
Quantization offers three core benefits: smaller size, faster speed, and lower hardware requirements. A 7B parameter model needs 14 GB of memory in FP16, but only 4.1 GB when quantized to Q4_K_M — an ordinary laptop can run it.
The core idea of quantization is simple: map continuous floating-point numbers to a limited set of integers. This process requires two key parameters — scale (the scaling factor) and zero_point (the zero-point offset).
The formula is intuitive: q = round(x / scale) + zero_point
To dequantize: x' = (q - zero_point) * scale
Some error is inevitable, but as long as the scale is chosen well, the error remains small.
Watch how floating-point numbers (orange) get mapped to the INT8 grid (teal). Drag the slider to adjust the scaling factor.
● Original float values ■ INT8 quantization grid Notice how nearby values merge into the same integer
Different quantization levels make different trade-offs between model size and output quality. Using the Perplexity metric, we can precisely measure this trade-off. The chart below shows the cost-effectiveness of each quantization level — Q6_K sits on the Pareto frontier as the optimal balance between quality and size.
Data from AtomGradient's Apple Silicon LLM inference benchmarks. Y-axis shows perplexity increase percentage — lower is better.
"Quantization brings large models from the cloud to your pocket — the same intelligence at one-tenth the size."