Quantizing DistilBERT

Most of my everyday use of quantization is simple. I pull a local coding model with Ollama, see a label like qwen3-coder-next:q4_K_M or int8 in the name, and use it. If the model fits on my machine and responds at a usable speed, I usually do not think much more about it.

That makes quantization look like one command: shrink the model, run it locally.

But that one-step view hides a few questions. Which parts of the model changed? Which parts stayed the same? Did the saved model get smaller? Did inference get faster, or did I only make a smaller file that runs more slowly?

I made this notebook to answer those questions with a model small enough to understand in one sitting.

The model is DistilBERT, a smaller BERT-style encoder. The task is SST-2 sentiment classification: give the model a sentence, get back negative or positive. It is not a local coding model, but it has enough transformer plumbing to make quantization real: embeddings, attention layers, feed-forward layers, and a classifier head.

Here, quantization means either storing model weights with fewer bits or deliberately rounding tensors to see what breaks. The common hope is: turn fp32 weights into int8, get a smaller model, and maybe get faster inference.

The measurements below focus on that last part.

TL;DR: I ran this on a MacBook Pro with an M4 Max and 64 GB of memory. Dynamic int8 quantization cut the saved model size by about 48%, but on my machine it was slower than the fp32 baseline. Fake activation quantization did not test real quantized speed; it only showed which tensors changed accuracy when rounded and converted back to float.

This is not a "quantization makes models faster" post. It is a notebook walkthrough of the places where that expectation cracked. One change at a time, same three columns every time:

• accuracy
• runtime
• serialized model size

I keep those columns separate because they do not always move together. In this run, the model got smaller. It did not get faster.

The Model

The exact model is distilbert-base-uncased-finetuned-sst-2-english. The dataset is SST-2, the same binary sentiment task the model was fine-tuned for.

DistilBERT is a smaller version of BERT. BERT is an encoder model: it reads the whole input at once and builds a contextual representation for each token. For this classifier, the path looks like this:

Take this sentence:

the movie was slow, but the ending worked

The tokenizer splits it into pieces. The encoder blocks let those pieces look at each other. The classifier head turns the final representation into two scores, one for NEGATIVE and one for POSITIVE. The experiments below mostly poke the encoder blocks and classifier path, not the tokenizer or final labels.

For this sentence, the fp32 model predicts:

label: POSITIVE
score: 0.991

That is the output the quantized model should preserve. If a smaller model flips ordinary mixed-sentiment examples like this, the smaller file is not useful for this task.

Inside each encoder block are two big pieces:

• self-attention, where tokens exchange information
• a feed-forward network, where each token representation is transformed

Both pieces contain many Linear modules. PyTorch's dynamic quantization API can quantize those modules directly, so the first experiment starts there.

What Quantization Changes

The model stores many weights as floating-point numbers. Quantization stores some values with fewer bits. The common target is int8: store an 8-bit integer plus scale information instead of a 32-bit float.

A tiny version:

float weights:  [-0.81, -0.12, 0.03, 0.77]
int8 values:    [-127,  -19,   5,  121]
scale:          0.0064

Those numbers are illustrative, not a claim that every quantizer picks exactly that scale or signed range.

The smaller-file part is the easy promise. The faster-runtime part has a catch: your hardware and backend need kernels that make the new math worth it.

The first notebook experiment uses real dynamic int8 quantization:

quantized_model = torch.ao.quantization.quantize_dynamic(
    model.cpu().eval(),
    {torch.nn.Linear},
    dtype=torch.qint8,
)

This changes supported Linear modules and can affect model size and CPU runtime.

The Setup

The notebook runs on CPU. I ran the results below on a MacBook Pro with an M4 Max and 64 GB of memory. The notebook printed Python 3.9.6, PyTorch 2.8.0, up to 8 CPU threads, and qnnpack as the quantized engine.

For the quick pass, the notebook uses:

QUICK_EVAL_SIZE = 256

Every experiment uses the same 256 SST-2 validation examples. That keeps the loop short. It also means small accuracy changes are easy to overread: one changed prediction is about 0.39 percentage points.

For a real accuracy pass, use the full validation split:

QUICK_EVAL_SIZE = None

The evaluation helper records accuracy, total eval time, examples per second, average batch latency, and serialized state_dict size. In the tables, "size" means the saved model weights, not the live Python object in memory. The benchmark rows come from the linked notebook, not a PyTorch or Hugging Face published benchmark.

Step 1: Get The Boring Row

Before touching quantization, run the original fp32 model.

This row is the baseline. Change the machine, backend, batch size, dataset slice, or PyTorch version and you should rerun it. Otherwise the later rows are not being compared against the same setup.

Step 2: Quantize Every Linear Layer

The first real run quantizes every supported Linear layer. That includes a lot of the attention and feed-forward weights, which is where a large chunk of DistilBERT lives.

The file size drops from 255.5 MB to 132.3 MB.

The eval time goes from 0.55s to 1.40s. With qnnpack on this machine, the quantized model was slower.

The accuracy change, +0.0039, is one extra correct prediction on this 256-example slice. That is too small to treat as a real improvement.

That raises a more useful question: did I quantize too much?

Step 3: Quantize Smaller Parts

All Linear layers made the model smaller but slower, so the notebook tries narrower targets: feed-forward layers for the big dense chunk, attention modules for the token-mixing chunk, and the tiny classifier head as a control.

The classifier row barely changes size because the classifier head is tiny compared with the transformer body.

The feed-forward cut saves much more size. The attention cut saves less. Both are still slower than the fp32 baseline on my machine.

"Quantize the model" is incomplete. Quantize which part? For accuracy, eval time, or size? On what backend? There is no single row that answers all of that.

Step 4: Quantize One Block At A Time

Now slice the model a different way. Instead of grouping by component, group by depth.

DistilBERT has six encoder blocks. The notebook quantizes one block at a time to see whether any layer loses accuracy more than the others.

This is still quick mode, so the claim stays small: no single block obviously damaged accuracy on this slice.

The block scan did not point to one obvious layer to keep in float. The next thing to check is activations: the tensors created while the model runs.

Step 5: Round The Activations

So far we mostly changed weights. Activations are the tensors created while the model runs. If you want to go beyond weight-only quantization later, you need to know which activations are sensitive to rounding.

The notebook fake-quantizes a few activation points to int8 and int4. This is not a real quantized-runtime speed test. It is a sensitivity test: does accuracy change when this tensor is rounded and passed to the next module?

Here is the helper:

def fake_quantize_tensor(x: torch.Tensor, bits: int = 8) -> torch.Tensor:
    # This symmetric helper uses -127..127 for int8 and -7..7 for int4.
    qmax = (2 ** (bits - 1)) - 1
 
    # Pick one scale for this tensor. The largest absolute value maps to qmax.
    scale = x.detach().abs().amax().clamp(min=1e-8) / qmax
 
    # Move from float values to integer buckets, then clip to the valid range.
    quantized = torch.round(x / scale).clamp(-qmax, qmax)
 
    # Move back to float so the normal PyTorch model can keep running.
    return quantized * scale

That function rounds a tensor to a low-precision grid, then returns a float tensor. The model still runs through the normal floating-point path, with extra hook and rounding work added on top. This does not prove a speed win or create a smaller model. It answers a narrower question: does this activation survive being rounded?

Dynamic quantization measures size and CPU runtime. Fake quantization measures precision sensitivity.

The targets are plain PyTorch module outputs. I attach a forward hook to a module and intercept its output. The hook rounds that tensor, converts it back to float, and hands the rounded float tensor to the next module.

Here is what each target means:

I also tried lower-level attention projection outputs: Q, K, V, and the output projection. Those are useful when tuning a real inference stack, but they would turn this post into an attention-projection audit. The cleaner module-level targets are enough: they show whether embeddings, attention, feed-forward layers, or the classifier path is more sensitive to rounding.

The row to watch is FFN outputs | int4. Accuracy drops from 0.8672 to 0.6289. Even with a small validation slice, that is a large drop.

The int4 embedding row goes up, but this is a 256-example slice. A few changed predictions can move the number. I would rerun the full split before treating that as a real improvement.

Step 6: Quick Run, Then Full Run

The first pass used 256 examples because I wanted the notebook to stay fast while I was still choosing targets. That is fine for exploration. It is not enough for trust.

On 256 examples, one changed prediction moves accuracy by about 0.39 percentage points. A +0.0039 row can be one sentence changing sides. A 24-point drop, like FFN outputs | int4, is large enough to keep checking.

For the full SST-2 validation split, I did not rerun every row. I reran the ones that could change the conclusion:

FULL_EVAL_SIZE = None

That is the same idea as QUICK_EVAL_SIZE, but the notebook uses a separate name because the full pass builds a fresh validation loader for only the rows I promoted.

That means the baseline, the big size-reduction row, the classifier-only row that looked faster, the clearly bad FFN int4 row, and the embeddings int4 row that looked suspiciously better in quick mode.

The full run makes the quick table calmer.

The all-Linear int8 row is still the real size win: 255.5 MB down to 132.3 MB. It is also still slower on this CPU path, and it loses about 1.15 percentage points of accuracy in this run.

The int4 embeddings bump disappears. On the full split it lands exactly on the baseline accuracy. That quick-run improvement was subset noise.

The FFN int4 row stays bad. The fake-quant timing numbers are notebook loop timings, not deployment timings, because they include hook and rounding overhead. The accuracy signal is clear enough: these activations do not like being rounded that hard.

What This Run Says

I started with a simple habit: pull a quantized local model, see that it fits, and move on. This notebook slowed that habit down.

On this small model, "quantized" was not one thing. Quantizing all Linear layers cut the saved file almost in half, but made this CPU run slower. Quantizing only the classifier ran faster, but barely changed the file size. Rounding FFN activations to int4 broke accuracy. The int4 embedding result looked good in the quick run, then disappeared on the full split.

That is the useful connection back to local LLMs. A q4_K_M or int8 label tells me the model was compressed. It does not tell me whether it will be faster on my machine, whether the quality tradeoff is acceptable for my prompts, or which part of the model paid the cost.

The rule I would reuse is small: measure size, latency on the target backend, and task quality separately. A suffix is a hint. The workload gets the vote.