{ "cells": [ { "cell_type": "markdown", "id": "21873377", "metadata": {}, "source": [ "# Quantizing DistilBERT on SST-2\n", "\n", "I usually see quantization as a label on a local model: q4, int8, quantized. The label is useful, but it hides what changed inside the model.\n", "\n", "This notebook keeps the problem small. I use a DistilBERT sentiment model on SST-2, then change one quantization target at a time. For each run I track three separate things:\n", "\n", "- accuracy\n", "- runtime\n", "- serialized model size\n", "\n", "The model is `distilbert-base-uncased-finetuned-sst-2-english`. It is small enough for quick CPU experiments, but it still has the transformer pieces that matter here: embeddings, attention projections, feed-forward blocks, layer norm, and a classifier head.\n", "\n", "There are two kinds of experiments:\n", "\n", "1. **Real dynamic int8 quantization**: PyTorch stores supported `Linear` weights as int8 and uses the supported CPU path.\n", "2. **Fake activation quantization**: selected activation tensors are rounded to fewer bits, then converted back to float. This tests accuracy sensitivity. It is not a real int8 runtime speed test.\n", "\n", "DistilBERT is encoder-only, so there is no KV cache in this notebook. KV-cache quantization belongs with decoder models such as Llama, Mistral, or Qwen." ] }, { "cell_type": "markdown", "id": "47bb70ee", "metadata": {}, "source": [ "## 1. Setup\n", "\n", "This notebook runs on CPU. That keeps the experiment focused on PyTorch dynamic quantization, which is mainly a CPU feature.\n", "\n", "The defaults are for a quick first pass:\n", "\n", "- `QUICK_EVAL_SIZE = 256` keeps the run short.\n", "- Set `QUICK_EVAL_SIZE = None` to use the full SST-2 validation split.\n", "- `BATCH_SIZE` affects timing. Do not compare latency numbers from different batch sizes as if they are the same experiment.\n", "\n", "If imports fail, uncomment the install cell. `urllib3<2` avoids a common macOS LibreSSL warning." ] }, { "cell_type": "code", "execution_count": 1, "id": "2ea9aee3", "metadata": { "execution": { "iopub.execute_input": "2026-07-08T06:02:15.118654Z", "iopub.status.busy": "2026-07-08T06:02:15.118407Z", "iopub.status.idle": "2026-07-08T06:02:15.123305Z", "shell.execute_reply": "2026-07-08T06:02:15.122638Z" } }, "outputs": [], "source": [ "# Run this only if the notebook environment is missing dependencies.\n", "# %pip install -q torch transformers datasets pandas numpy tqdm psutil matplotlib \"urllib3<2\"\n", "" ] }, { "cell_type": "code", "execution_count": 2, "id": "161c990b", "metadata": { "execution": { "iopub.execute_input": "2026-07-08T06:02:15.125688Z", "iopub.status.busy": "2026-07-08T06:02:15.125499Z", "iopub.status.idle": "2026-07-08T06:02:17.679694Z", "shell.execute_reply": "2026-07-08T06:02:17.679399Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Device: cpu\n", "Quantized engine: qnnpack\n", "PyTorch: 2.8.0\n" ] } ], "source": [ "import copy\n", "import gc\n", "import os\n", "import random\n", "import tempfile\n", "import time\n", "import warnings\n", "from contextlib import contextmanager\n", "from typing import Iterable\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pandas as pd\n", "import psutil\n", "import torch\n", "import torch.nn as nn\n", "from datasets import load_dataset\n", "\n", "try:\n", " from IPython.display import display\n", "except ImportError:\n", " # Fallback for running this notebook code as a plain Python script.\n", " def display(value):\n", " print(value)\n", "\n", "from torch.utils.data import DataLoader\n", "from tqdm.auto import tqdm\n", "from transformers import AutoModelForSequenceClassification, AutoTokenizer, DataCollatorWithPadding\n", "\n", "# Keep experiment knobs in one place so reruns change only these values.\n", "SEED = 7\n", "MODEL_NAME = \"distilbert-base-uncased-finetuned-sst-2-english\"\n", "QUICK_EVAL_SIZE = 256 # Use None for the full SST-2 validation split.\n", "BATCH_SIZE = 16 # Timing numbers only compare fairly at the same batch size.\n", "MAX_LENGTH = 128 # Truncate long examples so every run has the same input cap.\n", "DEVICE = torch.device(\"cpu\")\n", "\n", "\n", "def seed_everything(seed: int) -> None:\n", " random.seed(seed)\n", " np.random.seed(seed)\n", " torch.manual_seed(seed)\n", "\n", "\n", "seed_everything(SEED)\n", "\n", "# Cap CPU threads so timing is less noisy and the notebook stays responsive.\n", "torch.set_num_threads(max(1, min(8, os.cpu_count() or 1)))\n", "\n", "# qnnpack is commonly available on Apple Silicon for quantized CPU ops.\n", "# If it is not available, PyTorch keeps its default backend.\n", "if \"qnnpack\" in torch.backends.quantized.supported_engines:\n", " torch.backends.quantized.engine = \"qnnpack\"\n", "\n", "print(f\"Device: {DEVICE}\")\n", "print(f\"Quantized engine: {torch.backends.quantized.engine}\")\n", "print(f\"PyTorch: {torch.__version__}\")\n", "" ] }, { "cell_type": "markdown", "id": "d7cbbbbe", "metadata": {}, "source": [ "## 2. Load SST-2\n", "\n", "SST-2 is a binary sentiment task: each sentence is labeled negative or positive. That gives us a simple output to preserve while changing the model.\n", "\n", "In quick mode I shuffle and slice the validation split once. Every experiment uses the same examples, so differences come from the model variant, not from a different dataset slice." ] }, { "cell_type": "code", "execution_count": 3, "id": "c2760d33", "metadata": { "execution": { "iopub.execute_input": "2026-07-08T06:02:17.681044Z", "iopub.status.busy": "2026-07-08T06:02:17.680852Z", "iopub.status.idle": "2026-07-08T06:02:24.705744Z", "shell.execute_reply": "2026-07-08T06:02:24.705422Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Validation examples: 256\n", "Batches: 16\n" ] } ], "source": [ "tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)\n", "raw_validation = load_dataset(\"glue\", \"sst2\", split=\"validation\")\n", "\n", "if QUICK_EVAL_SIZE is not None:\n", " # Use a fixed shuffled subset so quick experiments stay comparable.\n", " eval_size = min(QUICK_EVAL_SIZE, len(raw_validation))\n", " raw_validation = raw_validation.shuffle(seed=SEED).select(range(eval_size))\n", "\n", "\n", "def tokenize_batch(batch: dict) -> dict:\n", " # Convert raw sentences into token IDs that DistilBERT can read.\n", " return tokenizer(batch[\"sentence\"], truncation=True, max_length=MAX_LENGTH)\n", "\n", "\n", "tokenized_validation = raw_validation.map(tokenize_batch, batched=True)\n", "tokenized_validation = tokenized_validation.remove_columns([\"sentence\", \"idx\"])\n", "tokenized_validation = tokenized_validation.rename_column(\"label\", \"labels\")\n", "tokenized_validation.set_format(\"torch\")\n", "\n", "# Pad each batch to its longest example instead of padding the whole dataset.\n", "collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors=\"pt\")\n", "validation_loader = DataLoader(\n", " tokenized_validation,\n", " batch_size=BATCH_SIZE,\n", " shuffle=False,\n", " collate_fn=collator,\n", ")\n", "\n", "print(f\"Validation examples: {len(tokenized_validation)}\")\n", "print(f\"Batches: {len(validation_loader)}\")\n", "" ] }, { "cell_type": "markdown", "id": "c3c415a4", "metadata": {}, "source": [ "## 3. Measurement Helpers\n", "\n", "Before changing the model, define one evaluation path and reuse it everywhere.\n", "\n", "The helper records accuracy, total runtime, average batch latency, memory movement, and serialized model size. The size and timing numbers are the main ones for this notebook. The memory number is only a rough process-level check." ] }, { "cell_type": "code", "execution_count": 4, "id": "73fa71e0", "metadata": { "execution": { "iopub.execute_input": "2026-07-08T06:02:24.707106Z", "iopub.status.busy": "2026-07-08T06:02:24.707021Z", "iopub.status.idle": "2026-07-08T06:02:24.714968Z", "shell.execute_reply": "2026-07-08T06:02:24.714593Z" } }, "outputs": [], "source": [ "process = psutil.Process(os.getpid())\n", "\n", "\n", "def model_size_mb(model: nn.Module) -> float:\n", " \"\"\"Measure the saved weights, not the live Python object.\"\"\"\n", " with tempfile.NamedTemporaryFile(suffix=\".pt\") as tmp:\n", " # state_dict is what we would usually save or ship.\n", " torch.save(model.state_dict(), tmp.name)\n", " tmp.flush()\n", " return os.path.getsize(tmp.name) / (1024 ** 2)\n", "\n", "\n", "def load_base_model() -> nn.Module:\n", " return AutoModelForSequenceClassification.from_pretrained(MODEL_NAME).to(DEVICE).eval()\n", "\n", "\n", "@torch.inference_mode()\n", "def evaluate_model(\n", " model: nn.Module,\n", " name: str,\n", " quantization_type: str,\n", " target: str,\n", " bit_width: str,\n", ") -> dict:\n", " \"\"\"Run one model variant on the fixed validation loader.\"\"\"\n", " model.eval().to(DEVICE)\n", "\n", " correct = 0\n", " total = 0\n", " batch_latencies = []\n", " memory_before = process.memory_info().rss / (1024 ** 2)\n", " started = time.perf_counter() # Wall-clock time for the whole validation pass.\n", "\n", " for batch in tqdm(validation_loader, desc=name, leave=False):\n", " batch = {key: value.to(DEVICE) for key, value in batch.items()}\n", " labels = batch.pop(\"labels\")\n", "\n", " batch_started = time.perf_counter()\n", " outputs = model(**batch)\n", " batch_latencies.append(time.perf_counter() - batch_started)\n", "\n", " # The larger logit chooses NEGATIVE or POSITIVE.\n", " predictions = outputs.logits.argmax(dim=-1)\n", " correct += (predictions.cpu() == labels.cpu()).sum().item()\n", " total += labels.numel()\n", "\n", " elapsed = time.perf_counter() - started\n", " memory_after = process.memory_info().rss / (1024 ** 2)\n", "\n", " return {\n", " \"experiment\": name,\n", " \"quantization_type\": quantization_type,\n", " \"target_component\": target,\n", " \"bit_width\": bit_width,\n", " \"accuracy\": correct / total,\n", " \"eval_seconds\": elapsed,\n", " \"examples_per_second\": len(tokenized_validation) / elapsed,\n", " \"avg_batch_latency_ms\": np.mean(batch_latencies) * 1000,\n", " \"model_size_mb\": model_size_mb(model),\n", " \"memory_delta_mb\": memory_after - memory_before,\n", " }\n", "\n", "\n", "def finish_result_table(rows: list[dict]) -> pd.DataFrame:\n", " \"\"\"Add deltas against the first row, which should be the fp32 baseline.\"\"\"\n", " df = pd.DataFrame(rows)\n", " baseline = df.iloc[0]\n", "\n", " df[\"accuracy_delta\"] = df[\"accuracy\"] - baseline[\"accuracy\"]\n", " df[\"speedup\"] = baseline[\"eval_seconds\"] / df[\"eval_seconds\"]\n", " df[\"size_reduction\"] = 1 - (df[\"model_size_mb\"] / baseline[\"model_size_mb\"])\n", "\n", " columns = [\n", " \"experiment\",\n", " \"quantization_type\",\n", " \"target_component\",\n", " \"bit_width\",\n", " \"accuracy\",\n", " \"accuracy_delta\",\n", " \"eval_seconds\",\n", " \"speedup\",\n", " \"model_size_mb\",\n", " \"size_reduction\",\n", " \"examples_per_second\",\n", " \"avg_batch_latency_ms\",\n", " \"memory_delta_mb\",\n", " ]\n", " return df[columns].sort_values([\"quantization_type\", \"target_component\", \"experiment\"]).reset_index(drop=True)\n", "\n", "\n", "def cleanup(*models: nn.Module) -> None:\n", " for model in models:\n", " del model\n", " gc.collect()\n", "" ] }, { "cell_type": "markdown", "id": "6ca2f324", "metadata": {}, "source": [ "## 4. Baseline\n", "\n", "Start with the unmodified fp32 model. Every later row compares against this one.\n", "\n", "If I change the dataset size, batch size, backend, or machine, I rerun the baseline too.\n" ] }, { "cell_type": "code", "execution_count": 5, "id": "69318db6", "metadata": { "execution": { "iopub.execute_input": "2026-07-08T06:02:24.716261Z", "iopub.status.busy": "2026-07-08T06:02:24.716175Z", "iopub.status.idle": "2026-07-08T06:02:25.899347Z", "shell.execute_reply": "2026-07-08T06:02:25.899084Z" } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c9773af5b2fb485886b76ad9e30d57b2", "version_major": 2, "version_minor": 0 }, "text/plain": [ "baseline fp32: 0%| | 0/16 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
experimentquantization_typetarget_componentbit_widthaccuracyeval_secondsexamples_per_secondavg_batch_latency_msmodel_size_mbmemory_delta_mb
0baseline fp32noneentire modelfp320.8671880.548506466.72222732.822719255.452064367.25
\n", "" ], "text/plain": [ " experiment quantization_type target_component bit_width accuracy \\\n", "0 baseline fp32 none entire model fp32 0.867188 \n", "\n", " eval_seconds examples_per_second avg_batch_latency_ms model_size_mb \\\n", "0 0.548506 466.722227 32.822719 255.452064 \n", "\n", " memory_delta_mb \n", "0 367.25 " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results = []\n", "\n", "baseline_model = load_base_model()\n", "baseline_row = evaluate_model(\n", " baseline_model,\n", " name=\"baseline fp32\",\n", " quantization_type=\"none\",\n", " target=\"entire model\",\n", " bit_width=\"fp32\",\n", ")\n", "results.append(baseline_row)\n", "\n", "pd.DataFrame(results)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### How to read the result columns\n", "\n", "- **Accuracy** is the fraction of examples predicted correctly. Higher is better.\n", "- **Accuracy delta** is variant accuracy minus baseline accuracy. On this 256-example subset, `+0.0039` is one extra correct prediction.\n", "- **Eval seconds** is wall-clock time for this validation pass. Lower is better.\n", "- **Speedup** is `baseline eval seconds / variant eval seconds`. `1.00x` means same speed as baseline. Values below `1.00x` are slower.\n", "- **Size reduction** is how much smaller the serialized model got.\n", "\n", "Keep size and speed separate. A smaller model is not automatically a faster model." ] }, { "cell_type": "markdown", "id": "7751f4b0", "metadata": {}, "source": [ "## 5. Real Dynamic Weight Quantization\n", "\n", "This is real quantization for supported CPU operators.\n", "\n", "PyTorch dynamic quantization stores supported `Linear` weights as int8 and uses quantized CPU kernels for those modules. It does not make every tensor int8, and it does not guarantee speed.\n", "\n", "I test a few cuts:\n", "\n", "- all `Linear` layers\n", "- the classifier head\n", "- the feed-forward blocks\n", "- the attention modules\n", "\n", "The selective runs answer a practical question: which part gives size reduction, and what happens to runtime and accuracy?" ] }, { "cell_type": "code", "execution_count": 6, "id": "45d2248d", "metadata": { "execution": { "iopub.execute_input": "2026-07-08T06:02:25.900592Z", "iopub.status.busy": "2026-07-08T06:02:25.900507Z", "iopub.status.idle": "2026-07-08T06:02:31.095381Z", "shell.execute_reply": "2026-07-08T06:02:31.095117Z" } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "79f6c91670374a0cb33e83c2ea03370e", "version_major": 2, "version_minor": 0 }, "text/plain": [ "dynamic int8 all Linear: 0%| | 0/16 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
experimentquantization_typetarget_componentbit_widthaccuracyaccuracy_deltaeval_secondsspeedupmodel_size_mbsize_reductionexamples_per_secondavg_batch_latency_msmemory_delta_mb
0baseline fp32noneentire modelfp320.8671880.0000000.5485061.000000255.4520640.000000466.72222732.822719367.250000
1dynamic int8 all Linearreal dynamicall Linear layersint80.8710940.0039061.4004950.391651132.2883290.482140182.79245385.96757348.203125
2dynamic int8 attention modulesreal dynamicattention modulesint80.863281-0.0039060.8702060.630318214.9701600.158472294.18334352.91028988.390625
3dynamic int8 classifierreal dynamicclassifierint80.8671880.0000000.5042971.087666255.4520640.000000507.63778430.233432-9.875000
4dynamic int8 FFNreal dynamictransformer FFNint80.8710940.0039061.1366260.482574174.4609590.317050225.22800869.59305552.765625
\n", "" ], "text/plain": [ " experiment quantization_type target_component \\\n", "0 baseline fp32 none entire model \n", "1 dynamic int8 all Linear real dynamic all Linear layers \n", "2 dynamic int8 attention modules real dynamic attention modules \n", "3 dynamic int8 classifier real dynamic classifier \n", "4 dynamic int8 FFN real dynamic transformer FFN \n", "\n", " bit_width accuracy accuracy_delta eval_seconds speedup model_size_mb \\\n", "0 fp32 0.867188 0.000000 0.548506 1.000000 255.452064 \n", "1 int8 0.871094 0.003906 1.400495 0.391651 132.288329 \n", "2 int8 0.863281 -0.003906 0.870206 0.630318 214.970160 \n", "3 int8 0.867188 0.000000 0.504297 1.087666 255.452064 \n", "4 int8 0.871094 0.003906 1.136626 0.482574 174.460959 \n", "\n", " size_reduction examples_per_second avg_batch_latency_ms memory_delta_mb \n", "0 0.000000 466.722227 32.822719 367.250000 \n", "1 0.482140 182.792453 85.967573 48.203125 \n", "2 0.158472 294.183343 52.910289 88.390625 \n", "3 0.000000 507.637784 30.233432 -9.875000 \n", "4 0.317050 225.228008 69.593055 52.765625 " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def get_submodule(root: nn.Module, path: str) -> nn.Module:\n", " \"\"\"Resolve dotted module paths like distilbert.transformer.layer.0.ffn.\"\"\"\n", " module = root\n", " for part in path.split(\".\"):\n", " module = module[int(part)] if part.isdigit() else getattr(module, part)\n", " return module\n", "\n", "\n", "def set_submodule(root: nn.Module, path: str, replacement: nn.Module) -> None:\n", " \"\"\"Replace a nested module after locating it by dotted path.\"\"\"\n", " if \".\" not in path:\n", " setattr(root, path, replacement)\n", " return\n", "\n", " parent_path, child_name = path.rsplit(\".\", 1)\n", " parent = get_submodule(root, parent_path)\n", " if child_name.isdigit():\n", " parent[int(child_name)] = replacement\n", " else:\n", " setattr(parent, child_name, replacement)\n", "\n", "\n", "@contextmanager\n", "def suppress_known_quantization_warnings():\n", " \"\"\"Keep expected PyTorch eager/qnnpack warnings out of result tables.\"\"\"\n", " with warnings.catch_warnings():\n", " warnings.filterwarnings(\n", " \"ignore\",\n", " message=\"torch.ao.quantization is deprecated.*\",\n", " category=DeprecationWarning,\n", " )\n", " old_stderr_fd = os.dup(2)\n", " with open(os.devnull, \"w\") as devnull:\n", " try:\n", " os.dup2(devnull.fileno(), 2)\n", " yield\n", " finally:\n", " os.dup2(old_stderr_fd, 2)\n", " os.close(old_stderr_fd)\n", "\n", "\n", "def dynamic_quantize_module(module: nn.Module) -> nn.Module:\n", " \"\"\"Apply real dynamic int8 quantization to Linear layers inside one module.\"\"\"\n", " with suppress_known_quantization_warnings():\n", " return torch.ao.quantization.quantize_dynamic(\n", " copy.deepcopy(module).cpu().eval(),\n", " {nn.Linear},\n", " dtype=torch.qint8,\n", " )\n", "\n", "\n", "def quantize_selected_paths(model: nn.Module, paths: Iterable[str]) -> nn.Module:\n", " \"\"\"Copy a model, then dynamically quantize only the requested submodules.\"\"\"\n", " quantized = copy.deepcopy(model).cpu().eval()\n", " for path in paths:\n", " original_submodule = get_submodule(quantized, path)\n", " set_submodule(quantized, path, dynamic_quantize_module(original_submodule))\n", " return quantized\n", "\n", "\n", "def transformer_layer_paths() -> list[str]:\n", " return [f\"distilbert.transformer.layer.{i}\" for i in range(6)]\n", "\n", "\n", "# These paths define the model parts we quantize separately.\n", "classifier_paths = [\"pre_classifier\", \"classifier\"]\n", "ffn_paths = [f\"distilbert.transformer.layer.{i}.ffn\" for i in range(6)]\n", "attention_paths = [f\"distilbert.transformer.layer.{i}.attention\" for i in range(6)]\n", "\n", "base_for_quant = load_base_model()\n", "\n", "dynamic_experiments = [\n", " (\n", " \"dynamic int8 all Linear\",\n", " \"all Linear layers\",\n", " lambda model: torch.ao.quantization.quantize_dynamic(\n", " copy.deepcopy(model).cpu().eval(),\n", " {nn.Linear},\n", " dtype=torch.qint8,\n", " ),\n", " ),\n", " (\"dynamic int8 classifier\", \"classifier\", lambda model: quantize_selected_paths(model, classifier_paths)),\n", " (\"dynamic int8 FFN\", \"transformer FFN\", lambda model: quantize_selected_paths(model, ffn_paths)),\n", " (\"dynamic int8 attention modules\", \"attention modules\", lambda model: quantize_selected_paths(model, attention_paths)),\n", "]\n", "\n", "for name, target, build_variant in dynamic_experiments:\n", " with suppress_known_quantization_warnings():\n", " quantized_model = build_variant(base_for_quant)\n", " # This row measures a real quantized model variant.\n", " results.append(evaluate_model(quantized_model, name, \"real dynamic\", target, \"int8\"))\n", "\n", "finish_result_table(results)\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Reading the dynamic quantization result\n", "\n", "- The all-`Linear` variant gives the biggest size reduction because most DistilBERT parameters live in linear layers.\n", "- The classifier-only row barely changes size because the classifier head is tiny compared with the transformer body.\n", "- In this run, the broad quantized variants reduced size but were slower than fp32 with `qnnpack`.\n", "- Small positive accuracy deltas are not quality wins. On 256 examples, one changed prediction moves accuracy by about 0.39 percentage points." ] }, { "cell_type": "markdown", "id": "6627e65c", "metadata": {}, "source": [ "## 6. One Transformer Layer at a Time\n", "\n", "Now quantize one DistilBERT block at a time.\n", "\n", "This checks whether one layer is much more fragile than the others. If one layer caused a large accuracy drop, I would keep that layer in float and try mixed precision.\n", "\n", "In quick mode, small positive accuracy changes are usually noise. With 256 examples, one changed prediction already moves accuracy by about 0.39 percentage points." ] }, { "cell_type": "code", "execution_count": 7, "id": "3d7987bf", "metadata": { "execution": { "iopub.execute_input": "2026-07-08T06:02:31.096748Z", "iopub.status.busy": "2026-07-08T06:02:31.096659Z", "iopub.status.idle": "2026-07-08T06:02:37.491102Z", "shell.execute_reply": "2026-07-08T06:02:37.490835Z" } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d44b43f4a8ec410fa53d89f284fc696b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "dynamic int8 transformer layer 0: 0%| | 0/16 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
experimentquantization_typetarget_componentbit_widthaccuracyaccuracy_deltaeval_secondsspeedupmodel_size_mbsize_reductionexamples_per_secondavg_batch_latency_msmemory_delta_mb
0baseline fp32noneentire modelfp320.8671880.0000000.5485061.000000255.4520640.000000466.72222732.822719367.250000
1dynamic int8 transformer layer 0real dynamictransformer layer 0int80.8710940.0039060.6874960.797832235.2066640.079253372.36607441.50758647.187500
2dynamic int8 transformer layer 1real dynamictransformer layer 1int80.8710940.0039060.6751310.812444235.2066640.079253379.18585740.86360718.421875
3dynamic int8 transformer layer 2real dynamictransformer layer 2int80.8710940.0039060.6738530.813985235.2067250.079253379.90484040.73848758.500000
4dynamic int8 transformer layer 3real dynamictransformer layer 3int80.8671880.0000000.8880570.617648235.2067250.079253288.26993353.96397469.453125
5dynamic int8 transformer layer 4real dynamictransformer layer 4int80.8710940.0039060.8539600.642309235.2067250.079253299.77982051.7779191.750000
6dynamic int8 transformer layer 5real dynamictransformer layer 5int80.8671880.0000000.8903710.616043235.2067250.079253287.52074354.0848360.031250
\n", "" ], "text/plain": [ " experiment quantization_type target_component \\\n", "0 baseline fp32 none entire model \n", "1 dynamic int8 transformer layer 0 real dynamic transformer layer 0 \n", "2 dynamic int8 transformer layer 1 real dynamic transformer layer 1 \n", "3 dynamic int8 transformer layer 2 real dynamic transformer layer 2 \n", "4 dynamic int8 transformer layer 3 real dynamic transformer layer 3 \n", "5 dynamic int8 transformer layer 4 real dynamic transformer layer 4 \n", "6 dynamic int8 transformer layer 5 real dynamic transformer layer 5 \n", "\n", " bit_width accuracy accuracy_delta eval_seconds speedup model_size_mb \\\n", "0 fp32 0.867188 0.000000 0.548506 1.000000 255.452064 \n", "1 int8 0.871094 0.003906 0.687496 0.797832 235.206664 \n", "2 int8 0.871094 0.003906 0.675131 0.812444 235.206664 \n", "3 int8 0.871094 0.003906 0.673853 0.813985 235.206725 \n", "4 int8 0.867188 0.000000 0.888057 0.617648 235.206725 \n", "5 int8 0.871094 0.003906 0.853960 0.642309 235.206725 \n", "6 int8 0.867188 0.000000 0.890371 0.616043 235.206725 \n", "\n", " size_reduction examples_per_second avg_batch_latency_ms memory_delta_mb \n", "0 0.000000 466.722227 32.822719 367.250000 \n", "1 0.079253 372.366074 41.507586 47.187500 \n", "2 0.079253 379.185857 40.863607 18.421875 \n", "3 0.079253 379.904840 40.738487 58.500000 \n", "4 0.079253 288.269933 53.963974 69.453125 \n", "5 0.079253 299.779820 51.777919 1.750000 \n", "6 0.079253 287.520743 54.084836 0.031250 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "layer_rows = []\n", "\n", "for layer_index, layer_path in enumerate(transformer_layer_paths()):\n", " # Quantize one encoder block and leave the other five unchanged.\n", " layer_quantized = quantize_selected_paths(base_for_quant, [layer_path])\n", " row = evaluate_model(\n", " layer_quantized,\n", " name=f\"dynamic int8 transformer layer {layer_index}\",\n", " quantization_type=\"real dynamic\",\n", " target=f\"transformer layer {layer_index}\",\n", " bit_width=\"int8\",\n", " )\n", " results.append(row)\n", " layer_rows.append(row)\n", "\n", "layer_df = finish_result_table([results[0], *layer_rows])\n", "layer_df\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Reading the layer-wise result\n", "\n", "- No single transformer layer obviously broke accuracy on this quick subset.\n", "- Each one-layer experiment has a similar size reduction because the six DistilBERT blocks are roughly the same size.\n", "- The timing rows are backend-dependent. Use them as a reason to measure latency on your own machine, not as a universal layer ranking.\n", "- The next useful follow-up is: which layers can I quantize together before accuracy or latency becomes unacceptable?" ] }, { "cell_type": "code", "execution_count": 8, "id": "5f68a1d0", "metadata": { "execution": { "iopub.execute_input": "2026-07-08T06:02:37.492325Z", "iopub.status.busy": "2026-07-08T06:02:37.492250Z", "iopub.status.idle": "2026-07-08T06:02:37.563480Z", "shell.execute_reply": "2026-07-08T06:02:37.563174Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAEiCAYAAABkykQ1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8ekN5oAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA+8ElEQVR4nO3dCbiM5f/H8a99y75GQpYQWbP0U1oUJVsl/CpLQtqIiLIkpBRZSxKlEilpkxL1VxHZU5GKKNkSStnnf33u//XMf2bOnOPMmXMcZ+b9uq5xzDP3PHM/28zzfe77/j6ZfD6fzwAAAAAgCpmjeTMAAAAACIEFAAAAgKgRWAAAAACIGoEFAAAAgKgRWAAAAACIGoEFAAAAgKgRWAAAAACIGoEFAAAAgKgRWAAAAACIGoEFgLPOSy+9ZJkyZbJt27ZF/N5HH33UvfdM+Pvvv+3OO++0EiVKuM/s3bv3GfnceFW2bFnr3LnzWbs/IHU99dRTdsEFF1iWLFmsZs2aFuvYVxELCCyAVPDss8+6H4T69eund1UQxuOPP27z589Pk/kqCOrZs6e98sordvvtt6f6Z8SbZcuWuROsAwcOpHdVYs6CBQvcus0IPv74Y+vfv7/95z//sRkzZrhjDcDZj8ACSAWvvfaau5q6cuVK+/HHH9O7OjhDgcWSJUusQYMGNnToULvtttusTp06qf4Z8RhYDBs2LGxgsXnzZnvhhRcimt+gQYPs33//TcUaZuzAQus2I9CxlTlzZnvxxRetY8eOdv3116d3lQAkA4EFEKWtW7e6k6GxY8da0aJFXZBxtjp8+HB6VyGm7NmzxwoUKHDackeOHLFTp06dkTrFshw5cli2bNkiek/WrFktZ86caVanWHXixAk7duxYuh5buXLlsuzZs6fK/Hw+31kRYP7zzz+WUWXkuuPMIbAAoqRAomDBgta8eXO7+eabEw0sdAX2gQcecC0bOkE677zz3JW4ffv2BZ2AqqtCpUqV3MnQueeeazfeeKP99NNP7vXPPvvMdbnS30Aai6Dp6pbjUV/0c845x71XV/vy5s1rt956q3vt888/t7Zt29r555/v6lK6dGlXt3A/vJs2bbJbbrnFBU36ob/wwgvtkUceca99+umn7nPffvvtBO+bNWuWe2358uVJrr9vv/3WrrrqKjdvrZMRI0YkehL+4Ycf2mWXXWZ58uRxy6N1rvcnRXVQQPXyyy+7/+vh9dP/5Zdf7O6773bLpM8vXLiwWy+nG9vhbQcFlR988IF/vnqf99rs2bPd1fJSpUpZ7ty57dChQ+69c+fOdS0b+rwiRYq4lo7ffvstaP7ettu+fbvdcMMN7v+az+TJk93r33zzjVtnWg9lypRx6zo5tA9q3vnz53cBUadOnWzdunUJ9p0rrrjCPULpvdp/Az399NN26aWXunWnZdKyvfnmm2G3w7333utajqpVq+b2u4suusgWLlzoL6N9v1+/fu7/5cqVC1qv4cZYeK+He3jvCddvPTl18Wh71q1b1x2P5cuXt+effz6ivvCRbG9Nb926tfu/jrcHH3zQTp48GVRWx8a4ceNcfVWn4sWLW48ePezPP/9Msh76DG//CVxPgd8f2paat5ZT6+S7775zwcWQIUPcMmi/0T6nY1DHfqDAeUydOtU/j0suucS+/vrroLK7du2yLl26uONdZfQ916pVK/8203zU/UnHrVdPb/9UwDN8+HD//LVPPPzww3b06NGgz9B0HTsfffSR235a/9p23vH5xhtvuNYbHVf6LtF398GDB918NFaqWLFibjuonqHzlldffdW/XQsVKmTt27e3HTt2BJXRMaT9a/Xq1Xb55Ze77wHVNRJaDzrWVR8tb9WqVe25554LKqPjWPvW8ePHE7z/2muvdd9v6VF3xKes6V0BIKNTIKGTf11Z69Chg/vS1w+pflADB/nqx/j777+3O+64w2rXru0Cinfffdd+/fVX96OgEwj9EC5evNh90ffq1cv++usvW7RokW3cuNH9kEZKP8JNmza1Ro0auR98/Th4Jzu6+qSxATohVBeuiRMnurroNc+GDRtcvXWVuHv37u7HWoHKe++9ZyNHjnQ/PgpKtA7atGmTYL2ozg0bNky0fjrBuPLKK109BwwY4E5adFKiH7xQGsOgH1Atz5NPPunqr3WtZVu7dm2CE97A92mAdb169dwyiLcutZ3U2qT1rZMcndhonlounVR56ytUlSpV3HwVjOl9ffv2ddN1MuidHOnkR/uETg51YqL/6+RIJyraN0aNGmW7d++28ePH25dffumWIbD1Q/vDdddd537UR48e7danToa1jhTYKUjUfjdlyhQXoGo962Q8qSu2Onn74osv7K677nLLoIBQ6zQaqn/Lli1dfXQSqoBKwdn777/vAr9A+ux58+a5YE4ncxMmTLCbbrrJBVDaD7U8P/zwg73++uv2zDPPuOPCW6/haBuEUjCnq906KUzK6eoi2ibNmjVzJ746CdU2eeyxxxKtT6hIt7f2bY3T0rH6ySef2JgxY9y+quPUoyDCm+/999/vgttJkya5+Wm+ibXo6H07d+503yfh1pt3EquLGzpOdBKrk04FxNOmTXPfbd26dXPfSeqepLrqeyN0ULWCXJXR5+kEXvuutuvPP//sr5vWsy4I3Hfffe641fZSvbTu9Vz10/eA5q/PFgWvomNZFwkUCOi4W7FihVu3+m4NvcChrnOqt+qiugeeYOs9+p7R9466r+r7T/VT9ysFaQoev/rqK7eudVwpuPLou2/w4MHugovqs3fvXvd+Hauh2/WPP/5wx7G+YxRUKhCMhL6PFETqGFPrm757tc8qwLznnntcGY3tmjlzpgui9BsS+P2qLmXqqpkedUec8gFIsVWrVvl0GC1atMg9P3XqlO+8887z9erVK6jckCFDXLl58+YlmIfeI9OnT3dlxo4dm2iZTz/91JXR30Bbt25102fMmOGf1qlTJzdtwIABCeb3zz//JJg2atQoX6ZMmXy//PKLf9rll1/uy5s3b9C0wPrIwIEDfTly5PAdOHDAP23Pnj2+rFmz+oYOHepLSu/evV0dV6xYEfTe/Pnzu+laLvnrr798BQoU8HXr1i3o/bt27XJlA6frM0O/2vLkyePWR3LWw/Lly937Z86c6TudMmXK+Jo3bx40zdtGF1xwQdD8jx075itWrJivWrVqvn///dc//f3333fltY+EbrvHH3/cP+3PP//05cqVy22j2bNn+6dv2rTJlT3dup4/f74rN3r0aP+0EydO+C677LIE+07jxo3dI5TqpWVOah1qObWMV111VdB0fUb27Nl9P/74o3/a+vXr3fSJEyf6pz311FNB2z6QPjvcdvRo2UK3Xbj9Ibl1adGihS937ty+3377zT9ty5Ytbt8+3c9nSrb3Y489FjSPWrVq+erUqeN//vnnn7tyr732WlC5hQsXhp0e6p577glbb+/7I1++fO74C6R95OjRo0HTtC8WL17cd8cddySYR+HChX379+/3T3/nnXfc9Pfee8//Xj3Xdk6K1omO20Dr1q1z773zzjuDpj/44INu+pIlS4L2FU3Tugl3fGq7aBt5OnTo4I6t6667Lqh8w4YNg/b5bdu2+bJkyeIbOXJkULlvvvnG7ReB03UM6bOmTJniS45w+2q476imTZu67xfPyZMn3e9Ou3btgsrpt0TL9PPPP6d53QEPXaGAKOgqsq7i6Kq76Apdu3bt3FXbwC4Mb731ltWoUSPBVX3vPV4ZXaHVVbzEyqRE4NVOT2CLgLobqPVEVwR1zqWrVqIrWUuXLnUtLOoylVh9dLVcV+QDu7/MmTPHtULoKtfpBpNq8LNaEzy6Gux12fLoaqa68ejqo+rqPZSGUld4Q7tlJFfgelA3Al2hq1Chgrtqt2bNGouGWgIC579q1Sp3ZVZXGwP7/OuqfuXKlV2XqlC6ouhRnXTFVS0Wutro0TS9pivCp1vXuuIZuD9o/YXb3yIRuIy60qvuJGrlCrf+mjRpEtTydvHFF1u+fPlOW/fk0D4wcOBAtzzJyc51urro+FWrgbomlSxZ0l9O+4eu4p5OSra3WpICaT0Grhu1Jqo70jXXXBN0HKhbi1poUnoceNSSENoao33EG+egq+T79+93x7a6F4Xbxvr+U9fQwGUQbzm8cRPqknS67lvh9mHp06dP0HSvxTB0naqlQS0r4eh7K7B1R98j+v7T910gTVc3IS2zqJVL60HHYOA2UMrpihUrJtgGavlR61JqHF86tvRZjRs3dutTz0WtLPrOVAu4WosCf5/0ve61ZJ7puiM+EVgAKaQTDwUQCirUHUHN6Xroh0hdHtSlyaPuQ+qvmhSV0UmiTv5Si+alrjqh1OVAfa7V1cHrz60fK/F+rLwTgdPVWydJ6uoROLZE/1fAoJOwpGiMg37QQoX2Cd6yZYv7q77GqmvgQ2kpdQKXEhpToi4O6s6lH1EFdpqnghhvPaRUaLckLWu4ZfPWofe6RyejoSd5OqnU9gwNNDX9dCdpmr+69IR2EQpXn0ioy5O2teqr/Ul1VveNcOsvNEAVnYRGeoIZSl34dEKr1KRKopAcp6uL9intH+H24dPt16m1vUPXjY4DrVf1tw89DtTdMqXHgSexrnTqeqTAS3VUNzF9nk7ik7ONvSDDWw4dZ+rKqPFSuijjdfVTt53T0TrTSXTo+teJsYLr0HWaVNfA0HrqGBJ9F4RO18m4t6zaBgpA9L0Vug3UHSt0G2gMRzQD0NW9TUGwLihoGfU53liHwPWvQEn7q9cdTN3AND4iMMg+03VHfGKMBZBC6rv6+++/u+BCj1A6udbAudSUWMtF6ABPj37E9UMcWlZXPHXl8aGHHnInOfrR0sBRBRspyV6kHzWNCdEJnlov1DdZ/b5Ti1cn9b3WSUSolAZjurqtfuUarKkxCjqJ0DpWn+JosziFGycSCV0pjmT6//XwSR1aB+HmF7qfKQmA+n7r5FD3clHgoqvAWqfhBpSnRd01rkP97bWva0BucveFM7EeI5FYfQJpn1RQkViCiOSO/Yhkn9VAX30vqOVGA+v1+aqrxih4SSUiXa863lq0aOEGz2tcgPr8a376Tq1Vq9Zp65ncFtykjsGUHl/aBvp8BUbhyoYG7tF8D2j9Xn311e47WgGzgh6d6KvlRmOQAr+jNKhbLVfaXvo+1l+VDWzdPJN1R/wisABSSD/u+pH1Mq0EUpOzrhxpYK2+nNXlQgOwk6IyGoioLjmJDcD0rv6F5vgPvVKXFGUU0gBZXYXUD1Bgd6NAuuOtnK7eohNxdU/QoFtdNVP9dQX5dJTRyGuNCKSrbYG8Lita37p6F6nETkTUfUtdljRI1qPBq2lxczYtq7dsankJpGne62lF81crmq5sB55AhK5rbz8L1z0pdD9T9z1dxdbJoU7sPQosUirSbn8awKzMVuq2l5qDS7WvadnC3ZcmOfeqSYvtreNA3bPUMpOSk76UdKnUMaLvAn2nBb4/cEBwSmhZ1IVJD30HaBC4jkOdECdG60wnxyqv5AMetRDrmE3rY8irt4IMtYYoe19a0kBtXahRF6fAFpbEurzp+1zfw7rgpcBe3e4Cu6WdybojftEVCkgBnTzrh1YZOHS1NPSh7D3q66ofBK/v8vr168OmZfWuhKmM+ruGu9LvldEPp6406SQqkK4WJ5d3pSrwCqL+r2w1oVc/dSV6+vTprutUuPp41IVI/c51UqCAS5l0vIw+SVEaXLVuKPuLR2M7Qq/Iqp+0+r/rRnfhUirqPUlRi0y4YEHrInRZlCElsRagaKhPuk5WFWwGpq/U1UN1QwjNoJTatK7VTzwwVaWWU8sbSicgSjMcuF61/6pbRuj608lm4PpSVqxobkaobSXJCe4UwCiFqIL7wHE6qUHLpiBWy6JsSoFBhbZZemxvXX3WulbGsVDatqdbZ5Gs26S+L3QB5HRppBOjbG4K3kP3N2XmCpfWNZB3kzylxA3kdX9L62NIlOFK60RZwkK/O/Rc47RSS7h1r+5PiQXuGoOm41Gtx7owEDrG7UzWHfGLFgsgBbxBcuoGEo76nHs3y9OVe3Uh0JU/peHU4EA1WasrkuajEw8N7NbVJqUM1BUnnWhr0KMGVusKpQaAKlWouupoHjoZ1A+IfpDVxz2SvtVqVtf7lAZV3Z90wq4rz+H6uSsFp9K5Kj2uUlDqSpdOHNW/WleJA6n+Cqok3IlPOP3793fdmxSI6MfQSzerAEqpbj2qo06I1V9YdVELidavAh7VRVdwk+p6pfWt9agTEA3E1XJoLIwCQ32+1qu6EuhkSeW8dKOpSa046luuwZAaz6KTAC/9qFJsKnVtWlLXE60npdfUNtTyKjgO109e+6jWlQK6rl27uv1L+6nSXnr34/BO5FRO2++///2vK6eTfPWBD9x+kfDuXq6UutrOWm+qu3dS7FEQruNCy6HWktAr3UqUEPqeSCnlqMbwaL1p0LtO6rWfadxR6P5/Jra35qPUqeo2pM9XV0t9jq7ga2C35u0dg0mtW7XyaNvqJFPrOCk6RrSfaH1qe2s8mfYFrXe1fkVKraXq3qMgSfNQ1zVdcNG6OV1d9D2pFkZ9Ryg40vrQd6VaX9VVy0uikZb03al77ShRgI4jfa6CIq0XLYe+J/Xdmhq0fdWdSfu/trvWt+48r4BVrRKh9J2oY1H7gsZjhAZaZ7LuiGP+/FAAkk1pKHPmzOk7fPhwomU6d+7sy5Ytm2/fvn3u+R9//OG79957faVKlXKpLpUeUCkVvde91IKPPPKIr1y5cu69JUqU8N18882+n376yV9m7969vptuusmlwSxYsKCvR48evo0bN4ZNNxuartHz3Xff+Zo0aeI755xzfEWKFHHpWr10m4HzEM27TZs2Lt2rlvnCCy/0DR48OME8lZJS9VH618D0mqezYcMGl9pQ89a6GT58uO/FF18Mm3JUqSKValGfofLly5d361lpf5NK2aiUrEqdq3Stes1LWarUl126dHHrQOtC81bZ06U1TU662blz54Z9z5w5c1waUaXoLVSokO/WW2/1/frrr0FlEtt2Wk8XXXRRsuoRjvbB22+/3aUV1TrU/9euXRt2u7/66qsupaX21Zo1a/o++uijsOlmta0qVqzolqdy5cpuPomleFW603B1D13X2ge0L2TOnDloPwgs66U3TezhvSfauixevNhtL60H7W/Tpk3z9e3b1+1/yRHN9g5Xd5k6dapLQ6v9Wemgq1ev7uvfv79v586dSdZFqWPvu+8+X9GiRV0aUm/e3roMlwJWqaWV9ljrRsugZVHK3NB9Ial5BKZD1ved1r32FS2z9sP69ev73njjjWStk+PHj/uGDRvm/44sXbq0S3l95MiRZB0TiR2f2m81/euvvw67DfS9G+itt97yNWrUyNVRDy2Plmvz5s2nPV4TE257v/vuu76LL77Y7W9ly5b1Pfnkk/7U5OFSMms96rXu3bsn+jlpUXfAk0n/pHdwAyDjU1cMtQbo6ppuoIWMQVcu1YKj7hWBd7VG4nSlVzd4Czc+CEhP77zzjts/1V3WS/ULnEmMsQCQKtQXXX3yAweEA7EwniqQggll5dHd2YGzjbpKabC9urAC6YExFgCiooGc6k+vcRVKFendDwOIBTpJU0uO/iorlsb6qN+7xgcBZwulPNf3sMacaaxNNDdVBaJBYAEgKjrR0sBZpYt86aWX0rs6QKrSYFilUdYN3DRIXPc7UXaycDd2BNKLkgMojbSSLSipAZBeGGMBAAAAIGqMsQAAAAAQNQILAAAAAFFjjEUKnTp1yt2NVTeXYZAUAAAAYpFGTeimwEopnzlz0m0SBBYppKCidOnS6V0NAAAAIM3t2LHDzjvvvLM/sJg8ebI99dRTLutGjRo1bOLEiVavXr1Ey+t29YMHD3Y3dlJmjieffNKuv/76oMhq6NChLp/zgQMH7D//+Y/LXBMui8fRo0etfv36tn79elu7dq3LbJMcaqnwVnK+fPlStNwAAADA2ezQoUPuYrp37ntWBxZz5syxPn362JQpU9wJ/rhx46xp06a2efNmK1asWILyy5Ytc2nVRo0aZTfccIPNmjXL3WVyzZo1Vq1aNVdm9OjRNmHCBHv55ZfdHWUVhGie3333neXMmTNofspFrqYdBRaR8Lo/KaggsAAAAEAsS07X/3RPN6tg4pJLLrFJkyb5xy4oKrrvvvtswIABCcq3a9fODh8+bO+//75/WoMGDVxLg4ITLY4Chb59+9qDDz7oXj948KAVL17c5dhv3769/30ffvihC2reeustu+iiiyJqsVD0lj9/fjdvAgsAAADEokjOedM1K9SxY8ds9erV1qRJk/+vUObM7vny5cvDvkfTA8uLWiO88lu3bnVdqgLLaGUogAmc5+7du61bt272yiuvWO7cudNg6QAAAID4ka6Bxb59++zkyZOuNSGQnis4CEfTkyrv/U2qjFo1OnfubHfddZfVrVs3WXXVWAxFbIEPAAAAAHF8HwsNDlfarIEDByb7PRrToZYP70FGKAAAAOAsCSyKFCliWbJkcd2SAul5iRIlwr5H05Mq7/1NqsySJUtct6gcOXJY1qxZrUKFCm66Wi86deoU9nMVhKhvmfdQNigAAAAAZ0FgkT17dqtTp44tXrzYP02Dt/W8YcOGYd+j6YHlZdGiRf7yygKlACKwjLotrVixwl9GGaOUBWrdunXusWDBAn+GqpEjR4b9XAUhXgYoMkEBAAAAZ1m6WWVlUiuBWgt07wqlm1XWpy5durjXO3bsaKVKlXJdkaRXr17WuHFjGzNmjDVv3txmz55tq1atsqlTp/pTYfXu3dtGjBjh7lvhpZtVpiilpZXzzz8/qA7nnHOO+1u+fPnT3vgDAAAAwFkYWCh97N69e23IkCFucLXSvS5cuNA/+Hr79u1Btw+/9NJL3b0rBg0aZA8//LALHubPn++/h4V3bwoFJ927d3c3yGvUqJGbZ+g9LDKisgM+sHiw7YnmFi/YprGHbQqc3eLlGBWOU5xJ6X4fi4wqve5jES9fhvH0Rcg2jT1sU+DsFi/HqHCcIm7uYwEAAAAgNhBYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIgagQUAAACAqBFYAAAAAIiNwGLy5MlWtmxZy5kzp9WvX99WrlyZZPm5c+da5cqVXfnq1avbggULgl73+Xw2ZMgQO/fccy1XrlzWpEkT27JlS1CZli1b2vnnn+/moXK333677dy5M02WDwAAAIh16R5YzJkzx/r06WNDhw61NWvWWI0aNaxp06a2Z8+esOWXLVtmHTp0sK5du9ratWutdevW7rFx40Z/mdGjR9uECRNsypQptmLFCsuTJ4+b55EjR/xlrrzySnvjjTds8+bN9tZbb9lPP/1kN9988xlZZgAAACDWpHtgMXbsWOvWrZt16dLFqlat6oKB3Llz2/Tp08OWHz9+vDVr1sz69etnVapUseHDh1vt2rVt0qRJ/taKcePG2aBBg6xVq1Z28cUX28yZM11rxPz58/3zeeCBB6xBgwZWpkwZu/TSS23AgAH21Vdf2fHjx8/YsgMAAACxIl0Di2PHjtnq1atdVyV/hTJnds+XL18e9j2aHlhe1Brhld+6davt2rUrqEz+/PldF6vE5rl//3577bXXXICRLVu2VFo6AAAAIH5EHFicPHnSnn76aatXr56VKFHCChUqFPSIxL59+9z8ihcvHjRdzxUchKPpSZX3/iZnng899JDrJlW4cGHbvn27vfPOO4nW9ejRo3bo0KGgBwAAAIAUBhbDhg1z3ZfatWtnBw8edOMjbrzxRtfS8Oijj1pGou5UGqfx8ccfW5YsWaxjx46uK1U4o0aNci0f3qN06dJnvL4AAABAzAQW6jL0wgsvWN++fS1r1qxuIPW0adNcFiaNUYhEkSJF3An97t27g6bruVpDwtH0pMp7f5MzT31+pUqV7JprrrHZs2e77FKJLcPAgQNdIOU9duzYEdGyAgAAALEs4sBC3YmU4lXOOeccd5ItN9xwg33wwQcRzSt79uxWp04dW7x4sX/aqVOn3POGDRuGfY+mB5aXRYsW+cuXK1fOBRCBZdRtSdmhEpun97lel6dwcuTIYfny5Qt6AAAAAPg/WS1C5513nv3+++/uHhDly5d33YiUlenrr792J9+RUleqTp06Wd26dd24DWV0Onz4sMsSJeqeVKpUKdcVSXr16mWNGze2MWPGWPPmzV1Lw6pVq2zq1Knu9UyZMlnv3r1txIgRVrFiRRdoDB482EqWLOnS0oqCDNW3UaNGVrBgQZdqVmW0PEkFHwAAAABSKbBo06aNaw1QlqX77rvPbrvtNnvxxRfd4GelcI2Uxmrs3bvXdaVSa0jNmjVt4cKF/sHXmq/Gb3iUuWnWrFkunezDDz/sggelka1WrZq/TP/+/V1w0r17dztw4IALIDRP3QxPlM523rx57t4ZKqcb5CmFreaZkuAIAAAAiHeZfImNVk4mjUnQTet0gt+iRQuLF+pepUHc6gp2JrtFlR0QWXezjGrbE80tXrBNYw/bFDi7xcsxKhynOJPnvBG3WCxdutS1Gmjgtugmc3qcOHHCvXb55ZenvOYAAAAA4mPw9pVXXuluKBdKUYxeAwAAABB/Ig4s1HNKA6RD/fHHH+5mcwAAAADiT7K7QukmeKKgonPnzkGDnHX37A0bNrguUgAAAADiT7IDCw3a8Fos8ubNa7ly5Qq6H4XGWXTr1i1tagkAAAAgNgKLGTNmuL9ly5a1Bx98kG5PAAAAAFKeFUr3fgAAAACAiAOLWrVqhR2wHc6aNWuSVQ4AAABAnAUWrVu3TvuaAAAAAIjtwILuTwAAAABS9T4WcuDAAZs2bZoNHDjQf7M8dYH67bffUjI7AAAAAPE2eFv3q2jSpIlLP7tt2zaXYrZQoUI2b9482759u82cOTNtagoAAAAgdlos+vTp426Qt2XLFsuZM6d/+vXXX29Lly5N7foBAAAAiMXA4uuvv7YePXokmF6qVCnbtWtXatULAAAAQCwHFjly5LBDhw4lmP7DDz9Y0aJFU6teAAAAAGI5sGjZsqU99thjdvz4cfdc97fQ2IqHHnrIbrrpprSoIwAAAIBYCyzGjBljf//9txUrVsz+/fdfa9y4sVWoUMHy5s1rI0eOTJtaAgAAAIitrFDKBrVo0SL74osvXIYoBRm1a9d2maIAAAAAxKeIAwtPo0aN3AMAAAAAkhVYTJgwIdkzvP/++6OpDwAAAIBYDSyeeeaZoOd79+61f/75xwoUKOC/E3fu3LnduAsCCwAAACD+JGvw9tatW/0PDdCuWbOmff/997Z//3730P81zmL48OFpX2MAAAAAGT8r1ODBg23ixIl24YUX+qfp/2rVGDRoUGrXDwAAAEAsBha///67nThxIsH0kydP2u7du1OrXgAAAABiObC4+uqrrUePHrZmzRr/tNWrV1vPnj1JOQsAAADEqYgDi+nTp1uJEiWsbt26liNHDveoV6+eFS9e3KZNm5Y2tQQAAAAQW/exKFq0qC1YsMC2bNniBm1L5cqVrVKlSmlRPwAAAACxfIO8ihUrugcAAAAARNwVCgAAAABCEVgAAAAAiBqBBQAAAICoEVgAAAAAOPOBRdmyZe2xxx6z7du3R//pAAAAAOIzsOjdu7fNmzfPLrjgArvmmmts9uzZdvTo0bSpHQAAAIDYDSzWrVtnK1eutCpVqth9991n5557rt17771Bd+MGAAAAED9SPMaidu3aNmHCBNu5c6cNHTrU3XX7kksusZo1a7q7c/t8vtStKQAAAIDYu0He8ePH7e2337YZM2bYokWLrEGDBta1a1f79ddf7eGHH7ZPPvnEZs2albq1BQAAABAbgYW6OymYeP311y1z5szWsWNHe+aZZ6xy5cr+Mm3atHGtFwAAAADiQ8SBhQIGDdp+7rnnrHXr1pYtW7YEZcqVK2ft27dPrToCAAAAiLXA4ueff7YyZcokWSZPnjyuVQMAAABAfIh48PaePXtsxYoVCaZr2qpVq1JUicmTJ7v7Y+TMmdPq16/vMk4lZe7cua7rlcpXr17dFixYEPS6Bo4PGTLEZavKlSuXNWnSxLZs2eJ/fdu2bW48iFpW9Hr58uXdAPRjx46lqP4AAABAvIs4sLjnnntsx44dCab/9ttv7rVIzZkzx/r06eNO7DV+o0aNGta0aVMXwISzbNky69ChgwsM1q5d67pj6bFx40Z/mdGjR7uMVVOmTHEBj1pQNM8jR4641zdt2mSnTp2y559/3r799ls3RkRlNegcAAAAwBkILL777juXajZUrVq13GuRGjt2rHXr1s26dOliVatWdSf4uXPndilrwxk/frw1a9bM+vXr5+6jMXz4cFefSZMm+Vsrxo0bZ4MGDbJWrVrZxRdfbDNnznRpcefPn+/K6P3qqnXttde6G/21bNnSHnzwQXfjPwAAAABnILDIkSOH7d69O8H033//3bJmjWzIhroerV692nVV8lcoc2b3fPny5WHfo+mB5UWtEV75rVu32q5du4LK5M+f33WxSmyecvDgQStUqFBE9QcAAACQwsBCV/kHDhzoTsQ9Bw4ccN2IlC0qEvv27bOTJ09a8eLFg6bruYKDcDQ9qfLe30jm+eOPP9rEiROtR48eidb16NGjdujQoaAHAAAAgBRmhXr66aft8ssvd5mh1P1J1q1b507cX3nlFctoNDZEXaPatm3rumQlZtSoUTZs2LAzWjcAAAAgZlssSpUqZRs2bHADpDUmok6dOm7cwzfffGOlS5eOaF5FihSxLFmyJOhapeclSpQI+x5NT6q89zc589S4iyuvvNIuvfRSmzp1apJ19VppvEe4AewAAABAvIq4xUKUZal79+5Rf3j27NldYLJ48WKX2UmUrUnP77333rDvadiwoXu9d+/e/mmLFi1y00UpZBVAqEzNmjXdNHVbUnaonj17BrVUKKjQ52sgt8Z2nG5siR4AAAAAUimwEGWA2r59e4J7PyjDUiSUarZTp05Wt25dq1evnsvodPjwYZclSjp27OhaSdQVSXr16mWNGze2MWPGWPPmzW327Nnu/hlei0OmTJlc0DFixAirWLGiCzQGDx5sJUuW9AcvCiquuOIK151LXbv27t3rr09iLSUAAAAAUvnO223atHFdn3QSr/Suov+LBmNHol27du7EXje00+BqtTIsXLjQP/hawUtga4K6Lc2aNculk9WAcQUPSiNbrVo1f5n+/fu74EStKhpY3qhRIzdP3VDPa+HQgG09zjvvvKD6eMsDAAAAIPky+SI8k27RooUbFzFt2jTXGqC7ZP/xxx/Wt29fd/X/sssus3ig7lVKY6vxFvny5Ttjn1t2wAcWD7Y90dziBds09rBNgbNbvByjwnGKM3nOG3GLhe4FsWTJEjfwWi0JeqhFQF2V7r//fnc3bAAAAADxJeKsUOrqlDdvXvd/BRfKrCQar7B58+bUryEAAACAs17ELRYay7B+/XrXDUp3s1baWWV30uDpCy64IG1qCQAAACC2AgsNmtbAaHnsscfshhtucOMqChcubHPmzEmLOgIAAACItcCiadOm/v9XqFDBNm3aZPv377eCBQv6M0MBAAAAiC8RjbE4fvy4Zc2a1TZu3Bg0vVChQgQVAAAAQByLKLDIli2bnX/++RHfqwIAAABAbIs4K9Qjjzzibkyn7k8AAAAAkKIxFpMmTXJ3rC5ZsqRLMZsnT56g19esWcOaBQAAAOJMxIFF69at06YmAAAAAOInsBg6dGja1AQAAABA/IyxAAAAAICoWywyZ86cZGpZMkYBAAAA8SfiwOLtt99OcG+LtWvX2ssvv2zDhg1LzboBAAAAiNXAolWrVgmm3XzzzXbRRRfZnDlzrGvXrqlVNwAAAADxNsaiQYMGtnjx4tSaHQAAAIB4Cyz+/fdfmzBhgpUqVSo1ZgcAAAAg1rtCFSxYMGjwts/ns7/++sty585tr776amrXDwAAAEAsBhbPPPNMUGChLFFFixa1+vXru6ADAAAAQPyJOLDo3Llz2tQEAAAAQPyMsZgxY4bNnTs3wXRNU8pZAAAAAPEn4sBi1KhRVqRIkQTTixUrZo8//nhq1QsAAABALAcW27dvt3LlyiWYXqZMGfcaAAAAgPgTcWChlokNGzYkmL5+/XorXLhwatULAAAAQCwHFh06dLD777/fPv30Uzt58qR7LFmyxHr16mXt27dPm1oCAAAAiK2sUMOHD7dt27bZ1VdfbVmz/t/bT506ZR07dmSMBQAAABCnIg4ssmfPbnPmzLERI0bYunXrLFeuXFa9enU3xgIAAABAfIo4sPBUrFjRPQAAAAAg4jEWN910kz355JMJpo8ePdratm2bWvUCAAAAEMuBxdKlS+36669PMP26665zrwEAAACIPxEHFn///bcbZxEqW7ZsdujQodSqFwAAAIBYDiw0UFuDt0PNnj3bqlatmlr1AgAAABDLg7cHDx5sN954o/3000921VVXuWmLFy+2119/3ebOnZsWdQQAAAAQa4FFixYtbP78+e6eFW+++aZLN3vxxRfbJ598Yo0bN06bWgIAAACIvXSzzZs3d49QGzdutGrVqqVGvQAAAADE8hiLUH/99ZdNnTrV6tWrZzVq1EidWgEAAACIj8BCqWU7duxo5557rj399NNuvMVXX32VurUDAAAAEHtdoXbt2mUvvfSSvfjiiy617C233GJHjx51Yy7ICAUAAADEr8yRDNq+8MILbcOGDTZu3DjbuXOnTZw4MW1rBwAAACC2Wiw+/PBDu//++61nz55WsWLFtK0VAAAAgNhssfjiiy/cQO06depY/fr1bdKkSbZv376oKzB58mQrW7as5cyZ08135cqVSZbXvTIqV67syutmfQsWLAh63efz2ZAhQ9zYD6XCbdKkiW3ZsiWozMiRI+3SSy+13LlzW4ECBaJeBgAAACDeJTuwaNCggb3wwgv2+++/W48ePdydtkuWLGmnTp2yRYsWuaAjUrqDd58+fWzo0KG2Zs0al1WqadOmtmfPnrDlly1bZh06dLCuXbva2rVrrXXr1u6hNLee0aNH24QJE2zKlCm2YsUKy5Mnj5vnkSNH/GWOHTtmbdu2da0vAAAAANIhK5RO1O+44w7XgvHNN99Y37597YknnrBixYpZy5YtI5rX2LFjrVu3btalSxc3+FvBgFoRpk+fHrb8+PHjrVmzZtavXz+rUqWKDR8+3GrXru1aT7zWCo3/GDRokLVq1crduG/mzJluPIgGmHuGDRtmDzzwgGvxAAAAAJDO97HQYG61EPz666/2+uuvR/RetRqsXr3adVXyVyZzZvd8+fLlYd+j6YHlRa0RXvmtW7e6zFWBZfLnz++6WCU2TwAAAADpdOftUFmyZPF3S0oujc84efKkFS9ePGi6nm/atCnsexQ0hCuv6d7r3rTEyqSU0urq4VG6XQAAAACpdOfteDFq1CjX+uE9Spcund5VAgAAAM4a6RZYFClSxLV07N69O2i6npcoUSLsezQ9qfLe30jmmVwDBw60gwcP+h87duyIan4AAABALEm3wCJ79uwude3ixYv905RhSs8bNmwY9j2aHlhelJHKK1+uXDkXQASWUZclZYdKbJ7JlSNHDsuXL1/QAwAAAEAqjrFIKaWa7dSpk9WtW9fq1avnMjodPnzYZYmSjh07WqlSpVw3JOnVq5c1btzYxowZY82bN3cpb1etWmVTp051r2fKlMl69+5tI0aMcDfxU6AxePBglxY3cPzH9u3bbf/+/e6vxnmsW7fOTa9QoYKdc8456bIuAAAAgIwsXQOLdu3a2d69e90N7TS4umbNmrZw4UL/4Gud+CtTlEc3tZs1a5ZLJ/vwww+74EFpZKtVq+Yv079/fxecdO/e3Q4cOGCNGjVy89QN9Tz6vJdfftn/vFatWu7vp59+aldcccUZWnoAAAAgdmTy6eYPiJi6WGkQt8ZbnMluUWUHfGDxYNsTzS1esE1jD9sUOLvFyzEqHKc4k+e8ZIUCAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAABRI7AAAAAAEDUCCwAAAACxEVhMnjzZypYtazlz5rT69evbypUrkyw/d+5cq1y5sitfvXp1W7BgQdDrPp/PhgwZYueee67lypXLmjRpYlu2bAkqs3//frv11lstX758VqBAAevatav9/fffabJ8AAAAQKxL98Bizpw51qdPHxs6dKitWbPGatSoYU2bNrU9e/aELb9s2TLr0KGDCwTWrl1rrVu3do+NGzf6y4wePdomTJhgU6ZMsRUrVliePHncPI8cOeIvo6Di22+/tUWLFtn7779vS5cute7du5+RZQYAAABiTboHFmPHjrVu3bpZly5drGrVqi4YyJ07t02fPj1s+fHjx1uzZs2sX79+VqVKFRs+fLjVrl3bJk2a5G+tGDdunA0aNMhatWplF198sc2cOdN27txp8+fPd2W+//57W7hwoU2bNs21kDRq1MgmTpxos2fPduUAAAAAZKDA4tixY7Z69WrXVclfocyZ3fPly5eHfY+mB5YXtUZ45bdu3Wq7du0KKpM/f34XQHhl9Ffdn+rWresvo/L6bLVwAAAAAIhMVktH+/bts5MnT1rx4sWDpuv5pk2bwr5HQUO48pruve5NS6pMsWLFgl7PmjWrFSpUyF8m1NGjR93Dc/DgQff3yy+/dF2tzpR/t/9/l69YtnRpXosXbNPYwzYFzm7xcowKxymidfjwYX+voLM6sMhIRo0aZcOGDUsw/frrr0+X+sS6xq+ndw2Q2timsYdtCpz9OE6RWv766y/XC+isDSyKFCliWbJksd27dwdN1/MSJUqEfY+mJ1Xe+6tpygoVWKZmzZr+MqGDw0+cOOEyRSX2uQMHDnSDzD2nTp1y5QsXLmyZMmWyWHXo0CErXbq07dixw2XQQsbHNo09bNPYwzaNPWzT2BMv29Tn87mgomTJkqctm66BRfbs2a1OnTq2ePFil9nJO2HX83vvvTfsexo2bOhe7927t3+aMjtpupQrV84FByrjBRLa8Bo70bNnT/88Dhw44MZ36PNlyZIl7rM1FiOcHDlyuEcgjdOIFzpgYvmgiUds09jDNo09bNPYwzaNPfGwTfOfpqXirOkKpVaATp06uYHU9erVcxmd1JdLWaKkY8eOVqpUKdcVSXr16mWNGze2MWPGWPPmzV0mp1WrVtnUqVPd62o9UNAxYsQIq1ixogs0Bg8e7KIsL3hRNillllI2KmWhOn78uAtk2rdvn6xoDAAAAMBZFli0a9fO9u7d625op4HTamVQKlhv8PX27dtdtibPpZdearNmzXLpZB9++GEXPCiNbLVq1fxl+vfv74IT3ZdCLRNKJ6t56oZ6ntdee80FE1dffbWb/0033eTufQEAAAAgcpl8yRnijbilTFhqLdIYk9CuYMiY2Kaxh20ae9imsYdtGnvYpgkRWAAAAADI+HfeBgAAAJDxEVgAAAAAiBqBBQAAAICoEVgAAACkMYa0Ih6ke7pZnF327dtn06dPt+XLl7v0v6IbDirNb+fOna1o0aLpXUUAADIcZQ1av369u5cWEKvICgW/r7/+2po2bWq5c+e2Jk2a+O8lsnv3bncn83/++cc++ugjdzNDxI4dO3bY0KFDXUCJjOHff/+11atXW6FChaxq1apBrx05csTeeOMNd3NRZBzff/+9ffXVV9awYUOrXLmybdq0ycaPH+/SWd5222121VVXpXcVEcGNf8PR9tS2LFy4sHs+duzYM1wzpCbdL03ftT/++KOde+651qFDB/+2jWcEFvBr0KCB1ahRw92NXHcwD6Td5K677rINGza41gzEDl1Bq127tp08eTK9q4Jk+OGHH+zaa691Nw/VcaobgM6ePdv9sHkXAkqWLMn2zEB0A9dWrVrZOeec4y7gvP322y4w1PfxqVOn7H/+53/s448/JrjIIHTTXW27AgUKBE3XdtSFuTx58rhjd8mSJelWR0ROF3G++OILd0FHF+Quv/xy+/PPP61SpUr2008/WdasWd3FgXLlylk8I7CAX65cuWzt2rXualk4uoJWq1Ytd7UUGce7776b5Os///yz9e3blxPRDKJNmzZ2/Phxe+mll+zAgQPWu3dv++677+yzzz6z888/n8AiA1JXUwUNI0aMcEHi3XffbT179rSRI0e613XzLbVQKbjA2e+JJ56wqVOn2rRp04KCwWzZsrkLOaGtjMg4AaO6iBcrVsy1PG3dutUWLFhg+fPnt7///tt9NxctWtRmzZpl8YzAAn6KsocNG5ZoF4qZM2fakCFDbNu2bWe8bojuy1BXx5I61PU6J6IZg7oofvLJJ1a9enX3XNtVJ6L6gfv000/d1VACi4xFJyYKHCpUqOBaKNQXf+XKle5CjmzcuNF1T/XGvSFjdC3WyWeLFi3cnZkVVBBYxE5gUb58ede745prrvG/vmzZMmvfvr1rTY5nZIWC34MPPmjdu3e3Xr16uavcK1ascA/9X9PUFap///7pXU1ESF1k5s2b505Ywj3WrFmT3lVEBNRiqCb3wKDwueeecycwjRs3dl2lkPF43U918pIzZ04XbHjy5s1rBw8eTMfaIVKXXHKJCxb37t3ruj8pOAztYoyMx9uGGsvmdT/1lCpVym3veEdWKPjdc889VqRIEXvmmWfs2Wef9V/xzJIli9WpU8d1vbjlllvSu5qIkLadfuDUhzuc07Vm4OyiroqrVq1KkFlm0qRJ7m/Lli3TqWZIqbJly9qWLVvcVVDRODZ1a/PoCmjoSQzOfhoz8/LLL7vubWpxohUx47v66qvdhZ1Dhw7Z5s2brVq1av7XfvnlFwZvE1ggVLt27dxDfbiVelYUbKgJFxlTv379XPaKxKj7hbrQIGNQP97XX3/dbr/99gSvKbhQK5Sa6JFxaDxF4Eln4MmKfPjhhwzczsDUPUZJFnSBp0yZMuldHaSQsieGBo6B3nvvPbvsssss3jHGAgAAAEDUGGMBAAAAIGoEFgAAAACiRmABAAAAIGoEFgAAAACiRmABAMiwHn30UatZs2aqpHwdN25cqtQJAOIVgQUAIEjnzp2tdevWllFu7Ll48eL0rgYAgPtYAAAygmPHjln27NkTTFcu+dB88gCA9EGLBQAgImPHjrXq1atbnjx5rHTp0nb33Xfb33//7V7TzRjz5ctnb775ZtB75s+f78r/9ddf7vmOHTvslltusQIFClihQoXcneG3bduWoNVk5MiRVrJkSbvwwguT1RXKe9/TTz/t7latO+Hec8897qafnj179liLFi0sV65cVq5cOXvttdcSzPfAgQN25513WtGiRd3y6AZ169evd6/t3bvXSpQoYY8//ri//LJly1zgQ+sJgHhGYAEAiEjmzJltwoQJ9u2339rLL79sS5Yssf79+7vXFDzoTsMzZswIeo+e33zzzZY3b153kt+0aVP3/88//9y+/PJL1+rQrFkz1zLh0Un65s2bbdGiRfb+++8nu366k/xPP/3k/qp+L730knsEBh8KbPS6AqBnn33WBRuB2rZt66bprte6Y3Lt2rXt6quvtv3797tgY/r06S6oWbVqlQuWdCf0e++915UBgLilO28DAODp1KmTr1WrVskuP3fuXF/hwoX9z1esWOHLkiWLb+fOne757t27fVmzZvV99tln7vkrr7ziu/DCC32nTp3yv+fo0aO+XLly+T766CN/HYoXL+6mJ2Xo0KG+GjVqBNW9TJkyvhMnTvintW3b1teuXTv3/82bN/v007dy5Ur/699//72b9swzz7jnn3/+uS9fvny+I0eOBH1W+fLlfc8//7z/+d133+2rVKmS77///a+vevXqCcoDQLyhxQIAEJFPPvnEXZkvVaqUa3XQ1fo//vjD/vnnH/d6vXr17KKLLnKtBfLqq69amTJl7PLLL3fP1aXoxx9/dO/1xkioO9SRI0dcS4NH3a3Cjas4HX12lixZ/M/VJcprkfj+++8ta9asVqdOHf/rlStXdl2yPKqfunapG5VXPz22bt0aVD91tzpx4oTNnTvXdafKkSNHxHUFgFjC4G0AQLJpHMQNN9xgPXv2dOMfFBB88cUX1rVrV9eNKXfu3K6cxidMnjzZBgwY4LpBdenSxTJlyuRe00m7TuzDjW1QNyOPulWlRLZs2YKe63NPnTqV7PerfgpGPvvsswSvBQYgCjJ27tzp5q31okAIAOIZgQUAINk03kAn0mPGjHFjLeSNN95IUO62225z4y40FuO7776zTp06+V/TeIU5c+ZYsWLF3MDoM0mtE2pl0HJccsklbprGcWiwdmD9du3a5Vo2dH+LcBREaRnbtWvnBpYrkPrmm2/cMgFAvKIrFAAggYMHD9q6deuCHhrwXKFCBTf4euLEifbzzz/bK6+8YlOmTEnw/oIFC9qNN95o/fr1s2uvvdbOO+88/2u33nqrFSlSxGWC0uBtdTFS68D9999vv/76a5oul4IADRLv0aOHrVixwgUYCgqUIcrTpEkTa9iwocsu9fHHH7vWCGV9euSRR9xgbdH/tY4UOD300ENWqVIlu+OOO9K07gBwtiOwAAAkoBP9WrVqBT2GDRtmNWrUcOlmn3zySatWrZrrzjRq1Kiw8/C6R4WecKu71NKlS+388893wUeVKlVcWY2xOBMtGOqapRS2jRs3dp/fvXv3oJYGdZ1asGCBGxOiLlwKGpTp6pdffrHixYu7daO7dCuoUn3VcqP/K0h67rnn0rz+AHC2yqQR3OldCQBA7NHJ9gMPPODGIaRkEDYAIGNhjAUAIFUpO9Tvv/9uTzzxhOtyRFABAPGBrlAAgFQ1evRoN0had6ceOHBgelcHAHCG0BUKAAAAQNRosQAAAAAQNQILAAAAAFEjsAAAAAAQNQILAAAAAFEjsAAAAAAQNQILAAAAAFEjsAAAAAAQNQILAAAAAFEjsAAAAABg0fpfMD6Gv1MHRNYAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plot_df = layer_df[layer_df[\"experiment\"] != \"baseline fp32\"].copy()\n", "plot_df[\"layer\"] = plot_df[\"target_component\"].str.extract(r\"(\\d+)\").astype(int)\n", "\n", "ax = plot_df.sort_values(\"layer\").plot(\n", " x=\"layer\",\n", " y=\"accuracy_delta\",\n", " kind=\"bar\",\n", " legend=False,\n", " figsize=(8, 3),\n", ")\n", "ax.axhline(0, color=\"black\", linewidth=1)\n", "ax.set_title(\"Accuracy delta from quantizing one transformer layer\")\n", "ax.set_ylabel(\"Accuracy delta\")\n", "ax.set_xlabel(\"Layer index\")\n", "plt.tight_layout()\n", "plt.show()\n" ] }, { "cell_type": "markdown", "id": "0d8330ec", "metadata": {}, "source": [ "## 7. Fake Activation Quantization\n", "\n", "Weights are only part of inference. Activations matter too: they are the tensors produced while the model is running.\n", "\n", "This section uses fake quantization because arbitrary activation tensors inside this model do not automatically run through real int8 CPU kernels. The hook does something smaller and easier to inspect:\n", "\n", "1. take a floating-point module output\n", "2. round it to an int8 or int4 grid\n", "3. convert it back to floating point\n", "4. pass that rounded tensor to the next module\n", "\n", "This does not make the model faster. It asks whether accuracy changes when a specific activation is rounded.\n", "\n", "The targets below are module outputs:\n", "\n", "| Target | What the hook changes |\n", "|---|---|\n", "| embeddings output | Rounds the token and position vectors returned by `distilbert.embeddings`, before the first encoder block sees them. |\n", "| attention module outputs | Rounds the output returned by each block's `attention` module, after self-attention has mixed token information. |\n", "| FFN outputs | Rounds the output returned by each block's `ffn` module, after the feed-forward network transforms each token representation. |\n", "| classifier input | Rounds the tensor returned by `pre_classifier`, right before the final sentiment classifier. |\n", "\n", "I also tried lower-level attention projection outputs while exploring. Those are useful when tuning a real inference stack, but they are too much detail for this pass. The cleaner module-level targets are enough to show which part of the model looks fragile when rounded." ] }, { "cell_type": "code", "execution_count": 9, "id": "3a56450b", "metadata": { "execution": { "iopub.execute_input": "2026-07-08T06:02:37.564821Z", "iopub.status.busy": "2026-07-08T06:02:37.564721Z", "iopub.status.idle": "2026-07-08T06:02:44.837506Z", "shell.execute_reply": "2026-07-08T06:02:44.837252Z" } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "21ac0f831b824817b98b33c91579b5d5", "version_major": 2, "version_minor": 0 }, "text/plain": [ "fake int8 embeddings output: 0%| | 0/16 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
experimentquantization_typetarget_componentbit_widthaccuracyaccuracy_deltaeval_secondsspeedupmodel_size_mbsize_reductionexamples_per_secondavg_batch_latency_msmemory_delta_mb
0fake int4 FFN outputsfakeFFN outputsint4 simulated0.628906-0.2382810.5523710.993004255.4520640.000000463.45701133.0808850.000000
1fake int8 FFN outputsfakeFFN outputsint8 simulated0.855469-0.0117190.7840220.699606255.4520640.000000326.52150247.5171330.046875
2fake int4 attention module outputsfakeattention module outputsint4 simulated0.828125-0.0390620.5177011.059504255.4520640.000000494.49384631.069675-0.078125
3fake int8 attention module outputsfakeattention module outputsint8 simulated0.8671880.0000000.7739100.708747255.4520640.000000330.78801446.723597168.250000
4fake int4 classifier inputfakeclassifier inputint4 simulated0.8671880.0000000.5030701.090318255.4520640.000000508.87584130.170487168.203125
5fake int8 classifier inputfakeclassifier inputint8 simulated0.8671880.0000000.4950071.108078255.4520640.000000517.16444829.6708280.000000
6fake int4 embeddings outputfakeembeddings outputint4 simulated0.8867190.0195310.5195511.055730255.4520640.000000492.73275331.094424152.718750
7fake int8 embeddings outputfakeembeddings outputint8 simulated0.8671880.0000000.7040350.779089255.4520640.000000363.61807142.4403650.296875
8baseline fp32noneentire modelfp320.8671880.0000000.5485061.000000255.4520640.000000466.72222732.822719367.250000
9dynamic int8 all Linearreal dynamicall Linear layersint80.8710940.0039061.4004950.391651132.2883290.482140182.79245385.96757348.203125
10dynamic int8 attention modulesreal dynamicattention modulesint80.863281-0.0039060.8702060.630318214.9701600.158472294.18334352.91028988.390625
11dynamic int8 classifierreal dynamicclassifierint80.8671880.0000000.5042971.087666255.4520640.000000507.63778430.233432-9.875000
12dynamic int8 FFNreal dynamictransformer FFNint80.8710940.0039061.1366260.482574174.4609590.317050225.22800869.59305552.765625
13dynamic int8 transformer layer 0real dynamictransformer layer 0int80.8710940.0039060.6874960.797832235.2066640.079253372.36607441.50758647.187500
14dynamic int8 transformer layer 1real dynamictransformer layer 1int80.8710940.0039060.6751310.812444235.2066640.079253379.18585740.86360718.421875
15dynamic int8 transformer layer 2real dynamictransformer layer 2int80.8710940.0039060.6738530.813985235.2067250.079253379.90484040.73848758.500000
16dynamic int8 transformer layer 3real dynamictransformer layer 3int80.8671880.0000000.8880570.617648235.2067250.079253288.26993353.96397469.453125
17dynamic int8 transformer layer 4real dynamictransformer layer 4int80.8710940.0039060.8539600.642309235.2067250.079253299.77982051.7779191.750000
18dynamic int8 transformer layer 5real dynamictransformer layer 5int80.8671880.0000000.8903710.616043235.2067250.079253287.52074354.0848360.031250
\n", "" ], "text/plain": [ " experiment quantization_type \\\n", "0 fake int4 FFN outputs fake \n", "1 fake int8 FFN outputs fake \n", "2 fake int4 attention module outputs fake \n", "3 fake int8 attention module outputs fake \n", "4 fake int4 classifier input fake \n", "5 fake int8 classifier input fake \n", "6 fake int4 embeddings output fake \n", "7 fake int8 embeddings output fake \n", "8 baseline fp32 none \n", "9 dynamic int8 all Linear real dynamic \n", "10 dynamic int8 attention modules real dynamic \n", "11 dynamic int8 classifier real dynamic \n", "12 dynamic int8 FFN real dynamic \n", "13 dynamic int8 transformer layer 0 real dynamic \n", "14 dynamic int8 transformer layer 1 real dynamic \n", "15 dynamic int8 transformer layer 2 real dynamic \n", "16 dynamic int8 transformer layer 3 real dynamic \n", "17 dynamic int8 transformer layer 4 real dynamic \n", "18 dynamic int8 transformer layer 5 real dynamic \n", "\n", " target_component bit_width accuracy accuracy_delta \\\n", "0 FFN outputs int4 simulated 0.628906 -0.238281 \n", "1 FFN outputs int8 simulated 0.855469 -0.011719 \n", "2 attention module outputs int4 simulated 0.828125 -0.039062 \n", "3 attention module outputs int8 simulated 0.867188 0.000000 \n", "4 classifier input int4 simulated 0.867188 0.000000 \n", "5 classifier input int8 simulated 0.867188 0.000000 \n", "6 embeddings output int4 simulated 0.886719 0.019531 \n", "7 embeddings output int8 simulated 0.867188 0.000000 \n", "8 entire model fp32 0.867188 0.000000 \n", "9 all Linear layers int8 0.871094 0.003906 \n", "10 attention modules int8 0.863281 -0.003906 \n", "11 classifier int8 0.867188 0.000000 \n", "12 transformer FFN int8 0.871094 0.003906 \n", "13 transformer layer 0 int8 0.871094 0.003906 \n", "14 transformer layer 1 int8 0.871094 0.003906 \n", "15 transformer layer 2 int8 0.871094 0.003906 \n", "16 transformer layer 3 int8 0.867188 0.000000 \n", "17 transformer layer 4 int8 0.871094 0.003906 \n", "18 transformer layer 5 int8 0.867188 0.000000 \n", "\n", " eval_seconds speedup model_size_mb size_reduction \\\n", "0 0.552371 0.993004 255.452064 0.000000 \n", "1 0.784022 0.699606 255.452064 0.000000 \n", "2 0.517701 1.059504 255.452064 0.000000 \n", "3 0.773910 0.708747 255.452064 0.000000 \n", "4 0.503070 1.090318 255.452064 0.000000 \n", "5 0.495007 1.108078 255.452064 0.000000 \n", "6 0.519551 1.055730 255.452064 0.000000 \n", "7 0.704035 0.779089 255.452064 0.000000 \n", "8 0.548506 1.000000 255.452064 0.000000 \n", "9 1.400495 0.391651 132.288329 0.482140 \n", "10 0.870206 0.630318 214.970160 0.158472 \n", "11 0.504297 1.087666 255.452064 0.000000 \n", "12 1.136626 0.482574 174.460959 0.317050 \n", "13 0.687496 0.797832 235.206664 0.079253 \n", "14 0.675131 0.812444 235.206664 0.079253 \n", "15 0.673853 0.813985 235.206725 0.079253 \n", "16 0.888057 0.617648 235.206725 0.079253 \n", "17 0.853960 0.642309 235.206725 0.079253 \n", "18 0.890371 0.616043 235.206725 0.079253 \n", "\n", " examples_per_second avg_batch_latency_ms memory_delta_mb \n", "0 463.457011 33.080885 0.000000 \n", "1 326.521502 47.517133 0.046875 \n", "2 494.493846 31.069675 -0.078125 \n", "3 330.788014 46.723597 168.250000 \n", "4 508.875841 30.170487 168.203125 \n", "5 517.164448 29.670828 0.000000 \n", "6 492.732753 31.094424 152.718750 \n", "7 363.618071 42.440365 0.296875 \n", "8 466.722227 32.822719 367.250000 \n", "9 182.792453 85.967573 48.203125 \n", "10 294.183343 52.910289 88.390625 \n", "11 507.637784 30.233432 -9.875000 \n", "12 225.228008 69.593055 52.765625 \n", "13 372.366074 41.507586 47.187500 \n", "14 379.185857 40.863607 18.421875 \n", "15 379.904840 40.738487 58.500000 \n", "16 288.269933 53.963974 69.453125 \n", "17 299.779820 51.777919 1.750000 \n", "18 287.520743 54.084836 0.031250 " ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def fake_quantize_tensor(x: torch.Tensor, bits: int = 8, symmetric: bool = True) -> torch.Tensor:\n", " \"\"\"Round a float tensor to a low-precision grid, then return it as float.\"\"\"\n", " if not torch.is_floating_point(x):\n", " return x\n", "\n", " if symmetric:\n", " # int8 uses values from -127 to 127. int4 uses -7 to 7.\n", " qmax = (2 ** (bits - 1)) - 1\n", "\n", " # Use one scale for the whole tensor. The largest absolute value maps to qmax.\n", " scale = x.detach().abs().amax().clamp(min=1e-8) / qmax\n", "\n", " # Convert float values to integer buckets, then clip to the valid range.\n", " quantized = torch.round(x / scale).clamp(-qmax, qmax)\n", "\n", " # Convert back to float so the normal model can keep running.\n", " return quantized * scale\n", "\n", " qmin, qmax = 0, (2 ** bits) - 1\n", " xmin, xmax = x.detach().amin(), x.detach().amax()\n", "\n", " # Asymmetric mode also learns a zero point so the integer range can shift.\n", " scale = (xmax - xmin).clamp(min=1e-8) / (qmax - qmin)\n", " zero_point = torch.round(qmin - xmin / scale).clamp(qmin, qmax)\n", " quantized = torch.round(x / scale + zero_point).clamp(qmin, qmax)\n", " return (quantized - zero_point) * scale\n", "\n", "\n", "def add_output_fake_quant_hook(module: nn.Module, bits: int):\n", " \"\"\"Attach fake quantization to a module's output tensor.\"\"\"\n", " def hook(_module, _inputs, output):\n", " if isinstance(output, tuple):\n", " return tuple(fake_quantize_tensor(item, bits) if torch.is_tensor(item) else item for item in output)\n", " return fake_quantize_tensor(output, bits) if torch.is_tensor(output) else output\n", "\n", " return module.register_forward_hook(hook)\n", "\n", "\n", "@contextmanager\n", "def fake_quant_hooks(model: nn.Module, module_paths: Iterable[str], bits: int):\n", " # Hooks let us test selected activation points without rewriting DistilBERT.\n", " handles = [add_output_fake_quant_hook(get_submodule(model, path), bits) for path in module_paths]\n", " try:\n", " yield\n", " finally:\n", " # Always remove hooks. Otherwise later experiments would reuse old fake quantization.\n", " for handle in handles:\n", " handle.remove()\n", "\n", "\n", "def evaluate_with_fake_quant(\n", " base_model: nn.Module,\n", " paths: list[str],\n", " bits: int,\n", " target: str,\n", " name: str,\n", ") -> dict:\n", " model = fresh_model_copy(base_model)\n", " with fake_quant_hooks(model, paths, bits):\n", " # This timing includes hook and rounding overhead. Do not read it as real int8 latency.\n", " return evaluate_model(model, name, \"fake\", target, f\"int{bits} simulated\")\n", "\n", "\n", "activation_experiments = [\n", " # Output of token + position embedding before the first transformer block.\n", " (\"embeddings output\", [\"distilbert.embeddings\"]),\n", "\n", " # Output of self-attention after tokens have mixed information in each block.\n", " (\"attention module outputs\", attention_paths),\n", "\n", " # Output of the feed-forward network inside each transformer block.\n", " (\"FFN outputs\", ffn_paths),\n", "\n", " # Output just before the final classifier maps the representation to labels.\n", " (\"classifier input\", [\"pre_classifier\"]),\n", "]\n", "\n", "for bits in [8, 4]:\n", " for target, paths in activation_experiments:\n", " results.append(evaluate_with_fake_quant(\n", " base_model,\n", " paths,\n", " bits,\n", " target,\n", " name=f\"fake int{bits} {target}\",\n", " ))\n", "\n", "activation_df = finish_result_table([\n", " result for result in results\n", " if result[\"kind\"] == \"fake\" and result[\"target\"] in {target for target, _ in activation_experiments}\n", "])\n", "\n", "cols = [\"target\", \"precision\", \"accuracy\", \"accuracy_delta\", \"eval_seconds\", \"speedup\"]\n", "display(activation_df[cols].sort_values([\"target\", \"precision\"]))" ] }, { "cell_type": "markdown", "id": "7674ddd0", "metadata": {}, "source": [ "## 8. Read the Quick Results\n", "\n", "The final quick-run table is not a leaderboard. Read the columns together:\n", "\n", "- `accuracy_delta`: did the model still solve the task?\n", "- `speedup`: did this run faster in this CPU setup?\n", "- `size_reduction`: did the serialized model get smaller?\n", "\n", "A row can be smaller and slower. A row can keep accuracy and still be a bad deployment choice. That is why the columns stay separate." ] }, { "cell_type": "code", "execution_count": null, "id": "be33d8b7", "metadata": { "execution": { "iopub.execute_input": "2026-07-08T06:02:48.303137Z", "iopub.status.busy": "2026-07-08T06:02:48.303059Z", "iopub.status.idle": "2026-07-08T06:02:48.320825Z", "shell.execute_reply": "2026-07-08T06:02:48.320607Z" } }, "outputs": [], "source": [ "final_df = finish_result_table(results)\n", "\n", "display(final_df.style.format({\n", " \"accuracy\": \"{:.4f}\",\n", " \"accuracy_delta\": \"{:+.4f}\",\n", " \"eval_seconds\": \"{:.2f}\",\n", " \"speedup\": \"{:.2f}x\",\n", " \"model_size_mb\": \"{:.1f}\",\n", " \"size_reduction\": \"{:.1%}\",\n", " \"examples_per_second\": \"{:.1f}\",\n", " \"avg_batch_latency_ms\": \"{:.1f}\",\n", " \"memory_delta_mb\": \"{:+.1f}\",\n", "}))\n" ] }, { "cell_type": "code", "execution_count": null, "id": "9f1a32aa", "metadata": { "execution": { "iopub.execute_input": "2026-07-08T06:02:48.322186Z", "iopub.status.busy": "2026-07-08T06:02:48.322098Z", "iopub.status.idle": "2026-07-08T06:02:48.326436Z", "shell.execute_reply": "2026-07-08T06:02:48.326224Z" } }, "outputs": [], "source": [ "summary_cols = [\"experiment\", \"accuracy_delta\", \"speedup\", \"size_reduction\"]\n", "display(final_df[summary_cols].sort_values(\"accuracy_delta\"))\n", "\n", "print(\"How I read this table:\")\n", "print(\"1. Did accuracy move enough to matter, or is it probably subset noise?\")\n", "print(\"2. Did the serialized model get smaller by a meaningful amount?\")\n", "print(\"3. Did this CPU/backend actually run faster, not just store fewer bits?\")\n", "print(\"4. Is this a real quantized path or only a fake-quant sensitivity test?\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9. Quick Run, Then Full Run\n", "\n", "The first pass used 256 examples because I wanted the notebook to stay fast while I was still choosing targets. That is fine for exploration. It is not enough for trust.\n", "\n", "On 256 examples, one changed prediction moves accuracy by about 0.39 percentage points. A `+0.0039` row can be one sentence changing sides. A 24-point drop, like `FFN outputs | int4`, is large enough to keep checking.\n", "\n", "For the full SST-2 validation split, I did not rerun every row. I reran the ones that could change the conclusion:\n", "\n", "- the baseline\n", "- the big size-reduction row\n", "- the classifier-only row that looked faster\n", "- the clearly bad FFN int4 row\n", "- the embeddings int4 row that looked suspiciously better in quick mode" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Optional full-validation rerun.\n", "# This cell is intentionally not executed in the saved notebook. Run it when you want\n", "# better accuracy numbers for the few rows that mattered in the quick scan.\n", "\n", "FULL_EVAL_SIZE = None # Use None for the full SST-2 validation split.\n", "\n", "# Rebuild the validation loader. By default this uses all SST-2 validation rows.\n", "full_raw_validation = load_dataset(\"glue\", \"sst2\", split=\"validation\")\n", "if FULL_EVAL_SIZE is not None:\n", " full_raw_validation = full_raw_validation.shuffle(seed=SEED).select(range(FULL_EVAL_SIZE))\n", "\n", "full_tokenized_validation = full_raw_validation.map(tokenize_batch, batched=True)\n", "full_tokenized_validation = full_tokenized_validation.remove_columns([\"sentence\", \"idx\"])\n", "full_tokenized_validation = full_tokenized_validation.rename_column(\"label\", \"labels\")\n", "full_tokenized_validation.set_format(\"torch\")\n", "\n", "full_validation_loader = DataLoader(\n", " full_tokenized_validation,\n", " batch_size=BATCH_SIZE,\n", " shuffle=False,\n", " collate_fn=data_collator,\n", ")\n", "\n", "# Temporarily swap the globals used by evaluate_model so the helper stays simple.\n", "quick_loader = validation_loader\n", "quick_tokenized_validation = tokenized_validation\n", "validation_loader = full_validation_loader\n", "tokenized_validation = full_tokenized_validation\n", "\n", "full_results = []\n", "try:\n", " full_results.append(evaluate_model(base_model, \"full baseline fp32\", \"baseline\", \"all\", \"fp32\"))\n", " full_results.append(evaluate_dynamic_quantization(\n", " base_model,\n", " \"full dynamic int8 all Linear\",\n", " \"all Linear\",\n", " lambda model: quantize_all_linear(model),\n", " ))\n", " full_results.append(evaluate_dynamic_quantization(\n", " base_model,\n", " \"full dynamic int8 classifier\",\n", " \"classifier\",\n", " lambda model: quantize_selected_paths(model, classifier_paths),\n", " ))\n", " full_results.append(evaluate_with_fake_quant(\n", " base_model,\n", " ffn_paths,\n", " 4,\n", " \"FFN outputs\",\n", " \"full fake int4 FFN outputs\",\n", " ))\n", " full_results.append(evaluate_with_fake_quant(\n", " base_model,\n", " [\"distilbert.embeddings\"],\n", " 4,\n", " \"embeddings output\",\n", " \"full fake int4 embeddings output\",\n", " ))\n", "finally:\n", " validation_loader = quick_loader\n", " tokenized_validation = quick_tokenized_validation\n", "\n", "full_df = finish_result_table(full_results)\n", "display(full_df[[\"experiment\", \"accuracy\", \"accuracy_delta\", \"eval_seconds\", \"speedup\", \"size_reduction\"]])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Full-validation results from my run\n", "\n", "These are the rows I got from the full SST-2 validation split on the same MacBook Pro run used in the post.\n", "\n", "| Experiment | Accuracy | Delta | Eval Time | Speedup | Size |\n", "|---|---:|---:|---:|---:|---:|\n", "| baseline fp32 | 0.9106 | +0.0000 | 2.15s | 1.00x | 255.5 MB |\n", "| dynamic int8 all Linear | 0.8991 | -0.0115 | 5.00s | 0.43x | 132.3 MB |\n", "| dynamic int8 classifier | 0.9106 | +0.0000 | 1.81s | 1.18x | 255.5 MB |\n", "| fake int4 FFN outputs | 0.6904 | -0.2202 | 1.91s | 1.13x | 255.5 MB |\n", "| fake int4 embeddings output | 0.9106 | +0.0000 | 1.82s | 1.18x | 255.5 MB |\n", "\n", "The full run makes the quick table calmer.\n", "\n", "The all-Linear int8 row is still the real size win: 255.5 MB down to 132.3 MB. It is also still slower on this CPU path, and it loses about one percentage point of accuracy.\n", "\n", "The int4 embeddings bump disappears. On the full split it lands exactly on the baseline accuracy. That quick-run improvement was subset noise.\n", "\n", "The FFN int4 row stays bad. The fake-quant timing numbers are not deployment timings because they include hook and rounding overhead, but the accuracy signal is clear enough: these activations do not like being rounded that hard." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 10. What This Run Says\n", "\n", "I started with a simple habit: pull a quantized local model, see that it fits, and move on. This notebook slowed that habit down.\n", "\n", "On this small model, \"quantized\" was not one thing. Quantizing all `Linear` layers cut the saved file almost in half, but made this CPU run slower. Quantizing only the classifier ran faster, but barely changed the file size. Rounding FFN activations to int4 broke accuracy. The int4 embedding result looked good in the quick run, then disappeared on the full split.\n", "\n", "That is the useful connection back to local LLMs. A q4 or int8 label tells me the model was compressed. It does not tell me whether it will be faster on my machine, whether the quality tradeoff is acceptable, or which part of the model paid the cost. For that, I still have to measure the workload I care about." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.6" } }, "nbformat": 4, "nbformat_minor": 5 }