CloudSignal AI

Blog

Shipping vision to the edge: quantization, distillation, and the latency budget

By Ashwin Rajendraprasad Computer Vision

The hardest part of production computer vision usually isn’t accuracy — it’s fitting the model into a real device’s latency, memory, and power budget. A model that misses the budget ships nothing. This is the field guide to the techniques that get you under it, and what each one costs.

The budget that decides everything

The constraint that matters is measured three ways (latency, memory, and power) and always camera-to-decision, on the device you actually ship, not on your workstation. For interactive video the frame budget is tight: hold 30 fps and you have roughly 33 ms per frame for capture, inference, and everything after it. Memory and power are just as binding on a phone or an embedded SoC, where a model that runs but drains the battery or blows the thermal envelope is still a failure.

Everything below is about buying headroom against that budget without giving away the accuracy that justified the model in the first place. Our companion post on production computer vision covers the surrounding system; this one is about the model itself.

Quantization: fewer bits per number

Most models train and run in 32-bit floating point (FP32). Quantization drops that numerical precision (FP32 to INT8, sometimes lower), which shrinks the model and speeds inference, usually at some cost in accuracy. It’s the highest-leverage technique on the list and the first one to reach for.

There are two standard approaches. Post-training quantization (PTQ) quantizes an already-trained model; it’s fast and needs only a small calibration set, but accuracy can slip at low bit-widths. Quantization-aware training (QAT) simulates quantization during training so the model learns to tolerate it: more work, but usually better accuracy when you push to INT8 and below. The caveat: INT8 isn’t free. Calibrate on representative data, and check per-class accuracy afterward, because quantization error doesn’t fall evenly across classes. An aggregate that looks fine can hide one important class that degraded.

Which approach to pick is mostly a question of how far you’re pushing precision and how much accuracy headroom you have. If INT8 with good calibration lands inside your accuracy budget, PTQ is the cheap win. Do that first. If it doesn’t, or if you need to go below eight bits, QAT is the tool that buys the accuracy back, at the cost of a training run. Either way the calibration set has to look like production traffic; calibrate on the wrong distribution and you’ve optimized for a scene you’ll never see.

Pruning: cut what isn’t earning its place

Pruning removes weights or whole channels to cut size and compute. The practical distinction is between unstructured and structured pruning. Unstructured pruning zeros out individual weights and can compress a model aggressively, but you only turn that sparsity into speed if your hardware and runtime support it. Structured pruning removes whole channels or filters. It compresses less, but the speedup is real on ordinary hardware because you’ve actually made the tensors smaller. For edge work, structured pruning is usually the one that pays off at inference time.

Distillation: a small student, a big teacher

Knowledge distillation (Hinton, Vinyals & Dean, 2015) trains a compact “student” model to reproduce a larger “teacher” model’s outputs. The subtlety is that the student learns from the teacher’s full soft output distribution, not just the hard labels. That distribution carries more information than a one-hot label, which is why a distilled student often behaves far better than the same architecture trained from scratch.

Distillation is different in kind from quantization and pruning. Those two shrink a model you already have; distillation lets you train a fundamentally smaller architecture that punches above its parameter count. In practice the three compose: you might distill to a smaller model, then quantize it for the target device.

The catch is that distillation needs a good teacher and a representative transfer set: the student can only learn the behavior the teacher demonstrates on the data you show it. Get those right and it’s the most powerful lever of the three, because you are not trimming a model down to size, you are training the right-sized model from the start.

The runtimes that actually run on-device

A shrunk model still needs a runtime that exploits the hardware, and the runtime choice matters as much as the model. The standard toolchains are ONNX Runtime (broad hardware coverage), NVIDIA TensorRT (NVIDIA GPUs and Jetson boards), TensorFlow Lite / LiteRT (mobile and embedded), and Apple Core ML (Apple silicon). Modern phones and edge SoCs also ship NPUs (dedicated neural accelerators), and the real win comes from matching the model, the runtime, and the accelerator, not from optimizing any one of them alone. A model exported to the wrong runtime for its target hardware can leave most of the device’s performance on the table. The same network can run markedly faster simply by targeting the accelerator the device actually has, with no change to the weights at all.

Measure on the device, then co-design

The last piece is a discipline: benchmark on-device, not on your workstation, because the workstation number is a fiction for anything you ship to the edge. Watch two things the mean will hide. Tail latency: p95 and p99, not just the average, because a smooth mean with ugly spikes still feels broken to a user. Per-class accuracy: an aggregate that holds while one class you care about degrades is the classic quantization trap, and it won’t show up in a single top-line number.

Our rule at CloudSignal is to pick the smallest model that clears the bar, then co-design model, runtime, and hardware together rather than optimizing them in sequence. Optimizing in sequence is how you end up with a beautifully quantized model the target runtime can’t accelerate, or a fast runtime starved by a model that was never shrunk to fit it. The three decisions constrain each other, so they belong on the table at the same time. And be skeptical of any speedup quoted in the abstract: a number only means something once you name the model and the device it ran on. That’s also why we won’t promise you one here: it’s exactly the figure that has to be measured on your target, not borrowed from someone else’s.

If you’re getting a vision model onto real hardware and the latency budget is fighting you, that’s a build and architecture-review conversation to have while the model choice is still open, and it pairs directly with picking the right real-time detector for the job.

Sources / further reading

Written by Ashwin Rajendraprasad for CloudSignal AI.