CloudSignal AI

Blog

Production computer vision: why the demo is the easy 20%

By Ashwin Rajendraprasad Computer Vision

Most computer-vision projects are pitched the same way: a demo that works. The demo is real, and it’s also roughly 20% of the work. The other 80% — the part that decides whether the system survives real users — is almost never in the demo. The perception systems Ashwin designed and led in prior roles reach more than 100 million people across 60+ countries, and the pattern below is consistent enough to state plainly.

The lab is a lie

A demo runs on one device, in one room, under one set of lighting. Production runs on a long tail of hardware you don’t control. New phones ship new cameras with new image signal processors every autumn, each making its own contrast and white-balance decisions before your code sees a pixel. Fluorescent and LED lighting flickers at frequencies your exposure control fights. The model that looked flawless on a developer’s flagship phone meets a three-year-old mid-ranger under a buzzing office light, and the distribution it was tuned on no longer holds. The first job of a production CV team is to make the lab less of a lie: test across real devices and real lighting, not the one on the desk.

Latency is a product decision, not an engineering detail

For anything real-time and interactive (try-on, AR overlays, live tracking), the budget is roughly 33 ms per frame to hold 30 fps, and that’s camera-to-screen, everything included: capture, preprocessing, inference, post-processing, render. Miss it and the experience feels broken regardless of accuracy. That budget is a product constraint, so it belongs in the architecture from day one. The systems that hold up at scale almost always split the work (a fast, cheap model on the hot path for every frame, with heavier computation run less often or asynchronously) rather than trying to force one big model into the frame budget.

Fidelity has a price curve, and you should know where you sit

More detail costs more compute, and the right amount is task-dependent, not “as much as possible.” Face-based try-on for beauty or eyewear genuinely needs dense geometry (400–500+ landmarks) to place a product convincingly. Attention detection or a simple game filter is well served by a classic 68-point model at a fraction of the cost. Picking the heaviest model available “to be safe” is how you blow the latency budget in section two. Know where on the fidelity/cost curve your task actually sits, and buy only the precision the product needs.

Production has no ground truth

In the lab you have labels, so you can measure accuracy. In production you have a live camera and no answer key. You cannot compute accuracy on traffic you never labeled, which means you’re flying on proxies: distribution drift, confidence trends, dropout and failure rates, user-behavior signals (do people retry, abandon, correct?). Building that observability, and alerting on the proxies before users feel the problem, is as much of the job as training the model, and it’s the part that’s missing from every demo.

Design for graceful failure

Every perception system fails sometimes: a face leaves the frame, the light dies, the subject moves too fast. The question is what happens next. A system that freezes, snaps a box to the wrong place, or overlays a product onto empty air feels broken. One that degrades gently (drops the overlay, holds the last good state, tells the user what’s wrong) feels solid even when the model is momentarily lost. Failure handling isn’t an edge case to bolt on later; it’s a first-class feature.

A buyer’s checklist for any CV demo

If you’re evaluating a computer-vision demo, from a vendor or your own team, a short list separates the ones that will scale from the ones that won’t:

  1. Which devices and lighting was this tested on beyond the one in the demo?
  2. What’s the end-to-end latency (camera-to-screen) on the worst target device, not the best?
  3. Where does it sit on the fidelity/cost curve, and why that point?
  4. How is quality measured in production, with no labels?
  5. What happens when it fails, and have you watched it fail on purpose?

Ask those five and the conversation moves from the demo to the system. The model is the only part they’ll show you unprompted.

Four of those five have answers a team can reach before a single user sees the product. The fourth does not. Nothing you can measure in advance tells you how a model behaves on traffic nobody labeled, so that answer arrives after launch or it never arrives at all. Under all of it sits a floor that belongs to physics rather than to engineering effort, and Seeing at the limit sets out where that floor is.

If you’re holding a demo that works and a ship date that doesn’t, the five questions above are the place to start. A CloudSignal architecture review is what to do with the ones that come back blank.

Sources / further reading

Written by Ashwin Rajendraprasad for CloudSignal AI.