CloudSignal AI

Blog

Object detection in the browser with Transformers.js (no server)

By Ashwin Rajendraprasad TutorialsComputer Vision

Same ethos as our face, hand, and pose tutorials (one HTML file, a webcam, a working result), but this time a full object-detection model runs entirely client-side. No server, no build step, and the camera frames never leave the device.

1. Why run detection in the browser

Three reasons. Privacy: the frames never leave the device, which means nothing to upload, nothing to store, and a much smaller compliance surface. Cost: inference runs on the user’s hardware, so there’s no GPU box to rent or autoscale, and no per-frame network round-trip to add latency. Deployment: it’s a static file. Ship it on any CDN or static host, and once the weights are cached the page keeps working with no network at all. The trade you’re accepting is that you inherit the user’s hardware and browser, which is what sections 5 and 6 are about.

2. Transformers.js in one paragraph

Transformers.js is Hugging Face’s JavaScript library for running models directly in the browser with no server. It executes them through ONNX Runtime Web (the browser build of Microsoft’s ONNX Runtime), using WebGPU when the browser supports it and falling back to WebAssembly (WASM) otherwise. It covers a wide range of tasks, object detection among them, and downloads models (usually quantized) to run client-side, so the data stays on the device.

If you’ve used the Python transformers library, the shape will look familiar: you name a task and a model, and a pipeline handles the pre-processing, inference, and post-processing for you. Transformers.js keeps that same task-first API in JavaScript and picks the execution provider (WebGPU or WASM) underneath, so the code you write barely changes between “toy” and “fast.”

3. The whole detector, from a CDN

One module script. Import pipeline, choose a device based on whether the browser exposes WebGPU, and load a detection model straight from the Hugging Face Hub. Pin the version (supply-chain reasons; see section 6). This, and the webcam code in section 4, slot into a <script type="module"> inside the usual single-file skeleton (a button, a <video>, and a <canvas id="overlay"> on top):

import { pipeline, env } from
  "https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.0.0";

env.allowLocalModels = false;                       // fetch weights from the HF Hub

const device = navigator.gpu ? "webgpu" : "wasm";   // fast path, or broad fallback
const detector = await pipeline(
  "object-detection",
  "Xenova/detr-resnet-50",
  { device }
);

Xenova/detr-resnet-50 is a standard detection model packaged for Transformers.js. The first call downloads and caches the weights; after that it’s local.

4. Webcam in, boxes out

Wire getUserMedia (secure context, as ever), copy each video frame to an offscreen canvas at native resolution, run the detector on it, and draw the returned boxes and labels on the overlay. Because the detector returns box coordinates in the input image’s pixel space, and our overlay is sized to match, they map straight across:

const video  = document.getElementById("video");
const button = document.getElementById("start");
const canvas = document.getElementById("overlay");
const ctx    = canvas.getContext("2d");

const frame = document.createElement("canvas");     // offscreen: the model's input
const fctx  = frame.getContext("2d");

button.addEventListener("click", async () => {
  try {
    const stream = await navigator.mediaDevices.getUserMedia({
      video: { facingMode: "environment", width: { ideal: 1280 } },
      audio: false,
    });
    video.srcObject = stream;
    button.remove();
    document.querySelector(".stage").hidden = false;
    video.addEventListener("loadeddata", loop, { once: true });
  } catch (err) {
    button.textContent =
      err.name === "NotAllowedError"
        ? "Camera blocked: check site permissions"
        : "No camera found";
  }
});

async function loop() {
  canvas.width = frame.width  = video.videoWidth;
  canvas.height = frame.height = video.videoHeight;

  while (document.body.contains(canvas)) {
    fctx.drawImage(video, 0, 0, frame.width, frame.height);
    const results = await detector(frame.toDataURL("image/jpeg"),
                                   { threshold: 0.5 });

    ctx.clearRect(0, 0, canvas.width, canvas.height);
    ctx.lineWidth = 2;
    ctx.strokeStyle = ctx.fillStyle = "#2dd4bf";
    ctx.font = "16px system-ui";

    for (const { label, score, box } of results) {
      const { xmin, ymin, xmax, ymax } = box;       // pixels in the input frame
      ctx.strokeRect(xmin, ymin, xmax - xmin, ymax - ymin);
      ctx.fillText(`${label} ${Math.round(score * 100)}%`, xmin + 4, ymin + 18);
    }
  }
}

The while loop awaits each detection, so it paces itself to the model’s speed instead of piling up a backlog. No manual throttling needed.

5. WebGPU vs. WASM, and model size

WebGPU is the fast path (it runs the model on the GPU), but browser and device support varies, so we feature-detect navigator.gpu and fall back to WASM, which runs almost everywhere and is slower. I won’t quote an FPS: it depends on the device, the model, and the precision, so measure on the machines you actually target.

The other dial is model size. Quantized models (smaller-precision weights) download faster and run lighter for a modest accuracy cost, the same size/accuracy trade we cover in shipping vision to the edge. Transformers.js downloads quantized weights for the web by default and exposes options to select a lower or higher precision when you want to trade accuracy against size and speed. The download size matters twice over here (it’s both your first-load latency and, on mobile, the user’s data), so pick the smallest model that clears your accuracy bar. A detector nobody waits for beats a heavier one they close the tab on.

6. Where it breaks

  • Secure context: getUserMedia needs https:// or localhost; nothing happens on plain http://.
  • WebGPU availability varies by browser and device, so always keep the WASM fallback in place.
  • First load pays a download. Detection models run to tens of megabytes; show a loading state and let the browser cache do its job. After the first load it’s local.
  • A CDN import is a live supply-chain dependency: executable code you don’t control, re-resolved on every load. Fine for prototypes; for production, pin exact versions (here, 3.0.0, but check the repo and pin what you test) and vendor the library and the model weights into your own hosting.

That’s a real object detector (model, webcam, and boxes) in one file the user’s browser runs by itself. On-device computer vision, and the model-and-runtime choices that decide whether it’s actually usable, are core to what CloudSignal builds. If you’re taking something like this to production, that’s a conversation worth having.

Sources / further reading

Written by Ashwin Rajendraprasad for CloudSignal AI. The code above is free to reuse.