CloudSignal AI

Blog

Segment anything, now just say it: SAM 3 and open-vocabulary vision

By Ashwin Rajendraprasad Computer Vision

SAM 1 and SAM 2 segmented what you pointed at. Meta’s SAM 3, released on November 19, 2025, segments what you describe: type “red baseball cap” and it finds every match, in images and in video. That shift from pointing to naming reorganizes how you build labeling, search, and editing pipelines, and it’s worth being precise about what it does and doesn’t buy you.

Update, March 2026: Meta has since shipped SAM 3.1 as a drop-in replacement for SAM 3. The headline change is object multiplexing, which tracks up to 16 objects in a single forward pass and, on video with a medium number of objects, lifts throughput from 16 to 32 frames per second on one H100.

What “segment anything” used to mean

The original Segment Anything Model and its successor (Meta) were built around visual prompts. You gave the model a point, a box, or a rough mask, and it returned a clean segmentation of the thing you indicated. It was genuinely useful, a general-purpose “cut out the thing I’m pointing at” tool that dropped into annotation and editing workflows.

But the interaction had a ceiling: you had to point at each thing. To segment every red cap in a crowded scene, someone or some upstream detector had to locate each one first and hand SAM a prompt. The model was a brilliant executor of “this one,” not an answer to “all of these.”

The SAM 3 leap: prompts you can type

SAM 3 adds text-promptable (Meta calls them concept) detection, segmentation, and tracking, across both images and video. In Meta’s own example, you “type in ‘red baseball cap’ and SAM 3 will segment all matching objects.” It handles a far wider range of text prompts than earlier versions.

That is a different tool, not a nicer version of the old one. The old question was “where I point, what’s the mask?” The new question is “everywhere in this image or video, where is the concept I named?” Detection, segmentation, and tracking collapse into one open-vocabulary step. For anything where you know what you’re looking for but not where it is (which is most real applications), that removes an entire stage of the pipeline.

Composing with language models

SAM 3’s prompts also compose. Because the interface is text, you can pair it with a large language model to handle compound or negated requests. Meta’s own worked example is “people sitting down, but not holding a gift box in their hands.” The language model parses and reasons about the phrase; SAM 3 does the grounding into pixels.

For a practitioner this is the interesting seam. It means a segmentation stage can accept the kind of messy, qualified instruction a human would actually give, rather than a rigid class list fixed at training time. It also means your failure modes now include the language model’s, which is worth remembering when you’re debugging why “not holding a gift box” didn’t quite land.

SAM 3D: shape from a single photo

Alongside SAM 3, Meta released SAM 3D, two open-source models. SAM 3D Objects handles object and scene reconstruction; SAM 3D Body handles human body and shape. The pitch is single-image 3D: infer plausible geometry from one ordinary photo rather than from a rig of cameras or a depth sensor.

Meta states that SAM 3D Objects “significantly outperforms existing methods,” and (the part practitioners should note) introduced a new evaluation dataset, SAM 3D Artist Objects, built with artists. A purpose-built, artist-made evaluation set is a meaningful detail: single-image reconstruction is notoriously easy to demo and hard to measure, and a good eval set is how you keep the claim honest.

What it changes in a pipeline

Put together, open-vocabulary segmentation and single-image 3D touch a lot of real workflows:

  • Auto-labeling. Describe a class in words and pre-segment a dataset, then have humans correct rather than draw from scratch, a natural complement to the frozen self-supervised backbones we’ve written about.
  • Visual search. Find every instance of a described concept across a large image or video library.
  • Video object tracking. Name an object once and follow it through a clip.
  • AR try-on and “view in room.” Text-driven segmentation plus single-image 3D is the substrate for placing products into a user’s space. Meta has announced integrations along these lines: Instagram Edits, Facebook Marketplace “View in Room,” and Meta AI experiences.

The through-line is that a stage which used to require a trained, fixed-vocabulary detector in front of it now takes a phrase. That collapses build time and, more subtly, changes who can specify a task: someone can describe a new concept in words instead of commissioning a labeling run and a retrain. The cost moves from up-front data work to inference-time evaluation. You spend less getting to a first result and more making sure the result is right.

For teams doing product recognition or virtual try-on, this is squarely on the critical path, not a side curiosity.

The limits

Two calibrations before anyone rewrites a pipeline. First, “open-vocabulary” means it handles a broad range of described concepts, not that it understands any concept perfectly. Rare, fine-grained, or ambiguous descriptions will still miss, and promptable segmentation still needs a human in the loop wherever correctness is load-bearing. Treat SAM 3 as a very strong first pass, not an oracle. The useful mental model is a fast, broad annotator that is occasionally wrong in ways that look right. That is exactly when you want review in the loop, not blind trust.

Second, read the headline number for what it actually measures. Meta does publish one: SAM 3 surpasses baselines on its new SA-Co benchmark by at least 2x, and on the SA-Co/Gold split it scores 54.1 cgF1 against 24.6 for the strongest baseline the paper tests. The comparison set is other open-vocabulary systems (OWLv2, GroundingDINO, LLMDet, APE, DINO-X, Gemini 2.5 Flash), not SAM 2, which has no way to take a concept prompt in the first place. So “twice as good” means twice as good as the systems that already did this job, and it tells you nothing about the point-and-click SAM you may be replacing. The same table puts human annotators at 72.8 cgF1. That is the more useful anchor: a real jump that still lands well short of a person, scored on Meta’s data rather than yours. Measure it on your own. And for video, budget for latency and cost: per-frame concept segmentation and tracking is not free, and the difference between a demo clip and a production video stream is exactly the kind of gap that decides whether this ships.

Promptable-by-text segmentation is a step change in how CV pipelines get built. If you’re thinking about where it fits in your product, that’s a build worth scoping deliberately — start from the pipeline stage it removes, not from the demo.

Sources / further reading

Written by Ashwin Rajendraprasad for CloudSignal AI.