Image segmentation is a computer vision task that labels an image pixel by pixel. Instead of naming what a photo shows or drawing a box around an object, it marks exactly which pixels belong to each object or region. That precision lets software measure shapes, areas and edges.
Picture a radiotherapy plan. A box around a tumour on a CT scan is not enough, because the treatment must follow the tumour’s outline and spare the tissue beside it. A webshop has the same problem on a smaller scale: to place a mug on a white background, software needs its exact edge, handle included.
This article covers the types of segmentation, how it works, which models to test in November 2026 and how to judge the results. It builds on our introductions to computer vision and object detection.
Key takeaways
- Segmentation gives you a mask per object or region, so you can measure area and shape. Detection only gives a box.
- No model wins everywhere. SAM 3 suits prompting, YOLO26 and RF-DETR suit real-time work, nnU-Net suits medical scans, and their licences differ sharply.
- A mask that looks right can still be wrong, so test it against human masks on your own images.
What is image segmentation?
Image segmentation divides an image into regions by assigning every pixel to a class, an object or the background. The result is a mask: a map the size of the image in which each pixel carries a label. A binary mask, for example, marks each pixel as “mug” or “not mug”.
Take a product photo of three mugs on a wooden table, two of them touching. Image classification tells you whether the photo contains a mug, with one label for the whole image. Object detection tells you how many mugs there are and where, with a box and a confidence score for each. Segmentation tells you which pixels are mug, so software can cut each one out or measure it.
Boxes are coarse: a box around a diagonal crack is mostly not cracked. Segmentation earns its extra cost when the answer depends on area, shape or an exact edge. When a count or a position is enough, detection is cheaper.
What are the main types of image segmentation?
The three main types are semantic, instance and panoptic segmentation. They differ in whether they separate objects and label the background.
Semantic segmentation
Semantic segmentation gives every pixel a class, such as mug, table or wall, but does not tell objects of one class apart. The two touching mugs merge into one region. That is fine when you need amounts, such as the share of a field covered by weeds.
Instance segmentation
Instance segmentation gives each object its own mask, so mug 1, 2 and 3 stay separate. The background usually stays unlabelled. Use it when you need to count, measure or follow individual objects.
Panoptic segmentation
Panoptic segmentation does both: every pixel gets a class, and every countable object also gets an ID. The paper that introduced it calls countable objects “things” and shapeless regions such as sky or road “stuff”. A robot needs exactly that combination: the floor plus every separate person.
| Type | Output | In the mug photo | Typical use |
|---|---|---|---|
| Semantic | A class for every pixel | Touching mugs merge into one region | Road surfaces, land cover, tissue types |
| Instance | A separate mask per object | Three separate mugs; background ignored | Counting and measuring items |
| Panoptic | A class for every pixel, plus an ID per object | Three separate mugs, plus table and wall | Driving and robotics scenes |
Where do promptable and open-vocabulary segmentation fit in?
These two terms describe how you ask for a mask, not what kind of map you get. A promptable model segments whatever you click or box, an approach Meta’s Segment Anything models made popular. An open-vocabulary model accepts a concept in text, such as “red mug”, instead of a fixed list like the 80 classes of the public COCO dataset. Meta’s SAM 3 combines both in what it calls promptable concept segmentation: you name a concept, and it returns an instance mask for every match.
How does image segmentation work?
A segmentation project runs in five steps, and choosing the model is only one of them.
- Collect images that reflect real conditions, such as your cameras and lighting.
- Annotate examples by drawing masks or polygons. This is slow work: for the Cityscapes street dataset, fine annotation took more than 1.5 hours per image on average.
- Pick a pretrained model, or fine-tune one on your own labelled images.
- Generate masks on new images and clean them up, for example by removing tiny fragments.
- Evaluate against human masks on unseen images, then retrain with the hard cases.
Which architectures sit behind the models?
Convolutional neural networks (CNNs) slide small filters over an image and excel at local detail. Encoder-decoder networks compress an image into features, then expand them back to full resolution to label each pixel. U-Net, published in 2015 for biomedical images, is the classic example.
Transformers use attention to relate every part of an image to every other part. Mask transformers such as Mask2Former predict a set of labelled masks, so one design covers all three types. Foundation models such as SAM 3 learn from huge, broad datasets and need little or no extra training.
How do automatic and prompted masks differ?
An automatic model segments a fixed set of classes in every image without any input. A YOLO26 model trained on COCO, for example, outlines every person, car or cup it recognises. A prompted model waits for an instruction, and the supported prompts vary. SAM 2 takes points, boxes and masks, while SAM 3 adds short text phrases and example images. Meta notes that SAM 3 cannot handle long descriptions such as “the second to last book from the right on the top shelf” without help from a multimodal language model.
What are the best image segmentation models in 2026?
There is no single best image segmentation model. The right choice depends on your task, data, accuracy target, speed, hardware and licence. The table compares eight options, checked on 12 November 2026. Each model family usually ships several checkpoints, the downloadable weight files, in different sizes.
| Model and version | Task and inputs | Best suited for | Main strengths | Limitations and compute | Licence |
|---|---|---|---|---|---|
| SAM 3 and 3.1 (Meta; Nov 2025, Mar 2026) | Promptable instance segmentation and tracking; text, examples, points, boxes | Annotation, prototypes, video | Finds every instance of a named concept without training | 3.45 GB; needs a strong GPU; weak on niche concepts until fine-tuned | SAM License: commercial use allowed, some uses excluded |
| SAM 2.1 (Meta; Sep 2024) | Promptable; points, boxes, masks; video | Click-to-segment tools, video tracking | Lighter than SAM 3; widely integrated | No text prompts; one object per prompt | Apache 2.0 |
| Mask2Former (Meta; 2022) | Semantic, instance and panoptic; fixed classes | Training one model on your own classes | One architecture for all three tasks | Repository archived since January 2025; not real-time | Mostly MIT |
| EoMT (TU Eindhoven and partners; 2025) | Semantic, instance and panoptic; fixed classes | Universal segmentation at higher speed | Plain vision transformer; up to 4x faster, say the authors | Needs a large backbone and a GPU; research code | Code MIT; check backbone weights |
| SegFormer (NVIDIA; 2021) | Semantic; fixed classes | Light semantic models, research | Efficient B0 to B5 sizes; in Hugging Face Transformers | Semantic only; older design | Official code and weights are non-commercial |
| YOLO26-seg (Ultralytics; Jan 2026) | Instance; 80 COCO classes | Real-time and edge use | Nano model is 6.4 MB; easy export | New classes need labelled data | AGPL-3.0 or a paid licence |
| RF-DETR-Seg (Roboflow; Jan 2026) | Instance; COCO classes | Real-time use with a permissive licence | Six sizes; DINOv2 backbone | Vendor-reported benchmarks; larger than YOLO nano models | Apache 2.0 |
| nnU-Net v2 (DKFZ) | Semantic; 2D and 3D; trained per dataset | Medical and specialist imaging | Configures itself to your data | Needs labelled data and training time | Apache 2.0 |
What are the main trade-offs?
Four trade-offs decide most choices.
- Prompted or automatic. SAM 3 needs a prompt but no training for common concepts. Automatic models need labelled examples, then run unattended.
- Size and speed. In an Ultralytics benchmark on one RTX PRO 6000 GPU, SAM 3 took 2,921 milliseconds per image and YOLO26n-seg took 8.4 milliseconds. Ultralytics makes YOLO, so treat this as a vendor test. Meta reports 30 milliseconds for SAM 3 on its own H200 setup, which shows how much settings matter.
- Licence. Apache 2.0 and MIT allow commercial use with few conditions. AGPL-3.0 requires you to open-source your project unless you buy a commercial licence, and SAM 3 has its own SAM License. Check code and weights separately.
- Pretrained or domain-specific. Meta notes that SAM 3 struggles with niche medical or scientific concepts, but adapts quickly after fine-tuning on a small labelled set. In a MICCAI 2024 benchmark, well-configured CNN U-Nets in nnU-Net still matched or beat newer designs on 3D medical scans.
Which model should you start with?
Start from the job, not from a leaderboard.
- For annotation and cut-outs, start with SAM 3. Pick SAM 2.1 if you need Apache 2.0 or lighter compute.
- For real-time instance segmentation on cameras or edge devices, test YOLO26-seg and RF-DETR-Seg. Choose RF-DETR if AGPL-3.0 is a problem.
- For medical imaging, train nnU-Net on your own scans and use SAM to speed up labelling.
- For semantic or panoptic scenes such as roads or land cover, fine-tune Mask2Former or EoMT on your classes.
Expect to fine-tune when your images look unlike public datasets, such as X-rays or your own production line.
How can image segmentation be used in practice?
Segmentation pays off where a decision depends on an exact area, shape or edge. The mask is an input: the diagnosis, measurement or action comes later.
Medical imaging
CT and MRI scans become masks of organs, vessels or tumours, so volumes can be measured rather than estimated. TotalSegmentator, built on nnU-Net, segments 104 anatomical structures in CT and reached a Dice score of 0.943 on its test set. A mask is not a diagnosis, though, and clinical software usually falls under medical device rules.
Autonomous driving and robotics
Camera frames become maps of road, lanes and every pedestrian, which planning software uses to choose a path. Night, rain and rare situations remain hard, so segmentation is one input among several.
Manufacturing inspection
Images from the line become masks of scratches, cracks or missing coating. Software then checks each defect’s area or length against a tolerance. Defects are rare, so examples are scarce, and stable lighting often matters more than the model.
Agriculture and satellite imagery
Drone and satellite images become masks of crops, weeds, fields or buildings for area statistics and targeted spraying. One satellite pixel can cover several metres, so small objects disappear.
E-commerce and creative editing
A product photo becomes a mask for background removal or effects on one object. Meta uses SAM 3 for the View in Room feature on Facebook Marketplace, which lets buyers preview a lamp or table in their own room. Glass, hair and shiny packaging still produce ragged edges.
Video analysis and object tracking
Video becomes a mask per object per frame, linked so each object keeps its identity. Meta says SAM 3.1 tracks up to 16 objects in one pass, doubling throughput to 32 frames per second on one H100 GPU for a medium number of objects. Identities can still switch when objects pass behind each other.
How do you get started with image segmentation?
Start by testing a pretrained model on a handful of your own images. The route depends on your role.
For non-technical users
Meta’s Segment Anything Playground lets you upload an image or video and try SAM models in the browser. Annotation tools such as CVAT build SAM 2 and SAM 3 into their editors, with limited free tiers. Check the terms before you upload confidential images to either.
For developers
The Ultralytics package runs YOLO26-seg and SAM 3, Hugging Face Transformers covers SAM 3, EoMT and SegFormer, and Roboflow’s rfdetr package runs RF-DETR-Seg. These lines run YOLO26-seg on a photo:
from ultralytics import YOLO
model = YOLO("yolo26n-seg.pt") # pretrained on COCO
results = model("three-mugs.jpg") # run instance segmentation
print(results[0].masks.data.shape) # one mask per object found
results[0].show() # view the masks
COCO has no “mug” class, so the model calls the mugs “cup”: a first lesson in closed-set models. The Ultralytics package is AGPL-3.0 licensed.
For teams with specialised data
Collect images that cover every camera, shift and defect type, and keep images from separate days aside for testing. Write a short labelling guide, then let a model draft masks for people to correct. The TotalSegmentator team retrained its model after 5, 20 and 100 corrected patients to speed up labelling. Compare any fine-tuned model with a pretrained baseline.
Walkthrough: how do you cut out a product photo?
- Choose a tool. A browser tool with a SAM model suits a few images, while a catalogue calls for a script.
- Load the mug photo and click on the mug you want. Add a click on the handle if the mask misses it, or a negative click on a neighbour that bleeds in. With SAM 3, you can also type “mug”.
- Inspect the mask at full zoom, especially the hole in the handle, reflections, the shadow and the point where two mugs touch.
- Fix errors with extra clicks or by editing the outline.
- Export a black-and-white PNG mask, a transparent PNG for the webshop or COCO JSON for training data.
How do you evaluate segmentation results?
Compare the model’s masks with careful human masks on images it has never seen, using a metric that fits the task. A convincing mask can still be wrong where precision matters.
Four metrics cover most projects. Each suits a different type of segmentation.
| Metric | What it measures | Suits |
|---|---|---|
| IoU and mean IoU | Overlap divided by combined area; mean IoU averages over classes | Semantic segmentation |
| Dice score | Twice the overlap divided by both areas; never below IoU | Medical imaging |
| Mask AP | Objects found and outlined, across confidence and overlap thresholds | Instance segmentation |
| Panoptic quality | Recognition and mask quality for things and stuff together | Panoptic segmentation |
The choice changes the number. If a tumour covers 1,000 pixels and the model predicts 1,000, of which 800 overlap, IoU is 0.67 and Dice is 0.80. It is the same mask with a different score, so always compare like with like. For mask AP, COCO averages over ten overlap thresholds from 0.50 to 0.95. The Metrics Reloaded framework helps you pick a metric.
Which mistakes should you avoid?
- Thin structures. A one-pixel shift costs a crack three pixels wide much of its IoU, and downscaling can erase it.
- Occlusion. Touching objects merge or split, so check object counts as well as overlap.
- Small or low-contrast objects. Report scores per object size, because averages hide these misses.
- Domain shift. A new camera, scanner or lighting setup can cut accuracy sharply, whatever the benchmark says.
- Annotation quality. When the Cityscapes team annotated 30 images twice, 96 percent of pixels got the same label, so check how well your annotators agree.
- Hardware. Large promptable models need a strong GPU, and video costs grow with every tracked object. See our comparison of cloud, edge and local AI.
- Privacy. Scans, faces and licence plates are personal data, and health data gets extra protection under the GDPR. Prefer local processing.
A mask with an IoU of 0.9 can still clip a margin that matters, or include a shadow that doubles a defect’s measured area. Set pass marks in your own units, such as millimetres, and test on held-out images from different days or sites.
Frequently asked questions about image segmentation
Is image segmentation the same as background removal?
Background removal is one application of image segmentation. A binary mask separates the subject from the background, and the background pixels are then made transparent. Segmentation can do much more, such as separate several objects, label every region or measure areas. Every background-removal tool uses segmentation, but not every segmentation task is background removal.
How does deep learning segmentation differ from classic methods such as thresholding?
Classic methods such as thresholding, edge detection and watershed split an image by brightness, colour or edges. They are fast, need no training data and work well under fixed lighting, such as a dark part on a bright conveyor. Deep learning models learn what objects look like, so they cope better with shadows, clutter and variation.
What image resolution do segmentation models need?
Most models first resize every image to a fixed input size. RF-DETR-Seg, for example, works at 312 × 312 pixels for its smallest checkpoint and 768 × 768 for its largest. A small defect in a large photo then shrinks to a few pixels, so crop large images or split them into tiles before segmenting them.
Can image segmentation run on a phone or without a GPU?
Compact segmentation models can run without a GPU. YOLO26 was built for edge devices and exports to formats that run on CPUs and edge hardware. Large promptable models such as SAM 3 do need a strong GPU. Measure speed on the device you will actually use, because the same model is usually slower on a CPU.
How long does it take to train a segmentation model?
Training takes from hours to days, depending on model size, resolution and data volume. The EoMT authors report about six hours on four NVIDIA H100 GPUs for one COCO configuration. Fine-tuning a pretrained model on a small dataset is usually much faster, but drawing the masks can take longer than the training itself.
What does an image segmentation project cost?
The model is rarely the biggest cost. Annotation, integration with your own systems, compute and maintenance usually weigh more. The exact amount depends on the number of classes, the accuracy you need and how often your images change. Start with a pilot on existing images and a pretrained model, so you can budget the rest on real numbers.
Where should you start with image segmentation?
Image segmentation turns pictures into measurements. Pick one use case where an area, a shape or an edge drives a decision. Test a pretrained model on a few dozen representative images and measure its masks against careful human ones. DataNorth’s computer vision team can help you assess feasibility and run a small pilot.