Computer Vision: How it works and what it does

Computer vision is the field of artificial intelligence that teaches computers to extract useful information from images and video, and to act on it. It already runs in phones, factories, hospitals and warehouses.

Your phone unlocks the moment it sees your face. A car brakes because its camera spotted a pedestrian a split second before the driver did. Both are computer vision at work.

This guide explains how it works, what it can do, where it falls short and how to start. Last updated October 2026.

In short

• Computer vision recognizes visual patterns. It does not understand a scene the way a person does.
• Projects succeed or fail on data and testing in the real environment, more than on the model.

What is computer vision?

A computer vision system takes in pixels and returns something useful: “this X-ray may show a fracture”, “there are 14 pallets in bay 3” or “this parcel label reads 9712 AB”.

Many modern systems use machine learning: instead of writing rules, you show a system examples and it learns the patterns. Deep learning, which uses many-layered neural networks, drives most recent progress. Others use traditional image-processing and geometric methods, for example to measure parts under controlled lighting, or combine the two.

How is computer vision different from image processing and image generation?

Image processing changes an image without trying to understand it, for example by sharpening or cropping it. Computer vision often uses such steps as preparation, but its goal is to extract meaning: which objects are present and where.

Image generation works the other way round. Tools like Midjourney create new pixels from a text prompt, while computer vision analyses pixels that already exist.

Does computer vision understand what it sees?

Computer vision does not understand images in the human sense. A model that labels a photo “dog” has learned patterns that went with dogs in its training data, nothing more. A driver who sees a stop sign half covered in snow still knows to stop. A model may fail if snowy signs were missing from its training data. So treat its output as a strong signal, not as comprehension.

How does computer vision work?

Most computer vision systems follow the same five steps, whether they inspect welds or read invoices.

1. Capture. A camera, scanner or drone records images or video. Position, resolution and lighting are fixed here, and they limit everything that follows.
2. Prepare. The images are resized, normalised and sometimes cropped, so they match what the model expects.
3. Run the model. A trained model analyses each image and returns raw output, such as probabilities or box coordinates.
4. Interpret. Software turns that output into a decision. It might only raise a defect alert above a confidence threshold, or when the same defect shows up in three frames in a row. A confidence score is not a guarantee, so set the threshold by testing it against real errors.
5. Act. The result feeds a process: a reject arm pushes a product off the line, or a specialist gets a case to review.

Training and inference explained

Training is the phase in which the model learns. In supervised training, the most common approach, you feed it thousands of labelled examples, such as photos with every defect marked, until its predictions match the labels. Other approaches learn from unlabelled images or from images paired with text. Training takes a lot of computing power.

Inference is the phase in which the trained model handles new images. It needs far less computing power, which is why a model can run on a phone or a small device next to a production line.

The models behind modern computer vision

Three approaches matter most today. The first two are network architectures; the third is a broader type of system built on them.

Convolutional neural networks (CNNs) drove the breakthrough of the 2010s. They scan an image with small filters that detect edges, then combine those into shapes and whole objects. Many production systems still use them.

Vision transformers (ViTs) cut an image into patches and learn how all the patches relate to each other. Google researchers showed in 2020 that this approach can match or beat the best CNNs when it is pretrained on enough data.

Vision-language models connect images to words, usually with transformers inside. OpenAI’s CLIP model learned to link pictures and descriptions from 400 million image and text pairs. The same idea now lets assistants such as ChatGPT, Gemini and Claude answer questions about your photos.

Model	Type	Developer	Year	Best at	Typical use
ResNet	CNN	Microsoft Research	2015	Reliable image classification	Backbone for many classification and detection systems
EfficientNet	CNN	Google	2019	High accuracy with little computing power	Classification on phones and edge devices
YOLO family	CNN-based detector	Redmon et al., later versions by Ultralytics	2016 onwards	Fast object detection	Real-time detection on cameras and production lines
Vision Transformer (ViT)	Transformer	Google	2020	Classification when pretrained on large datasets	Backbone for modern vision and multimodal models
DETR	Transformer detector	Meta	2020	Object detection without hand-tuned steps	Detection in research and newer production systems
CLIP	Vision-language model	OpenAI	2021	Linking images to text descriptions	Image search, zero-shot classification, part of multimodal systems
OpenPose / MediaPipe Pose	Pose estimation	Carnegie Mellon / Google	2017 / 2020	Locating body key points	Fitness apps, ergonomics, motion analysis
Tesseract / PaddleOCR	OCR	Google (open source) / Baidu	2006 / 2020	Extracting printed text	Document scanning, invoice and form processing
Depth Anything	Depth estimation	HKU and TikTok	2024	Depth from a single image	Robotics, AR, 3D scene understanding
DINOv3	Self-supervised transformer	Meta	2025	Visual features learned without labels	Starting point for custom models with few labelled images
SAM 3	Segmentation foundation model	Meta	2025	Segmenting objects from a text or visual prompt	Labelling data, video editing, inspection
GPT, Gemini, Claude, Qwen-VL	Multimodal assistants	OpenAI, Google, Anthropic, Alibaba	2023 onwards	Answering open questions about images	Document understanding, visual Q&A, low-volume tasks

Reliable results depend on representative data

A model is only as good as the images it learns from and meets in daily use. If the training photos were taken in daylight and the factory also runs night shifts under orange lamps, accuracy can drop sharply.

Research shows how sensitive models are. In a 2019 study, UC Berkeley researchers built a fresh ImageNet test set by following the original procedure, and the accuracy of the classifiers they tested fell by 11 to 14 percentage points on images that were only slightly harder. So you need training data that covers real variation, and tests on images from different products, sessions or dates, not near-identical frames from the same video.

What can you do with computer vision?

Nine common capabilities cover most use cases, and many systems combine several of them.

Image classification

Classification gives a whole image one or more labels. Your photo app uses it to group pictures of beaches or receipts, and an insurer might use it to sort damage photos into “repairable” and “write-off”.

Object detection

Detection finds each object in an image, draws a box around it and labels it. A traffic camera that counts cars, bikes and buses per lane relies on detection.

Image segmentation

Segmentation labels individual pixels, so you get an estimated pixel-level boundary for each object instead of a rough box. Video-call apps use it to blur your background. Meta’s SAM 3, released in November 2025, outlines every instance of a concept from a short text prompt such as “yellow school bus”.

Object tracking

Tracking follows the same object across video frames and keeps its identity. Sports broadcasts use it to show how far a player has run.

Optical character recognition (OCR)

OCR extracts printed or handwritten text from images. Parking garages use it to read number plates at the barrier. Document-understanding systems combine that text with layout and other cues to identify fields, such as the total on an invoice.

Pose estimation

Pose estimation locates key points such as shoulders, elbows and knees, and links them into a simple skeleton. Fitness apps use it to count repetitions and check your squat.

Anomaly detection

Anomaly detection learns what normal looks like and flags anything that deviates. That helps when defects are rare and varied, because you do not need an example of every fault. A model trained on thousands of good castings can flag a crack it has never seen.

Depth estimation and 3D perception

Depth estimation works out the distance to each point in a scene, sometimes from a single camera, and builds a 3D picture of the surroundings. Robot vacuums use it to steer around furniture, and phones use it for portrait mode.

Multimodal image understanding

Multimodal models combine vision and language, so you can ask open questions about a picture, such as which switch in a photographed fuse box is off. This is flexible and quick to try, but less predictable than a dedicated model for a narrow, high-volume task.

Where is computer vision used today?

Computer vision is established in several sectors, while other uses are still experimental. This is where each sector stands in October 2026.

Sector	Problem	How computer vision helps	Maturity
Healthcare	Specialists cannot review every scan quickly	Flags possible findings on X-rays, CT scans and retinal images for priority review	Established and regulated
Manufacturing	Manual inspection is slow and inconsistent	Checks every product for defects, missing parts or wrong labels	Established
Retail	Empty shelves and checkout queues cost sales	Monitors shelf stock and recognises loose produce at self-checkout	Established; cashierless stores still limited
Agriculture	Spraying whole fields wastes chemicals	Spots weeds, disease or ripe fruit for targeted spraying and picking	Targeted spraying commercial; robotic harvesting emerging
Transportation	Drivers miss hazards	Detects lanes, vehicles and pedestrians for driver assistance	Assistance established; driverless emerging in limited areas
Logistics	Manual scanning and damage checks slow the flow	Reads labels, measures parcels and records damage on arrival	Established
Accessibility	Blind and low-vision people miss visual information	Describes scenes, reads text aloud and identifies objects	Established and improving fast
Document processing	Staff retype data from invoices and forms	Extracts fields from scans and photos into business systems	Established

Healthcare shows both the scale and the care involved. The US Food and Drug Administration listed more than 1,600 AI-enabled medical devices in September 2026. That count covers all AI, not only computer vision, but imaging tools make up most of it. Each device is authorised for a defined use. Most support clinicians, while a few screen on their own within narrow limits, such as the IDx-DR diabetic eye screening system authorised in 2018.

What are the benefits and limitations of computer vision?

The main benefits are speed, scale, consistency and automation. A system can analyse thousands of images per hour, with the same criteria at 3 a.m. as at 3 p.m., which frees people for cases that need judgement.

The limitations are just as real, because performance depends heavily on conditions:

• Lighting and camera angle: glare, shadows or a camera knocked out of position change what the model sees.
• Occlusion: partly hidden objects are harder to detect and count.
• Unfamiliar environments: a model trained in one warehouse may struggle in another.
• Biased datasets: groups or conditions under-represented in training get worse results.
• False positives and false negatives: every system makes both, so decide which costs more and tune for it.

From a successful demo to reliable deployment

A demo usually runs on clean, carefully chosen images. Real use brings dusty lenses, new product variants, seasonal light and cameras that get moved, so the images drift away from what the model learned. That is why a pilot should run in the real environment for weeks, with monitoring after go-live.

Privacy, security and responsible use

Images of identifiable people can be personal data under the GDPR, and a facial template used to identify someone is biometric data with stricter rules. A camera project needs a legal basis, a clear purpose and notices for the people filmed. A data protection impact assessment is required when the processing is likely to create a high risk, such as large-scale monitoring of public spaces.

Facial recognition and surveillance carry the highest risk. Since 2 February 2025, the EU AI Act has banned building face databases by untargeted scraping, emotion recognition at work and in education (except for medical or safety reasons), and, with narrow exceptions, real-time remote biometric identification by police in public spaces. Certain other biometric uses are classified as high-risk. Our EU AI Act guide covers the current timeline.

Accuracy can also differ between groups. A 2019 study by the US National Institute of Standards and Technology found demographic differences in most of the face recognition algorithms it tested, including higher false positive rates for some groups. Camera feeds and stored footage also need the same security as any other sensitive data.

When do you need human oversight?

You need human oversight whenever an error can harm a person or lead to a costly decision, as in medical findings, hiring, access control and law enforcement. A common design passes predictions above a tested threshold and routes uncertain ones to a person. In the first weeks, reviewers also catch patterns averages hide, such as a model that fails on one product colour.

How do you get started with computer vision?

Start with one specific problem, not with the technology.

• Define the problem and how you will measure success. “Use AI on our production line” is too vague. “Detect scratches longer than 2 mm on at least 95 percent of panels, with at most one false alarm per 200 panels” is a target you can test (a hypothetical example).
• Check your images, permissions and data quality. You need images from the real environment in every condition that occurs, the right to use them, and a plan for who labels them consistently.
• Choose between an existing tool and a model adapted to your data. For common tasks such as OCR, an existing tool is often enough. For your own products or defects, you usually fine-tune a pretrained model. Training a model from scratch is a separate and much rarer decision.
• Run a small pilot, evaluate it honestly and keep monitoring. Test on unseen images from the real setting and compare the results with your current process.

A good beginner project is a classifier you train in your browser. With Google’s free Teachable Machine, you can teach a model to tell recyclable packaging from general waste with your webcam, without code, and see how lighting and background change the results.

A business pilot can stay just as small. Take a hypothetical furniture maker that wants to catch surface defects on panels before packing. It fine-tunes a pretrained detection model on about 2,000 labelled photos from one camera station. For four weeks the system runs alongside the inspectors without rejecting anything, and then the team compares both against the agreed criteria.

The cost of such a project depends on five factors more than on the model: data collection, labelling (often the largest effort), computing power for training and inference, integration with machines or business systems, and maintenance, including retraining when conditions change.

What’s next for computer vision?

Three developments stand out, each partly real today and partly our forecast.

Multimodal systems come first. You can already ask AI assistants about photos, charts and documents, and we expect them to take over more tasks that used to need a dedicated model.

On-device processing is second. Phones already run face unlock locally, without sending images to the cloud. We expect more inspection and safety systems to run on the camera itself, which helps both speed and privacy.

Learning from fewer labels is third. Meta’s DINOv3, released in August 2025, learned visual features from 1.7 billion images without human labels, so it adapts to new tasks with few labelled examples.

The most useful next step is to pick one visual task in your organisation that is repetitive, measurable and costly when it goes wrong. DataNorth’s computer vision team can help you scope that task and test it in a small pilot.

Frequently asked questions about computer vision