Computer vision is the branch of AI focused on getting machines to extract useful information from images and video: what’s in a photo, where it is, and what shape it takes. It’s the technology behind everything from photo tagging and medical imaging diagnostics to self-driving car perception stacks and the camera app on your phone that blurs the background automatically. Three tasks, in increasing order of detail, cover most of what the field does: classification, detection, and segmentation.
Image Classification
Classification is the simplest task: given an image, assign it one label from a fixed set of categories, “cat,” “dog,” “not a dog,” and so on. The field’s modern era is usually dated to 2012, when Krizhevsky, Sutskever, and Hinton’s AlexNet (Krizhevsky et al., 2012) won the ImageNet Large Scale Visual Recognition Challenge by a wide margin, cutting the top-5 error rate from 26.2% to 15.3% using an eight-layer convolutional neural network (CNN) trained on GPUs. That result is widely credited with kicking off the broader deep learning boom, not just in vision.
For over a decade after that, the standard architecture for classification was the CNN, which processes an image through a stack of learned filters (convolutions) that each detect increasingly abstract patterns, edges and textures in early layers, shapes and object parts in deeper ones. The reason CNNs dominated vision for so long is a property called translation invariance: a convolutional filter that detects an edge works the same way whether that edge appears in the top-left or bottom-right of the image, because the same filter weights are applied everywhere via sliding a small window across the whole image. That’s dramatically more parameter-efficient than a fully-connected network trying to learn a separate weight for every pixel position.
A recurring problem with early deep CNNs was that simply stacking more layers made training worse, not better, past a certain depth, not from overfitting, but because gradients had to survive backpropagating through too many sequential layers. ResNet (He et al., 2015) solved this with residual connections: shortcut paths that let a layer’s input skip ahead and be added directly to its output, giving gradients a direct route back to earlier layers. That single idea made it practical to train networks with over a hundred layers, and ResNet-based architectures are still widely used as lightweight baselines today.
Vision Transformers (ViT), which apply the same self-attention mechanism used in language models to image patches, were introduced in 2020 (Dosovitskiy et al.) and now match or beat CNNs on most large-scale classification benchmarks, provided there’s enough training data. ViTs give up the built-in translation invariance of convolutions and instead have to learn spatial relationships from data, which is why they tend to need more training examples to reach the same accuracy, but scale better once you have that data. For a deeper look at how the attention mechanism itself works, see our transformers and attention mechanisms explainer.
Object Detection
Detection goes a step further than classification: instead of one label for the whole image, the model has to find every object present and draw a bounding box around each one, along with a label and confidence score. This is the technology behind things like automatic photo tagging of multiple people in a group shot, or a self-driving car identifying every pedestrian, vehicle, and traffic sign in its field of view simultaneously.
Two broad approaches have dominated detection. Two-stage detectors like Faster R-CNN (Ren et al., 2015) first use a Region Proposal Network to suggest candidate regions that might contain an object, then classify and refine each region separately, trading speed for accuracy. Single-stage detectors like the YOLO (“You Only Look Once”) family (Redmon et al., 2015) predict all boxes and classes in a single pass over the image, running the original version at 45 frames per second on the hardware available at the time, which is dramatically faster and makes real-time detection on video feasible, at some cost to accuracy on small or overlapping objects.
| Approach | How it works | Strength | Tradeoff |
|---|---|---|---|
| Two-stage (Faster R-CNN) | Propose candidate regions, then classify/refine each | Higher accuracy, especially on small/overlapping objects | Slower, harder to run in real time |
| Single-stage (YOLO family) | Predict all boxes and classes in one pass | Real-time speed, simpler pipeline | More localization errors on small or crowded objects |
The tradeoff between these two families, roughly, accuracy versus latency, is still the central design decision when choosing a detection model for a given application, whether that’s a security camera system that needs to run on cheap edge hardware or an offline pipeline processing satellite imagery where latency doesn’t matter.
Semantic and Instance Segmentation
Segmentation pushes detail to the pixel level: instead of a bounding box, the model assigns a label to every individual pixel in the image. Semantic segmentation labels pixels by class, every pixel belonging to any car gets the label “car,” without distinguishing between individual cars. Instance segmentation goes further and separates individual objects of the same class, so each of three cars in a photo gets its own distinct mask.
This level of detail matters wherever the exact boundary of an object is the point: medical imaging (outlining a tumor precisely rather than just detecting “there is a tumor somewhere in this region”), autonomous driving (knowing the exact drivable road surface, not just “there is road somewhere in this image”), and background removal tools. Mask R-CNN (He et al., 2017) extends Faster R-CNN with an extra branch that predicts a pixel mask for each detected object, adding only a small amount of overhead to the existing detection pipeline.
More recently, Meta AI’s Segment Anything Model (SAM) (Kirillov et al., 2023) shifted the segmentation task itself: rather than training a model on a fixed set of categories, SAM is trained to segment arbitrary objects in an image given just a point, box, or rough prompt, using a dataset of over one billion masks across 11 million images. That makes it usable zero-shot on object categories it never saw labeled examples of during training, turning a task that used to require a custom-trained model per use case into something closer to a general-purpose tool. Its 2024 successor, SAM 2 (Ravi et al.), extends the same idea to video: a streaming memory module lets it track a selected object across frames, even through brief occlusion, and Meta reports it’s roughly 6x faster than the original SAM on image segmentation while needing 3x fewer user interactions for comparable video segmentation accuracy.
Evaluation Metrics: How These Models Are Actually Scored
Benchmark tables in computer vision papers almost always report a metric called mean Average Precision (mAP), and it’s worth knowing what it actually measures. For any detection or segmentation model, a predicted box or mask only “counts” as correct if it overlaps sufficiently with the ground truth, measured via Intersection over Union (IoU), the ratio of the overlapping area to the total combined area of the two boxes. Average Precision summarizes how well a model balances precision (not producing false detections) and recall (not missing real objects) at a given IoU threshold, and mAP averages that score across all object categories.
The most widely used benchmark, Microsoft’s COCO dataset (328,000 images, 80 object categories, 1.5 million labeled instances), goes a step further: instead of scoring at a single fixed IoU threshold, its standard mAP metric averages performance across ten thresholds from 0.5 to 0.95, which rewards models for producing tightly-fitting boxes and masks rather than just roughly-correct ones. When a computer vision paper or a multimodal model’s technical report cites a “COCO mAP” number, this is the calculation behind it, and it’s the main way detection and segmentation models get compared apples-to-apples across different research groups.
Why This Keeps Showing Up in AI News
Vision capability is quietly one of the biggest differentiators between current-generation multimodal models. When a new frontier LLM release claims improved “vision” or “multimodal” capability, it’s usually some combination of better image classification and description (captioning what’s in an image), visual question answering (answering specific questions about image content), and increasingly, the ability to read and reason about screenshots, charts, and documents, not fundamentally new capabilities so much as the classification/detection/segmentation building blocks above getting integrated directly into a language model’s input pipeline via a vision encoder, often itself a ViT.
Practical Takeaways
- Classification is the cheapest and most mature of the three tasks; if a product only needs to answer “what is this,” it doesn’t need the complexity of detection or segmentation.
- Detection is the right choice when you need to count or locate objects, not just confirm they’re present, and single-stage detectors are almost always the right default unless accuracy on small/overlapping objects is critical.
- Segmentation is expensive to label and train for, which is exactly why prompt-based segmentation models like SAM, which need no per-category training data, have been such a significant practical shift: they turn a task that used to require a custom-trained model per use case into something usable out of the box.
- The CNN-versus-ViT choice increasingly comes down to data volume, not raw capability: with a large enough training set, ViT-based approaches tend to win; with a limited or specialized dataset, a CNN backbone (often a ResNet variant) can still be the more practical, faster-to-train choice.
