You may have wondered what is computer vision when your phone sorted photos by face. It is how machines learn to understand images and video, mostly by training neural networks on millions of labelled examples.
Computer vision is one branch of AI, so if you want the bigger picture, start with what is artificial intelligence.
| Item | Value |
|---|---|
| Images in the ILSVRC-2010 contest set | 1.2 million images in 1000 classes |
| Size of the 2012-era SuperVision network | 60 million parameters, 650,000 neurons |
| Winning top-5 error in ILSVRC-2012 | 15.3% (second-best entry: 26.2%) |
| Size of the ImageNet dataset | A little more than 14 million images |
| Years the ImageNet challenge ran | 2010 to 2017 |
These figures come from the ImageNet classification paper by Krizhevsky, Sutskever and Hinton and a Machine Learning Mastery overview of the ImageNet challenge. Figures can change, so check the sources before you rely on them.
What does a computer vision system actually do?
Computer vision covers tasks that take an image or video as input, according to the ML Compiled reference. That reference lists image captioning, image classification, image segmentation, image-to-image translation and object detection as core examples.
Three of those tasks do most of the everyday work:
- Image classification: predict the classes of objects present in an image.
- Object detection: classify the image and also draw a bounding box around each object present.
- Image segmentation: partition an object into meaningful parts with associated labels, which is sometimes called per-pixel classification.
The ImageNet challenge shows how demanding this can be. According to MIT Technology Review, contestants faced 1,000 different categories of objects ranging from abacus to zucchini.
Face recognition is a special case. The normal pipeline starts with face detection, then face alignment using landmarks like the nose, eyes and mouth, then feature extraction and similarity calculation, as described by ML Compiled.
Why learning replaced hand-built vision rules
For years, most researchers believed a vision system needed to be carefully hand-designed using a detailed understanding of the task. The authors of the ImageNet paper argued the opposite: with enough computation and enough data, learning beats programming for complicated tasks.
The core ideas were not new. Researchers got excited about backpropagation in the 1980s, yet the same paper says deep networks needed far more labeled data and hugely more computation.
Data came first. Until recently, labeled image datasets were on the order of tens of thousands of images, as the paper notes.
The ImageNet dataset changed that scale. A Machine Learning Mastery overview reports a little more than 14 million images in a little more than 21 thousand classes, annotated by humans using crowdsourcing platforms such as Amazon’s Mechanical Turk.
What happened at the 2012 ImageNet challenge?
The ImageNet Large Scale Visual Recognition Challenge was an annual competition that ran between 2010 and 2017, built on subsets of the ImageNet dataset, per Machine Learning Mastery. Its goal was to promote better computer vision techniques and to benchmark the state of the art.
In 2012, a team from the University of Toronto entered an algorithm called SuperVision, which MIT Technology Review says swept the floor with the opposition.
In the team’s paper, a variant of the model achieved a winning top-5 test error rate of 15.3% in ILSVRC-2012, compared to 26.2% for the second-best entry. The authors say it almost halved the error rate for recognizing objects in natural images.
How the winning network was built
SuperVision was a convolutional neural network. Per MIT Technology Review, such networks consist of several layers of small neuron collections that each look at small portions of an image, and each layer repeats the process on the representation from the one before.
The paper describes the design in these terms:
- Five convolutional layers, some followed by max-pooling layers, and three fully connected layers.
- 60 million parameters and 650,000 neurons.
- A very efficient GPU implementation of the convolution operation to make training faster.
- A method called dropout to reduce overfitting in the fully connected layers.
Progress continued quickly. As of that September 9, 2014 report, an algorithm called GoogLeNet had reached an error rate of only 6.7 percent. The same report quotes the challenge’s reviewers saying a trained human annotator could still outperform GoogLeNet by approximately 1.7%.
Where computer vision still struggles
Real scenes are messy, and ML Compiled lists the standing challenges:
- Parts of the object may be obscured.
- Photos can be taken at different angles.
- Lighting conditions differ in direction, amount and number of light sources.
- Objects belonging to one class can come in a variety of forms.
As of the 2014 review, the best algorithms still struggled with small or thin objects, such as a small ant on a stem of a flower, and with images distorted by filters. The same MIT Technology Review report adds that humans are weaker at fine-grained categories, such as the particular species of dog or bird.
Faces add their own problems: changes in facial hair, glasses that may not always be worn, and people aging over time, according to ML Compiled.
One common remedy is data augmentation. The training images are randomly altered to improve generalization, for example by flipping an image with probability 0.5, as the same page explains.
Datasets to know, and neighbouring fields
Beyond ImageNet, a few benchmark datasets come up again and again in the ML Compiled reference:
- CIFAR-10: 60000 32×32 colour images in 10 classes (ML Compiled).
- MNIST: 70000 28×28 pixel grayscale images of handwritten digits in 10 classes (ML Compiled).
- SVHN: Street View House Numbers, with over 600,000 images of the numbers 0-9 (ML Compiled).
- COCO: Common Objects in COntext, a dataset for image recognition, segmentation and captioning (ML Compiled).
Vision is only one corner of AI. If your interest is language rather than pictures, read what is natural language processing.
If you are curious about systems that create images rather than analyse them, see what is generative ai.
Frequently asked questions
What is computer vision in simple terms?
Show answer
It is the area of AI where a computer takes an image or video as input and produces an answer, such as a label, a box around an object or a caption.
Is computer vision the same as deep learning?
Show answer
No. Computer vision is the set of tasks, while deep convolutional neural networks are one technique for solving them. That technique became dominant after the 2012 ImageNet result described in the MIT Technology Review report.
Can computers see better than people?
Show answer
It depends on the task. As of the September 2014 review, a trained human annotator still beat GoogLeNet by approximately 1.7%, but machines handled fine-grained categories such as dog or bird species with ease, according to MIT Technology Review.
What is ImageNet?
Show answer
It is a large dataset of human-annotated photographs designed for computer vision research. A Machine Learning Mastery overview reports a little more than 14 million images in the dataset.
