Riiven Threads
Facial Recognition
The Skewed Mirror
The story you know
Facial recognition is an AI breakthrough from the last decade: someone trained a neural network on millions of photos and computers suddenly learned to see faces.
What the record shows
Compact face representations date to eigenface projection methods in the late 1980s, and the neural networks of 2015 were judged against standardized testing methods built over the preceding decades. Only later demographic analysis of benchmark datasets showed that their reported accuracy hid large skews in who was measured.
A phone lies face-down on a café table until its owner picks it up and looks at it. In under a second it matches that face against a stored set of numbers and opens. The match feels personal, but the numbers come from decisions made in labs decades apart. Someone had to shrink a face into a few dozen values. Someone had to teach a network what makes two photos the same person. Someone had to agree on how to score the mistakes, and on whose faces to score them. By 2015 one system was matching faces on a hard public benchmark 99.63% of the time. That figure was true. The question this piece follows is the one the figure left out: accurate for whom, and who checked?
- 40eigenfaces
- Enough blurred template faces to classify 150 test images with a single error in 1991.
- 99.63%
- FaceNet's 2015 accuracy on Labeled Faces in the Wild, up from 95.17% for handcrafted features.
- 0.5%
- Best NIST-tested algorithms wrongly rejected true matches less often than this at strict settings.
- 34.7%
- Peak error for darker-skinned women in Gender Shades, versus 0.8% for lighter-skinned men.
How each idea was handed down
Facial Recognition inherited its parts in sequence. Each field handed its result to the next.
The inheritance, in order
- 1987 Principal Component Analysis and Linear Subspace Methodshanded down ↓
- 2008 Biometric Performance Evaluation and Benchmarking Metrologyhanded down ↓
- 2015 Convolutional Neural Networks and Metric Learninghanded down ↓
- 2018 Population Statistics and Dataset Demographics
Follow the inheritance, link by link.
Each link hands its result to the one after it. The chart above places them by date.
Keystone
The blurry faces every match still descends from
Ghostly averaged faces, blurred and nameless, were the first useful face keys. Every real face was a mix of them.
Picture a few dozen ghostly, blurred faces, each a pattern of light and shadow that no real person wears. In 1987 Lawrence Sirovich and Michael Kirby showed that real faces could be rebuilt by mixing such patterns. Their tool was principal component analysis, a method that finds the few directions along which a pile of images varies most. Matthew A. Turk and Alex Pentland turned that into a recognizer in 1991 and called the patterns eigenfaces. A face became a short list of mixing weights instead of thousands of pixels. With 40 eigenfaces, the approach made 1 error in 150 images, a 99.33% rate. Every face key since, including the one in your phone, stores numbers like these rather than a picture.
Without this link
PCA represents face images in a lower-dimensional space of eigenfaces, so recognition compares compact projection weights instead of every pixel. Without this dimensionality reduction, that compact representation would be lost, although the sources do not establish that matching across millions of faces would have become computationally intractable.
Using 40 eigenfaces, the method classified faces correctly 99.33% of the time, with 1 error in 150 images.
How we know
Source: Eigenfaces for Recognition (1991) · tier1
Once a face was a list of numbers, anyone could build a matcher, so the field needed one fair way to judge them.
Agreeing on how to count a mistake
Every face matcher makes two kinds of mistakes. For years, no two labs counted them the same way.
A passport gate can fail by turning away its rightful owner or by letting in someone else. Metrology, the science of measurement, gave those two failures fixed names and fixed thresholds so rival systems could be compared. A team including Gary B. Huang and Eric Learned-Miller assembled Labeled Faces in the Wild, a public set of everyday photos for scoring face matching. NIST later tested vendors at strict settings on mugshot photos. At a false match rate of 0.00001, one stranger accepted per 100,000 tries, the best algorithms wrongly rejected the true person less than 0.5% of the time.
Without this link
Without standardized biometric testing, developers would lack comparable error-rate measurements at stated thresholds, which would make it hard to compare algorithms or predict how they behave once deployed. NIST's FRVT results show why operating conditions matter: the best algorithms had false non-match rates below 0.5% at a false match rate of 0.00001 in one-to-one mugshot verification.
At a false match rate of 0.00001, NIST's best mugshot verifiers rejected true matches less than 0.5% of the time.
How we know
Source: Face Recognition Vendor Test (FRVT), Part 3: Demographic Effects (2019) · tier1
With a shared scoreboard in place, a better matcher could prove it was better, and in 2015 one did.
Teaching a network what same person means
Three photos at a time: two of one person, one of a stranger. FaceNet learned from that comparison, repeated endlessly.
Show a network three photos: a person, the same person again, and a stranger. In 2015 Florian Schroff, Dmitry Kalenichenko, and James Philbin trained a convolutional neural network, a stack of learned filters that spot edges and then shapes, on exactly that setup. Their triplet loss rewarded the network whenever the matching pair landed closer together than the mismatch. The output was an embedding, a compact list of coordinates for each face, much like eigenface weights but learned rather than computed. On Labeled Faces in the Wild, FaceNet reached 99.63% accuracy, against 95.17% for hand-designed texture codes known as LBP.
Without this link
CNNs learned face representations directly from image data, and FaceNet's triplet loss trained a compact embedding that placed matching faces closer than non-matching ones. Without these learned representations and metric learning, reported LFW performance would have stayed substantially lower: handcrafted LBP features were reported at 95.17% accuracy, compared with FaceNet's 99.63%.
FaceNet's triplet-trained embedding scored 99.63% accuracy on LFW, a benchmark of unposed, everyday face photos.
How we know
Source: FaceNet: A Unified Embedding for Face Recognition and Clustering (2015) · tier1
The scoreboard said the problem was nearly solved, but it never reported who was in the photos being scored.
Asking who the benchmark had counted
A test score near perfect can hide a group doing far worse, if that group barely appears in the test.
Sort the photos in the standard face benchmarks by who is in them, and the pile leans heavily toward lighter-skinned faces. In 2018 Joy Buolamwini and Timnit Gebru built a test set balanced by gender and skin type, then ran commercial gender classification systems on it. These products guessed whether a face was male or female, a narrower task than identifying someone. Error rates reached 34.7% for darker-skinned women, while lighter-skinned men never saw more than 0.8%. A single aggregate score had averaged that gap out of sight.
Without this link
Without demographic characterization of benchmark datasets, their heavy skew toward lighter-skinned subjects could go unnoticed, and aggregate evaluations could conceal much higher error rates for darker-skinned women. Gender Shades measured this disparity with a benchmark balanced by gender and skin type, showing why subgroup representation matters when evaluating facial analysis systems.
Gender Shades found errors up to 34.7% for darker-skinned women, against a maximum of 0.8% for lighter-skinned men.
How we know
Source: Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification (2018) · tier1
Watch
A visual companion to the fields above.
Takeaway
Line the four links up and the face key on a phone starts to make sense. Eigenfaces turned a face into a short list of numbers. Shared error metrics made those lists comparable across labs and vendors. FaceNet learned better lists, good enough to post 99.63% on a public benchmark in 2015. Each link was real progress, and each one inherited the photos the previous one had been tested on. What the finished chain produced was a key that works astonishingly well on average and unevenly by face, along with, after 2018, the tools to see exactly where it fails. Gender Shades did not make any system worse. It changed what counts as a result, because a single accuracy number can no longer stand alone. NIST's 2019 demographic-effects report now measures those splits at vendor scale. The unresolved part is the photo sets themselves: who rebuilds them, and whether the phone in your hand was trained before or after anyone asked who was in them.
References
- Eigenfaces for Recognition (1991) tier1
Matthew A. Turk and Alex Pentland, Journal of Cognitive Neuroscience, 1991
- Face Recognition Vendor Test (FRVT), Part 3: Demographic Effects (2019) tier1
Patrick Grother, Mei Ngan, Kayee Hanaoka, NISTIR 8280, 2019
- FaceNet: A Unified Embedding for Face Recognition and Clustering (2015) tier1
Florian Schroff, Dmitry Kalenichenko, James Philbin, CVPR, 2015
- Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification (2018) tier1
Joy Buolamwini and Timnit Gebru, Proceedings of Machine Learning Research, 2018
