Estimate project
← All posts

Face recognition in the new era of AI

This article examines face recognition – its fascinating and controversial uses, and the potential problems that people working in the field deal with every day.

11 min read
Machine LearningComputer VisionFace RecognitionBiometrics

This article takes a look at face recognition: where it is used, what it is made of, which problems the field faces, and a few ways of solving them.

What is Face recognition?

Face recognition is a Computer Vision task in which people are identified by the features of their faces. Biometrics also offers other techniques, such as fingerprints and iris scanning, which spy movies love to show. The big benefit of face recognition is that, as a biometric method, it is less invasive than either of those two.

Face recognition systems, however, may deliver lower quality. Even so, progress in Deep Learning over recent years has let the technology reach a much broader audience, i.e., aceId on Apple or Android smartphones.

Face Recognition and AI Act

Under the plans set out in the EU AI Act:

  • face recognition systems working in real time will be prohibited,
  • using face recognition systems retroactively will be allowed only for prosecuting serious crimes, and only with judicial authorization,
  • systems that categorize people by biometrics (gender, age, race, etc.) will be prohibited,
  • building face recognition datasets from social media or CCTV cameras (which breaches the human right to privacy) will be prohibited.

Applications

Let’s now see where face recognition systems appear in our surroundings. Their use is sometimes contentious, and people disagree on whether it is ethical.

Law enforcement

The use of face recognition by law enforcement comes up often. In films, investigators run algorithms over CCTV footage, compare the results against their “database”, and get a “match”.

This kind of use is frequently disputed: some places (such as San Francisco in 2019) have banned it, whereas others (such as Australia) have welcomed it, particularly since the Covid-19 pandemic.

As noted earlier, the EU AI Act also intends to restrict how face recognition systems may be used.

User authentication

User authentication is another common use of face recognition systems. You may come across them when entering an airport, when trying to get into a research facility with tighter access rules, or when logging into an online course.

At airports, they have sped up passport control and removed human mistakes from the process.

Account Verification

Account verification is yet another use, typically when you have to confirm your identity with a bank, an investment platform or a social network. This step is also known as KYC (“Know your Customer”). Verifying accounts reduces banking fraud, i.e., someone attempting to take out a loan under your name. Fraud on social media can be serious too, since people impersonating influential accounts can sway stock markets or political campaigns.

You may have gone through it when opening an account with Revolut, Bunq or another online bank. Investment platforms such as Degiro or Coinbase rely on comparable services.s. Social networks like Twitter, Facebook and Tinder verify accounts during user onboarding as well. The process tends to look alike: you usually show your face straight to the camera and then turn to the left and to the right. One part of it is a liveness test, in which you have to make particular movements to show you are a real human and not a generated face video or just a photo.

Attendance

Face recognition systems are also used to track class attendance, particularly now that online classes, where cameras are mandatory, have become common. In this setting, a face recognition system can identify students as they enter the exam hall, reducing the chance of cheating or of someone taking an exam for another person. It can also serve as an automatic way to record who was present at a given class and who was absent.

Once more, the use of such systems is disputed: in Sweden, a primary school was fined tens of thousands of dollars for using cameras in a classroom without students knowing about it and agreeing to it.

Problems with face recognition

Having covered potential applications, let’s look at the problems face recognition systems can run into.

The first group is systematic occlusions (see examples in the graphic below). They appear when something covers part of the face (glasses, hair, facial hair such as a moustache or beard, scars, or clothing such as headwear).

Examples of systematic occlusion

A second group of problems is temporary occlusion (i.e. changes in pose, an object or body part covering the face, or environmental occlusions such as shadows cast by light). Examples of these occlusions are shown below.

Examples of temporary occlusion

Because face recognition systems rely on fine details of the face to identify or verify someone correctly, they are sensitive to low-quality images and bad lighting.

Finally, there are problems caused by certain groups being underrepresented. For a long time, face recognition datasets were dominated by middle-aged Caucasian men, so systems performed worse for other races, and poorly for women, black and Asian people, and the elderly. To reduce this, datasets were expanded, and performance for these groups has improved considerably, yet these aspects still deserve attention whenever a dataset is built. This fits well with what the AI Act requires of datasets.

Branches of face recognition

Now we can describe the various branches of face recognition. The first distinction is the type of task we are solving. When the question is “Is this that person?”, we speak of face verification. Face verification is commonly used to limit access or confirm someone’s identity (i.e., at the airport, our face is compared with the face in our passport, and the system checks whether we are the person shown in the passport photo).

When, instead, the question is “Who is that person”, this branch is called face identification. One example is a face recognition system that suggests the person in your Facebook photos is in fact Harry Potter.

A further split depends on whether we want to freeze the database of identities and compare only within it (subject dependent), or build a universal system that works out of the box even for faces the model has never encountered (subject independent).

To picture this more clearly, imagine a system controlling access to Hogwarts. The model was built when Harry and Ron were first-years, and both should be let in. A year after the model was created, Harry and Ron are still let in. The catch appears with Ginny Wesley, Ron’s younger sister. With a subject-dependent model, she will be refused, since her photo was missing from the database at training time. With a subject-independent model, she would be let in as long as her photo is in the current database. And then there are criminals such as Lord Voldemort: he was given access as a student, but we would want to take it away once he became the most wanted person in the Wizarding World.

Unsurprisingly, subject-independent models are used much more widely, because databases and access rights change over time and retraining the model each time is impractical.

Face recognition pipeline

Now let’s go through how a face recognition pipeline works. First, there is camera input in the form of an image or video. Next, we run face detection or tracking (depending on the input) and obtain a bounding box around a candidate face. Then we align the image so that it contains only the selected face, discarding everything else because it could add noise. At that stage we have a clean image showing a single face. We then extract features from this face and compare them against the faces stored in the database. If the incoming face is similar enough to one in the database, we report a match; otherwise, no match is found.

Face processing

One to many face processing examples from Deep Face Recognition: A Survey

Many to one face processing example from Deep Face Recognition: A Survey

Augmentations known as “one-to-many” and “many-to-one” are one way of making the system more robust. Their goal is to counter systematic and temporary occlusions so that all the key facial features are represented and modelled. The “One-to-many” algorithm generates many face images from one image. As a result, the model can get all the specialized information from each image, and we can also feed it many more faces without adding.

“Many-to-one” methods build a single super-face out of several faces seen from different angles. As mentioned earlier, this approach makes the system even more robust to occlusions than “one-to-many” augmentation.

The classical approach to biometrics

Let’s now look at how biometrics were traditionally used to recognize individual people by the features of their faces.

Typical facial biometrics include:

  • The distance separating the eyes.
  • The distance from forehead to chin.
  • The distance separating the nose and the mouth.
  • How deep the eye sockets are.
  • How the cheekbones are shaped.
  • The outline of the lips, ears and chin.

Example of facial landmarks presented in Harry Potter

As you already know, the problems that come up in face recognition make classical methods inaccurate, because some of these characteristics can change over time.

Machine learning solutions

Datasets

Many face recognition datasets have been gathered, each with a different goal. I will describe the three best known: VGGFace, MSCeleb-1M and Megaface.

VGGFace

This dataset contains faces in a variety of poses and at a variety of ages, with an emphasis on making face recognition systems robust to changing conditions. It holds 3.3M photos of 9k unique identities.

MSCeleb-1M

This dataset includes roughly 4 million instances covering about 87k unique ids. As a supplement to MSCelb-1M, the authors released a dedicated dataset, AsianCeleb, with nearly 94k unique identities and 3 million instances. The initial version suffered from rather noisy labels and dataset biases. The authors therefore cleaned up the labels, dropped low-quality examples and rebalanced the datasets, paying more attention to a better balance of race, gender and age.

Megaface

This is likely among the biggest face recognition datasets available, containing over 4.7 million images of 670k unique identities. It addresses a different problem: how to identify faces across a large number of identities when each identity has only a few images.

Approaches

Face recognition approaches throughout the years.

Holistic approaches

Among machine learning approaches, we can trace the order in which they emerged. The first were holistic approaches, which treated every available pixel as a feature and classified using all of those features.

Eigenfaces, introduced in 1991, applies Principal Component Analysis to describe a face with a smaller number of features. The name’s first part reflects the fact that PCA relies on eigenvectors and eigenvalues computed over all faces in the trainset, so that, approximately, “any new face can be expressed as a linear combination of the faces in the train set”.

Fisherface followed Eigenfaces, replacing PCA with Linear Discriminant Analysis. LDA is a dimensionality reduction technique chosen for keeping classes separable, in contrast to PCA, whose mechanism ignores labels.

Approaches based on local handcrafted features

A different branch relies on local handcrafted features, such as Gabor filters, which carry out spatial pooling or compute histograms. A classifier can then be applied to these specialized features to decide whether the user is the person they claim to be.

Gabor filters are a technique frequently used for texture analysis; in practice, they are a Gaussian kernel function modulated by a sinusoidal signal.

Deep learning approaches

DeepFace architecture from the original paper

‍DeepFace (released in 2014) was among the earliest deep learning approaches. It applies 3D face alignment (based on 67 fiducial points), frontalizes the face (with a classical Computer Vision method) and then performs face recognition on 152x152 RGB images with an 8-layer network (made of convolutional, locally connected and fully connected layers); the architecture was strongly influenced by AlexNet.

DeepID (released in 2015) took inspiration from VGGNet and GoogleNet, combining stacked convolutions with inception layers. It relies on contrastive loss, which reduces the distance between the features produced for positive pairs (faces that match) and increases the distance for negative pairs (faces that do not match).

FaceNet (released in 2015) likewise works in hyperspace (Euclidean space) and draws on GoogleNet. It uses triplet loss, which always takes three examples together (anchor, positive and negative) and at the same time shrinks the distance to the positive example and enlarges the distance to the negative one.

Comparison of losses, Triple loss, Tuplet loss, and ArcFace loss

A newer approach, ArcFace (released in 2018), built on the ResNet-100 backbone, relies on a purpose-designed loss. Whereas triplet loss maximized distances between individual examples, ArcFace loss applies more sophisticated measures and works with the centres of the positive and negative populations. These more refined loss computations appear to make models more robust to noisy and difficult faces.

Current SOTA results build on the approaches described above, with all available hyperparameters fine-tuned.

Conclusions

This article examined face recognition, its fascinating and contentious applications (particularly given the regulations now on the way) and the potential problems that people working in the field deal with every day. We also went through most branches of face recognition (including how face identification – “Who is in the photo?” – differs from face verification – “Are you really who you say you are?”), what the classical and machine learning approaches looked like, which clever tricks they relied on, and where they stand today. I hope this overview has given you a better understanding of face recognition.