How Facial Recognition Technology Works

By: Kevin Bonsor , Ryan Johnson & Zach Taras  | 
Even smartphones make use of facial recognition technology. Witthaya Prasongsin / Getty Images

Anyone who has seen the TV show "Las Vegas" has seen facial recognition technology in action. In any given episode, the security department at the fictional Montecito Hotel and Casino uses its video surveillance system to pull an image of a card counter, thief or blacklisted individual.

It then runs that image through the database to find a match and identify the person. By the end of the hour, all bad guys are escorted from the casino or thrown in jail. But what looks so easy on TV doesn't always translate as well in the real world.


Humans have always had the innate ability to recognize and distinguish between faces, yet computers only recently have shown the same ability. In the mid 1960s, scientists began using computers to recognize human faces. Since then, facial recognition software has come a long way.

Here, we will look at the history of face recognition technology, the changes that are being made to enhance their capabilities and how governments and private companies use (or plan to use) them.


Facial Recognition Technology

FaceIt software compares the faceprint with other images in the database.
Photo © Identix Inc.

Identix®, a company based in Minnesota, was one of many early developers of facial recognition systems. Its software, FaceIt®, could pick someone's face out of a crowd, extract the face from the rest of the scene and compare it to a database of stored images.

In order for this software to work, it had to know how to differentiate between a basic face and the rest of the background. A face recognition system is based on the ability to recognize a face and then measure the various features of the face.


Every face has numerous, distinguishable landmarks: the different peaks and valleys that make up facial features. FaceIt defined these landmarks as nodal points. Each human face has approximately 80 nodal points. Some of the ones measured by the software were:

  • Distance between the eyes
  • Width of the nose
  • Depth of the eye sockets
  • Shape of the cheekbones
  • Length of the jaw line

FaceIt used these measured nodal points to create a numerical code, called a faceprint, representing the face in the database.

In the past, face recognition systems relied on a 2D image to compare or identify another 2D image from the database. To be effective and accurate, the image captured needed to be of a face that was looking almost directly at the camera, with little variance of light or facial expression from the image in the database.

This created quite a problem.

In most instances the images were not taken in a controlled environment. Even the smallest changes in light or orientation could reduce the effectiveness of the system, so they couldn't be matched to any face in the database, leading to a high rate of failure. In the next section, we will look at ways to correct the problem.


3D Facial Recognition

A relatively new trend in facial recognition technology uses a 3D model, which claims to provide more accuracy.

Capturing a real-time 3D image of a person's facial surface, 3D facial recognition identifies the subject by using distinctive features of the face — where rigid tissue and bone is most apparent, such as the curves of the eye socket, nose and chin. These areas are all unique and don't change significantly over time.


Using depth and an axis of measurement that is not affected by lighting, 3D facial recognition can even be used in darkness and has the ability to recognize a subject at different view angles with the potential to recognize up to 90 degrees (a face in profile).

Using the 3D software, the system goes through a series of steps to verify the identity of an individual.


Acquiring an image can be accomplished by digitally scanning an existing photograph (2D) or by using a video image to acquire a live picture of a subject (3D).


Once it detects a face, the facial recognition system determines the head's position, size and pose. As stated earlier, the subject has the potential to be recognized up to 90 degrees, while with 2D, the head must be turned at least 35 degrees toward the camera.


The system then measures the curves of the face on a sub-millimeter (or microwave) scale and creates a template.


The system translates the template into a unique code. This coding gives each template a set of numbers to represent the features on a subject's face.


If the image is 3D and the database contains 3D images, then matching will take place without any changes being made to the image. However, there is a challenge currently facing databases that are still in 2D images.

3D provides a live, moving variable subject being compared to a flat, stable image. New facial recognition technology is addressing this challenge. When a 3D image is taken, different points (usually three) are identified.

For example, the outside of the eye, the inside of the eye and the tip of the nose will be pulled out and measured. Once those measurements are in place, an algorithm (a step-by-step procedure) will be applied to the image to convert it to a 2D image.

After conversion, the software will then compare the image with the 2D images in the database to find a potential match.

Verification or Identification

In verification, an image is matched to only one image in the database (1:1). For example, an image taken of a subject may be matched to an image in the Department of Motor Vehicles database to verify the subject is who he says he is.

If identification is the goal, then the image is compared to all images in the database, resulting in a score for each potential match (1:N). In this instance, you may take an image and compare it to a database of mug shots to identify who the subject is.

Next, we'll look at how skin biometrics can help verify matches in facial recognition technology.


Biometric Facial Recognition

The surface texture analysis (STA) algorithm operates on the top percentage of results as determined by the local feature analysis. STA creates a skinprint and performs either a 1:1 or 1:N match depending on whether you're looking for verification or identification.

The surface texture analysis (STA) algorithm operates on the top percentage of results as determined by the local feature analysis. STA creates a skinprint and performs either a 1:1 or 1:N match, depending on whether you're looking for verification or identification.

The image may not always be verified or identified in facial recognition alone, so Identix® created another product to help with precision. FaceIt®Argus used skin biometrics, the uniqueness of skin texture, to yield even more accurate results.


The process, called Surface Texture Analysis, works much the same way facial recognition does. A picture is taken of a patch of skin, called a skinprint. That patch is then broken up into smaller blocks.

Using facial recognition algorithms to turn the patch into a mathematical, measurable space, the system would then distinguish any lines, pores and the actual skin texture. It could identify differences between identical twins, which had not been possible using facial recognition software alone.

FaceIt used three different templates to confirm or identify the subject: vector, local feature analysis and surface texture analysis.

  • The vector template is very small and is used for rapid searching over the entire database primarily for one-to-many searching.
  • The local feature analysis (LFA) template performs a secondary search of ordered matches following the vector template.
  • The surface texture analysis (STA) is the largest of the three. It performs a final pass after the LFA template search, relying on the skin features in the image, which contains the most detailed information.

By combining all three templates, FaceIt® offered an advantage over other facial recognition systems.

It was relatively insensitive to changes in expression — including blinking, frowning or smiling — and had the ability to compensate for mustache or beard growth and the appearance of eyeglasses. The system was also uniform with respect to race and gender.

Poor lighting can make it more difficult for facial recognition software to verify or identify someone.
Photo © Identix Inc.

However, it was not a perfect system. There were some factors that could get in the way of recognition, including:

  • Significant glare on eyeglasses or wearing sunglasses
  • Long hair obscuring the central part of the face
  • Poor lighting that would cause the face to be over- or under-exposed
  • Lack of resolution (image was taken too far away)

While most facial recognition systems work the same way FaceIt does, there are some variations. For example:

  • A company called Animetrics, Inc. has a product called FACEngine ID® SetLight that can correct lighting conditions that cannot normally be used, reducing the risk of false matches.
  • Sensible Vision, Inc. has a product that can secure a computer using facial recognition. The computer will only power on and stay accessible as long as the correct user is in front of the screen. Once the user moves out of the line of sight, the computer is automatically secured from other users.

Due to these strides in technology, modern facial recognition systems are more widely used than they were just a few years ago. In the next section, we'll look at where and how they are being used and what's in store for the future.


Facial Recognition Systems Uses

Jim Williams, former director of US-VISIT, former Secretary Tom Ridge and former Commissioner Robert Bonner launch US-VISIT in Atlanta, Georgia.
Photo courtesy U.S. Department of Homeland Security

In the past, the primary users of facial recognition technology have been law enforcement agencies, who used the system to capture random faces in crowds. Now, other government agencies are getting in on the action.


In the early 2000s, the U.S. government began a program called US-VISIT (United States Visitor and Immigrant Status Indicator Technology), aimed at foreign travelers gaining entry to the United States. When a foreign traveler receives his visa, they submit fingerprints and have their photograph taken.


The fingerprints and photograph are checked against a database of known criminals and suspected terrorists. When the traveler arrives in the United States at the port of entry, those same fingerprints and photographs will be used to verify that the person who received the visa is the same person attempting to gain entry.


As facial recognition systems become less expensive, their use becomes more widespread. They are now compatible with cameras and computers that are already in use by banks and airports.

Time Tracking

A4Vision, a creator of facial recognition technology, is currently marketing a system that will keep track of employees' time and attendance. Their Web site states that it will prohibit "buddy punching," which will cut down on security risks and decreased productivity.


Other potential applications include ATM and check-cashing security. The software is able to quickly verify a customer's face. After a customer consents, the ATM or check-cashing kiosk captures a digital image of him. The software then generates a faceprint of the photograph to protect customers against identity theft and fraudulent transactions.

By using the facial recognition software, there's no need for a picture ID, bankcard or personal identification number (PIN) to verify a customer's identity. This way businesses can prevent fraud from occurring — or, if it does, law enforcement agencies can swiftly respond.

But, of course, there's also a risk of false negatives in this scenario: What if the software doesn't recognize you trying to access your own account? No one wants to be locked out of accessing their own cash.


Concerns About Privacy in Facial Recognition Systems

While all the examples above work with the permission of the individual, sometimes the use of facial recognition is done without a person's consent or knowledge. Opponents of the systems note that while the technology does provide security in some instances, it is not enough to justify overriding a sense of liberty and freedom.

Many feel that privacy infringement is too great with the use of these systems, but their concerns don't end there. They also point out the risk involved with identity theft. Even facial recognition corporations admit that the more use the technology gets, the higher the likelihood of identity theft or fraud.


As with many developing technologies, the incredible potential of facial recognition comes with some drawbacks, but manufacturers are striving to enhance the usability and accuracy of the systems.

Lots More Information

  • ACLU. "Q & A on Face-Recognition." September 2, 2003.
  • Beach, Cindie. "The Face of Security." Military Information Technology. December 22, 2005.
  • Electronic Privacy Information Center. "Face Recognition." January 19, 2006.
  • Gupta, A. "Biometrics: Eyes Don't Lie." DataQuest. October 14, 2006. /2006/106101402.asp
  • Identix. "FaceIt® G6 Frequently Asked Technical Questions"
  • Kimmel, Ron and Guillermo Sapiro. "The Mathematics of Face Recognition." SIAM. April 30, 2003.
  • Moyer, Paula. "New Technology Targets Skin as Valid Biometric Identification, Security." Dermatology Times. June 1, 2004.
  • The University of York, Department of Computer Science. "Biometrics: Face Recognition."
  • Woodward, John D, et al. "Biometrics: A Look at Facial Recognition." RAND Public Safety and Justice. 2003.