How to Spot a Liar: Experts Uncover the Signs of Deception—Can you See Them?

Can you spot a liar by looking at their facial quirks alone? That's the question one team of researchers in the U.S. is trying to answer, and the results look promising. Using a massive "deception dataset" containing 1.3 million frames of expressions from 302 individuals split into pairs, experts are using data science and machine learning technology to separate the fibs from the truth.

Researchers from New York's University of Rochester are using an online crowdsourcing framework called Automated Dyadic Data Recorder (ADDR) to study facial and verbal cues. One aim of the project, the team said in a release, is to help "minimize instances of racial and ethnic profiling" by authorities who frequently question subjects, including the U.S. Transportation Security Administration (TSA).

They discovered that the so-called Duchenne smile—a smile that extends to the muscles of the eye—is most frequently associated with lying. In comparison, witnesses who are responding honestly will often contract their eyes while trying to truthfully recall the information they are being asked. Anyone who has seen 1998's The Negotiator action flick will already know this, of course. We see you, Niebaum.

"A lot of times people tend to look a certain way or show some kind of facial expression when they're remembering things," commented Tay Sen, a PhD student working in the lab of Ehsan Hoque, an assistant professor of computer science. "When they are given a computational question, they have another kind of facial expression." Using a machine learning tool, the researchers found patterns.

"It told us there were basically five kinds of smile-related 'faces' that people made when responding to questions," Sen continued. The one that was most frequently associated with lying was a smile involving both cheek/eye and mouth muscles. It was consistent with what is known as the "Duping Delight" theory which says that "when you're fooling someone, you tend to take delight in it," Sen added.

Reliable witnesses would often contract their eyes, but not smile at all with their mouths like their deceptive counterparts. "We found that this often happened when people were trying to remember what was in an image," Sen said. "This showed they were concentrating and trying to recall honestly."

The new study involves pairs of subjects, one person in the role of "describer" and the other the interrogator. The describer is shown an image and is instructed to memorize as many details as possible. A computer then instructs the describer to either lie or tell the truth about what they saw.

The interrogator, who has not been privy to the instructions to the describer, then asks the describer a set of baseline questions that are not relevant to the image. This is done, the team said, to capture individual behavioral differences which could be used to develop a "personalized model." The routine questions included "what did you wear yesterday?"—to provoke a mental state relevant to retrieving a memory— and "what is 14 times 4?"—to provoke a mental state relevant to analytical memory.

There are also questions that the witness would have no incentive to lie about and that provide a baseline of that individual's "normal" responses when answering honestly. And, of course, there are questions about the image itself, to which the witness gives either a truthful or dishonest response.

The entire exchange is recorded on a separate video for later analysis.

So, will these findings simply tip off liars to change their facial expressions? Fortunately not, experts said. The tell-tale strong Duchenne smile that has been associated with lying involves "a cheek muscle you cannot control," Hoque said. "It is involuntary."

Can you spot the liars in the video below?

Sen collaborated closely with Karmul Hasan, another PhD student in the group, on two papers for the Institute of Electrical and Electronics Engineers (IEEE): Automated Face and Gesture Recognition and the Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquituous Technologies.