A.I. Helps Paralyzed Woman Speak for First Time in 18 Years

Artificial intelligence is helping a severely paralyzed woman to speak again for the first time in 18 years.

The patient, Ann, is participating in research at the University of California (UC) where a novel new technology is being developed.

The system, described in a study published in the journal Nature, involves a brain implant that connects to computers and enables Ann to communicate via a human-like digital avatar on a screen.

The brain-computer interface decodes Ann's brain signals and turns them into synthesized speech that the digital avatar speaks out loud while simultaneously recreating natural facial movements.

The UC scientists hope their research could one day lead to an FDA-approved system that would enable patients like Ann—who are cognizant but physically unable to speak—to communicate more naturally in near real-time.

"Our goal is to restore a full, embodied way of communicating, which is the most natural way for us to talk with others," Dr. Edward Chang, chair of neurological surgery at UC San Francisco (UCSF) and an author of the study, said in a statement.

"These advancements bring us much closer to making this a real solution for patients."

Eighteen years ago, when Ann was 30 years old, she suffered a brainstem stroke that left her severely paralyzed. She lost control of all the muscles in her body and was unable to even breathe on her own initially. Doctors still do not know the cause of the stroke, which came on suddenly.

"Overnight, everything was taken from me," Ann wrote, with the help of a device that enables her to type slowly on a computer screen by making small head movements. "I had a 13-month-old daughter, an 8-year-old stepson and 26-month-old marriage."

"Locked-in syndrome (LIS) is just like it sounds," Ann said using the device. "You're fully cognizant, you have full sensation, all five senses work, but you are locked inside a body where no muscles work."

Over the next few years, Ann went through painstaking physical rehabilitation, during which she learned to breathe on her own again and move her neck. She also learned to move some facial muscles, enabling her to laugh, cry, smile and wink, among other actions. But despite all the rehabilitation, she is still not able to speak.

Ann found out about the research being conducted by Chang and colleagues in 2021 after reading about a paralyzed man named Pancho.

Pancho had also experienced a brainstem stroke many years earlier, and Chang's research team had been trying to translate his brain signals into text. For this to work, Pancho actually had to attempt to speak in order for the system to register what he meant.

This research was the first successful demonstration of the direct decoding of full words from the brain activity of someone who is paralyzed and cannot speak, according to Chang.

However, the scientist and his team wanted to go a step further with Ann. So for the latest study, the scientists decoded her brain signals into actual speech, as well as recreating the associated facial movements with an avatar.

First, the researchers implanted a set of more than 250 electrodes onto the surface of her brain, covering a region that is critical for speech. These electrodes intercept brain signals that Ann produces when she is trying to speak.

The implant is connected to computers via a cable plugged into a specially designed port that protrudes from her skull.

Over a period of several weeks, Ann worked with the researchers to train an artificial intelligence system to recognize her unique speech brain signals.

To do this, Ann repeated different phrases from a set of more than 1,000 words so that the AI algorithm could recognize the patterns of brain activity associated with the basic sub-units of speech, known as phonemes.

The system the UC team have developed, at present, can decode brain signals into text at a rate of nearly 80 words per minute. This is significantly faster than her current text-based communication system, which can only produce around 14 words per minute.

"The accuracy, speed and vocabulary are crucial," said UCSF researcher Sean Metzger, another author on the Nature study. "It's what gives Ann the potential, in time, to communicate almost as fast as we do, and to have much more naturalistic and normal conversations."

The voice that the avatar speaks in is personalized, based on Ann's own voice. The team recreated the voice using language learning AI, which analyzed footage of a wedding speech that she gave in 2005.

The avatar, meanwhile, simulates the muscle movements of Ann's face with the help of AI that harnesses the signals being sent from her brain.

"We're making up for the connections between her brain and vocal tract that have been severed by the stroke," Kaylo Littlejohn, a UC Berkeley graduate student who is another author on the study, said in the statement.

The next step for the team is to create a wireless version of their system that would enable Ann to communicate with it without the implant being physically connected

The next goal for the team is to create a wireless version of their system that would not require Ann to be physically connected to any computers.

"Giving people like Ann the ability to freely control their own computers and phones with this technology would have profound effects on their independence and social interactions," David Moses, an adjunct professor in neurological surgery at UCSF and co-first author of the study, said in the statement.

Ann said the process of helping the team to develop this novel communication technology has been life-changing for her.

"When I was at the rehab hospital, the speech therapist didn't know what to do with me," she said. "Being a part of this study has given me a sense of purpose, I feel like I am contributing to society. It feels like I have a job again. It's amazing I have lived this long; this study has allowed me to really live while I'm still alive!"