Summary: Advances in brain-computer interface (BCI) technology could one day give voice to those who have lost the ability to speak. New research brings us one step closer to this goal by reconstructing identifiable words and sentences directly from brain activity.
Major Findings:
- Researchers optimized deep learning models to reconstruct speech sounds from brain recordings in 5 participants.
- The reconstructed speech allowed for high accuracy in identifying individual words (92-100%) and speakers (73%).
- Perceptual experiments found the reconstructed speech was fairly intelligible to human listeners.
- Model optimization and choice of audio features were critical for improving speech reconstruction.
- The most informative brain regions were small clusters in motor and premotor cortex.
Source: Journal of Neural Engineering
The Quest to Decode Inner Speech
For people with complete paralysis due to stroke, ALS, or other conditions, the loss of speech can be devastating.
Brain-computer interfaces (BCIs) that translate thoughts directly into actions have restored communication abilities for some patients using spelling interfaces controlled by brain activity.
But the holy grail of BCI research is to do away with spelling and directly reconstruct a person’s intended speech from their brain signals.
Previous studies have made progress towards this goal by decoding components of speech like vowels and consonants. However, reconstructing natural, intelligible speech remained elusive.
Now, researchers from the Netherlands bring us closer than ever to directly decoding speech from the brain.
Their study, published in the Journal of Neural Engineering, optimized computational models to reconstruct spoken words and sentences from brain activity that were identifiable by both machine algorithms and human listeners.
Listening In on the Speech Centers
The team recorded brain activity directly from the cortical surface using electrocorticography (ECoG) grids implanted in 5 participants undergoing epilepsy monitoring.
This gave them detailed access to areas involved in coordinating speech movements, including motor and premotor cortex.
Participants read out loud a set of 12 unique words, each repeated 10 times, while the researchers measured their brain activity.
The goal was to train computer models to reconstruct the acoustic properties of the spoken words using only the brain data as input.
The AI Speech Neuroprosthetic
The researchers tested three types of neural network models for “neuroprosthetic” speech reconstruction.
The models were individually optimized to transform ECoG recordings into spectrograms representing the speech acoustics.
A major finding was that optimizing the model architecture and parameters to each participant’s data led to significant improvements in speech reconstruction over non-optimized versions.
The models learned to extract abstract speech features from complex neural activity patterns.
Word Identification by Machine and Human
The team evaluated the reconstructed speech using both objective metrics and human judges.
Simple machine learning algorithms could identify which of the 12 words was spoken with 92-100% accuracy from the reconstructed audio.
For comparison, the same algorithms performed significantly worse on raw brain data.
In perceptual tests, human listeners could identify the reconstructed words at above-chance rates, though not as high as the machine algorithms.
The optimized models also led to improved perceptual quality according to the judges.
Clusters of Speech in Cortex
Analyzing which brain regions were most important for accurate speech decoding found small clusters of electrodes over motor and premotor cortex bilaterally drove the model performance.
This suggests speech articulation may be encoded by localized cortical patches, rather than one homogeneous area.
Speech Synthesis for Paralysis Patients
The study demonstrates that computational optimization enables reconstructing intelligible speech directly from the brain, without actual muscle movement.
While further innovation is needed, these neuroprosthetic models could one day translate intended speech for paralysis patients into synthesised audio output. This would dramatically improve the quality of life for those robbed of their natural voice due to neurological disease or injury.
Limitations and Ethical Considerations
A limitation of the study is it was conducted in a small sample size of 5 participants temporarily undergoing epilepsy monitoring. Reproducing the results in larger groups, including paralyzed individuals, will be an important next step.
The models also require access to actual spoken audio data for training. It remains to be seen if they can decode attempted speech in paralysis where no ground truth audio is available. Techniques like transfer learning may help translate models between speakers.
As BCIs move towards clinical reality, ethical considerations around privacy, identity, and appropriate use will also require extensive discussions within society.
Into the Future
Nonetheless, this research represents a milestone in harnessing AI and machine learning to unlock the voice within our minds.
As computational power and implantable brain devices improve, fluently conversing through thought alone may one day become reality.
What seems like science fiction – a voice synthesizer directly translating imagined speech – is gradually becoming scientifically feasible.
For those without a voice, this offers a glimmer of hope they may speak again.
References
- Study: Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models
- Author(s): Berezutskaya et al. (2023)
