Summary
Researchers used a 253-electrode cortical implant and artificial intelligence to decode intended speech and gestures simultaneously in two people with severe paralysis. The proof-of-concept system displayed words as text and represented gestures through a personalised animated avatar.
A single brain–computer interface (BCI) has decoded intended speech and non-verbal gestures at the same time, converting neural activity into on-screen text and movements by a personalised animated avatar. The system produced outputs within seconds of the participants’ intended communication and was tested in two people with severe paralysis.
The work, described in Nature Neuroscience and reported by Nature on 14 September 2026, is presented as a proof of concept for BCIs that can convey more than one form of communication at once.
How one implant handled two communication channels
The device used a surgically implanted array containing 253 electrodes. It was placed on the surface of the cortex and covered a relatively large area of the sensorimotor cortex, a region involved in planning and controlling speech and body movements.
The electrodes recorded electrical activity while participants attempted to speak, gesture or combine the two. An artificial-intelligence system translated those patterns into two kinds of output: intended words appeared as text on a screen, while intended gestures prompted a personalised avatar to move.
The researchers tested communication in separate and combined forms. Examples included pairing the word “hello” with a wave and pairing “yes” with a head nod. This allowed the system to process verbal and non-verbal signals as part of the same communication event rather than treating speech and gesture as unrelated tasks.
The first participant had impaired speech and movement after a brainstem stroke. He silently attempted to speak five phrases and tried to perform gestures including waving, nodding, shaking his hands and clapping.
The second participant had amyotrophic lateral sclerosis, or ALS, a motor-neuron disease that had caused progressive speech paralysis and motor impairments. He could still type messages on a computer and phone during the study. The researchers asked him to vocalise 10 phrases and imagine 10 gestures, including a shrug and a thumbs-up, while keeping his body still apart from his facial muscles.
Why simultaneous decoding is difficult
Speech and gestures can involve overlapping neural signals. In addition, the patterns produced while speaking and gesturing together can differ from the patterns recorded during speech-only or gesture-only tasks. A system designed for just one output therefore cannot simply be assumed to handle both channels accurately at the same time.
The significance of the demonstration is that one cortical recording array was used to interpret these overlapping patterns and generate coordinated verbal and visual outputs. For people who cannot reliably produce speech or body movement, the avatar could provide non-verbal expression such as a nod, wave or thumbs-up alongside text.
The report describes the system as the first demonstration of a BCI translating verbal and non-verbal communication simultaneously. It also frames the result as a step towards more multifunctional neural prostheses that can represent a wider range of human expression.
The evidence remains an early-stage demonstration involving two participants and different testing conditions. The supplied account describes the timing qualitatively as occurring within seconds of intent, but it does not provide numerical accuracy, error-rate or latency results. Performance across a larger vocabulary, a broader set of gestures and additional users will require further study.