Researchers at the University of Texas at Austin have developed a new AI system that can translate a person’s brain activity into text. This system called a semantic decoder, can translate brain activity while listening to a story or imagining telling a story into a continuous stream of text. It uses a transformer model, similar to the ones that power Open AI’s ChatGPT and Google’s Bard.
The system is designed to help people who are mentally conscious yet unable to physically speak, such as those who have suffered strokes, to communicate again. Unlike other language decoding systems, the semantic decoder does not require subjects to have surgical implants, making the process noninvasive. Participants also do not need to use only words from a prescribed list.
To use the system, brain activity is measured using a functional MRI scanner after extensive training of the decoder, in which the individual listens to hours of podcasts in the scanner. Later, provided that the participant is open to having their thoughts decoded, their listening to a new story or imagining telling a story allows the machine to generate corresponding text from brain activity alone.
The results are not word-for-word transcripts but capture the gist of what is being said or thought. About half the time, when the decoder has been trained to monitor a participant’s brain activity, the machine produces text that closely (and sometimes precisely) matches the intended meanings of the original words.
The researchers have also addressed concerns about the potential misuse of the technology. Decoding worked only with cooperative participants who had willingly participated in training the decoder. Results for individuals on whom the decoder had not been trained were unintelligible, and if participants on whom the decoder had been trained later put up resistance, for example, by thinking other thoughts, results were similarly unusable.
The system is not yet practical for use outside of the laboratory due to its reliance on the time needed on an fMRI machine. However, the researchers think this work could transfer to other, more portable brain-imaging systems, such as functional near-infrared spectroscopy (fNIRS). The study was published in the journal Nature Neuroscience.
About Author
