A Computer Program That Sees What You See

Scientists at the University of California-Berkeley have developed a “visual decoder” which employes a computational algorithm to identify what someone saw just by examining their brain activity. The success of the study represents an advance in the scientific understanding of how the brain processes images, but could also have potential ramifications for mind-reading technology.

Using functional magnetic resonance (fMRI) scanners to record a person’s brain activity while they looked at thousands of pictures, researchers created a model to predict the mental activity patterns elicited by looking at specific images. Then individual subjects were asked to look at a set of novel images and the computer was able to identify which images they were looking at with a significant amount of accuracy. According to the report (subscription) published in Nature, “Results suggest that it may soon be possible to reconstruct a picture of a person’s visual experience from measurements of brain activity alone.”

While such mind-reading and dream re-creation may be a still be in the world of science fiction, the results are a big step forward in demonstrating the huge amount of information available in fMRI signals, which could go a long way in helping understand how the brain works.

The study consisted of two stages and only two subjects, both volunteers from the research team. The first stage involved showing the subjects 1,750 grayscale pictures of “natural” images–houses, trees, plants, animals–and recording their mental activity with fMRI scanners. The imaging data and mental activity patterns were then used to create a predictive model of activity in the visual areas of the brain. In the second stage, the researchers tested the model by showing subjects 120 new images. The model attempted to identify which image the subjects were focused on by comparing the measured mental activity against predicted activity. Based on the 120 possible images in the experimental set, the probability of guessing an image correctly each time is 0.8 percent. The model identified the correct image 92 percent and 72 percent of the time for each subject.

Any potential “visual decoder” would need to handle a larger set of images. So to test the capabilities of their model, researchers increased the set size to 1,000 new images and found that identification performance dropped only slightly. Extrapolating from these results, the scientists believe that identification accuracy would remain above ten percent even up to a set size a hundred times greater than the amount of pictures indexed by Google (880 million images). This amount of accuracy happens to be significantly above chance, demonstrating the potential of this model. The researchers even tested the performance of their model over time, bringing one of the subjects back after two months and achieving 82 percent correct identification.

Media outlets who covered the study choose to focus on the possible applications of the technology and the ethical questions it raises:

• Brandon Keim of Wired lays out the ethical dilemmas of the study. He writes about the possibility of such technology invading mental privacy, as well as how it could be used in the courtroom.

• Nikhil Swaminathan at Scientific American spoke to John-Dylan Haynes, a neuroscientist at the Max-Planck Institute, who said the predictive model used in the study is currently limited to sensory inputs and that high-level mental functions such as memories and emotions are still a bit far from being codified into a mathematical model.

• James Randerson at the Guardian outlines the study and highlights the futuristic potential of visualizing dreams and memories and helping understand the mental state of coma patients.

• Emily Singer at Technology Review points out a growing trend: the use of technology to analyze and study the brain’s neural processing pathways.

• Science Progress Editor-in-Chief Jonathan Moreno raises ethical and policymaking questions about the growing interest of Federal defense agencies in neuroscience in his book, Mind Wars. Case in point, this study was partially funded by the National Defense Science and Engineering Fellowship, whose website explains the purpose of these grants is “a means of increasing the number of U.S. citizens and nationals trained in science and engineering disciplines of military importance.”

Image: flickr.com/tico_bassie

Tags: bioethics, Neuroscience

Comments on this article

You can skip to the end and leave a response. Pinging is currently not allowed.