Reading visually embodied meaning from the brain: Visually grounded computational models decode visual-object mental imagery induced by written text. Neuroimage 2015 Oct 15;120:309-22
Date
07/19/2015Pubmed ID
26188260DOI
10.1016/j.neuroimage.2015.06.093Scopus ID
2-s2.0-84939450021 (requires institutional sign-in at Scopus site) 52 CitationsAbstract
Embodiment theory predicts that mental imagery of object words recruits neural circuits involved in object perception. The degree of visual imagery present in routine thought and how it is encoded in the brain is largely unknown. We test whether fMRI activity patterns elicited by participants reading objects' names include embodied visual-object representations, and whether we can decode the representations using novel computational image-based semantic models. We first apply the image models in conjunction with text-based semantic models to test predictions of visual-specificity of semantic representations in different brain regions. Representational similarity analysis confirms that fMRI structure within ventral-temporal and lateral-occipital regions correlates most strongly with the image models and conversely text models correlate better with posterior-parietal/lateral-temporal/inferior-frontal regions. We use an unsupervised decoding algorithm that exploits commonalities in representational similarity structure found within both image model and brain data sets to classify embodied visual representations with high accuracy (8/10) and then extend it to exploit model combinations to robustly decode different brain regions in parallel. By capturing latent visual-semantic structure our models provide a route into analyzing neural representations derived from past perceptual experience rather than stimulus-driven brain activity. Our results also verify the benefit of combining multimodal data to model human-like semantic representations.
Author List
Anderson AJ, Bruni E, Lopopolo A, Poesio M, Baroni MAuthor
Andrew J. Anderson PhD Assistant Professor in the Neurology department at Medical College of WisconsinMESH terms used to index this publication - Major topics in bold
Brain MappingCerebral Cortex
Concept Formation
Female
Humans
Image Processing, Computer-Assisted
Imagination
Magnetic Resonance Imaging
Male
Models, Theoretical
Pattern Recognition, Visual
Reading
Semantics
