From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs
Motivation
Audio-visual LLMs answer questions about video with sound. How the audio and the video reach the answer inside the model has not been studied. There is interpretability work on vision-language models and on video LLMs, but not on models that take both audio and video.
We ask three questions. Does attention indicate where information flows? In a single audio-visual video, how do the two modalities interact, and how does their information reach the prediction? And with several interleaved images and audio clips, how is information routed and how is the answer selected?
Attention does not show where information goes
Attention is the obvious place to start. It can be read directly, without any intervention, and most interpretability work uses it to say where information goes. So we check first whether it does.
Looking at how the generated token allocates attention across layers, one position dominates. The last token attends heavily to video tokens at layer 31, and in the attention map these appear as bright vertical stripes that almost every query attends to.
The hidden states show the same thing. A small number of video tokens carry much larger norms than their neighbours, and that magnitude sits in a few dimensions rather than being spread across them. These are sinks, with very large activations, similar to the language sinks reported in LLMs.
If attention indicated information flow, removing it should reduce accuracy. It does not. Masking video and audio attention from layer 31 onward leaves accuracy unchanged or slightly higher, 42.24 to 42.52. The large attention weights are on tokens that are no longer contributing.
The rest of the analysis therefore uses causal interventions rather than attention maps.
Method
We use attention knockout, a causal intervention that blocks selected attention edges inside a sliding window of seven layers. If blocking an edge changes the answer, that edge was carrying information. We measure the relative change in the probability of the predicted answer letter, and run the intervention only on samples the model already answers correctly.
Nothing blocked. The last token reaches the answer through the question, which has already taken what it needs from video and audio.
Video can no longer reach the question. Accuracy drops, most on tasks that depend on what is seen.
Audio can no longer reach the question. Accuracy drops, most on tasks that depend on what is heard.
The question can no longer reach the last token. This edge carries the whole route, so accuracy drops sharply.
Little changes. There is no direct route from the video tokens to the last token.
Little changes. There is no direct route from the audio tokens to the last token either.
The two modalities can no longer inform each other. The cost falls on tasks that need them aligned, such as speech recognition and speaker detection.
The main model is Qwen2.5-Omni 3B [3]. Qwen2.5-Omni 7B and Video-SALMONN2 Plus at 3B and 7B are used to check that the findings hold across models and scales.
Single video: one sequential route
For a single audio-visual video we use AV-SpeakerBench [1], a four-way multiple-choice benchmark with five task categories grouped by which modality informs which.
Attention across frames matters for nearly every task. Two-way interaction between audio and video matters most on tasks that need fine-grained alignment between them, such as speech recognition and speaker detection. Interaction between audio tokens has little effect.
The route to the prediction is sequential. The modalities pass information to the question tokens, and the question passes it to the last token. Which modality contributes depends on the task: visual tasks route mostly through the video, and audio-heavy tasks draw on both.
Interleaved inputs: two parallel routes
For multiple interleaved inputs we use the matching subset of AV-Odyssey [2], where one reference item is matched against four candidates.
The routing differs from the single-video case. Candidates and question pass information to the reference, and the reference passes it to the last token. The candidates also exchange information among themselves at the middle layers.
Answer selection runs through the option letters. The correct option letter aggregates information from the correct candidate. At the late layers the correct and incorrect option letters compete, and blocking the incorrect ones increases accuracy.
The sequential route found for single videos is therefore not the general case. With interleaved inputs the model uses two parallel paths.
BibTeX
@inproceedings{suharitdamrong2026senses,
title = {From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs},
author = {Suharitdamrong, Wish and Awais, Muhammad and Zhu, Xiatian and Atito, Sara},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
} References
- Nguyen, P. L., et al. See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models (AV-SpeakerBench). 2025. arxiv.org/abs/2512.02231
- Gong, K., et al. AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information? 2024. arxiv.org/abs/2412.02611
- Qwen Team. Qwen2.5-Omni Technical Report. 2025. arxiv.org/abs/2503.20215