NeurIPS 2026 (Main Conference)

👁️👂From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs

Motivation

Audio-visual LLMs answer questions about video with sound. How the audio and the video reach the answer inside the model has not been studied. There is interpretability work on vision-language models and on video LLMs, but not on models that take both audio and video.

We ask three questions. Does attention indicate where information flows? In a single audio-visual video, how do the two modalities interact, and how does their information reach the prediction? And with several interleaved images and audio clips, how is information routed and how is the answer selected?

Attention does not show where information goes

Attention is the obvious place to start. It can be read directly, without any intervention, and most interpretability work uses it to say where information goes. So we check first whether it does.

Looking at how the generated token allocates attention across layers, one position dominates. The last token attends heavily to video tokens at layer 31, and in the attention map these appear as bright vertical stripes that almost every query attends to.

Left: how the generated token allocates its attention across system, video, audio, user instruction and generated tokens, layer by layer. Right: the full attention map at the same layer, with query and key positions coloured by token type.
Left: where the generated token puts its attention, layer by layer, split by token type. Right: the attention map at the same layer. Bright vertical stripes mark positions that attract attention from almost every query.

The hidden states show the same thing. A small number of video tokens carry much larger norms than their neighbours, and that magnitude sits in a few dimensions rather than being spread across them. These are sinks, with very large activations, similar to the language sinks reported in LLMs.

Left: hidden-state L2 norm at each token position across layers, with system, video, audio and user instruction tokens coloured separately. Right: per-dimension magnitude for a system token and a video token, showing the video token concentrating its magnitude in a few hidden dimensions.
Left: hidden-state L2 norm at each token position, layer by layer. Right: magnitude across hidden dimensions for a system token and a video token. The video token puts most of its magnitude into a small number of dimensions.

If attention indicated information flow, removing it should reduce accuracy. It does not. Masking video and audio attention from layer 31 onward leaves accuracy unchanged or slightly higher, 42.24 to 42.52. The large attention weights are on tokens that are no longer contributing.

The rest of the analysis therefore uses causal interventions rather than attention maps.

Method

We use attention knockout, a causal intervention that blocks selected attention edges inside a sliding window of seven layers. If blocking an edge changes the answer, that edge was carrying information. We measure the relative change in the probability of the predicted answer letter, and run the intervention only on samples the model already answers correctly.

block:
V A Question Last p(answer)

Nothing blocked. The last token reaches the answer through the question, which has already taken what it needs from video and audio.

Video can no longer reach the question. Accuracy drops, most on tasks that depend on what is seen.

Audio can no longer reach the question. Accuracy drops, most on tasks that depend on what is heard.

The question can no longer reach the last token. This edge carries the whole route, so accuracy drops sharply.

Little changes. There is no direct route from the video tokens to the last token.

Little changes. There is no direct route from the audio tokens to the last token either.

The two modalities can no longer inform each other. The cost falls on tasks that need them aligned, such as speech recognition and speaker detection.

The main model is Qwen2.5-Omni 3B [3]. Qwen2.5-Omni 7B and Video-SALMONN2 Plus at 3B and 7B are used to check that the findings hold across models and scales.

Single video: one sequential route

For a single audio-visual video we use AV-SpeakerBench [1], a four-way multiple-choice benchmark with five task categories grouped by which modality informs which.

System V1 A1 V2 A2 ... VC AC Question + options Last interleaved in time windows answer predicted here
Token layout for a single audio-visual video, from AV-SpeakerBench. Video and audio tokens alternate in time windows, followed by the question and its options, then the last token.

Attention across frames matters for nearly every task. Two-way interaction between audio and video matters most on tasks that need fine-grained alignment between them, such as speech recognition and speaker detection. Interaction between audio tokens has little effect.

The route to the prediction is sequential. The modalities pass information to the question tokens, and the question passes it to the last token. Which modality contributes depends on the task: visual tasks route mostly through the video, and audio-heavy tasks draw on both.

Video Audio Question Last dashed: flow strength is task-dependent
The route in a single audio-visual video. Each modality attends within itself, audio and video exchange information with each other, both pass information to the question tokens, and the question passes it to the last token.

Interleaved inputs: two parallel routes

For multiple interleaved inputs we use the matching subset of AV-Odyssey [2], where one reference item is matched against four candidates.

System img img img img Q audio O1 ... O4 Last 4 candidates reference + option letters
Token layout for interleaved inputs, from the AV-Odyssey matching subset. Four candidate images, then the question, the reference item and the option letters, then the last token.

The routing differs from the single-video case. Candidates and question pass information to the reference, and the reference passes it to the last token. The candidates also exchange information among themselves at the middle layers.

Answer selection runs through the option letters. The correct option letter aggregates information from the correct candidate. At the late layers the correct and incorrect option letters compete, and blocking the incorrect ones increases accuracy.

C1 C2 C3 C4 Question Reference Options Last
The route with interleaved inputs. Candidates exchange information among themselves, then pass it to the reference and to the options, while the question also feeds the reference. The reference and the options then reach the last token by separate paths.

The sequential route found for single videos is therefore not the general case. With interleaved inputs the model uses two parallel paths.

BibTeX

NeurIPS 2026
@inproceedings{suharitdamrong2026senses,
  title     = {From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs},
  author    = {Suharitdamrong, Wish and Awais, Muhammad and Zhu, Xiatian and Atito, Sara},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}

References

  1. Nguyen, P. L., et al. See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models (AV-SpeakerBench). 2025. arxiv.org/abs/2512.02231
  2. Gong, K., et al. AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information? 2024. arxiv.org/abs/2412.02611
  3. Qwen Team. Qwen2.5-Omni Technical Report. 2025. arxiv.org/abs/2503.20215

← Back to projects