Naive stream
A guess enters memory as if observed.
Free-form think-while-watching merges what the model saw with what it merely expected. The error can survive compression and compound across the stream.
visual certaintyearly
The failure mode
Seeing arrives early.
Hearing unfolds.
Visual evidence is dense and often locally decidable. Auditory evidence is sparse, temporally extended, and vulnerable to eviction. Under bounded memory, this asymmetry creates premature cross-modal commitment: a plausible visual explanation becomes a working fact before audio can settle it.
OST stream
A guess becomes a future-testable claim.
Each unresolved interpretation names a verifying modality, an evidence interval, and the states that depend on it. The answer gate waits while an answer-critical review remains scheduled.
audio maturityunfolding
≠
Physical synchronization is not evidential synchronization. Audio and video can arrive at the same timestamp while becoming decisive at very different rates.
The OST loop
Reason, then audit
the reasoning.
OST runs one closed loop at every decision step. New evidence first checks earlier assumptions; corrections propagate through the exact reasoning they influenced; only then can the model decide whether the stream is sufficient.
01
Retain
Keep visual and audio native tokens in separately budgeted buffers.
02
Claim
Record the prediction, verifying modality, future interval, and support scope.
03
Verify
Check the claim against realized evidence from its specified modality.
04
Correct
Reduce refuted influence through recorded dependencies, then rewrite the state.
05
Answer
Wait while any answer-critical claim remains unsettled.
Figure 2 · Omni-Streaming Thinking. Separate audio and visual retention supplies evidence for the claim–verify–retract loop. A pending claim records its expected evidence, verifying modality, and future interval; when later audio refutes it, OST reduces its influence through recorded dependencies, rewrites the Omni-State, and answers only after answer-critical reviews are settled.Open image to inspect details
State, not narration
Every chunk says exactly what it knows.
The six-field Omni-State makes omission visible. Silence is a valid fact; uncertainty is legal; fabricated audio is not required. Claims preserve their modality and provenance even after history is folded into summaries.
visual_evidenceaudio_stateaudio_evidenceconflictforecastsufficiency
omni_state @ 8.0s
{
"visual_evidence": "Gate 6 sign visible",
"audio_state": "uncertain",
"audio_evidence": "announcement incomplete",
"conflict": null,
"forecast": {
"claim": "announcement names Gate 6",
"modality": "audio",
"deadline": "t + 8s",
"status": "pending"
},
"sufficiency": false
}
{
"visual_evidence": "Gate 6 sign visible",
"audio_state": "uncertain",
"audio_evidence": "announcement incomplete",
"conflict": null,
"forecast": {
"claim": "announcement names Gate 6",
"modality": "audio",
"deadline": "t + 8s",
"status": "pending"
},
"sufficiency": false
}
A benchmark that changes the evidence
Lock the video.
Intervene on audio.
OST-DiagBench diagnoses whether an omni-modal model actually uses auditory evidence. Its 510 matched Core groups keep video fixed while audio is retained, muted, swapped, or mixed. A separate set of 510 reviewed CLASH examples leaves an old fact visible in subtitles while speech states a mutually exclusive new fact.
/blog/ost/OST-DiagBench-pastel.html
Matched acoustic Core · subtitle–speech CLASH · paired inference pathsOpen OST-DiagBench ↗
Native streaming results
Earlier when possible.
Correct when it matters.
With a frozen Qwen3-Omni-30B-A3B-Instruct backbone and lightweight adaptation, OST outperforms the strongest open baselines on five streaming and audio-visual benchmarks by more than 10% relative on average.
Audio-visual + QA context
88.4
SOVBench-O average
Best online result across real-time, recall, and proactive questions.
87.8
Audio-visual contextSOVBench-O average under causal, prefix-only inference.
92.4
SOVBench-T F1Respond/wait trigger quality under native streaming.
2.95 d′Auditory sensitivity, versus at most 1.38 for open baselines.
95.49%CLEAN source recall when audio and video agree.
10.39%MUTE visual hallucination rate; lower is better.
81.76%SWAP faithful attribution under contradiction.
51.37%MIX joint recall for coexisting source and donor.
46.47%CLASH accuracy on the new spoken fact.
Reporting note. SOVBench values reproduce Table 1 of the current arXiv manuscript; diagnostic values reproduce Table 2 on 510 matched Core groups and 510 reviewed CLASH examples.
What OST adds
Three contributions for
trustworthy streams.
The latest manuscript connects a failure mode, a mechanism, and an evaluation: premature commitment becomes measurable, future evidence can revise memory, and the effect is tested across both standard benchmarks and controlled audio interventions.
Premature cross-modal commitment
OST identifies how an early visual interpretation can become a working fact and persist in streaming memory after later audio contradicts it.
Claim → verify → retract
Future-testable claims connect a modality-specific review to dependency-aware retraction, corrected state decoding, and answer timing.
Five-benchmark open lead
OST leads open systems across five streaming and audio-visual benchmarks and exposes audio use under agreement, absence, contradiction, coexistence, and fact conflict.
Abstract
Streaming memory as a causal audit loop.
Streaming omni-modal models must decide what and when to answer from the video chunks and synchronized audio observed so far. Visual cues often support an interpretation before an utterance or sound event is complete. If that interpretation enters memory as a fact, later reasoning can keep relaying it even after audio contradicts it. We call this failure premature cross-modal commitment. We propose Omni-Streaming Thinking (OST), which generates structured outputs that include evidence observed so far, forecasts of future evidence, and claims based on this evidence. Each claim is initially marked as pending and linked to a future verification interval. Audio and visual evidence are stored separately, and OST checks a claim against the evidence from the specified modality at the end of the verification interval. When contradictory evidence is detected, a refutation process reduces the influence of the claim and its dependent states, and then guides a state update using the new evidence. An answer gate decides whether the answer-critical claims meet the conditions for giving a response. Using a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation, OST outperforms the strongest open baselines on five streaming and audio-visual benchmarks by more than 10% relative on average. We also introduce OST-DiagBench, which holds video fixed and edits audio to test agreement, absence, contradiction, coexistence, and subtitle-speech conflict. OST reaches d′ = 2.95, compared with at most 1.38 for open baselines, while reducing vision-induced auditory hallucinations.
@article{du2026omni,
title={Omni-Streaming Thinking},
author={Enjun Du and Siyi Liu and Ziyu Zheng and Jingyu Li and Yiwen Guo and Yongqi Zhang and Difan Zou},
year={2026}
}