Hearing the Truth: VeriSpeak Exposes the Text-Speech Modality Gap in Fact-Checking
聽見真假：VeriSpeak 揭示語音語言模型在事實查核中的「模態差距」
To combat spoken misinformation, researchers developed "VeriSpeak," a benchmark with 3,879 spoken claims. Tests show that while models verify text well, they fail on equivalent speech due to a modality gap. Simple retrieval often fails as models conflate evidence with speech, but combining retrieval with explicit reasoning (via thinking-tuned LALMs) boosts accuracy to 86.1%.
Key points
Introducing VeriSpeak Benchmark
Contains 3,879 spoken claims spanning temporal, geographical, and relational facts, designed with balanced true/false labels for speech fact verification.
Text-Speech Modality Gap
Experiments reveal that LALMs capable of reliably verifying written claims often fail when testing the exact same claims in spoken format.
Conflation in Simple RAG
Retrieval alone yields limited gains because models frequently conflate the retrieved text evidence with the input spoken claim.
Explicit Reasoning Boosts Accuracy
Combining retrieval with explicit reasoning, such as in a thinking-tuned LALM, drastically improves comparison and reaches 86.1% accuracy.
How it works
Why it matters
Misinformation runs rampant in spoken formats like podcasts and social media videos. VeriSpeak addresses this critical gap, proving that simple speech-to-text or basic retrieval is insufficient. By demonstrating the necessity of explicit reasoning (thinking-tuned models) to overcome the modality gap and prevent evidence conflation, this work paves the way for reliable, multimodal misinformation defense systems.
Who it affects
- AI Researcher
- AI Developer
- Policy Maker
How to use it
- 1Automated speech and video misinformation screening platforms
- 2Real-time fact-checking features for multimodal conversational AI assistants
- 3Automated content moderation and trust-scoring for podcasts and broadcast news
Limitations & caveats
- Models tend to conflate retrieved textual evidence with the original input spoken claims.
- Utilizing thinking-tuned models for explicit reasoning increases inference latency and computational cost.