Your Prompt Should Do More: Effects of Retrieval Instructions in Embedding Models
提示詞還能做更多：探討嵌入模型中的檢索指令效應
While prompted embedding models are designed to follow detailed instructions for retrieval, this paper reveals they often fail to follow simple instructions when query-side distractors are introduced. The authors attribute this to current training and evaluation designs, and show that fine-tuning models with added query-side distractors substantially improves instruction-following performance with minimal impact on other tasks.
Key points
Instruction Failure
Current embedding models struggle to follow simple retrieval instructions when query-side distractors are introduced.
Mechanism Analysis
The paper investigates the mechanisms of how instructions affect query representations in asymmetric retrieval tasks.
Training Deficiencies
This unreliable behavior is driven by the training setup and evaluation design of current embedding models.
Distractor Fine-Tuning
Fine-tuning embedding models with added query-side distractors leads to substantial improvements with minimal side effects.
How it works
| 標準嵌入模型 (Standard Embedding) | 干擾微調模型 (Distractor Fine-Tuned) | |
|---|---|---|
| Training Data Style | 缺乏查詢端干擾因子 (No query-side distractors) | 加入查詢端干擾因子 (Includes query-side distractors) |
| Robust Instruction Following | 遇到干擾容易失效 (Fails under distractor presence) | 顯著提升且不易受影響 (Substantial improvement, highly robust) |
| Other Task Performance | 基準水準 (Baseline) | 幾乎無負面影響 (Minimal negative effect) |
Why it matters
For robust RAG and semantic search, embedding models must reliably parse instruction prompts. This research exposes a critical vulnerability to query-side distractors and provides a practical, low-overhead fine-tuning solution that prevents models from being easily confused in complex retrieval tasks.
Who it affects
- AI Developer
- AI Researcher
- Product Manager
How to use it
- 1Optimizing complex, instruction-based semantic search within RAG pipelines.
- 2Improving fine-tuning workflows for prompted embedding models to resist distractor queries.
Limitations & caveats
- The study focuses specifically on asymmetric retrieval tasks; generalizability to other task types requires further validation.
- Standard pre-trained embedding models inherently lack native defenses against query-side distractors due to baseline training limitations.
Related
Building Persistent 3D Object Memory: How Ledger Tracks Objects from Egocentric Videos
打造過目不忘的 3D 空間記憶：Ledger 如何透過第一人稱影片追蹤隱形物體
Researchers introduce Ledger, a framework that builds a persistent 3D object memory from egocentric videos, significantly improving spatial question-answering accuracy for embodied agents.
Decoupling Exploration from Optimization: How ExpDis Boosts LLM Reasoning and Solution Diversity
探索與優化解耦：全新強化學習框架 ExpDis 提升大語言模型的推理多元性
The ExpDis framework decouples exploration from optimization in RLVR. By training explorers with novelty bonuses and distilling filtered trajectories into a student model, it prevents model degradation while fostering diverse reasoning.
Clipped Decentralized SGD: Achieving Optimal Convergence and Linear Speed-Up Under Heavy-Tailed Noise
去中心化 SGD 克服重尾雜訊：梯度裁剪如何實現最佳收斂與線性加速
This study proves that clipped decentralized SGD (DSGD) achieves order-optimal convergence rates and linear speed-up under heavy-tailed noise for non-convex optimization.