Why Forget-Only Unlearning Needs Memorization
「僅去記憶」的兩難：為什麼機器去學習反而需要模型「記住更多」？
Machine unlearning aims to delete specific training data so the model behaves as if it were retrained from scratch. This paper investigates "forget-only" unlearning, where the algorithm only has access to the trained model and the forget targets. The authors demonstrate that because different datasets can yield identical models but require distinct outputs after deletion, forget-only unlearning has fundamental limitations. To succeed, the model must intentionally memorize more training information during its initial learning phase than standard training requires.
Key points
Forget-Only Constraint
In forget-only unlearning, the deletion algorithm receives only the trained model and the forget targets, with no retained training data or extra training information.
Same Model, Different Outputs
Different datasets can produce the exact same trained model but require very different outputs after the same forget examples are removed.
The Memorization Paradox
To handle arbitrary deletions, models must retain more information than usual. For threshold learners, the required info can be the entire dataset, though standard training keeps only one boundary point.
How it works
| 標準訓練 (Standard) | 支援「僅去記憶」訓練 (Forget-Only Ready) | |
|---|---|---|
| Stored Information | 僅保留決策邊界 (例如單一邊界點) | 需額外記住多餘的資料細節 (甚至整個資料集) |
| Unlearning Accuracy | 極低，無法精準還原重新訓練的效果 | 極高，可精準模擬從頭重新訓練 |
| Space & Privacy Trade-off | 節省空間，但難以完全抹除特定資料的影響 | 空間開銷大，但能隨時安全且精準地執行刪除 |
Why it matters
This work exposes a fundamental paradox: to successfully "forget" later without keeping the original dataset, a model must "memorize" more details during initial training. This provides crucial theoretical bounds for designing privacy-preserving and compliant machine learning systems.
Who it affects
- AI Researcher
- AI Developer
- Product Manager
How to use it
- 1Designing privacy-compliant ML models and training algorithms that satisfy GDPR "Right to be Forgotten" mandates.
- 2Assessing the feasibility and accuracy limits of unlearning on edge devices without access to the full training set.
Limitations & caveats
- The findings are primarily based on theoretical analysis and lower bounds (such as threshold learners); empirical overhead in complex deep models like LLMs remains to be explored.
Related
Building Persistent 3D Object Memory: How Ledger Tracks Objects from Egocentric Videos
打造過目不忘的 3D 空間記憶：Ledger 如何透過第一人稱影片追蹤隱形物體
Researchers introduce Ledger, a framework that builds a persistent 3D object memory from egocentric videos, significantly improving spatial question-answering accuracy for embodied agents.
Decoupling Exploration from Optimization: How ExpDis Boosts LLM Reasoning and Solution Diversity
探索與優化解耦：全新強化學習框架 ExpDis 提升大語言模型的推理多元性
The ExpDis framework decouples exploration from optimization in RLVR. By training explorers with novelty bonuses and distilling filtered trajectories into a student model, it prevents model degradation while fostering diverse reasoning.
Clipped Decentralized SGD: Achieving Optimal Convergence and Linear Speed-Up Under Heavy-Tailed Noise
去中心化 SGD 克服重尾雜訊：梯度裁剪如何實現最佳收斂與線性加速
This study proves that clipped decentralized SGD (DSGD) achieves order-optimal convergence rates and linear speed-up under heavy-tailed noise for non-convex optimization.