- formatting
- images
- links
- math
- code
- blockquotes
- external-services
•
•
•
•
•
•
-
OPD 随想
笔者最近在做 MOPD 研究时发现，在计算 Advantage（教师-学生预测概率的 log-ratio）时，仅用学生采样的 top-1 token 计算已经效果很好了。巧合的是，Kimi K3 技术报告中也称，top-k 采样的消融未见明显性能或效率提升。因此有感而发，在本文中讨论一个问题：在 On-Policy Distillation 中，为什么不应该多步在全词表上计算 Advantage？
-
LU-KV: KV Cache Optimization Based on Long-term Utility
When large language models process extremely long texts and run out of memory, how do we decide what to keep? LU-KV predicts the "long-term marginal utility" of each cached memory fragment to determine what to retain and what to discard, maintaining strong comprehension under tighter memory budgets.