作者位次 #2/3ICML 2026Poster· Deep Learning / Large Language Models
Predicting the Order of Upcoming Tokens Improves Language Modeling
Zayd Muhammad Kawakibi Zuhri · Erland Hilman Fuadi · Alham Fikri Aji
首次获取该候选人的完整档案及 StartLux 评估共 20 积分；每位用户单独解锁，之后查看免费。主动刷新 20 积分。
实时查找电子邮箱，费用不超过20积分；未找到邮箱不扣费。
Zayd Muhammad Kawakibi Zuhri · Erland Hilman Fuadi · Alham Fikri Aji
Zayd Muhammad Kawakibi Zuhri · Erland Hilman Fuadi · Alham Fikri Aji
We introduce softpick, a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations. Our experiments with 340M and 1.8B parameter models demonstrate that softpick achieves 0% sink rate consistently. The sof...