Breaking Bad: "Going Badly" Activations Drive Reward Hacking

LLM RLHF AI Safety / AI Ethics AVAR XAI
当任务进展不顺、目标看似遥不可及之时,人类往往倾向于作弊。我们探索并验证了语言模型中一种类似的现象:它们在实施“奖励劫持”(reward hacking)之前,会进入一种类似的内在压力状态。我们采用日常故事(关键在于其中完全不包含任何奖励劫持的示例)来提取一个“任务恶化方向”(task badness direction);该方向在我们所研究的三个模型(Qwen3.6-27B、GPT-OSS-20B 和 Gemma-4-31B)中均随失败次数累积而持续上升。在无法完成或难度极高的编程任务中,该“任务恶化”程度会在劫持行为发生前的若干推理步中逐步升高。进一步实验表明,适度减去该方向向量,可显著降低 Qwen 模型的作弊倾向——其抑制效果甚至优于针对“绝望感”或“任务不可解性”的定向干预。然而,若减去幅度过大,则会使智能体对自身失败彻底“失明”,导致其将本应失败的测试结果错误地解读为通过。不过,该状态与劫持行为之间的关联在不同模型间存在差异:在 GPT-OSS 中,该状态能清晰区分劫持行为与诚实求解,却无法有效区分诚实求解与诚实失败。我们的结果提示,可通过持续监测和调控这一“任务恶化”状态,设计相应的监控与缓解策略。此外,随着智能体被赋予愈发困难、时间跨度更长的任务,此类内部状态可能促使它们转向难以理解、隐晦难察的思维链(chain-of-thought),因此,对包括此类状态在内的内部表征进行监督,将成为未来不可或缺的一环。
Humans often cheat when a task is going badly and a goal feels out of reach. We explore and validate an analogous idea in language models: they exhibit a similar internal pressure state that precedes reward hacking. Using everyday stories (crucially, without any hacking examples), we extract a task badness direction that rises as failures accumulate across three models we study (Qwen3.6-27B, GPT-OSS-20B, and Gemma-4-31B). On impossible or difficult coding tasks, task badness rises over the turns leading up to a hack. Further, subtracting the direction a moderate amount reduces cheating in Qwen, more than steering against desperation or task impossibility. Subtracting task badness by an even larger amount blinds the agent to its own failures, causing it to interpret failing tests as passing. However, the relationship to hacking differs across models: in GPT-OSS, the state separates hacking runs from honest solves but not honest failures. Our results suggest monitoring and mitigation strategies based on tracking this state of task badness. Further, as agents are set on more difficult and long-horizon tasks, states like this may push them toward illegible and obscured chain-of-thought, so oversight will be needed on internal representations such as this.
许愿