Bernhard Schölkopf

The rise of autonomous AI agents suggests that dynamic benchmark envi- ronments with built-in feedback on scientifically grounded tasks are needed to evaluate the capabilities of these agents in research work. We introduce STARGAZER, a scalable environment for evaluating AI agents on dynamic, iterative physics-grounded model-fitting tasks using inference on radial- velocity (RV) time series data. STARGAZER comprises 120 tasks across three difficulty tiers, including 20 real archival cases, covering diverse scenarios ranging from high-SNR single-planet systems to complex multi-planetary configurations requiring involved low-SNR analysis. Our evaluation of eight frontier agents reveals a gap between numerical optimization and adherence to physical constraints: although agents often achieve a good statistical fit, they frequently fail to recover correct physical system pa- rameters, a limitation that persists even when agents are equipped with domain-expert skills. Furthermore, increasing test-time compute yields only marginal gains, with excessive token usage often reflecting recursive failure loops rather than meaningful exploration. STARGAZER presents an opportunity to train, evaluate, scaffold, and scale strategies on a model- fitting problem of practical research relevance today. Our simulation-driven methodology may also provide a useful design pattern for other scientific model-fitting domains.

Large language model (LLM) agents act on inputs users do not control, including tool returns, retrieved documents, and messages. Indirect prompt injection embeds instructions in these channels to redirect an agent toward unintended actions. The same tool call may be appropriate for one task and harmful for another, making this threat difficult to address with input-level defenses alone. We ask whether robustness against indirect prompt injection can be built into the LLM itself, and instantiate this idea with Odile, a low-rank adapter trained on paired benign and harmful traces generated by the base model. Odile is designed to convert attacker-directed trajectories into a no-action state before a valid malicious tool call is produced, while preserving ordinary behavior on benign inputs. In matched evaluations, this single model-level defense matches or improves upon the security–utility tradeoffs of prompt hardening, runtime detection and rewriting, and prior fine-tuning defenses. Across three prompt-injection benchmarks, eight model backbones, and two defense-aware adaptive settings, Odile consistently reduces attack success while retaining benign behavior and ordinary model capability. It transfers from a compact training dataset to longer trajectories, different tool interfaces, mixture-of-experts models, and hybrid attention architectures. We release Odile at https://github.com/memo-ozdincer/odile

A central premise in mechanistic interpretability is that meaningful concepts in language models are represented by linear features in activation space. For such features to support reliable interventions, manipulating one feature should not substantially alter the effects of others. In practice, however, feature entanglement leads to interference such that localized interventions can have unintended downstream effects. Motivated by the _Independent Causal Mechanisms_ principle, we propose to constrain internal features to be almost orthogonal. We argue that this promotes modular representations amenable to causal intervention. We formalize this problem by characterizing the gap between an idealized isolated intervention and its realized effect on model outputs in terms of feature interference. We upper-bound the propagation of feature interference in terms of the self-coherence of the feature dictionary, and relate this discrepancy to an explicit orthogonality regularization on the dictionary itself. Empirically, we show that this regularization enables more isolated interventions on mathematical reasoning concepts while preserving model performance. Our code is available under https://github.com/mrtzmllr/sae-icm.