Grounding latent algorithm routing in transformer reasoning
Xiangbo Zhang · Xiaoxu Ma
Transformers may first infer which solver family fits an in-context episode. On ROUTEBENCH, models track latent regime changes, resist formatting perturbations, and support causal route edits, supporting latent algorithm routing.
摘要
A central question in the in-context learning literature is whether transform- ers can organize episode-level adaptation around different inductive-bias families. We study this question in a controlled setting through latent algorithm routing: route-like behavior in which the solver-family prefer- ence changes with the latent data-generating regime while prompt form is held fixed, remains stable under nuisance perturbations, and is selectively influenced by targeted activation interventions without large losses in an- swer quality. We introduce ROUTEBENCH, a diagnostic benchmark whose regimes differentially favor global shrinkage, sparsity, robustness, and local- ity, operationalized by ridge-like, lasso-like, Huber-like, and kNN-like fam- ily representatives. Across dense decoder-only transformers trained from scratch at 44M–612M parameters, a 306M model closes 80.9% of the oracle- routing gap and achieves route F1 84.1. The effect remains substantial under natural-language renderings, shuffled supports, lexical paraphrases, and a unified four-way routing setting. Stronger adaptive alternatives—an input- conditioned soft mixture and an unsupervised Gumbel router—narrow the gap but remain below the 306M/612M models on route F1 and OOD per- formance. Probe controls and matched activation-patching controls further show that route-relevant internal directions are decodable and functionally involved in solver-family-consistent output behavior. These results pro- vide controlled evidence that dense transformers trained on ROUTEBENCH can develop route-like internal variables; they do not establish universal routing in pretrained language models or unrestricted natural-language reasoning.