read more: https://
alphaxiv.org/abs/2605.02105
@askalphaxiv
-
Sharpness-Aware Pretraining for Mitigating Catastrophic Forgetting in AI Models
By
–
“Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting” Better base models don’t always become better fine-tuned models, because post-training can overwrite pretrained capabilities. So this paper pretrain models into flatter loss regions, so later updates from
-
Manifold Steering for Neural Network Representation and Behavior
By
–
"Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior" Most activation steering treats concepts like directions, but model representations are often curved, so straight line steering can cut through unnatural internal states. The key idea
-
Research on the Effects of Iterative Finetuning on AI Model Behavior
By
–
"Iterative Finetuning is Mostly Idempotent" If a model has a behavioral tendency and you train the next generation on its outputs, does the tendency snowball? This paper actually finds that it's mostly no. So they seed models with traits like sycophancy and misalignment, then
-
Demystifying Manifold Constraints in LLM Pre-training
By
–
"Demystifying Manifold Constraints in LLM Pre-training" LLM pre-training depends on a stack of heuristic tricks, eg. normalization layers, weight decay, and optimizer tuning. This paper suggests that by constraining weight matrices to geometric manifolds like spectral or
-
Continuous Latent Diffusion Language Model Advances
By
–
“Continuous Latent Diffusion Language Model” Most diffusion language models still use diffusion to recover token-like states, just in a different generation order. However, this paper uses diffusion in a different way. It learns a continuous latent prior for global semantics
-
Sparser, Faster, Lighter Transformer Language Models
By
–
“Sparser, Faster, Lighter Transformer Language Models” LLMs are naturally sparse in their feedforward layers, but unstructured sparsity usually doesn’t get you real speed on GPUs, because the hardware stack is built for dense compute. The key idea of the paper is to redesign