In-context learning is just gradient descent without explicit training! This paper "Learning without training: The implicit dynamics of in-context learning" shows that ICL can be mathematically interpreted as an implicit low-rank weight update during inference.
In-Context Learning as Implicit Gradient Descent During Inference
By
–
