Check out the imaginary RoPE implementation here!
LLMS
-
RoPE++ Improves Long-Range Dependencies in Language Models
By
–
The "imaginary" part of RoPE contains valuable information for long-range dependencies RoPE++ reintroduces it in heads with slower decay curves that better capture distant tokens We provide implementations with one-command scripts for both the original codebase and Nanochat!
-
Model Reliability Over Raw Capability in AI Selection
By
–
The big reason why people choose better models isn’t so much IQ points, but reliability. This is why smaller models often suck – it isn’t that they literally can’t do the task, but that they don’t do it anywhere near reliably enough. If you used Sonnet 3.5 back in the day, and
-
Memory Systems for Long-Horizon Coding Agent Reasoning
By
–
On building memory systems for long-horizon reasoning tasks. This is a really important topic, especially for improving coding agents like Claude Code. Current Memory-Augmented Generation approaches rely on semantic similarity over monolithic memory stores. Everything gets
-
Apache 2.0 Licensed Model Available on AI21 SaaS and Hugging Face
By
–
4/4 Licensed under Apache 2.0 and available today on AI21 SaaS and Hugging Face:
-
Jamba2 Outperforms Competing Models in Enterprise Reliability
By
–
2/4 Grounding + instruction following
Jamba2 outperforms peer models from @MistralAI and @alibaba_cloud on IFBench, IFEval, and FACTS, core indicators of enterprise reliability. -
Jamba2: Open Source Model for Enterprise Reliability
By
–
1/4 Introducing Jamba2, a memory-efficient open source model family built for total enterprise reliability and steerability.
-
Small Models Over Large: Efficiency and Memory Footprint
By
–
Reject big models, embrace small memory footprint
-
End-to-End Test-Time Training Eliminates KV Cache Limitations
By
–
Farewell to the shackles of KV Cache, compressing long contexts into weights—is there hope for continuously learning large models? Researchers from Stanford, NVIDIA, UC Berkeley, and the Astera Institute present a new method called End-to-End Test-Time Training (TTT-E2E). They
-
Appreciation for nanochat upgrade and request for scaling laws video
By
–
Thanks for this solid upgrade on nanochat, Karpathy! Hoping you release a video for the science of scaling laws 🙂