The "imaginary" part of RoPE contains valuable information for long-range dependencies RoPE++ reintroduces it in heads with slower decay curves that better capture distant tokens We provide implementations with one-command scripts for both the original codebase and Nanochat!
OPEN SOURCE
-
Apache 2.0 Licensed Model Available on AI21 SaaS and Hugging Face
By
–
4/4 Licensed under Apache 2.0 and available today on AI21 SaaS and Hugging Face:
-
Jamba2: Open Source Model for Enterprise Reliability
By
–
1/4 Introducing Jamba2, a memory-efficient open source model family built for total enterprise reliability and steerability.
-
Small Models Over Large: Efficiency and Memory Footprint
By
–
Reject big models, embrace small memory footprint
-
Write High-Performance Hardware Kernels in Python with Pallas
By
–
You no longer need to leave Python to write high-performance hardware kernels. Learn how to use Pallas in Keras to author custom ops that lower to Mosaic for TPUs or Triton for GPUs:
-
NVIDIA’s Alpamayo-R1: Open Weights 4D Understanding Autopilot Model
By
–
As NVIDIA pushed its first ever open weights autopilot model alpamayo-R1 A key frontier is native 4D understanding: not just “describe the video” but reason about depth + motion + 3D interactions over time, even for a specific region. This paper introduces 4D-RGPT (also from
-
Open Source PR Contributor Receives Death Threat
By
–
Also, this is how the submitter of the PR was treated (literal death threat), so I can understand why some folks were feeling heated
-
Atom Tech Stack Rewritten from Scratch in Rust
By
–
Ambitious, dissatisfied with the performance of the Atom tech stack at the time, rewrote it from scratch in Rust, idealism
-
Open Source Motivation: Helping People Achieve Their Goals
By
–
I'm quite familiar with it. I wrote my first piece of open source software >30 years ago, and have written and released hundreds of libraries. And I still have plenty of incentive to write good OS – it's the same as I've always had: to help people do the stuff they want to do.
-
Chinese OSS Models Dominance and US Competitive Risk
By
–
All leading OSS foundation models are Chinese. I anticipate that a services company will grow around many of them. That company will get big (or will be existing big). If America is forced to tariff/ban that big co à la BYD, it will be ceding yet another global race. America