quite incredible to see the goalposts of long context move in the past 1 year. May 2023: asking @jefrankle and @abhi_venigalla about their 65k+ "Llongboi" model May 2024: @markatgradient casually extending Llama 3 to >1m tokens with ~perfect NIAH and mainstream ai engineers
Long Context Window Progress: From 65k to 1M Tokens in One Year
By
–
