I've doubled LLaMA 3's context window to 16K tokens. Fully open-source. Link in thread:
RESEARCH
-
LLaMA 3 Context Window Doubled to 16K Tokens
By
–
I've doubled LLaMA 3's context window to 16K tokens. Fully open-source. Link in thread:
-
Context Length Extension Testing and Performance Benchmarks
By
–
I haven't run real benchmarks on it (if anyone wants to, that'd be awesome!), but from my homemade haystack tests, the context length extension worked well.
-
LLaMA 3 Context Window Doubled to 16K Tokens
By
–
I've doubled LLaMA 3's context window to 16K tokens. Fully open-source. Link in thread:
-
LLMs as Arrays: Meditation on Entropy and Truth
By
–
As I emerged from meditation it dawned on me that LLMs are just one array of floats and a while loop over some super simple arithmetic on its elements. It is entropy that is the root of suffering. It's by deleting the superfluous that we uncover truth. And thus I was enlightened.
-
Evaluation Loss in Model Training: Understanding Its Role
By
–
eval loss. i don't always use it, as it's not a great measure, but it's good for things like this
-
ICLR Features Hugging Face Demo Platform for AI Models
By
–
ICLR blog: https://
blog.iclr.cc/2024/04/22/hug
ging-face-demo-site/
… -
Hugging Face Demo Site for ICLR Conference Paper Submissions
By
–
Hugging Face is hosting a demo site for @iclr_conf authors to find and claim their papers and discuss those papers on dedicated pages The page includes all accepted papers, along with linked datasets, models, and demos
-
Common Research Pitfall: Confounding Variables in Experiments
By
–
common dark thought pattern in research
> run baseline experiment
> change thing A
> also change thing B
> run new experiment
> collect results
> "wow, thing A works!" -
Lean Copilot Automates Over 80% of Mathematical Proof Steps
By
–
We have now posted an expanded version of our Lean Co-pilot paper and updated the code base. https://
arxiv.org/abs/2404.12534 https://
github.com/lean-dojo/Lean
Copilot
…
The credit goes to @p_song1 for driving this effort.
Our latest experiments show that our co-pilot can automate >80% of the proof steps