OPEN SOURCE

Extending LLaMA 3 Context Window with PoSE Method

By

–

26 April 2024 4h16

Starting on using PoSE to extend LLaMA 3's context window, with @winglian
's help. This one was really fun to work on. Eval loss started insanely high, let's see how far down we can get it.

Extending L3 70B context length with Unsloth on single A100

By

–

26 April 2024 0h16

I almost always use (and LOVE) Axolotl… but wanted to try Unsloth to try to extend L3 70B's ctxlen w/ just one A100.

LLaMA 3-Instruct Dataset Format Structure Guide

By

–

26 April 2024 0h00

Specifically, I have a dataset in the LLaMA 3-Instruct format:
“`
system {{ system_prompt }}user {{ user_message_1 }}assistant {{

SFTTrainer Response Template Training Logic Issue

By

–

25 April 2024 23h51

Is it just me, or does the @huggingface SFTTrainer's approach to training only on completions make no sense? What if your response template shows up multiple times in your example? Am I misunderstanding this?

→ View original post on X — @swyx

Postgres Renaissance: Vector Databases Give Way to SQL Evolution

By

@swyx

–

25 April 2024 22h07

if 2023 was the New Hope of vector dbs

then 2024 is Postgres Strikes Back@1st1 figured out how to add support for streaming RAG in 3LOC with deferred indexes in edgedb. pretty natural when you are already rewriting SQL for the AI/post-relational era! https://t.co/7iiY1XKNAa pic.twitter.com/5u7HFtpVG0
— swyx 🐣 (@swyx) 25 avril 2024

if 2023 was the New Hope of vector dbs then 2024 is Postgres Strikes Back @1st1 figured out how to add support for streaming RAG in 3LOC with deferred indexes in edgedb. pretty natural when you are already rewriting SQL for the AI/post-relational era!

→ View original post on X — @simonw

Improving Homebrew Documentation for Open Source Tools

By

@simonw

–

25 April 2024 20h44

I wish Homebrew was better about providing usage documentation! https://
formulae.brew.sh/formula/whispe
r-cpp
… has basically no clues at all – the best I could find was the test in the formula and that's still not enough information:

→ View original post on X — @langchain

Advanced Vector Retrieval with Graph-Based Metadata Using LangChain Neo4j

By

@langchain

–

25 April 2024 20h32

Optimizing vector retrieval with advanced graph-based metadata techniques using LangChain and Neo4j Text embeddings and vector similarity search help us find documents by understanding their meanings and how similar they are to each other. However, text embeddings aren’t

LLaMA 3 Extended Context Window Reaches 128K Tokens

By

–

25 April 2024 18h20

It's been a week since LLaMA 3 dropped. In that time, we've:
– extended context from 8K -> 128K
– trained multiple ridiculously performant fine-tunes
– got inference working at 800+ tokens/second If Meta keeps releasing OSS models, closed providers won't be able to compete.

→ View original post on X — @karpathy

Code Simplicity vs Compression: The LOC Paradox

By

@karpathy

–

25 April 2024 18h05

[gif] me trying to read tinygrad code earlier 😀

I think the LOC requirements (which are only a proxy for simplicity) led to too great compression. You wouldn't brag about your .min.js code being 1 LOC. Imo it would be a lot more simple if the code was given room to breathe and… pic.twitter.com/K4krMQdRkU
— Andrej Karpathy (@karpathy) 25 avril 2024

[gif] me trying to read tinygrad code earlier 😀 I think the LOC requirements (which are only a proxy for simplicity) led to too great compression. You wouldn't brag about your .min.js code being 1 LOC. Imo it would be a lot more simple if the code was given room to breathe and