OPEN SOURCE

–

03 May 2024 23h25

It's worth noting that this code specifically trains GPT-2.
PyTorch trains anything under the sun.

GPU and Disk Constraints: Need for Smaller Dataset Variants

By

–

03 May 2024 23h12

I'm not only GPU poor but disk poor too. 350GB?
(And ofc doing so wouldn't be representative of the full data distribution)
Also while replying, ideally there could be a "dataset miniseries", e.g. 1B, 10B, 100B, and then full. I think would be very helpful and bandwidth saving.

cuBLASLt and cuDNN Dependencies for Optimized Performance

By

–

03 May 2024 20h53

Yes, cuBLASLt for gemms, cuDNN for flash attention
The fp32 version will become more educational and will delete these dependencies. The "mainline" version we just want to be really fast, so we're less discriminating. cuBLASLt I think is ~ok dep, but cuDNN turned out surprisingly

llm.c Day 24: Multi-GPU Training in C/CUDA Outperforms PyTorch

By

–

03 May 2024 20h22

Day 24 of llm.c: we now do multi-GPU training, in bfloat16, with flash attention, directly in ~3000 lines of C/CUDA, and it is FAST! We're running ~7% faster than PyTorch nightly, with no asterisks, i.e. this baseline includes all modern & standard bells-and-whistles: mixed

→ View original post on X — @langchain

LangChain Weekly Release: LangSmith Improvements and LangGraph Tutorials

By

@langchain

–

03 May 2024 18h08

LangChain Release Notes, Week of 4/29 Improved regression testing experience in LangSmith Hotkeys and mustache support in LangSmith New Evaluations content Three new video tutorials using LangGraph Community created content! Read it all here:

→ View original post on X — @langchain

Build RAG System Llama 3B-Instruct PDFs FAISS

By

@langchain

–

03 May 2024 17h00

Build a RAG system with Llama 3B-Instruct for your PDFs Uses API for partitioning & chunking, FAISS for vector store, huggingface for the model It's a collab notebook so it's easy to get started! Thanks @mariaKhalusova for a great resource! https://
colab.research.google.com/drive/1BJYYyrP
Ve0_9EGyXqeNyzmVZDrCRZwsg?usp=sharing#scrollTo=Y2m2l-vt_RSp
…