GPT-2 is the "hello world" of LLMs I think (there must be a better analogy… err MOS 6502? xv6?), so that's why I started there. And it has a proper paper, weights released and available, and a lot is known about it. At this point it is an artifact of historical significance.
LLMS
-
GPT-2 Weight Initialization and Fine-tuning Strategy
By
–
atm we're doing init from gpt-2 weights and finetuning. this was very useful for debugging and when the code was slower. there is no code yet to init from scratch, so no code to warmup the lr etc. should be a very short addition though.
-
Synthesizing AI Responses with RAG Pipelines and SFT Models
By
–
It can be done: Gather any available real query examples, add in synthetic Q’s derived from docs. Synthesize responses using RAG pipeline. SFT (a smaller model) on resulting Q&A’s, without RAG context. Whether it saves any money is another question.
-
Meta AI vs ChatGPT: Internet Search Capabilities Compared
By
–
That's not an equal comparison, because Meta AI has the ability to search the internet and incorporate the search results into its responses – ChatGPT can do that too with Bing and GPT-4, but can't do that with 3.5
-
Fine-tuning GPT-3.5: Use Cases and Community Experiences
By
–
Using https://
platform.openai.com/docs/guides/fi
ne-tuning
… ? What have you used that for – I've not heard many examples of people fine-tuning 3.5 -
PyTorch Support and llm.c Project Development Status
By
–
Part agree! I love PyTorch ofc. But also llm.c is a ~2 week old project that is worked on by ~3 people as a hobby in spare time.
-
Meta AI Launches Llama 3 with Search and Summarization
By
–
If you just want to try it as a user https://
meta.ai is running Llama 3 with the ability for it to execute and summarize searches -
Llama 3 API Search Capabilities vs Perplexity Sonar
By
–
I've not see a Llama 3 API with built in search capacity yet – @perplexity_ai have sonar-medium-online but I think that's Mixtral 8x7B? They might upgrade to a Llama 3 online model soon though
-
llm.c Matches PyTorch Performance Training GPT-2 on GPU
By
–
llm.c update: Our single file of 2,000 ~clean lines of C/CUDA code now trains GPT-2 (124M) on GPU at speeds ~matching PyTorch (fp32, no flash attention) https://
github.com/karpathy/llm.c
/blob/master/train_gpt2.cu
… On my A100 I'm seeing 78ms/iter for llm.c and 80ms/iter for PyTorch. Keeping in mind this is fp32, -
Benchmarking Results: OpenAI and Anthropic Lead Agent Performance
By
–
Benchmarking Agents We’re happy to share updated benchmarking results using `langchain-benchmarks` to evaluate agents based on different models. @OpenAI and @AnthropicAI take the lead across a variety of tasks. For more explanation about the various benchmarking