This is a key point on AI agents "Brains" matter for agents. As long as the scaling law holds, you get much better results out of using bigger frontier models as the main brains of your agent, though it might be able to delegate to lesser AIs Labs with the best model get the
LLMS
-
User Requests Additional A100 GPUs for Bixtral Model Testing
By
–
Hey @ClementDelangue would you mind adding two extra A100s (total 4) to my account? Want to run some tests on Bixtral (big Mixtral) using HF Endpoints.
-
Explaining LLM.c: Simplifying Large Language Model Training
By
–
# explaining llm.c in layman terms Training Large Language Models (LLMs), like ChatGPT, involves a large amount of code and complexity. For example, a typical LLM training project might use the PyTorch deep learning library. PyTorch is quite complex because it implements a very https://
t.co/dkpZGRvkmU -
Gemini Pro Output Token Limit Impact on Model Performance
By
–
Yeah the cap on output tokens is a really important variable in all of these – Gemini Pro's limit is 8,192
-
Base Model Release Announcement Without Instruction Fine-Tuning
By
–
FYI: This is a base model and not instruction fine-tuned, please stay tuned!
-
Mixtral-8x22B Fine-tuning Configuration with LoRA Parameters
By
–
base_model: mistral-community/Mixtral-8x22B-v0.1
model_type: AutoModelForCausalLM
tokenizer_type: LlamaTokenizer
trust_remote_code: true load_in_8bit: false
load_in_4bit: false
strict: false unfrozen_parameters: – ^lm_head.weight$ – ^model.embed_tokens.weight$ – -
Selective Layer Freezing for Language Model Fine-tuning Strategy
By
–
Freezing all layers except these: – ^lm_head.weight$ – ^model.embed_tokens.weight$ – model.layers.4[4-9]+.block_sparse_moe.gate – model.layers.4[4-9]+.block_sparse_moe.experts – model.layers.5[0-5]+.block_sparse_moe.gate – model.layers.5[0-5]+.block_sparse_moe.experts
-
GPT-4 Turbo Needle-in-Haystack Long Context Benchmark
By
–
New GPT-4 Turbo needle-in-haystack long context benchmark:
-
CUDA kernels for GPT-2 forward pass implementation in llm.c
By
–
Okay I did a first quick pass of naive CUDA kernels for the forward pass of GPT-2 and pushed everything to one file in llm.c, Still only ~1000 lines of code: https://
github.com/karpathy/llm.c
/blob/master/train_gpt2.cu
… Current per iteration timings on my Lambda box <3 A100 40GB PCIe, B=4, T=1024:
– llm.c: 111ms
– -
ChatGPT Model Update Status Discussion
By
–
I don’t think ChatGPT has been updated to this model just yet