Dude…. big fan. Your character-level GPT models were how I initially got into training models
LLMS
-
Nostalgic complaint about modern token streaming versus sequential watching
By
–
ugh kids these days! back in my days we used to watch the tokens stream one at a time and wait for the output.
-
RoPE Scaling Enables 16K Token Context Window Models
By
–
Yeah — simply multiplied rope * 2, trained on long data with a MSL of 16K.
-
LLaMA 3 70B enables instant code refactoring on Groq
By
–
Using LLaMA 3 70B on @GroqInc to instantly refactor and document code.
— Matt Shumer (@mattshumer_) 22 avril 2024
The implications for software engineering are wild.
Gone are the days of waiting on an LLM for suggestions or code changes.
Now, it's an instant feedback loop.
Demo link in the comments: pic.twitter.com/joLJy4oJ3TUsing LLaMA 3 70B on @GroqInc to instantly refactor and document code. The implications for software engineering are wild. Gone are the days of waiting on an LLM for suggestions or code changes. Now, it's an instant feedback loop. Demo link in the comments:
-
LLM Plugin Directory Now Features 30 Available Plugins
By
–
The LLM plugin directory lists 30 plugins now! https://
llm.datasette.io/en/stable/plug
ins/directory.html
… If you want to try writing your own there's a comprehensive tutorial included in the docs: https://
llm.datasette.io/en/stable/plug
ins/tutorial-model-plugin.html
… -
LLaMA 3-based model deployment nearing completion
By
–
Very soon. This is one of the last things I have to get working well before we can push a LLaMA 3-based model live.
-
Extending Context Length for Base Model Development
By
–
Planning to do this for the base model, at least at first. It's more of a learning thing for me, to help me perfect the context length extension approach so I can do it for our internal HyperWrite stuff… so I don't want to do too many variations.
-
Llama 3 Announcement Highlights Data Quality’s Impact on AI Performance
By
–
Llama 3 announcement says the quality of both SFT and preference data have an “outsize influence” on performance. The first excerpt above alludes to an explanation for why preferences would help with reasoning. (This screenshot is the text just before the one above:)
-
LLaMA 3 8B Extended to 16K Token Context Window
By
–
Training a LLaMA 3 8B that supports 16K tokens, twice the current context window. If successful, will open-source it.
-
Mistral Mixtral 8x7B Mixture of Experts Architecture Course
By
–
New short course with @MistralAI !
— Andrew Ng (@AndrewYNg) 22 avril 2024
Mistral's open-source Mixtral 8x7B model uses a "mixture of experts" (MoE) architecture. Unlike a standard transformer, an MoE model has multiple expert feed-forward networks (8 in this case), with a gating network selecting two experts at… pic.twitter.com/VFOg1dDab8New short course with @MistralAI ! Mistral's open-source Mixtral 8x7B model uses a "mixture of experts" (MoE) architecture. Unlike a standard transformer, an MoE model has multiple expert feed-forward networks (8 in this case), with a gating network selecting two experts at