Yes, this is standard for general-purpose fine-tuning. Llama 3.1 uses 17% of code for pretraining and 14.89% for post-training.
LLMS
-
Code in Pretraining Data Improves Model Performance
By
–
Nice paper from @Cohere about the impact of code in the pretraining data: • Including code improves performance on non-code tasks. It would be nice to see the effect on math benchmarks. [1] • Optimal proportion is around 25% but there's not a lot of data points: what about
-
Llama 3.1 70B Performs Well on HuggingChat Platform
By
–
llama 3.1 70B on huggingchat nails it too: https://
hf.co/chat/r/d8VmAu9 -
OpenAI Fine-Tuning, Claude Copyright Suit, AI Image Tools
By
–
Top stories in AI today: -OpenAI adds free fine-tuning to GPT-4o
-Claude sued for copyright infringement
-Create AI images in real-time on WhatsApp
-Microsoft’s new AI beats larger models
-6 new AI tools & 4 new AI jobs Read more: http://
therundown.ai/p/openai-gpt-4
o-fine-tuning
… -
Gemini Becomes Unresponsive With Three Korean Characters
By
–
You can cause Gemini to become unresponsive by simply entering the three characters “자인이” @OfficialLoganK
-
Grok-2 Pushes AI Ethics And Innovation Boundaries
By
–
AI Gone Wild: How Grok-2 Is Pushing The Boundaries Of Ethics And Innovation Elon Musk's latest #AI model is breaking barriers and raising eyebrows. This article explores how #Grok-2 is challenging our understanding of AI #capabilities and #ethics and what it means for the
-
MagicDec: Speculative Decoding Breaks Latency-Throughput Tradeoff
By
–
MagicDec Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding discuss: https://
huggingface.co/papers/2408.11
049
… Large Language Models (LLMs) have become more prevalent in long-context applications such as interactive chatbots, document analysis, and -
Transfusion: Multi-Modal Model for Token Prediction and Image Diffusion
By
–
Transfusion Predict the Next Token and Diffuse Images with One Multi-Modal Model discuss: https://
huggingface.co/papers/2408.11
039
… We introduce Transfusion, a recipe for training a multi-modal model over discrete and continuous data. Transfusion combines the language modeling loss function -
GenAI Adoption Tools: Open Source LLMs and Python Packages
By
–
What tools are organizations using to execute their GenAI strategy?
— Databricks (@databricks) 21 août 2024
The new State of Data + AI report highlights trends in GenAI adoption, with a big focus on open source LLMs. This chart showcases the top GenAI Python Packages by use case.
Learn more: https://t.co/qgK12erMYm pic.twitter.com/l0WQL9n9tkWhat tools are organizations using to execute their GenAI strategy? The new State of Data + AI report highlights trends in GenAI adoption, with a big focus on open source LLMs. This chart showcases the top GenAI Python Packages by use case. Learn more: https://
dbricks.co/4dhJBjH