Have you tried seeing what impact different amounts of extra tokens has? Have you looked at the impact on downstream tasks too?
@jeremyphoward
-
Fine-tuning pretrained models with new tokens
By
–
That's interesting – I haven't seen that done before. So did you just add a few tokens to a pretrained model, and then fine-tune them somehow? Or left them as randomly initialised?
-
Are Thought Tokens Different Pause Tokens in LLMs?
By
–
Are thought1, thought2, thought3… actually different tokens? Or all copies of the same pause token?
-
fastai Updates for PyTorch 2.1 Compatibility
By
–
Just pushed out a new version of fastai — now compatible with the recently-released @PyTorch version 2.1.
-
LSH-Based IO-Optimized Algorithm for Machine Learning
By
–
Why not both? I wonder if an LSH-based IO-optimised algorithm would work well.
-
AI industry shifts toward closed source profit-driven projects
By
–
Not sure that'll happen — AFAICT it's largely moving to closed source secret projects now that are focused on making money, instead of creating things to benefit all society.
-
Whatever Happened to Reformer Efficient Transformer Architecture
By
–
Whatever happened to "Reformer: The Efficient Transformer"?
-
GPT unexpectedly repeated initial prompt in subsequent replies
By
–
It's actually never done that before – I put it in the first prompt, then GPT put it in following replies for some reason.
-
Understanding Wave Quantization Effect in AI Systems
By
–
Sorry what's the wave quantization effect?
-
FP8 Training: Understanding Gradient and Weight Data Types
By
–
Why does it matter for fp8? Are grads and weights different data types in that case? (Sorry if it's a dumb question – I've never done any fp8 training)