It's interesting that this "kernel" is designed for torch.compile(), so it's Python code but turns out faster! Conversely, some of the other Liger kernels are Triton and I measured them as slower than torch.compile'd versions.
@alexjc
-
Parameter Optimization in Large Language Models
By
–
Got it… Still, if the rule is you have to have <=124M parameters active per token, where did you save those 9216?
-
Model Design Trade-offs: Parameter Count and Efficiency Rules
By
–
Love the idea, fascinating design! But wait, it's almost 3.5x the number of extra active parameters? Doesn't that violate rule 2?
-
Token-level bit efficiency metric for model evaluation
By
–
I need to think about this more, but I'm proposing token-level calculation of that concept: sum(token_bits / token_bytes for _ in all_tokens) … and not: sum(tokens_bits) / sum(tokens_bytes) Because the second one would hide bad predictions for longer tokens in the sum(),
-
Tokenization Boundaries and Cross-Entropy Loss Optimization
By
–
I found the exact tokenization boundary of the text and the corresponding token index. I can do the normalization but it's not perfect… (Testing it shortly.) Computing the cross-entropy like this rewards vocabularies with more tokens and easy predictions (on average) vs.
-
Normalization and Token Counting in AI Model Evaluation
By
–
Isn't the point of normalization to control for the number of tokens (i.e. using mean rather than sum), so these 20% variations should be accounted for with acc_norm. I figured that's why most people use acc_norm these days!
-
Token Optimization and Model Performance Improvements Debate
By
–
Well, if you're willing to debate token equivalence I think that's a fair approach! The code itself hasn't changed and HellaSwag score is significantly higher given the same number of steps. It's not only fewer tokens on disk but they are "better" tokens too, IMHO +40% total.
-
Token Density Impact on Validation Metrics in Model Training
By
–
Nope, it wouldn't be the same; that's the catch! Because the token density is about 20% higher when the same text is processed, the validation metric ends up being around 4.18. Making the train/val boundary at exactly the same word is possible, but it won't change the mismatch.
-
GPU Training Issues with Latest PyTorch Nightly DDP
By
–
Glad to see the level of interest. I figured tokenization was too boring for most! The results on 8xH100 will have to wait until tomorrow; with the latest nightly torchrun is crashing in DDP with exit code -11.
-
HellaSwag Evaluation Script for Language Models Released
By
–
You can also find a HellaSwag evaluation script that works with one large batch item, EOT separators, and document_mask here: https://
github.com/alexjc/nanogpt
-speedrun/blob/master/eval_gpt2.py
… It's setup to use TokenMonster too, so you can check the scores on downstream benchmarks easily hopefully.