Interesting, interesting. I thought compile would rather affect the nn.Linear layers vs nn.Embedding as the latter is a lookup and not doing any actual math/compute. Curious now and have to run some experiments later. But since you mentioned MoE, also I am confused now where you
@rasbt
-
AI Systems Solving Voice Recognition and Cocktail Party Problem
By
–
maybe that’s the use case for current/next gen AI systems. I mean voice inputs have been pretty much solved years ago. But there are still false positives, and newer systems may solve the cocktail party problem here.
-
torch.compile Performance with nn.Embedding Layers
By
–
Interesting, do you mean torch.compiling here? And if yes, why/how does that work in terms of speed-up since nn.Embedding is basically just a tensor for cheap look-up? (But also, if you are already compiling the model, why would you skip the embedding layer in the first place?)
-
Different paradigms and audiences for proprietary versus open-weight models
By
–
We will see! They are both different paradigms and target different audiences (proprietary vs open-weight, unless they change it with V4).
-
Internal metrics and taxonomy in AI model evaluation
By
–
You are not wrong but that was a deliberate as I covered the internal metrics (loss, perplexity, rewards) many times before . Thanks for sharing the taxonomy paper btw!
-
DeepSeek V4 vs Gemini 3 Pro: October Release Race
By
–
It’s October. DeepSeek V4 or Gemini 3 Pro, who wants to go first?
-
Multiple Choice Questions as Model Training for Real-World Performance
By
–
What would be useful is when including more MC questions would also improve the answer performance on similar-topic questions where answer choices are not shown (like in a real world context). That being said I can see models who do well on MC settings being useful as verifiers
-
Internal Sanity Checks vs External Evaluations for LLM Benchmarking
By
–
Yes, it’s cheaper and easier, but it’s more of an internal sanity check than outward facing eval to report imho. Btw, spot on regarding including it for the sake of benchmarks. You can tell based on how sensitive some LLMs are to the exact MC prompt format.
-
Open-ended Evaluations: Free-form Answers and Conversational Performance
By
–
With open-ended evals you mean free-form answers (and measuring conversational performance)?
-
Verifier and LLM-as-Judge for Output Conformance
By
–
Good suggestions. I'd say those fall into the verifier category (perhaps also LLM-as-a-judge for output-conformance); or do you use something different?