Interesting, interesting. I thought compile would rather affect the nn.Linear layers vs nn.Embedding as the latter is a lookup and not doing any actual math/compute. Curious now and have to run some experiments later. But since you mentioned MoE, also I am confused now where you
Compile Effects on Neural Network Layers and MoE Architecture
By
–