hey @fchollet . I think this is tricky to do in general (like a no free lunch way).
I think the Mistral-7B implementation KerasNLP writes out a KVCache (for compiler optimization) and specifically writes code in a way that is needed for XLA Compilation to work well.
Do let me
Mistral-7B KVCache Optimization for XLA Compilation
By
–