1000 tokens of text compressed at 1.2 bits/char and 4 chars/token ≈ 600 bytes 1000 tokens of kvcache in LLAMA 70B in fp16 takes up 8192 dim x 80 layers x 2 x 16 bit ≈ 2.6 GB so the kv cache is 2.6 GB / 600 bytes = 4.4 million times larger than the input make this make sense
@jxmnop
-
Entropy Approximation Methods in Machine Learning Models
By
–
all the experiments in the paper approximate Hᴷ(x | θ) using bits under arithmetic coding, which is equivalent to cross-entropy here Hᴷ(x) is approximated using the formula for entropy of random uniform distribution for synthetic, and AC/xentropy of 'reference model' for text
-
Understanding Neural Tangent Kernels in Machine Learning
By
–
well jokes on both of you because i still don't understand neural tangent kernels
-
NeurIPS submissions surge 10x: Is AI science accelerating?
By
–
▂▃▆ is AI really experiencing explosive growth? this year over 25,000 papers were submitted to NeurIPS, up over 10x from 2,400 in 2016. poll for AI researchers: do you think AI *science* is progressing more quickly, more slowly, or at the same pace?
-
Experimental Results and Equations in AI Research
By
–
i tried this one first, it's more of a collection of equations; the experimental results are pretty important here…
-
Neural Tangent Kernel: Timeless Mathematical Foundation vs LLM Research
By
–
maybe, but unlike most research e.g. in LLMs, the neural tangent kernel is simply *true* and thus will still be relevant in 5 or 10 or 50 years
-
Neural Tangent Kernel: Understanding Deep Learning Foundations
By
–
most foundational concept in deep learning that no one understands is probably the Neural Tangent Kernel (NTK) this line of work studies neural networks of *infinite width*, which explain a lot about normal finite-width NNs and there is exactly one Very Good blog post on them:
-
Appreciation for inspirational open-source AI work
By
–
it is related, and thank you!! your open-source work is inspirational
-
Meta Employment: No Full-Time Position Guarantees
By
–
i love meta, but it's not like they guarantee you a full-time position
