But yeah, modeling performance-wise you'll probably take a hit. Same thing that happened when they distilled ChatGPT GPT-3.5 and later GPT-4
@rasbt
-
Making LLM serving affordable through pruning and quantization
By
–
Fair point, but I think the appeal is more in trying to make running LLM more affordable (via pruning and quantization) to reduce the number of GPUs required for serving.
-
Standard Practice for Pruning Linear Layers in Neural Networks
By
–
I haven't seen that yet. I think the standard is to prune all linear layers, this includes the QVK of the attention layers and the FC layers (except the embedding and output layers of course).
-
Pruning Large Language Models Without Retraining Using Activation Norms
By
–
LLMs are usually too large for most contexts, but creating pruned versions of a model usually require retraining. Here's a new straightforward alternative based on computing element-wise product between the weight magnitude and norm of input activations: https://
arxiv.org/abs/2306.11695 -
Praise for Deep Learning Focus Newsletter Quality
By
–
Likewise, your Deep (Learning) Focus newsletter is an absolute gem!
-
LLM Knowledge Retrieval Search Recommendation
By
–
Hm, maybe try to also search for "LLM knowledge retrieval". Off the top of my head, I can't make a particular recommendation, but @cwolferesearch probably has some resources handy there 🙂
-
SLURM cluster management: benefits and challenges for experimentation
By
–
That’s an interesting one. I have both fond and not so fond memories using SLURM on the departmental cluster. It’s actually great if you know exactly what you want to run. But yeah also make sure you have access to an interactive node for experimentation or you’ll go crazy 😛
-
PyTorch dominance in latest LLM and vision transformer research
By
–
That's totally true, but I think the problem is once you want to adopt or use more of the recent (research) stuff and/or need more customization. E.g., all the latest LLM and vision transformer is basically exclusively PyTorch.
-
PyTorch Preference: Balancing Ease of Use with Flexibility
By
–
For me it's PyTorch because I like the trade-off between being ease of use but also being flexible and customizeable. Plus almost all people I interact with (research and companies) are using PyTorch. But yeah, I think your mileage may vary based on your collaborators.
-
Fabric positioned between PyTorch and Trainer abstraction levels
By
–
Glad to hear it's useful! Btw if you are already using the Trainer, then you probably don't need Fabric. I would think of Fabric as a thing between pure PyTorch and using the Trainer 🙂