As a funny aside, one of the early private beta stories I circulated among friends a few months ago as I was testing formats was literally titled “a model in your pocket” and about low-param OSS models. Whoops.
GENERATIVE AI
-
JAX Omission Raises Questions About AI Training Technology Strategy
By
–
There’s no mention of JAX. That was a little weird as they own an extremely powerful next-generation training technology in the same way they owned TensorFlow at first.
-
Google’s Strategic Missteps in AI and Open Source Innovation
By
–
Google keeps falling asleep at the wheel here. It didn’t productize MapReduce, allowing Hadoop to happen. It let PyTorch blossom and take over the DS/DL community. And now it has basically let competitors run off with the Transformers technology.
-
Compare GPT Models Simultaneously with This New Tool
By
–
Ever wish you could compare GPT models simultaneously ? @zwack010 built a sample app so you can do it in seconds.
— Replit ⠕ (@Replit) 5 mai 2023
Just plug in your API key and start prompting. pic.twitter.com/fBCNbyR8PZEver wish you could compare GPT models simultaneously ? @zwack010 built a sample app so you can do it in seconds. Just plug in your API key and start prompting.
-
GPT-4 Architecture: Parameter Count and Competitive Moat Analysis
By
–
OpenAI is mentioned in here as having no moat, but OpenAI was also intentionally silent about the parameter count of GPT-4. I know some people have been flinging around a 1T parameter number for GPT-4 but it’s very possible that it isn’t a single 1T model and instead 1T total.
-
Meta’s LLaMA Leak: Legal Implications and Open Source Adoption
By
–
And you have to appreciate the stance here which is basically like “what is Meta going to do, sue all the people using the leaked LLaMA model?”
-
Meta’s FAIR Outpaces Google in Open Source AI Leadership
By
–
Meta comes out looking so good from this. Meta’s AI division FAIR has consistently outmaneuvered Google in the open source deep learning community, starting with PyTorch and now with LLaMA.
-
Custom Language Models Spawn New Generation of AI Startups
By
–
An entire generation of startups are going to be born and grow on the back of these technologies because they are so lightweight and cheap to make custom implementations. I have yet to talk to a company deploying language modeling that is not playing around with custom models.
-
Open Source LLaMA Models Achieving Foundation Model Performance
By
–
What is new is just how many times this has been applied to LLaMA and other models to squeeze near-equivalent foundation model performance out of a small number of parameters. But that’s basically what happens when the cat comes out of the bag in open source.
-
GPT Quantization and LoRA: Historical Context of Optimization Techniques
By
–
Those techniques are not explicitly new—GPT quantization goes back to October 2022 (and Google released a paper about leveraging sparsity in training in 2021). LoRA shows up in a paper in 2021.