We need a Consumer Reports or Underwriters Lab for AI testing. All the public benchmarks are game-able and mostly not useful measures of things LLMs do. We need secret test batteries for subject areas (coding, reasoning, human conversation, writing) and secret red team tests, too
AI
-

Claude 3 Creates Original Optical Illusions and Visual Patterns
By
–
I asked Claude 3 to create original optical illusions. You are supposed to see:
— Ethan Mollick (@emollick) 24 avril 2024
"A perception of tilted lines from a pattern of alternating rows of tiles"
"A mesmerizing pattern"
"A square that appears to be breathing or pulsating, even though it's actually just changing size" pic.twitter.com/FrMOAx7isiI asked Claude 3 to create original optical illusions. You are supposed to see:
"A perception of tilted lines from a pattern of alternating rows of tiles"
"A mesmerizing pattern"
"A square that appears to be breathing or pulsating, even though it's actually just changing size" -
Model License Terms and Commercial Use Clarification
By
–
As far as I can tell the license for this model is here – it looks to me like you can use it commercially but I don't recognize the general shape of the license and IANAL
-
OpenELM Model Weights Now Available on Hugging Face
By
–
Here are some OpenELM weights on a Hugging Face, anyone got these working yet?
-
OpenELM Training Data Sources Documentation
By
–
This is the best documentation I could find of the OpenELM training data – it looks like the bulk of it comes from RefinedWeb, RedPajama, The Pile and Dolma https://
github.com/apple/corenet/
blob/main/projects/openelm/README-pretraining.md
… -
Apple releases CoreNet: open neural network training library
By
–
CoreNet looks to be the parent project – brand new on GitHub as of a few hours ago, "a library for training deep neural networks" Includes:
– OpenELM: a brand new LLM with open training data
– CatLIP: a CLIP-like model with more efficient training? More: https://
github.com/apple/corenet -
Apple Releases Significant ML and LLM Research
By
–
Apple appear to have just released a huge volume of fascinating ML and LLM work in one go, I'm still trying to get my head around what they've published here
-

Claude 3 excels at creative puzzle-solving in teaching games
By
–
Claude 3 is pretty good at puzzles. One of our teaching games takes place on a doomed mission to Saturn, and a minor challenge is naming the ship where you are given an over-the-top manual and a set of criteria and have to come up with a name. People struggle, Claude nailed it.
-

Phi 3 Mini Model Issues: Looping and Autoregressive Problems
By
–
As I use Phi 3 mini more, it is clear that model is a bit weird. It is usually quite good to start, but tends to loop quite often, repeating things, and also sometimes descend into autoregressive madness.