Confabulations or #Hallucinations? Balancing creativity and accuracy with commercial #LLMs presents interesting challenges. Check out this insightful article via @benjedwards @arstechnica on some of the risks and limitations of #generativeAI. #chatgpt http://
ow.ly/qG2650NEJnb
LLMS
-
LLM Hallucinations: Balancing Creativity and Accuracy in Generative AI
By
–
-
Understanding in LLMs: Emergent Properties Beyond Token Prediction
By
–
Bullshit machines might be harsh. The real debate is if the predicting the next token requires these models to build some level of understanding — even if it isn’t reliable. The whole emergent property of LLMs is fascinating, even if they can’t hierarchically, decompose their
-
LLMs Cannot Verify Truth: The Persistent Problem
By
–
What LLMs say is sometimes true, sometimes not. They can’t tell the difference: they don’t know how to do validity checks (eg crossreferencing WIki or their own training corpus). That’s what makes it BS. First said it in @techreview 2020; still true: https://
technologyreview.com/2020/08/22/100
7539/gpt3-openai-language-generator-artificial-intelligence-ai-opinion/
… -
High Quality Corpus Training Efficiency for Language Models
By
–
There seems to be some growing hints that a few epochs of training on a high quality (and well deduplicated corpus) is just fine
-
ASICs for LLM Inference: Specialized Hardware Solutions
By
–
Who is working on ASICs for inference on pre-trained LLMs? LLMs with frozen weights seem like ideal candidate for specialized hardware that can accelerate the inference process, reduce energy consumption, and potentially lower costs, making them an attractive option for
-
SGD and Transformer Network Depth Rules Matter
By
–
No, I asserted here that SGD was *a* thing. I mention elsewhere that the fact that a classical 100-layer transformer network obeys its own "100-step rule" is another thing that matters (and also is not inaccessibly deep mathematics).
-
Dataset Filtering for Large Language Model Training Runs
By
–
Possibly yes. More broadly, if we're going to be doing more large training runs at all, I'd guess we should start filtering the datasets soon (presumably using a previous-generation LLM finetuned for that?). It's hard to finetune out a cognition once it's learned by the base.
-
Open Source Translation Models: Transformers vs English Pipeline
By
–
yes open source, vrai travail, par contre sur la fin il parle de traduction et qu'il dit que ca passe par de l'anglais et là c'est pas du tout ça, ils passent par les transformers et pour l'entraînement et Meta a les même résultats en terme de traduction
-
LLMs Confabulate Inherently Despite Infinite Data
By
–
Conjecture: even with infinite data, LLMs would still confabulate, because they blur the inputs and don’t reliably create precise representations of individuals and their properties. (see The Algebraic Mind, 2001 for related discussion which has thus far has held true)
-
Baby GPT as Finite State Markov Chain Visualization
By
–
This is a baby GPT with two tokens 0/1 and context length of 3, viewing it as a finite state markov chain. It was trained on the sequence "111101111011110" for 50 iterations. The parameters and the architecture of the Transformer modifies the probabilities on the arrows. E.g. we