This is quite interesting … 1) I would expect that the opposite is true for, e.g., RNN-based LLMs like RWKV (since it's processing information sequentially, it might rather forget early information) 3/5
RESEARCH
-
Transformer Architecture and Middle Document Retrieval Performance Bias
By
–
2) To my knowledge, there is no specific inductive bias in transformer-based LLM architectures that explains why the retrieval performance should be worse for text in the middle of the document. 4/5
-
LLM Attention Weights: Training Data Structure and Human Writing Patterns
By
–
I suspect it is all because of the training data and how humans write: the most important information is usually in the beginning or the end (think paper Abstracts and Conclusion sections), and it's then how LLMs parameterize the attention weights during training. 5/5
-
Long Context LLMs: Promise and the Middle Information Problem
By
–
We have seen a new wave of LLMs for longer contexts: 1) RMT, 2) Hyena LLM & 3) LongNet There are several use-cases for such long LLMs but the elephant in the room is: How well do LLMs use these longer contexts? Turns out not so well if info is in the middle of the input.
1/5 -
LLMs Struggle Retrieving Information from Document Middle
By
–
According to "Lost in the Middle: How Language Models Use Long Contexts" (
https://
arxiv.org/abs/2307.03172) LLMs are good at retrieving information at the beginning of documents. They do less well in terms of retrieving information if its contained in the middle of a document. 2/5 -
AI Model Decodes Thoughts into Images with 80% Accuracy
By
–
Two researchers have created a new #AI model that can draw what you’re thinking with 80% accuracy https://
bit.ly/3FTXOoD -
India’s AI and Machine Learning Initiative for Nuclear Power
By
–
@_DigitalIndia @GoI_MeitY @NeGD_GoI @nasscom #ai #NuclearPowerplant #MachineLearning #INDIAai
-
Emotional AI Recognition: Low-Level Contagion vs. Theory of Mind
By
–
Oooh. Strikes me that one use the same method while having people view emotional images, some where it's low level emotional contagion, and others where it's high level theory of mind emotion recognition? Or caption an image on its objective content vs. emotional content…
-
Will AI End Traditional School Classrooms?
By
–
Is #AI likely to spell end of traditional school classrooms? https://
theguardian.com/technology/202
3/jul/07/ai-likely-to-spell-end-of-traditional-school-classroom-leading-expert-says
…