Each one still has the softmax issue though and it all gets turned into one vector at the end. Learning to do addition in that projection feels hard, but I don’t know enough to say it can’t happen
RESEARCH
-
LLM Evaluations Grant Proposals Open Until September 6th
By
–
We're still accepting proposals for our LLM Evaluations Grant until this Friday, September 6th — recipients will get $200K in funding to support this work. More details https://
go.fb.me/82ft4d -
Paper on limitations of counting in single-token inference
By
–
Yes — and you can often achieve much more reliable counting by repeating an input a second time annotated with a running total, but that’s expensive and context-constrained; this paper is about limitations on counting directly in a single token inference
-
Why Transformers Struggle to Count, According to a Paper
By
–
The reason LLMs say there's two r's in "strawberry" isn't (just) tokenization — they struggle with counting generally, e.g. "horse" in the example shown. The paper in the quoted post below offers the best intuition I've seen for why this happens: Transformers can't count because
-
llms.txt Uses Markdown for Language Models and Agents
By
–
The llms.txt file is unusual in that it uses Markdown to structure the information rather than a classic structured format such as XML. The reason for this is that we expect many of these files to be read by language models and agents.
-
How Language Models Process Information Differently
By
–
Providing information for language models is different to providing information for humans. Language models generally like to have information in a more concise form, and they can ingest a lot of information quickly.
-
PhD Defense: Hardware-Efficient Training Pioneer Achievement
By
–
Congratulations @jiawzhao on an excellent PhD defense! Jiawei has been a pioneer in hardware-efficient training. When he started his PhD, everyone was focusing on inference efficiency, and training runs were small, Jiawei took the bold step to pursue training efficiency. Slides:
-
SubgoalXL: Advanced AI Reasoning and Proof Formalization Framework
By
–
Check out our work: GitHub: https://
github.com/zhaoxlpku/Subg
oalXL
… Datasets & Models: https://
huggingface.co/collections/xl
-zhao/subgoalxl-66c84b77e0eaf131c0cdac15
… Paper: https://
arxiv.org/abs/2408.11172
Join us in pushing the boundaries of AI reasoning! #AI #LLM #SubgoalXL #AIResearch #ProofFormalization #MachineAssistedProofs -
SubgoalXL Enhances Multi-Step Reasoning in Large Language Models
By
–
Key strengths of SubgoalXL: • Subgoal-based decomposition for improved reasoning
• Iterative refinement through expert learning
• Efficient use of limited formal and informal data
Result: Enhanced multi-step reasoning in LLMs! -
Subgoal-based framework improves theorem proving with expert learning
By
–
Our new framework combines subgoal-based proofs with expert learning for theorem proving in Isabelle. Subgoal-based proofs break complex reasoning into intermediate steps, aligning with formal proof structures. Expert learning iteratively refines formal statement, proof,