The idea is that llms.txt is manually created/curated to have the exact info an LLM would find most useful. It's a bit of work to get right, but in my experience it's extremely helpful in practice (all our projects have carefully written llms.txt files).
@jeremyphoward
-
Python Tool for Inlining Links in Documentation Files
By
–
If you want to full set of info, we recommend using the python tool we provide to create a version of the file with the links inlined, and tell your users how to use it. E.g we do this for fasthtml.
-
LLM Context Limits: Performance Degradation Beyond 100K Tokens
By
–
For small/simple tools, you could just dump all the concise needed info straight into llms.txt, but for larger systems it might not fit even in 200k tokens. Also, all current LLMs degrade a *lot* beyond 100k tokens in practice.
-
LLMs Agentic Search Design with Descriptive Links
By
–
It was written on the assumption that LLMs would use an agentic-search approach to follow the links that seem most useful. That's why each link has a description of what's there, so the LLM knows where to find the info it needs.
-
Government Should Not Restrict Educational Content Access Online
By
–
I'm not that keen on the idea of restricting my daughter from being able to learn from 3b1b, Professor Dave, Eddie Woo, etc… And especially not keen on the government trying to enforce that. If some parents want to do that for their kids, then sure go ahead.
-
Testing AI Critical Thinking Through Precise Question Asking
By
–
Or perhaps you're testing whether they are good at asking question – i.e "please define that term precisely and provide a clear example".
-
Inverting Binary Trees: Interview Problem Not CS Concept
By
–
In fact, "invert a binary tree" does *not* exist as a computer science concept at all. It was created entirely as an interviewing problem. If you ask someone "can you invert a binary tree" then you're testing whether they know the definition of this interview-specific term.
-
FastDiffusion course teaches comprehensive diffusion model knowledge
By
–
If you finish the fastdiffusion course you'll know a *lot* about diffusion 😀
-
The Truth Lies Between Extremes: Context and Misinformation
By
–
The reality is somewhere between your response, and what you're responding to. There's a lot of context you're missing, but also plenty of garbage in the "files".
-
Chinese Version Uses Different Task Counting Character Occurrences
By
–
The Chinese version is doing a fairly different task. It's using the character for "ha" (as in the English "haha" laughing sound) and asking how many times that appears in a sentence. Chinese doesn't really have many-character words, so the task doesn't quite fit here.