More experiments can be found in our TMLR paper: https://
arxiv.org/abs/2310.14540
Data and code: https://
huggingface.co/datasets/yyama
da/SpatialEvalLLM
…, https://
github.com/runopti/Spatia
lEvalLLM
…
Special thanks to @AndrewLampinen for helpful discussion and advice!
7/n
LLMS
-
TMLR Paper with Experiments, Data and Code Resources
By
–
-
Human Baselines Outperform GPT-4 on Non-Square Grid Tasks
By
–
We also compare LLMs with human baselines. Although human responses are not perfect, they outperform GPT-4 (0314) by a substantial margin. Furthermore, like GPT-4 (0314), non-expert humans struggle with non-square grid shapes.
6/n -
LLM Performance Variability Across Different Spatial Structures
By
–
We examine various spatial structures (squares, triangles, hexagons, rings, and trees), which reveal substantial variability in LLM performance across different structures. (GPT-4 here is 0314)
5/n -
Loop Closure in Square Grid: Claude 3 vs GPT-4 Comparison
By
–
Answering this question correctly demonstrates an understanding of loop closure in the square grid, which is a fundamental aspect of this spatial structure. The Claude 3 vs. GPT-4 comparison above is tested on a 3 by 3 square grid w/ 8 navigational steps. 4/n
-
Sequential Transitions Enable LLM Spatial Understanding
By
–
Motivated by this, we hypothesize that presenting sequential transitions might be enough for LLMs to achieve spatial understanding. e.g. if a model comprehends a square map’s structure, it should be able to answer the question shown in the image. 3/n
-
Evaluating Spatial Understanding in Text-Only Language Models
By
–
Evaluating the text-only models’ understanding of spatial information is tricky because text-only LLMs do not explicitly interact with the physical world. But humans can implicitly learn representations that mirror spatial structures only from sequential navigational data. 2/n
-
Claude 3 vs GPT-4: Spatial Reasoning Task Comparison
By
–
A quick comparison b/w Claude 3 and GPT-4 on a spatial reasoning task (n=100, 5 run average w/ temp=1.0). Seems like Claude 3 still beats GPT-4, and gpt-4-turbo performs worse than gpt-4-0613. Interesting contrast to their perf in chat & coding, where GPT-4 comes out ahead. 1/n
-
Iterative Fine-Tuning Impact on Pre-Training Data Contamination
By
–
How much does deliberately iterative fine-tuning on model outputs tell us about the impact of smaller amounts of generated data making it into larger pre-training datasets?
-
DSPy: Automatic Prompt Optimization Through Example Derivation
By
–
My current mental model of DSPy is that it leans into the thing where LLMs can produce much better output if you prime them with the right examples – then helps automatically derive the best examples based on running some evals I may have misunderstood it though
-
Base Models Unreliable Self-Assessment Capabilities
By
–
That said, asking a base model directly about its own capabilities feels even less useful to me than asking an instruction-tuned model – models have always been inherently unreliable when it comes to answering questions about themselves