Answering this question correctly demonstrates an understanding of loop closure in the square grid, which is a fundamental aspect of this spatial structure. The Claude 3 vs. GPT-4 comparison above is tested on a 3 by 3 square grid w/ 8 navigational steps. 4/n
MACHINE LEARNING
-
Sequential Transitions Enable LLM Spatial Understanding
By
–
Motivated by this, we hypothesize that presenting sequential transitions might be enough for LLMs to achieve spatial understanding. e.g. if a model comprehends a square map’s structure, it should be able to answer the question shown in the image. 3/n
-
Evaluating Spatial Understanding in Text-Only Language Models
By
–
Evaluating the text-only models’ understanding of spatial information is tricky because text-only LLMs do not explicitly interact with the physical world. But humans can implicitly learn representations that mirror spatial structures only from sequential navigational data. 2/n
-
Iterative Fine-Tuning Impact on Pre-Training Data Contamination
By
–
How much does deliberately iterative fine-tuning on model outputs tell us about the impact of smaller amounts of generated data making it into larger pre-training datasets?
-
Base Models Unreliable Self-Assessment Capabilities
By
–
That said, asking a base model directly about its own capabilities feels even less useful to me than asking an instruction-tuned model – models have always been inherently unreliable when it comes to answering questions about themselves
-
RLHF Fine-tuning Effects on Model Behavior Analysis
By
–
That example looks to me like it's more caused by RLHF fine-tuning than anything that was baked into the model in the pre-training phase
-
Artificial Data Risk at Pre-training Stage in Models
By
–
That seems like a slightly different aspect of this to me – we have seen tons of examples now of models being fine-tuned on carefully created artificial data, but to me that doesn't speak to the risk of unintentional artificial data affecting the models at the pre-training stage
-
DSPy Documentation Challenges for Deep Learning Practitioners
By
–
Yeah, I've done a little bit of deep learning in the past (took a Fast AI course a few years ago) and I still find that the DSPy documentation that compares how to use it to deep learning doesn't help me that much
-
Using DSPy Without Building Evaluations: A Learning Curve Challenge
By
–
Can you use DSPy without building evals? Because building evals is an enormous learning curve for most people (myself included)
-
Data AI Summit Early Bird Registration Discount Ends April 30
By
–
Early bird catches the worm Save $400 by registering for #DataAISummit before April 30! You’ll explore the latest advances in #ApacheSpark, Delta Lake, MLflow, LangChain, PyTorch, dbt, and more: https://
bit.ly/3UlaDA7