It was great to be back at @IITBayArea after nearly a decade, and seeing incredible talent and energy there. AI has come far and I have been lucky to be part of this revolution. My short interview:
RESEARCH
-

LLMs Memorizing Algorithms: Inefficient vs Coding Solutions
By
–
Look! We can get an LLM to memorize the multiplication algorithm at a million times the cost of just coding it!
-
LLM Capabilities Fragmented: Vision, Tools, and Heavy Thinking Models
By
–
The state of LLMs is messy: Some AI features (like vision) lag others (like tool use) while others have blind spots (imagegen and clocks). And the expensive "heavy thinking" models are now very far ahead of all the other AIs that most people use. None of this is well-documented.
-
Multiple Choice Questions as Model Training for Real-World Performance
By
–
What would be useful is when including more MC questions would also improve the answer performance on similar-topic questions where answer choices are not shown (like in a real world context). That being said I can see models who do well on MC settings being useful as verifiers
-
Internal Sanity Checks vs External Evaluations for LLM Benchmarking
By
–
Yes, it’s cheaper and easier, but it’s more of an internal sanity check than outward facing eval to report imho. Btw, spot on regarding including it for the sake of benchmarks. You can tell based on how sensitive some LLMs are to the exact MC prompt format.
-
Open-ended Evaluations: Free-form Answers and Conversational Performance
By
–
With open-ended evals you mean free-form answers (and measuring conversational performance)?
-
Technology adoption follows exponential acceleration patterns
By
–
Things take longer than you expect until they take less.
-

Scaling Parallel Agents for Computer Use Tasks
By
–
what if we stopped betting everything on one agent rollout? "The Unreasonable Effectiveness of Scaling Agents for Computer Use" Generates multiple trajectories in parallel & selects the best using "behavior narratives" 69.9% on OSWorld, nearly matching human-level 72%
-
Humans and Machines Coevolve in Dynamic Synergistic Existence
By
–
Humachineology and humachinekind studies show that as we have been making our machines, our machines have been making us. Humans and machines have been evolving together, and they will continue to further evolve to maintain their dynamic and synergistic existence.
-
Verifier and LLM-as-Judge for Output Conformance
By
–
Good suggestions. I'd say those fall into the verifier category (perhaps also LLM-as-a-judge for output-conformance); or do you use something different?