At today’s boot camp, HAI faculty affiliates @chrispiech and @EmmaBrunskill join congressional staffers in a session moderated by Shima Salehi. They look into how AI can be leveraged to improve the education system without causing harm to teachers and students.
RESEARCH
-
Learn Data Science Skills: Insights from Industry Experts
By
–
Love problem solving? There has never been a better time to learn data science skills! Hear from a few data scientists about their passion for their field, what drives them + the the future. How YOU can learn critical #DataScience skills. http://
2.sas.com/6014PbYUq -
Text-to-SQL Deep Dive: Steps and Implementation Guide
By
–
A great deep dive by @manuelsoria_ and @RLanceMartin on text-to-SQL and all the steps involved!
-
Understanding How Language Models Learn Beyond Training Data
By
–
While large language models appear to have a rich understanding of the world, how do we know they’re not simply regurgitating from training data? Check out the latest AI Explorable on a phenomenon called grokking to learn more about how models learn. → https://t.co/Okc9GvJjuN pic.twitter.com/StxRLwtSRT
— Google AI (@GoogleAI) 8 août 2023While large language models appear to have a rich understanding of the world, how do we know they’re not simply regurgitating from training data? Check out the latest AI Explorable on a phenomenon called grokking to learn more about how models learn. → https://
goo.gle/45ohnQh -
Training Token Influence Across Neural Network Layers
By
–
Influence can also be attributed to particular training tokens and network layers. On average, the influence is equally distributed over all layers (so the common heuristic of computing influence only over the output layer is likely to miss important generalization patterns).
-
Neural Layer Influence Patterns: Fine-grained Wording vs Semantics
By
–
On the other hand, individual influence queries show distinct influence patterns. The bottom and top layers seem to focus on fine-grained wording while middle layers reflect higher-level semantic information. (Here, rows correspond to layers and columns correspond to sequences.)
-
Analyzing Pretraining Finetuning Interactions and Neural Circuits
By
–
This work is just the beginning. We hope to analyze the interactions between pretraining and finetuning, and combine influence functions with mechanistic interpretability to reverse engineer the associated circuits. You can read more on our blog:
-
Cross-Lingual Influence Scales with Model Size
By
–
Another striking example occurs in cross-lingual influence. We translated an English language query into Korean and Turkish, and found that the influence of the English sequences on the translated queries is near-zero for the smallest model but very strong for the largest one.
-
Influence Functions Reveal AI Role-Playing Behavior Patterns
By
–
Influence functions can also help understand role-playing behavior. Here are examples where an AI Assistant role-played misaligned AIs. Top influential sequences come largely from science fiction and AI safety articles, suggesting imitation (but at an abstract level).
-
Training Data Influence Distributions Follow Heavy-Tailed Power Laws
By
–
The influence distributions are heavy-tailed, with the tail approximately following a power law. Most influence is concentrated in a small fraction of training sequences. Still, the influences are diffuse, with any particular sequence only slightly influencing the final outputs.