Maybe not one-shot — but models can allow an expert to test out ideas and iterate much quicker. It's certainly an advantage, even if it's not as strong as it is for solving new problems.
@thegautamkamath
-
AI-driven research accumulation leads to inscrutable solutions
By
–
You can see how after a few such AI-driven works that build on top of each other, entire solutions could become inscrutable. No individual paper may have committed a serious offense, but the obfuscation accumulates, even when there might have been a much more elegant soln. 5/n
-
AI models struggle to produce elegant, interpretable solutions
By
–
In my experience, (current) AI models are not great at producing "human interpretable" or elegant solutions, even after de-slopifying. They've produced technical modifications on top of previous technical solutions, which get increasingly hard (for me?) to intuit. 4/n
-
Using AI to simplify and clean up mathematical literature
By
–
Most of the focus on AI for math has been on producing new results. But surprisingly little on finding simpler or more elegant arguments for old results? I would love for folks to clean up the literature using these newfound superpowers. 1/n
-
AI automation of mathematical and ML research careers
By
–
I was getting at something slightly different. Many people say this is the end of mathematical research as a career. But if so, I don't see why it stops there. The same argument implies that any sort of intellectual work (e.g., ML research, etc.) will be automated away.
-
Adversarial examples: CIFAR-10’s unexpected importance
By
–
Remember how we never really solved adversarial examples on CIFAR-10 and dismissed it as a toy problem that didn’t matter? Turns out the fundamental ideas preventing test-time adversarial attacks are now quite important.
-
New paper and code on sequential poisoning attacks in AI
By
–
Lots more in the paper: how does DPO fit into the picture? What if attackers have different goals? etc. Paper: https://
arxiv.org/abs/2606.04929
Code: https://
github.com/jcksanderson/s
equential-poisoning
… Led by @jcksanderson
, w/ @YihanWww
, Xiaoqian Lu, co-supervised w/ @YiweiLu3r 6/6 -
0.5% poison breaks reward model, 5% needed for RLHF transfer
By
–
What about poisoning PPO? A remarkable paper of @javirandor and @florian_tramer (
https://
arxiv.org/abs/2311.14455) shows that just 0.5% poison is enough to break a reward model (L)! Again, fear not: somehow, it takes a (high) 5% poisoning before it transfers to the RLHF'd model (R). 4/n -
2% SFT poisoning gives 90% attack success; RLHF wipes it away
By
–
There's multiple post-training phases attackers can infiltrate: SFT, DPO, PPO. Let's start with SFT. With just 2% SFT poisoning, 90% attack success (L)! But not to worry, RLHF works as we hope (?): it wipes away the poison. An RM scores outputs just like a clean model (R). 3/n
-
Backdoor attacks on LLMs via untrusted training data
By
–
LLMs are trained on lots of data, often from untrusted sources. This is particularly true in safety post-training, where data is gathered from human responses. Attackers can try to sneak in a backdoor: if there's a trigger in the prompt, bypass safety guardrails. 2/n