They've only listed those that agree I told them "I don't agree. I don't think CoT has much of a useful role to play. It's only really showing something that appears to be a meaningful trace because they're trained to appear that way, but actually they're not faithful at all"
ETHICS
-
Publishing Research: Present Both Supporting and Critical Feedback
By
–
IMO if you're going to go to the trouble to get feedback about your paper before publishing (which is a great idea!) you should publish *both* sides. Just listing prominent supporters gives the wrong impression.
-
Publishing Research: Include Both Supporting and Opposing Perspectives
By
–
IMO if you're going to go to the trouble to get feedback about your paper before publishing (which is a great idea!) you should publish *both* sides. Just listed prominent supporters gives the wrong impression.
-
Redefining AI Misuse Requests: Safety vs Developer Intent
By
–
"requests for misuse" only makes sense, to me, if defined as "request to do a thing that the system prompt asks not to do". This is not necessarily harmful or unsafe. It's just not what the developer wanted.
-
User Intent Framing and AI Safety Safeguards Context
By
–
I think I disagree with your framing. "The user is intentionally asking for something that could lead to harmful outcomes if no safeguards are in place" could refer to no prompts, or all prompts, depending on context.
-
Meta’s Financial Influence on Content Creators and Speech
By
–
Most people won't post this, though, because they all want to get paid by Meta.
-
On-Policy Learning: Authenticity Over Imitation in Life
By
–
On-policyness is power. Walk your own trajectories and don't imitate. Such a big fan of ML wisdom for the day to day life.
-
On-Policy Learning in Life: Moving Beyond Imitation to Find Your Path
By
–
Becoming an RL diehard in the past year and thinking about RL for most of my waking hours inadvertently taught me an important lesson about how to live my own life. One of the big concepts in RL is that you always want to be “on-policy”: instead of mimicking other people’s successful trajectories, you should take your own actions and learn from the reward given by the environment. Obviously imitation learning is useful to bootstrap to nonzero pass rate initially, but once you can take reasonable trajectories, we generally avoid imitation learning because the best way to leverage the model’s own strengths (which are different from humans) is to only learn from its own trajectories. A well-accepted instantiation of this is that RL is a better way to train language models to solve math word problems compared to simple supervised finetuning on human-written chains of thought. Similarly in life, we first bootstrap ourselves via imitation learning (school), which is very reasonable. But even after I graduated school, I had a habit of studying how other people found success and trying to imitate them. Sometimes it worked, but eventually I realized that I would never surpass the full ability of someone else because they were playing to their strengths which I didn’t have. It could be anything from a researcher doing yolo runs more successfully than me because they built the codebase themselves and I didn’t, or a non-AI example would be a soccer player keeping ball possession by leveraging strength that I didn’t have. The lesson of doing RL on policy is that beating the teacher requires walking your own path and taking risks and rewards from the environment. For example, two things I enjoy more than the average researcher are (1) reading a lot of data, and (2) doing ablations to understand the effect of individual components in a system. Once when collecting a dataset, I spent a few days reading data and giving each human annotator personalized feedback, and after that the data turned out great and I gained valuable insight into the task I was trying to solve. Earlier this year I spent a month going back and ablating each of the decisions that I previously yolo’ed while working on deep research. It was a sizable amount of time spent, but through those experiments I learned unique lessons about what type of RL works well. Not only was leaning into my own passions more fulfilling, but I now feel like I’m on a path to carving a stronger niche for myself and my research. In short, imitation is good and you have to do it initially. But once you’re bootstrapped enough, if you want to beat the teacher you must do on-policy RL and play to your own strengths and weaknesses 🙂
→ View original post on X — @_jasonwei, 2025-07-16 01:26 UTC
-
Asymmetry of Verification and Verifier’s Law in AI Training
By
–
New blog post about asymmetry of verification and "verifier's law": jasonwei.net/blog/asymmetry-… Asymmetry of verification–the idea that some tasks are much easier to verify than to solve–is becoming an important idea as we have RL that finally works generally. Great examples of asymmetry of verification are things like sudoku puzzles, writing the code for a website like instagram, and BrowseComp problems (takes ~100 websites to find the answer, but easy to verify once you have the answer). Other tasks have near-symmetry of verification, like summing two 900-digit numbers or some data processing scripts. Yet other tasks are much easier to propose feasible solutions for than to verify them (e.g., fact-checking a long essay or stating a new diet like "only eat bison"). An important thing to understand about asymmetry of verification is that you can improve the asymmetry by doing some work beforehand. For example, if you have the answer key to a math problem or if you have test cases for a Leetcode problem. This greatly increases the set of problems with desirable verification asymmetry. "Verifier's law" states that the ease of training AI to solve a task is proportional to how verifiable the task is. All tasks that are possible to solve and easy to verify will be solved by AI. The ability to train AI to solve a task is proportional to whether the task has the following properties: 1. Objective truth: everyone agrees what good solutions are 2. Fast to verify: any given solution can be verified in a few seconds 3. Scalable to verify: many solutions can be verified simultaneously 4. Low noise: verification is as tightly correlated to the solution quality as possible 5. Continuous reward: it’s easy to rank the goodness of many solutions for a single problem One obvious instantiation of verifier's law is the fact that most benchmarks proposed in AI are easy to verify and so far have been solved. Notice that virtually all popular benchmarks in the past ten years fit criteria #1-4; benchmarks that don’t meet criteria #1-4 would struggle to become popular. Why is verifiability so important? The amount of learning in AI that occurs is maximized when the above criteria are satisfied; you can take a lot of gradient steps where each step has a lot of signal. Speed of iteration is critical—it’s the reason that progress in the digital world has been so much faster than progress in the physical world. AlphaEvolve from Google is one of the greatest examples of leveraging asymmetry of verification. It focuses on setups that fit all the above criteria, and has led to a number of advancements in mathematics and other fields. Different from what we've been doing in AI for the last two decades, it's a new paradigm in that all problems are optimized in a setting where the train set is equivalent to the test set. Asymmetry of verification is everywhere and it's exciting to consider a world of jagged intelligence where anything we can measure will be solved.
→ View original post on X — @_jasonwei, 2025-07-16 00:59 UTC
-
Hofstadter’s Metaphysics and Artificial Consciousness Debate
By
–
You seem to be possessed by a vision of Hofstadter hallucinating blue flames and having an imagined uncouth relationship to metaphysics