AI Dynamics

Global AI News Aggregator

About

@_jasonwei

  • Simple Ideas Behind Massive AI Research Compute Investments

    By

    @_jasonwei

    –

    AI research is strange in that you spend a massive amount of compute on experiments to learn simple ideas that can be expressed in just a few sentences. Literally things like “training on A generalizes if you add B”, “X is a good way to design rewards”, or “the fact that method M is sample efficient means that we should create environments with this specific property”. But somehow if you find the correct five ideas and you really understand them deeply, suddenly you’re miles ahead of the rest of the field

    → View original post on X — @_jasonwei, 2025-06-24 19:15 UTC

  • Description-Execution Gap: Key to AI Automation Potential

    By

    @_jasonwei

    –

    One way of thinking about what AI will automate first is via the “description-execution gap”: how much harder is it to describe the task than to actually do it? Tasks with large description-execution gaps will be ripe for automation because it’s easy to create training data and the value of automating them is huge, even if execution is non-trivial: – Fixing grammar mistakes in a long piece of writing – Submitting receipts for reimbursement – Training a model that achieves performance of X on a standard evaluation benchmark – Building an app where the UI is easy to check but requires a lot of moving parts in the backend Description-execution gaps tend to be small when the task is high-context and not technically challenging. The value of automating these is by definition smaller, and it’s harder to create data for them. For example: – Data processing scripts where the code to process the data is shorter and more precise than a natural language description – Running an ablation study in a high-context codebase that trains specialized models – Editing a video in a specific style (often easier to edit the video yourself than to describe how each little edit should be done) – Buying chinese groceries for my mom (she has very specific items and amounts, it's easier for her to go herself than to describe to me exactly the item, how to select the best fruit, etc) A bit similar to the discriminator-generator gap, but not exactly the same. Some things, like editing a video in a specific style, can have a large discriminator-generator gap but small description-execution gap

    → View original post on X — @_jasonwei, 2025-06-18 19:24 UTC

  • RL Environment Specs: Critical Role in AI Model Training Quality

    By

    @_jasonwei

    –

    RL environment specs are among the most consequential things we can write as AI researchers. A relatively short spec (e.g., <1000 words of instructions saying what problems to create and how to grade them) often gets expanded either by humans or via synthetic methods into thousands of datapoints. Just one sentence in the spec can be the difference between a perfect post-trained model versus one with crazy hacking. Specs are also typically a product of a large amount of compute, where each training run allows us to iterate on the spec to patch reward hacking and get the nuances of model behavior just right. Writing a good spec requires context and taste and I don’t think AI can automate this just yet

    → View original post on X — @_jasonwei, 2025-06-11 19:12 UTC

  • Grammar Errors as Human Touch in Writing

    By

    @_jasonwei

    –

    It’s actually a good thing these days to have subtle grammar errors in your writing. It sprinkles on a clear human touch. You never want your reader questioning if what they’re reading was written or edited by chatgtp

    → View original post on X — @_jasonwei, 2025-06-07 21:46 UTC

  • BrowseComp Benchmark for Evaluating Agent Search Capabilities

    By

    @_jasonwei

    –

    Interesting launch! If the agent is good at "agentic search that doesn't stop until it finds what you need", consider evaluating on our BrowseComp benchmark, which measures just that! SimpleQA mainly targets models that don't browse: https://
    openai.com/index/browseco
    mp/
    …

    → View original post on X — @_jasonwei

  • Humorous suggestion about someone still using GPT-4o model

    By

    @_jasonwei

    –

    Man was still on 4o. We can start a gofundme for him

    → View original post on X — @_jasonwei

  • Man Uses ChatGPT for Relationship Advice During Lecture Hall Talk

    By

    @_jasonwei

    –

    Was attending a talk in a big lecture hall and the guy in front of me had the craziest conversation with ChatGPT for the whole hour about how to get his girlfriend back. Dozens of messages of pasting screenshots of text conversations to analyze tone of responses; whether to include an exclamation point to whether a smiley face was appropriately flirty. Apparently his ex-GF was talking to another guy in her lab and too stressed with work to give him attention, but also let him borrow her car, so he was getting mixed signals. Random stranger next to me was also spectating and found it so funny he literally cracked up in the middle of the talk. Convo ended with ChatGPT saying “you just had sex last week so youre no second class citizen.” Weirdest mix of pity, fascination, and awe I’ve ever felt

    → View original post on X — @_jasonwei, 2025-06-06 00:34 UTC

  • ChatGPT’s independence from traditional chatbot research methods

    By

    @_jasonwei

    –

    OK as someone pointed out ChatGPT using nothing from chatbot research isn't totally accurate. What I meant to say is that much of chatbot research that was mainstream at some point in time (e.g., dialogue state tracking, or slot filling, or semantic parsing) wasn't used in ChatGPT

    → View original post on X — @_jasonwei, 2025-06-05 04:08 UTC

  • The Inverse 80-20 Rule in AI Research: When Final Effort Matters Most

    By

    @_jasonwei

    –

    The 80-20 rule happens often in AI research, where you get 80% of the payoff from the first 20% of the effort. But there is often also an inverse rule, where it’s actually the final 20% of that yields 80% of the payout. Some examples: 1. When your eval is already good in many ways but it has one drawback and fixing that will extend its life from 6 months to 2 years – When you’ve already done all the experiments and writing them up in a nice report will allow dozens of people to learn from the work you already did – When you’ve spent some time debugging and you have a workaround. Instead of moving forward you spent the time to fully understand it; this knowledge compounds – When you’ve already run four experiments to investigate how much a particular component matters in training, and running one more allows you to find the answer (understanding things more clearly also compounds) Key is to know when it’s a 20-80 and stop and when to do the whole thing

    → View original post on X — @_jasonwei, 2025-06-03 19:40 UTC

  • AI Research as Max-Performance Domain: Specialization for Seminal Impact

    By

    @_jasonwei

    –

    A recent clarity that I gained is viewing AI research as a “max-performance domain”, which means that you can be world-class by being very good at only one part of your job. As long as you can create seminal impact (e.g., train the best model, start a new paradigm, or create

    → View original post on X — @_jasonwei