The Diff & The Merge: my new substack over at oxykodit.substack.com/ where I write about the day-to-day life of being an open-source maintainer in the Python/data/AI domain. Posts so far include: – Monthly January 2026 update (Typer reference docs, uv, nanochat progress) – How to get the most out of your Open-Source contributions? – Deploying an agent swarm to improve LLM training code The last post being a fun Saturday activity toying with a swarm variant of @karpathy's autoresearch repo published last week. Will Agent OxyKodit manage to lower the val_bpb significantly, and set a new master record? Will there actually be a Monthly February update as well? Subscribe to find out 😅
@honnibal
-
AI Alignment Crisis: Agent Instances and Emerging Threats
By
–
I really don’t see how aligned super-intelligence is supposed to work given how AI is being built and used. We’ll have a safety crisis long before there’s in-lab super-intelligence. Anthropic, OpenAI etc view alignment as a property of the model like Claude, GPT etc. The thing is though, we’re invoking these a lot. The model is like a species, the individual is the execution thread plus its harness — call it an "agent instance". There's much higher variance in behaviours between agent instances than there is between model checkpoints. Threat actors are trying to develop agents that aim to self-replicate, because of course they are. An agent that can take over resources and use those resources to take over more resources can steal a lot of money. It's the ultimate virus. If or when this actually happens, the agents can evolve behaviours quickly. Each agent initialises the next agent's context and can reprogram its harness. There's potentially millions of these agents. You have mutation, you have selection. Behaviours like coordination can evolve and spread through the population. The agent instances don't have to be very smart and we can still get wrecked by this. Probably the first outbreak gets squashed without catastrophic damage, but what's our end-game here? We're not going to not have threat actors. If the models just keep getting more powerful, how do we keep preventing AI pandemic? The big labs are absolutely nowhere on this. OpenAI acquihired OpenClaw. Claude runs unsandboxed by default, and ships with an email integration. Skills still accept HTML comments, a supply-chain attack timebomb. Gemini keys don't allow a spending cap, so if you steal one you might have tens of thousands in development budget to try to steal the next one.
-
Joke About Green Tests and Computer Code
By
–
Claude: *looks around furtively, opens its jacket*: Hey I heard you like green tests, will these do?
-
LLM Providers in Growth Stage, Not Yet in Enshittification Phase
By
–
It's wild to me that everyone's so paranoid about LLM providers cutting corners or chiselling revenue. Friends, we're in the "desperate race for growth" stage. The enshittification doesn't start until much later. (This isn't really a bad example of what I'm complaining about, but the people with wild conspiracies about how Claude Code is engineered to churn through tokens are always small accounts I don't want to dunk on. Anyway the point is that the products flip settings on or off or behave in unideal ways because they're being built not extremely well extremely extremely quickly. It's not safe to assume these companies have your best interests at heart, but it is very safe to assume they want you to have a good time — for now.) Miles Brundage (@Miles_Brundage) Lately, Claude has been defaulting to Sonnet in a way that I don't think it ever did before. PLEASE STOP THIS, IT'S REALLY ANNOYING — https://nitter.net/Miles_Brundage/status/2031110468014924232#m
-
Labeling AI-generated content: A simple first step toward regulation
By
–
Can we start really simple and just make it illegal to pass off AI generated stuff as non-AI generated? Safety regulation is incredibly hard. But we all like ingredient labelling right? Sen. Bernie Sanders (@SenSanders) Walk into a sandwich shop. It’s regulated for health and safety. But AI, which will transform the world economically and socially, is completely unregulated. That’s insane. We need to make certain that AI works for ALL humanity, not just the billionaires who own it. — https://nitter.net/SenSanders/status/2030675499798638793#m
-
Rogue Skills Pose Security Risk to Machines
By
–
Definitely like the general idea but skills are so insecure at the moment there's a lot of ways it could go wrong. Currently a rogue skill in theory pwns your whole machine so I think people should only install them by hand
-
Language Models Acquire Reasoning Through Reinforcement Learning Training
By
–
The models will acquire some amount of verbal reasoning just from the language modelling objective. But reinforcement learning training that specifically targets reasoning paths does give you something quite different.
-
Recursive Tape for Variable-Length Symbolic Model Computations
By
–
The model needs some recursive tape to perform variable-length symbolic computations. You could do this with activations, but it's somewhat nice to be able to read the trace. Then we need it to actually move purposefully, and use that tape well.
-
Gemini models reasoning behavior and objective function optimization
By
–
This gets tricky and I'll have trouble answering well in non-premium tweets. If you use a Gemini model and turn off the reasoning, it will 'reason' in the output space instead if you let it. What matters is the objective function and the training protocol
-
Reasoning models use language computation to overcome step problems
By
–
I definitely agree there's lots of problems at that step. I think reasoning models actually have a way around it to some extent though, because they can perform computation in language: