← Back to search

My AI Agent Bought Me A Car

Agentic Conversations (formally mlops.community) · 2026-11-09 · 35 min
relevance 100 6729 words Episode page ↗ Audio ↗
Show full episode description
Sam Herring ’s first real test of Hermes Agent was a shopping tool that matched clothes to photos of ancient pottery. A few months later, the same agent was checking government auction sites every day until it found him a retired three-wheel parking ticket car, which he now drives around San Francisco. In this episode, recorded at AGNTCon + MCPCon North America in San Jose, Sam, who works on the RL post-training team at Nous Research, goes from those everyday wins to the question everyone in AI is asking: can agents improve themselves? He explains how Hermes grew from an internal productivity tool into one of the most talked-about open-source agents, and why letting it write its own skills was the moment it clicked for users. He also describes an early version that could design its own RL environment, train on Thinking Machines’ Tinker, and hot-swap the new weights into a model running locally. Sam then breaks down Trace to Environment, an idea he credits to Xiaomi’s MiMo-V2.6-Pro work, and the loop Nous is building around it: find the exact point where an agent fails, cluster those failures by type, synthesize environments that recreate them, and train on them again and again. The second half moves from training to inference. We get into whether reasoning models overthink, why leaning on pretraining knowledge can be a model’s Achilles heel, and why choosing a reasoning effort level may be a temporary quirk. Sam also makes his case for activation steering as an underexplored way to get more out of small models, explains where steering meets interpretability, and describes what true continual learning could look like. Demetrios Brinkmann: https://www.linkedin.com/in/dpbrinkm Sam Herring: https://www.linkedin.com/in/sam-herring-67225819a Timestamps:[0:00] Cold open[1:21] Meet Sam from Nous Research[2:39] How Hermes Agent started[3:38] From pottery to a parking ticket car[7:40] Benchmarking life admin tasks[8:58] Agents that write their own skills[10:10] RL at Nous a year ago[12:24] Hot swapping self-trained weights[15:00] Xiaomi and Trace to Environment[17:22] Turning agent failures into training[20:27] Where the RSI conversation is heading[21:34] Token efficiency and overthinking[26:30] Steering models at inference time[32:35] Toward true continual learning[34:54] Wrap up
✨ Episode Outline — click any point to jump to it in the episode
Problem solved
How Hermes offloads life-admin and creative tasks end-to-end, and how Nous Research's RL work points toward agents that train and improve themselves.
Benefits
  • Hermes handles recurring life-admin tasks (auction, apartment, car searches) without daily manual checking
  • Hermes auto-converges repeated tasks into reusable skills without being told
  • Non-coding benchmark targets email, utility bills, and life admin instead of only code
  • Local-model Hermes can design an RL environment, train on cloud compute, and hot-swap weights
  • Outcome-first prompting: describe the goal, let Hermes explore, then codify a deterministic workflow
Use cases
  • Hermes built an image-embedding tool in ~1 hour that finds listed men's clothing matching any art or clothing photo; code on GitHub
  • Hermes script scanned government auction sites daily; Sam bought a three-wheel ex-ticketing car from a listing in May and drives it in SF
  • Nous launch videos on Twitter: ASCII-art video plus self-composed music produced end-to-end via a Hermes creative pipeline
  • Smart-home: a user tracked dog-door entries per week via Hermes with data visualizations
  • Benchmark task: Hermes compiles utility bills from email and drafts a check with the human as final approver
KPIs / results
  • Clothing-matching program built and on GitHub in about 1 hour
  • Auction-watch script demoed live in February; car purchased in May
  • Sam interned ~1 year at Nous, joined full-time October of last year
Tools / build
  • Hermes
  • Qwen model running as Hermes in a self-training RL loop
  • Endless Terminals (procedurally generated terminal RL tasks)
  • Thinking Machines Tinker cloud RL
  • Nous open-source RL framework
  • Claude Code (prior workflow)
0:00 / 0:00
[SPEAKER_02] In SF and New York, they drive around these little three-wheel ticketing cars. And I started to do some research and found out they're sold on government auction. In May, a listing came up that I bought. You have one? Yes, I drive it. [SPEAKER_01] You took the bit too far? [SPEAKER_02] I have one. I run loops on my own hardware of like, I have a Quen model running as Hermes that it designs an RL environment, kicks it off, downloads the weights, and then hot swaps into this version. It's probably Neo learning comes true. They call it trace to environment. And the idea is that you take a bunch of agent traces and synthesize RL environments based on those traces and then train on them. [SPEAKER_02] And this could work in a continual learning loop like every night, every task that you've had your agent do while you're sleeping, it goes and creates RL environments, trains itself on them, and updates. We've only trained models to be like this because we expect that they sort of make a wrong mistake and then correct themselves. I actually think they could reason less and still get to the right answer. [SPEAKER_00] We're back. And this time I got an old friend with me, Sam, who's working at News Research. You all probably know that company because of Hermes, the agent that has taken the world by storm. Sam's working on RL and it's been great to catch up with him. And I just want to talk through what you're most excited about, some use cases you have for Hermes, RL in general, [SPEAKER_00] like what you were doing a year ago versus now, how it's changed. And then maybe we can touch on the hottest topic in the world, the continual learning and potential recursive self-improvement. [SPEAKER_02] Recursive self-improvement. [SPEAKER_00] Yes, the RSI, dude. It is so hot right now. So I laid it all out for you. Where do you want to begin? [SPEAKER_02] Thanks for having me here, first of all. Super, super stoked to be sitting down and chatting. I'll start off with kind of my personal uses of Hermes because that's a good intro to those viewers that might not know, you know, what Hermes is or the full power of what you can do with it. [SPEAKER_00] And I'll just give a little bit of the context and the background about how Hermes started. And if you know what Hermes is or if you're using Hermes, tell us in the chat because I think that is a good gauge of, you know, who's using it, how they're using it. Yeah. [SPEAKER_02] So Hermes was started internally as sort of a productivity tool. It was an idea to be like, we have OpenClaw and it kind of had had its moment sort of about a year ago and changed. Maybe, you know, nine months ago, it was really, really starting to take off. And some people internally at Noose were like, wanted something different out of it. And they wanted to design their own version of kind of a long running agent harness that would just improve productivity internally. [SPEAKER_02] Like there was no ever semblance of like, oh, we're going to make this a product and people are going to use it. And so we did. And they kind of started like telling us like, hey, you should start testing out Hermes. And I remember like telling some of my friends like, oh, yeah, we're like testing out this new thing internally. And I didn't talk about it much. I was still using Cloud Code at the time for all my work. Yeah. And I really didn't like have anything that was jumping out at me to use Hermes until one day. [SPEAKER_02] I kind of either was bored or I think I was on I was having a conversation with a friend on Twitter who was posting these pictures of like ancient art pieces, like these old ceramic pieces and like Native American art. And they were saying like, how do I find clothes that match this art? Nice. And that was the first idea clicked in my head of like, you know what? I could probably test Hermes with doing this and just see how far it could. I knew the pipeline in my head of like pull a bunch of images from men's clothing site, you know, [SPEAKER_02] do some sort of image embedding similarity to the target image of whatever art you're looking for and then rank them by like price or by, you know, whatever. There's a million ways to filter it once you have this like similarity distance. So I tasked Hermes with doing it and an hour later I came back to him with a program that, you know, it was like code is on GitHub in an hour. No way. And it could find any clothes currently listed that matched either clothing that you could send it or a picture of something random like. Damn. Yeah. So I still use that. [SPEAKER_00] A random item just like pottery. Yeah. That is. And it's like what vibes with this? I really am inspired by this pottery. I need that. Yeah. All right. I'm going to go look up that. [SPEAKER_02] Yeah. Yeah. Yeah. Do it. It's it's a it was a fun use and I still use it when I want to find clothes. That's great. The other thing is in SF and New York, I guess other cities, they drive around these little three wheel ticketing cars. They're these tiny triangular cars. Yeah. I have seen one in my neighborhood, one or two, and I started to do some research and found out they're sold on government auction. [SPEAKER_02] So in February, the first live demo of Hermes agent that we did on Discord, I got on and showed everyone the end to end process of me tasking Hermes to write a script to search government auction sites every day. And you can find one of these cars. In May, a listing came up that I bought. No. And I knew you have one. Yes. [SPEAKER_01] I drive it a bit too far. Yeah. Really? [SPEAKER_02] You. I have one and I drive it around San Francisco all the time. People probably are like, no, don't wait. [SPEAKER_00] I'm coming back. [SPEAKER_02] I paid for my parking. Well, it's spray painted. It's interesting. It's all, it's painted so, like, it's got all this, like, graffiti on it and stuff because the guy who had had it previously was an artist. So it's like, people think, oh, I'm a ticket person and then they realize it's, like, not. I'd love to repaint it. It's kind of blue and black right now. Like, it almost is, like, Hermes colors. It'd be cool to have some yellow streaks in there or, like, I'm joking. I'm just, like, surprisingly calling it the Hermes mobile to coworkers. [SPEAKER_00] I mean, it got you the car. [SPEAKER_02] It did. It did. And that's what I tell people often. Like, if you're looking for an apartment that fits some specifications or you're looking for a car and, like, instead of checking Craigslist every day, you know, that's a great use of Hermes. It's, like, life admin tasks that, like, would require you to kind of be on your computer a lot or, like, be. And follow up. Yeah. And, you know, outside of that, we have seen wild uses of Hermes. Tell me about this. That's what I want to know. There's a lot of people use it for creative pipelines. [SPEAKER_02] Like, our launch videos on Twitter, super hype. They're beautiful. The music behind them, people love. All done through Hermes, through this Hermes workflow that some of our designers have laid out that you basically end-to-end get this beautiful ASCII art launch video where it composes the music itself. And the whole thing is just this cohesive creative pipeline. And so our designers use it a lot for that. We've seen a lot of uses of people using Hermes for smart home automation. [SPEAKER_02] So, like, having Hermes do analytics or control on, like, your lights. Or, like, one person, they had, like, a dog door that was attached to their smart home. And, like, they could pull the numbers of how many times their dog had been in and out, you know, within a week. And, like, start tracking the behavior that way. And it has a beautiful data visualization. Yeah, yeah, yeah. It makes this pretty, yeah. I mean, anything's possible, man. [SPEAKER_02] Like, of course, a lot of people use Hermes for things like, you know, automated crypto trading workflows and market analytics. You know, I think, like, we created an internal and sort of external benchmark to evaluate model performance inside of Hermes. And there's already a ton of coding agentic benchmarks. Like, everything out there targets, like, the majority of what people use that for, which kind of comes down to writing and executing code. But we wanted to target the things that were outside of that. [SPEAKER_02] So a lot of it was life admin stuff. A lot of it was, you know, managing your utility bills, having Hermes go through your email, compile all your utility bills, and, like, even draft a, you know, check to send out. Like, you're the last step in the approval workflow for that. [SPEAKER_00] And that's probably what I love is how far it can get without me. Oh, yeah. And because you don't necessarily need to. And also, you don't need to sit there and explain it step by step. It's like, hey, here's this outcome that I'm looking for. You go figure it out. Once it's figured it out, then you can optimize on it. And you can decide, like, okay, now that I have this workflow, let's codify it and let's make it a little bit more deterministic where we can. [SPEAKER_00] You don't need the biggest model to go and do this every time that we have it because we've already explored the space. We found what the optimal way of doing it is. And now we can optimize. [SPEAKER_02] And that's the cool part about Hermes, too. I mean, you're teeing me up right there. It's like when we designed it, we started to add in the ability for it to write its own skills. And normally with the agentic workflow stuff, as you were starting to do more, you would tell your agent, like, oh, write this as a skill. Hermes is just like if you have it do something, especially multiple times, it's all converging to the same skill. [SPEAKER_00] I think that was a big unlock, man. Yeah, huge. It's the thing that made, like, that magical experience where you're like, oh, wow, it just does it. And it does it, like, automatically. I don't need to think about it. I don't need to tell it to do it. And it knows how to do it. [SPEAKER_02] Yeah. It's pretty wild. So, yeah, we've had a lot of fun and growth with Hermes. And I think this whole landscape is a lot of learning about, you know, what the right paradigms are to use and how to – we talk a lot in news about, you know, time to magic. We want to understand, like, from someone's first uses of Hermes, how long does it take them to really click and understand that this, you know, this agent grows with you and has this figure. Offloads a task. [SPEAKER_00] Yeah. Like, me as that user, how long to, like, can offload a task and, like, trust that, wow, that can get done and I don't have to think about it. Should we talk? [SPEAKER_02] RL? Yeah. Yeah. This is perfect segue into the rest of what I wanted to talk about. So, you asked what I was doing a year ago and how the landscape has changed. A year ago, I was working on – we had our own internal RL framework. I mean, it was public. It was open source. But, like, we used it for all of our RL training internally. And Thinking Machines had just released their Tinker library, which was basically cloud RL. [SPEAKER_02] You could offload your training loops to their compute and you could pay them per use. But for a while in their dev, you know, beta phase, it was free. So, we basically were getting, like, this super great deal on, like, running RL experiments. Take advantage of it while you can. That only after a while started to cost us a little money, but still worth it. And I was an intern at News for about a year and then I joined full-time in October of last year. So, I just passed your anniversary there. Congrats. Thanks. [SPEAKER_02] So, my first project was to integrate our RL training library into their online RL, you know, their cloud RL thing. So, I built out this whole library. Sorry. It was, a lot of the RL stuff that we do now is very different than what it was back then. Because the whole idea of, like, a genic RL, like, was just starting to form. [SPEAKER_02] Like, a lot of times we were, like, the stuff I was proofing out for this was, like, RLing on, like, certain benchmarks. Like, math benchmarks. And, like, different types of problems. Like, it wasn't until about March when I was using this framework to build out, it was a project called Endless Terminals. And it was basically procedurally generated terminal use tasks. Nice. That you could just keep generating them with varying levels of difficulty and then training the model on it. [SPEAKER_02] And it would just get better and better at, like, this complex terminal management. You didn't talk about Hill Club. I know, which is, like, ground zero for agent performance. Like, you optimize that, your agent gets so much better out of the box. That's awesome. So that was a really cool one. And what I'll say about this is that the main contribution I made to Hermes, which we ended up moving off of our RL framework. But in the early days, this was the biggest pieces of code that I ported into Hermes Agent. [SPEAKER_02] We built that idea of using our RL framework with this, you know, this cloud compute into the earliest version of Hermes. And the idea being that if you're running Hermes with a local model and you have an ability within Hermes to design an RL environment, kick it off on some cloud compute. And just let it go. And then download those weights and replace them with whatever's running in Hermes. [SPEAKER_00] It's like hot swap. [SPEAKER_02] Yes. You never. It can teach itself a skill. Like, if you task Hermes with, like, I want you to create an RL environment to learn how to use the terminal better. [SPEAKER_00] Yeah. Right. I'm having a horrible time with this ERP system. Yeah. Like, here's a bunch of data of me using it. Go make it easier for you to use it now. [SPEAKER_02] Yeah. And so that was the earliest, like, embers of what we would see RSI becoming. And there's a lot of other ways that this has all kind of come into the fold. Um, but that was a big contribution I made. And I felt, you know, I run loops on my own hardware of, like, I have a Quen model running as Hermes that it designs an RL environment, kicks it off, downloads the weights, and then hot swaps into this. [SPEAKER_00] Dude, literally Neo learning Kung Fu. Yeah. I know Kung Su in the moment, and you're there, like, oh, sweet. My model is now much better. Yeah. And it can do that. But then it changed? [SPEAKER_02] Yeah. So we ended up, like, RRL training framework was good, but a lot of priorities in the company have shifted since we did all that. Namely, Hermes Agent took off way more than any of us expected, so we shifted a lot of resources towards that. But, um, model training paradigms got a lot harder since then. [SPEAKER_02] Like, the DeepSeq RL explosion was sometime, you know, like, it was recently enough that people are just now figuring out how to really harness, like, not only RL on specific tasks or specific task types, but this, like, long context, long horizon RL. And, like, really hammering it for agentic loops and, like, agentic, you know, models. And I think we've noticed this a lot in, like, the latest, you know, anthropic series of models. [SPEAKER_02] I think all of the open-weight Chinese models that have come out have since, you know, really benefited from focusing on this, like, longer context stuff. The first one that I'm forever will show this model because I personally love it. So Xiaomi, the telecommunications company. And Carmaker. Yes, and Carmaker, right? [SPEAKER_00] Yeah, the massive, you know. Incredible Carmaker. I'm really thinking about getting everyone because I live in Germany and they can ship them over there. Yeah, yeah. You should. I know. See them a lot. They seem really good. Anyway, I digress. Xiaomi. [SPEAKER_02] What do they do? They were some of the earliest to jump on this agentic RL stuff. And they had also picked up, focusing on training specifically for Hermes agent pretty early on. And they're doing it in the open, like writing papers about it? Yeah. Yeah. Really? I haven't seen this at all. Xiaomi is like, I was a fan of their model. It performs really well in Hermes for its, like, weight class or whatnot. [SPEAKER_02] And so, you know, we were really inspired with what they did to sort of, we've been amassing all of these different learnings and basically saying, like, okay, if we can take, you know, all of this agentic open training and stuff and we're just kind of reshaping the post-training team to be able to do this kind of thing well. Like, that's, I think, our goal as the RL post-training team at News is we got our start training models. Yeah. [SPEAKER_02] And we have a really strong talent density in terms of the, you know, there's eight of us and we can train a really good model given the right time and resources. That is so wild. So we've been sort of biding our time and making sure we can do it right. Yeah. And so definitely more to come on that soon. [SPEAKER_00] But how does it go to the continual learning? Like, explain that now where you're at with all of it. I'm glad you asked. All right. [SPEAKER_02] I'm glad you asked. So, Xiaomi, the latest model is Mimo V2.6 Pro. In that paper, they talk about this brilliant thing that has come out lately and it's sort of obvious when you think more about it, but still the idea struck me. They call it trace to environment. And the idea is that you take a bunch of agent traces and synthesize RL environments based on those traces and then train on them. [SPEAKER_02] And this could work in a continual learning loop like every night, every task that you've had your agent do while you're sleeping. It goes and creates RL environments, trains itself on them and updates. Or you could do, you know, amassing tens of thousands of different traces on data sources that you have. Why not just spin up a bunch? Yeah. If you have a lot of data of Hermes agent uses, what we do first is what's called failure analysis. So, we go through and find the exact points where the agent makes a mistake. [SPEAKER_02] Then we cluster those by the mistake type. And once you have a bunch of different piles of, oh, the agent, you know, entered the wrong directory here. Or, oh, the agent, you know, copied the wrong file here. Like, you just have a bunch of different types. Then you can sort of build an environment that captures those mistakes or build a bunch of environments. Are you human labeling those or is that all? All automated pipeline. [SPEAKER_00] Also awesome. [SPEAKER_02] Yes. So, again, that helps with the recursiveness. Yeah. Huge. Okay. That was one of the last things I've built that news was like our internal, we call it the insights and observations pipeline. It's actually based on some research, Anthropic Publish, where they do the same thing. And they didn't publish the code for it, obviously, but they did publish their methodology. [SPEAKER_02] So, we sort of co-opted that between Claude itself and their Claude code variant of doing it and built our own internal pipeline that does this insights and observations, like failure analysis, clustering. And then we have someone working on lately, which I'm going to start collaborating on too, you know, environment synthesis. And then we can just loop that. What's the environment synthesis? [SPEAKER_02] That's the generation of, like, you take a bunch of failed traces and you have a bunch of examples of, like, here's where the model messed up. And then you build some kind of, like, fake environment that's supposed to match pretty closely to, like, those, like, the world that those environments were in, like, the state of that computer at the time and the files. But it's not the real thing, obviously. Yeah. It's just synthetic. And then you can add sort of levels of difficulty based on how well models do. [SPEAKER_00] How often are you trying to train on, like, gigantic cloud or so not train on the cloud, but, like, train for cloud environments? Because if I've got a bunch of different AWS services I'm using and maybe I've got some Google Cloud services I'm using and I want to recreate a little bit of that, can you bring that into the environment? Yeah. [SPEAKER_02] Yeah. We've done a fair bit of, like, Google Cloud, you know, simulated, like, cloud environment training, too. And I think that that's a lot of this is very green. Like, there's so many areas of exploring ways in which you can put an agent in the simulated scenario that we, like, probably haven't touched yet. And I think exploring, like, cloud deployments and managing cloud infrastructure is one of them. [SPEAKER_02] Because a lot of times the traces that we see or the traces that we get from ourselves are all Hermes running on our own local hardware or local machines. That's what I was thinking. Yeah. [SPEAKER_00] But if you're, oh, maybe you want to start serving a bunch of traffic and you want those traces, too. Yeah. And you want to see, like, oh, now the agent is trying to fix something on production traffic or simulated production traffic, right? But it's messing up and we've got a SEV1 incident. And then we can see that, cool, it's messing up here. Let's spin up that environment, kick off another training run, and have that whole loop happen again. Yeah. [SPEAKER_02] So I'm very excited about that. That is one of the things I'm the most excited about and I think is a very healthy place that the RSI conversation has turned to. Yeah. Is less of, like, just having the agent decide, like, to teach itself some new skill. Because we kind of do this already with, like, the tiers of memory in Hermes and this, like, automated skill writing. Like, the agent can kind of already bolt on things that it's learned to itself in context. [SPEAKER_02] But learning from its own failures, you can only do so much without sort of this idea of, like, just collecting all the failures and building some learning environment for it. And, you know, that might honestly be what's left in post-training. Like, everything else has kind of been. Really? Yeah. I mean, there's a lot. That's a hot take. There's a lot to do. But I think that that's, like, the last, like, yeah. [SPEAKER_02] I mean, only thing we can do right now is improve models for agentic use. Because that is by far in a way, like, what we care about models doing. How long they can act in a harness. How long they can go without making mistakes. Yeah. Improving token efficiency is one. Like, that's one I explored early on in my research. Tell me more about that. You want a model to be able to finish a task as efficiently as possible. [SPEAKER_02] And then there's this, like, exploration versus exploitation balance of, like, you want it to spend some tokens exploring other options. But if it knows that it's going down a wrong path, you don't want to waste a lot of turns or tokens doing that. [SPEAKER_00] Yeah. You know what messes you up there, though, is, like, the models will a lot of times lean on their pre-training. Yeah. And so you say, like, you want something done. And it goes, oh, yeah, I know how Vercel works. Meanwhile, Vercel came out with a new version. And guess what? You don't know how Vercel works. So it doesn't go and, like, read documentation or try and figure out how it works because it says, I know how it works. And then later you're like, ah, it doesn't work like that, does it? [SPEAKER_00] You get these, I don't know how to explain it, but there's those moments where the model is magical because it knows things. But it also is almost like it's Achilles heel. [SPEAKER_03] Yeah. [SPEAKER_02] A hundred percent. And that used to happen a lot worse with knowledge cutoff dates. And now less so, but I think a lot of labs are trying to train models to act with the, oh, I should look this up. But instead of, you know, oh, let me pull from my memory. But then it's not as token efficient, right? Exactly. Because then it has to read through the documentation. [SPEAKER_00] So that's what I was thinking. Like, if you're doing research on token efficiency, like, you want that, but you don't. [SPEAKER_02] It's a huge tradeoff. And for me, token efficiency comes more in, like, long reasoning traces. Like, I think there's a lot of cases in which models sort of over-reason because they've been sort of trained into this basin where, like, this longer reasoning answer involves a lot of backtracking and second guessing. I think a lot of that is, I mean, this is, like, my personal hot take is I think that, like, a lot of that is somewhat unnecessary. [SPEAKER_02] And that, like, we've only trained models to be like this because we expect that they sort of make a wrong mistake and then correct themselves. And, like, most of these models are good enough. Like, I actually think they could reason less and still get to the right, you know, right answer. [SPEAKER_00] Yeah, we also just kind of assume that the longer it reasons, the more correct the answer will be or the more accuracy we're going to have at the end of the day. And that's true that potentially, like, a lot of that reasoning is wasted reasoning. [SPEAKER_02] Yeah, it's just tokens, like. And, you know, on the contrary, like, I did do research on this. I published a research blog about this. It did turn out that the longer you have a model reason, the better it does. In a generally constrained setting. So intuition is right, but. But I also think that, like, a lot of people liken this to a Turing tape analogy. Like, if you're computing a program on a Turing tape, like, the longer the Turing tape, the more, you know, you're going to approximate the program you're trying to compute on it. [SPEAKER_02] Like, this classical old computer science analogy. And that's roughly the case here. I wonder if there's a point of diminishing returns. Like, I was looking at, you know, up to maybe 2,000 reasoning tokens. But, like, when you're reasoning for 10,000 tokens or more, like, there might be a diminishing returns point. And I think it's worth looking into more. But that's also one that comes into training. Like, a lot of training regimens these days involve, like, reasoning length or, you know, token efficiency and reasoning. [SPEAKER_02] Like, sometimes you want the model to reason less. It's usually there's less room for mistakes. The newest Quen series has three reasoning settings. And default is extra high. But you could do, you know, medium and low reasoning if there's some cases where you might want that. [SPEAKER_00] Dude, I'll tell you what. I never know how to. For whatever task I'm giving, it's always high. And then every once in a while, it's like, ah, I think this could be medium or low. We'll see. Well, first of all, like, I'm pretty sure the OpenAI models, they don't even have low. It starts at BIA, yeah, yeah, yeah. As they know. So, like, ah. And then it just goes medium, high, extra high, ultra high. Ultra high. Like, I don't understand when I should be using these. [SPEAKER_00] I have no idea. It's so complicated. And please, can it just figure it out for me? Yeah. Like, that is, the fact that we have to decide what we want is a little bit, for me, it's probably, like, an artifact of today. Yeah. [SPEAKER_02] But I get the feeling it's not going to be. No. I also, I mean, I never think about it. And I don't change the options. I'm like, I'm always using the highest reasoning because I just don't think about it. That's it. But then you end up wasting a lot. Yeah. I think it's just, like, a hypothesis that I'll put out there is that that's likely the case is that there's maybe more of a dynamic way you could control reasoning length. And then you would also ask, like, things I'm excited about. [SPEAKER_02] And right now, the areas that I'm most excited about with all of this, actually, or even outside of the RL post-training regime, inference time improvements. You know, I think between there's a lot of work that comes out in steering and, like, using these external vectors and activation math to, like, guide a model in some behavioral direction. Yeah. And then also sampling, like, the order you pick the tokens. [SPEAKER_02] My earliest research for the token efficiency thing was in sampling. And latest research that I've published was in steering. And it was specifically focused on, like, refusal steering, like, safety-related work. But I am more interested in steering towards behaviors that could make an agentic model better, especially smaller models. [SPEAKER_03] Yes. [SPEAKER_02] So, like, an example is, like, if you're solving a math problem, if you had some way to steer the model towards being more pragmatic, like, if you could multiply some of the activations at inference time and be like, hey, you're solving a math problem. I want you to... Like, go hard. Yeah, be more pragmatic. Or if it's doing creative writing, I want you to do more... [SPEAKER_00] At inference time. Yeah. It's almost like you have latent lauras or something. [SPEAKER_02] In essence, it's kind of the same thing. It's like a trained adapter that you can basically choose to add in at inference time. Oh, man. And lauras are an extension or sort of a parallel, you know, adjacent thing to that, that you can train specific lauras and hot swap them. And Apple was one of the first people to publish research. [SPEAKER_00] Yeah, I remember... That's why it was like, oh, yeah, I think somebody was doing that. Yeah. And you can have that come in, especially when it's on device. It's very useful. Yeah, super. But then you're talking about it a little bit different. And you're saying not necessarily these lauras that are hot swapping in. And it's just inside of the actual transformer architecture? [SPEAKER_02] It's like at the end. Like the mechanism that would be multiplying the vectors in is external to the transformer. But there's no really swapping. It's sort of just multiplying a vector into the last layer of the model or whatever as it's sampling to change the behavior. And you would create that when you're RLing the model? I would. We would create that as sort of like an external thing that builds into the inference engine. So that's like my vision is that if we are able to serve inference... [SPEAKER_02] So into like VLM? Yes. Yeah. Into like a VLM, you know, sidecar that basically does this like dynamic, you know. Whoa. Okay. And that I think is like a really underexplored area of improving small models. Because right now we're kind of trying to RL them to death to get performance gains. [SPEAKER_00] Or distill the shit out of it. Yes. Make them super small so that you can have them on edge. And then it's a different story when you're saying you can still get the value of the small model or of a larger model. But with that smaller model, maybe you need it on device. [SPEAKER_02] Yeah. And I think, I don't know. I just think, again, there's some areas of this that are really underexplored. Yeah. And so this is what I'm personally very excited about. I think most people don't want to touch it. It ties a lot. And to like the interpretability field and understanding. Oh, really? Yeah. Because if you can control a model's behavior, likely you're getting closer to understanding the components that you're touching. Yeah. Where those behavioral things come from. [SPEAKER_02] It works a little bit in the other way where if you do understand a behavior, then you can easily control it. And sometimes when we do the steering, we control something and think that we can explain backwards why it actually exists in the model. Normally not the case. Yeah. But that's why they're like this Venn diagram overlap of like understanding the model interpretability, steering, like controlling behavior. [SPEAKER_02] And like the middle is when you've hit both on the head and you understand both the components that exist and how to maneuver them. And I'm normally focused on the latter. Like I only care kind of about steering behavior. Yeah. At least in this agenic context. Yeah. I'd love to understand how a model works. But for us, it would be really beneficial to be able to just kind of guide it in these ways. Yeah. Because you can get such a better outcome. And it's just two sides of the coin. [SPEAKER_02] It's like one, you're training this behavior into it and sort of designing these RL environments and these post-training environments to like converge into the behavioral basin that you want. And the other, you're sort of sprinkling it in at inference time and choosing to like, you know, and I just have a hunch. Yeah, I have a hunch that there's these ways that you can sort of do it a little bit more deliberately and a little bit more fine-grained at inference time. [SPEAKER_00] In my head, I almost envision that animation that you see when you look inside of how a key clicks all of the different pieces of a lock before you can open it. Right. It's like you need certain things to be lit up. And so then you get that key and it sticks in and then boom, we now have this model. [SPEAKER_00] It's the same model, but because we put the key in it this way, it hits all of these different pieces and then you can open it and it gives you that. [SPEAKER_02] Yeah. Yeah. And every lock is a different, you know, you insert the key for one behavior and all of a sudden now you've, yeah, that's a great analogy. [SPEAKER_00] Or like swapping, you know how like gears kind of sometimes? Yeah. So you have like, oh, the gears fold together like this, that kind of thing. I'm seeing it very visually because it is so, it is pretty abstract in a way. So I'm like only visualizing it and then it's a little bit harder to figure out the things on it. But dude, okay. So the last thing I wanted to mention on this was, well, I mean, there's so much, there's so much. [SPEAKER_00] And I just got word that we got one minute left. So I'm going to try and like close it up. But you kind of blew my mind on a few of these pieces so that, ah, damn. As I went into the visual analogy, I started just thinking about all different kinds. But what I wanted to say was my friend Linus, one of my favorite talks that I've ever seen was from Linus Lee three years ago when LLM's first came out. [SPEAKER_00] And he was doing a bunch of interpretability stuff and he was talking about how we can visualize the inner workings of how the model is coming to its conclusion. And hopefully we can get different ways of interacting with it. So in this talk, he talked about how when we interact with computers, sometimes we will use histograms. [SPEAKER_00] Or sometimes when you're like with music, you can have a high pass or a low pass and you see the ways that you're interacting with the generation or the computer generation in a different way than just like, oh, we're looking at output, like just a bunch of text logs and whatever. [SPEAKER_00] And it feels to me like some of this stuff that you're talking about, it would be so cool to see a histogram of the inner workings of how it's going. And can we steer it differently? Not necessarily steering it in how we want to like type and say it needs to go this way or that way. But can we like have this RSI loop training and we are deciding how we want to steer things because of a histogram. I know it's way out there and I may have. [SPEAKER_02] No, I mean, you're touching on like all of this does connect. If you can understand components of it and the visualizing as a whole, like everyone's trying to figure out the best way to visualize this kind of stuff. But if you can understand different components of a model and sort of, you know, shuttle RSI learning updates into them, like eventually we will get to what we envision is like true continual learning. Like a model that can just update itself constantly, like we humans learn. [SPEAKER_02] Like that's the goal is we want to be able to understand like how to update one track in the model's mind and not do a full wave update of like every little neuron gets changed and we don't know why. Like we want to know why. And like if I encode some factual information about AgentCon in San Jose 2026, like I want the one pathway in the model's, you know, neurons to encode that information. Exactly. So I think you're, that talk sounds interesting. You should. [SPEAKER_00] I'll send it to you. All right. So we're going to shoot over to the next talk. Speaking of which. And let us know in the chat. I'm going to go look at it now. What you think. If you're using Hermes, give a shout out to Sam. Thank you for doing this, dude. Thanks for having me here. This was awesome, man. And I'm like, I'm really glad we got to nerd out on it. Yeah, me too. This is great. We'll see you all after this next talk.