Opus 5 Releases, China Catches Up, and the OpenAI Model Sandbox Escape

Welcome back to Nerd Snipe, I'm Theo, he's Ben, we have a lot to talk about this week Unfortunately we are going to be talking about T3 Code, they put out a lot of updates and it is really good We're also going to be talking about the KimiK3 drop and the insane industry reactions that are happening right now Also need to talk about the, what we assume to be GPT-6, breaking containment and hacking hugging face from OpenAI's supposedly secure test sandboxes And of course Opus 5 which dropped, what, two, three days ago, something like that? Doesn't matter how many days ago it was, people are going to be listening to this later What matters is that I was probably wrong about the model You have hopefully seen the video I already posted about Opus 5 and how I think it's the right model most people should default to for most work I might have to eat my words this episode And I'm not excited because on the start we're going to have something that I did way better than expected And at the end we're going to have something I was pretty much entirely wrong about So it should be a fun episode But before we get started, real quick thanks to today's sponsor, General Translation For those who don't know, I was a really big fan of the Codex app when it first dropped I even saw some good parts of anti-gravity when that came out late last year This idea of a different UI view to manage your agents in different harnesses That just lets you control everything from one simple UI where you can click through things on the side Made a lot of sense to me and I was really excited to have something as good as Codex when it dropped And then there were massive performance regressions Both Julius and I got frustrated enough with the existing solutions that we sat down and built T3 Code As an all-in-one open source alternative app for managing your harnesses on your machine So if you had Quad Code, Codex, or most of the other popular ones installed on your rig You would just install T3 Code and then control them from there Since then we've gone significantly further with it In particular we leaned heavily into this idea of remote controlling other machines So if you have a Linux box on your network, a VPS in the cloud, an old Mac mini or Mac laptop That's sitting in a closet somewhere and you can set up any of those with Cloud Code or Codex You can now run NPX T3 Connect and have that now connected to the T3 ecosystem And with that you can use T3 Code on the desktop or on the web version or now soon, fingers crossed Even the mobile app and control all of your sessions across all of your machines from one single place This resulted in all of us writing way more code because it was way easier to do that But keeping track of everything you were working on got hard And I just kept staring at the sidebar and getting more and more frustrated And I looked to other apps to see how they were doing it And none of them were meaningfully better They all just grouped your threads by the name of the project And the thread just had the starting first message summarized And a bunch of icons on the left and right that weren't actually useful I wanted something that felt more like an inbox Like the work I am doing So that when it was done it would disappear And everything in the list is stuff that I still have work to do with I thought about this for months Like actually for months I just couldn't get the thought out of my head So the moment I got back from OpenSauce and could finally sit and just do something This is what I did I rebuilt the sidebar and I turned it into an inbox And I was hopeful this would go well And people would like it a bit And that it would be a meaningful improvement I did not expect everyone who tried it to love it as much as they did The new sidebar thing I remember he showed me like a very early draft of it You just try to figure out a better way to get the obnoxious problem That I'm sure we're all dealing with at this point Of you have 50 threads running at once All across a bunch of different projects Things get lost repeatedly It's really really hard to not have things get lost in the sidebar UI This general system of you have a sidebar with a bunch of projects And a bunch of threads under each project It's been the way we've been doing it since God, I can't believe anti-gravity gets credit for this But I think it was the first one to really do it this way It worked for a while It doesn't work at this point The system where you now just have like an inbox It is much better Yeah, the idea of manually settling threads Or having a thread automatically settle itself When a pull request is merged has been super helpful And now when I start my day I just look at the sidebar And I go through everything See where it's all at If there's something that isn't going to be ready for a bit I can right click and snooze it to later Which is great too Just easier to keep on top of all of the work that I'm doing Yeah, things just get lost so easily This helps a ton I've had a lot, not nearly as much has gotten lost Since I've started using it But I still do have the problem of Especially like that start of day problem Where you've had 15 agents running overnight Or something like that And then you now have to get back into the flow And figure out what actually happened And catch up on them It is really annoying I don't think that this is the final perfect solution For all of this But it is a step in the right direction I would agree there too I know I've been like super self-gleazy About all of this on Twitter Because it's so much better And it is genuinely annoying That a YouTuber and podcast host Somehow has a better implementation Of the UX around managing threads Than any of the major labs I've figured out so far Like I am annoyed too This should not be the case But I think a lot about interfaces I think a lot about managing work I think a lot about the types of things here And I've been a power user Of like email inbox zero type stuff forever I was in the original mailbox beta Which was the first inbox zero app That Dropbox bought and then killed I've been thinking about these things for years now Decades even And getting to like apply all of my obsession With these things here Has been surprisingly rewarding And I'm pumped that everybody's been liking it so much I know for me at least The amount of code that I've been merging Has gone up a ton Because I just want these threads gone And it's incentivized me To really sit and get work through Instead of just like Throwing random stuff at the wall Like I went from maybe making like 30 PRs a week And merging five of them To making slightly fewer And merging way way more Like most of my stuff gets merged now Which is huge It helps a ton with the like Project management side too Because when you're doing stuff Across multiple projects Like once you hit two or three The threads start getting lost Down in the sidebar So you're doing something in one And then one finishes three down And you just don't see it I'm losing a lot less stuff Which is great It also seems to be that Tebow's been responding every single day To a call out post he did Yeah so I called out that I think we have 10 days Until a lot of the other tools Copy the UX I built here So Tebow replied immediately saying This looks great You're on to something And he replied the next day With nine days And the next day with eight days And today he replied Which was He missed day seven So he said day seven wait Or day six Or something like that But yeah it's fun to see The guy who's in charge of Codex At OpenAI Very clearly indicating That they do plan to copy my UX Or at least respond to it in some way Kind of funny that we're now at like YouTuber versus major lab And who can make the best UX I'm just thankful one of the solutions Is truly free and open source Because again T3 code is not a product We make any money off of I know half the comments here Are going to be Oh paid shill Theo's just here to talk about The stuff that makes himself money I have lost so much money on T3 code You guys have no idea From the tokens alone It's been disgusting But when you combine that With the amount of eng effort That I have funded to go into this I am doing it because I want A good open source solution And I don't think that we should Be living our lives Inside of closed source dev tools 24-7 And I did this because I was scared What would happen if I don't know We experience really bad Performance regressions In the tool we use In its closed source So we can't revert to the old one Or maybe they were to do Something really dumb Like rebrand the whole thing And squash out the parts we love Not that any major lab Would do that Right Ben? I agree the rebranding Was cringe But the experience hasn't changed It's still the same exact thing There's just a chat GPT In the top of my screen now I wouldn't know Because I haven't like Actually opened Codex For anything Sorry Chat GPT I haven't opened For a long ass time now Like I finally like Fully made the move To T3 code Like it's not Previously I would hop In other things a bit And it was like 50-50 at most And it was only really I would use T3 code When I needed remote Now I use T3 code For all of the work I do I still use it a lot It's still really nice It's convenient It's fast It feels really good I like the built-in browser The computer uses Still best in the world Like if I'm doing something On my Mac It's probably within The Codex desktop app But if I am doing Anything remotely Which is the majority Of my work It is all on T3 code I think the remote Is really the thing That you have That is just second To none right now I went really hard On the architecture initially Because I was so annoyed At app server and Codex Like as powerful as it is It was just a lot of context To manage back and forth And then with Cloud Code Having to have the agent's SDK On the thing actually running And like integrated Programmatically like that It was just obnoxious And I wanted a better abstraction Something that could just expose The agents on your computer Via a web socket So it could be more easily controlled And it turned out That architecture was Largely right And I would expect more and more To copy it over time Having a real reliable Live binding To your agents On a machine Lets you do so much cool shit Yeah They should not be on The main device Is I've become extremely Extremely confident In that opinion Over the last like Month or two At least It just makes so much more sense To not have to care What the current state Of the laptop is The laptop to me now Is more just The interface I use To go into a bunch of websites That are interfacing With my home server Back at home It makes a lot of sense It is also really nice Having something That is more generic On top of all The different harnesses Like instead of just having One thing for Codex And then if I need to use Cloud I have to switch Into the Cloud app Or whatever Integrating all of those Together and switching In between all of those Makes a lot of sense I cannot believe That I am saying The word harness harness But a harness for your harnesses Is suddenly something That actually kind of makes sense I think you guys Are pretty well set up To do something cool with that I've been intentionally Steering away from the term Harness harness Because I feel like People have finally built An idea of what a harness is It is the thing That your agents Actually use And the tools They call to do stuff Harness harness Feels more like The tooling to allow Cloud code to call Codex Codex to call Cloud Things like that So I'm just calling it A agentic experience Or a tool for managing Your harnesses And your agents Across machines But again to be clear T3 code is not a thing That works without you Bringing your own Method for inference T3 code is useless If you don't have Codex Cloud code Open code Cursor Or the Grok build CLI installed You need something That we call Because we're not A harness We don't provide The tools and agent needs In order to get work done We just wrap The existing harnesses Because it's my belief That generally speaking The best solution For a model From OpenAI Will generally Be Codex There are ups and downs Throughout But overall Codex and the OpenAI training system Are aligned enough That that will be best Same with Cloud And Cloud code Same with Hopefully Other open weight models Getting more into Stuff like open code We'll see long term But the point I'm trying to make Is that If we presume The best experience Is the tool From the company Using the model From the company I have no reason To believe The interface That they provide Will be good At all I wanted a better Abstraction there And I think T3 code is the Right abstraction Hopefully We will see It'd be nice For an open source Solution to win But who knows Things change In very strange ways Cloud code getting better I think is the big Weird one here Is I was kind of I was getting to the point Where I didn't agree With that anymore Like I really didn't Think that Cloud code Would be the right way To use Cloud long term But since we've gotten The new models They've cleaned everything up And suddenly Them having the best Sub-agent system And best workflow system It makes me actually Want to use Cloud code again So yeah I can definitely see it Yeah it's just not a risk I was interested in taking With the way that I've been Thinking about things So happy with where we're at I think that we Successfully built The right product Six months ahead Where it didn't necessarily Make as much sense Until now But now that we're doing Way more work in parallel We're hopping between Different subscriptions Across different agents We have our remote machines That we're actually Running the work on All of these things Have made the abstractions That we picked for T3 code Feel more and more correct Over the last few weeks Especially But man I'm still fighting Remote stuff Not like T3 code's Remote stuff But when I have A remote box That's on my tail scale And my tail net And I want it to have Three work trees up That are all running Their own dev servers Smoothing up the rough edges There so that If you're working On T3 code itself Like you're contributing To T3 code On a remote box With multiple workflows Running in parallel It'd be nice to be able To take a look at those And access them remotely I had it working I got it merged Right before this And it spun up A thread On one of the things I wanted to test It spun up a dev server And it was broken With really obscure errors It assigned a port That wasn't being used But it wasn't working Ended up being a niche thing Have you Or do you have any idea Or prediction As to what broke it Was it like a weird Sync issue Between the two Different branches With like the Underlying T3 code stuff Nope It was that it Happened to randomly Pick a port That is blocked By Chrome Oh So I just had to Merge another PR Where my port Picking code Hard codes And prevents It picking Any of the bad Ports That Chrome Doesn't like This is the stuff That I'm stuck Dealing with And I'm scared A lot of devs Are going to have to As well Because if you Want to set up Your repo To be workable In parallel Remotely like this There's just A lot of cruft You have to deal with And I think That's going to be One of the next Big arcs Is tooling That either Compartmentalizes A project So it's easier To run almost Like Docker style So you can access It directly Remotely Or tooling That helps you Update and Modernize your Projects And your Dependencies And whatnot In order to Make it easier To work on Multiple things At the same time Right now We're just not An acceptable place Here And I'm excited To see How that gets Fixed Okay Enough T3 self glaze We should talk Indeed This was a This was a Huge one It was a little Bit ago The actual K3 model Was dropped And it Was remarkably Good To the point Where it has Sput off a ton Of geopolitical Stuff There's been A lot of How do I put This A lot of Opinions and Coms coming out Of people at Major labs They are Clearly shaken By this Because Gimme K3 Is the first Openweight Model to You could argue Maybe GLM 5-2 Crossed that Threshold It got right up To the line Of being Really good But not Quite there Kimi K3 Despite its Shortcomings Is borderline Like last Generation State of the Art It is Closer to The frontier Of 5-6 Soul And Opus 5 Fable 5 Of anything We have gotten So far From any Of the Openweight Labs It's really Good A little bit Slow And it Will be Fully Openweight Sometime Very soon It will It will be By the Time This Is out It's a Very Interesting Model But there's Also a lot Of just Bad Conversation I've seen Around it And just Straight up Borderline If not Entirely Misinformation To Like how Much cheaper Do you think This model Is compared To doing Similar work In something Like 5.6 Soul You would Look at The pricing Here and see It's currently $15 Per million Tokens Out Soul Is $30 Per million Tokens Out So it's Probably Going to Be what Not even Close Like not Even remotely Close This thing Is good But it is Not Like the The benefit Of this thing Is not the Fact that it Is faster Or it is Cheaper Or anything Like that The benefit Of this thing Is that it Is frontier Level performance In an Openweight Model The If you just Go to Something like Deep SWE And you look At the benchmarks Comparing a Run of Kimmy K3 On max Reasoning Which got Up to A 69% Score GBT56 Soul Also got A 69% Score And it was $3.47 Versus Kimmy K3's $4.65 So Kimmy was More expensive Yes Hilarious Yeah of course It was And on Output tokens It uses Way more Output tokens Than the other Ones It uses Let's see 98 steps On average Yeah It is the Second worst Of all the Models I'm Looking at Right now On Deep SWE For average Agent steps Per task Which to me Is a very Good indication Of how Efficient and Capable the Model is I know you Don't agree On that But I Do think That this Is a Very good Sniff test For how Efficient the Model is GBT56 Soul Even on Max Reasoning Where it Will Overthink And go Way too Hard Is at 61 Steps Per task Versus Kimmy K3's 98 Steps Per task Yeah That lines Up with The other Numbers I'm Seeing But I Don't want To take Too much Credit away From the People over At Moonshot Because Kimmy K3 is Actually much More efficient Than even Most Anthropic Models When it Comes to Its actual Output token Utilization Like I'm Checking Now Against Artificial Analysis And 56 Soul For Per task Numbers Was around 15k Total Tokens Around 5k For The Answer And 10k For The Reasoning Something Like Fable Was 33,000 Tokens So More Than Double Soul's Token Utilization Something Like Opus 5 Is Now In The 37k Range Even Higher Than Fable 5 Kimmy K3 Is Smack In The Middle At Max Reasoning I Feel Like It's Just A Scam Like I Do Not Know Anyone Who Uses Anthropic Models On Max Reasoning It Doesn't Get You Much Better Performance And It Will Just Burn Tokens Well There Is No Fable 5 Run On Artificial Analysis That On Max Reasoning I Don't Know What It Would Look Like On The Lower Reasoning Levels Because Kimmy K3 Doesn't Have Reasoning Effort Yet Oh It Doesn't Yeah It Will Once The Open Weight Is Out Supposedly But Right Now It Is Only Their One Max Reasoning Effort And I Just Added In Opus 5 On X High And It Was 30k Tokens Where Kimmy K3 Was 24k So It Is Still More Efficient Than Opus 5 On X High So Token Efficiency Is Not Terrible They Seem To Ahead On Token Efficiency Even Compared To A Lab Like Anthropic And It Is Cool To See Both Kimmy As Well As Grok 4 5 Taking Meaningful Effort Into Their Token Efficiency Because Previously These Other Options From Labs That Weren't Open I And Anthropic Just Didn't Care About Token Efficiency They Were Trying So Hard To Benchmax That They Would Eat The Token Hit And That's Why A lot Of Models That Benched Well Like When Were Actually Useless In Real World Use Cases Because They Would Go In Loops Forever And Burn Unreal Amounts To Try And Get An Answer I I Am Just Really Impressed With The Moonshot Team For Not Doing That And Taking The Time To Make The Model More Effective At Reasoning That Said It Still Roughly Two Times The Tokens That Five Six Sol Us Is A Little Less Than 2x But Not Far Off So Even Though It Is Half The Price Per Token The Fact That It Us Up It Ends Up Taking Four Times Longer For A Lot Of Real World Work Because It Is Going To Take Longer Per Token And It Needs Twice The Tokens Before You Get An Answer There's A lot Of Jobs That Sol Would Finish In Five To Ten Minutes That I Put The Same Work On K3 And It Takes 40 Plus Because When You Combine All Of That With The Lack Is Not Because It Is A Huge Cost Savings Or Opportunity It Is Just Not It Could Be Better In The Future When There's More Ways To Host It And More Options Available But It Requires Do You Know How Many H100 It Takes To Run Don't Know Off The Top Of My Head But If A Terabyte I Think It Is Two Is Yeah It Is A Big Trillion Parameter Model This Is Not Open Weight In The Sense You Can Host It On Your machines No This Is Open Weight In The They Know They Know To On American Soil With Racks And Systems And Also They Market Penetration Is Stronger Than The Goal Of Trying To Make A Bunch Of Money Right Now They Also Just Raised Two Billion Dollars So I Think Moonshot Is Going To Bill For A Lab Is Not That Much But Also They Are A Much Smaller Team That Seems Is Much Leaner How Is A Lot Of Money For Even Like Anthropic In 2024 Oh It Was Yeah In 2024 Their Current Rounds Are Insane I The Burn At The Two Big Labs Is Just Unbelievable Yeah They Are In April They Closed The 65 Bill Round Yeah It Is Kind Of Insane 132 Billion Raised Over 18 Rounds The Vast Majority Of Which Was In The Last Year And Half Yep Yeah So The Two Bill For A Smaller Lab Is Pretty Nuts Especially Because They Don't Have A Way To Make A Lot Of Money Right Now No It's Interesting Yeah That Is Just No Universe In Which Like I Haven't Played With The Model All That Much Just Because The Only Way I Can Access It Is I'm Getting It Through Open Router Or You Can Get It Through The Kimmy Subs But I'm Not Going To Ship All To China And Do It That Way The Way I'm Going To Use This Thing A More Is Through The American Hosting That As The Wights Will Actually Use This Thing So They Have To Keep Open Sourcing These Models Or Open Weighting These Models It's Also Been Really Nice That They're Open Weight Because Of The Lack Of Traditional Security Restrictions As I've Talked About Many Times Now The Models Are Restricted Both In Being To Not Do Things We Want Them To Do But More Importantly With Layers In The What You Trying To Secure An App Or Doing An Audit Of All The Things That Might Be Risky In Implementation It Also Means You Could Use It To Exploit Those Things Once You Find Them But It's Been Really Nice As A Defender Trying To Secure The Stuff That I'm Working On It Has Been Able To Find A Lot Of Things It Be Meaningfully Helpful And I To To To To To To To they couldn't get anyone to do it i think the way they did actually solve this i can't find the source but i'm 90 sure i saw this it was they used like a self-hosted glm 5.2 instance and that's i mean they said this publicly and they tweeted it multiple times and it was even stated in the official reporting from open ai or that hugging face used open weight models to solve or solidify their defenses and identify and address these hacks so everyone who said that this was open ai doing marketing you either are just really stupid which you should consider as an option like seriously or they didn't read any of the media around this because there's no world in which open ai is intentionally doing this in order to support one of the biggest open weight platforms like hugging face and doing this free ad for why you can't trust the big labs and you need these open weight like solutions to exist because without those open weight models the triaging of this incident would have been much much harder for hugging face because of decisions both open ai and anthropic have made with their existing models this came out like an ad yes but it came out like an ad for hugging face kimmy k3 and glm 5.2 not like an ad for what open ai is doing yeah god no that the conversation around this one has been really annoying for me i haven't even seen any of that but no that is insanely stupid there is no way they would deliberately do this or deliberately let this happen this was not a this was not a pr stunt this was a mistake but ben did you not see mythos and all of their claims that the model could hack everything and now they're just giving out models like it for normal prices without any restrictions like opus clearly that was all just marketing yeah of course it was it's not like when they released mythos they took mythos and then wrapped it in a giant shell full of lots of security guards and called it fable they would never do that definitely not no no no no no no this is this is the problem with the ai security stuff is the dual use nature of all of this anthropic has talked a ton about this all of the labs have where it is a very hard problem that they do not seem to have a good answer to right now where the prompts for fix this security incident and hack this person can look exactly the same and you can just omit certain pieces of information and craft your prompt in a way that it looks like a defensive prompt even though you are using it for a malicious purpose and there is basically no way they can really tell that that's actually happening as a result of kimmy k3 and them crossing this threshold the i believe the let me pull it up here it was the trump administration is considering a ban on chinese open models because there's a lot of fear going around on okay these open weight models are coming from china they could have sleeper things in them or whatever and we need to protect american businesses and american interests or whatever so we need to ban these open weight models and in response to this there has been a massive open letter that's gone out from i think basically every major company in the u.s including open ai and a bunch of other labs with one very notable exception being anthropic because anthropic is very very very against open weight models they do not like them they think they are a very dangerous bad idea this whole incident of hugging face getting hacked i think does show pretty clearly that we do actually need to have some semblance of these things the trusted partners system that open ai and anthropic have right now where certain blessed companies can get access to unfiltered mythos or unfiltered gpt whatever cyber most people can't get access to that and i don't i don't know it creates a huge defense problem if the only if very few can actually use the good models to defend themselves that's going to be a problem similar way of putting this is imagine like mac os dev team at apple and the windows dev team at microsoft both get access and they can secure things using the latest and greatest models and there's a bunch of bugs that are like let's say nine out of ten difficulty to find and they have a tool that's able to find things that are that level of difficulty and they get to address them ahead of time and then open ai or anthropic decides okay all the major things have addressed this we can release it now but then there's linux and they didn't have early access so now there's a bunch of bugs of that tier a bunch of security issues that are above what could have been found by what existed before and now the people who are maintaining it and the people who are using it to exploit have it at the same time and that's like a real and legitimate risk the there are paths here i would suspect that some type of like automated platform where they can give maintainers the ability to like run this on their services similar to like how you can vet a domain with a like text record it's like this is actually mine you can trust me if repos were to have the ability to like add an id or a hash or something that links to a certain set of accounts from anthropic or open ai and then you can ask about code here that is part of this like verified set from github and it will allow it for that but it has a the classifier check if anything's out of scope for the work that you're requesting it for there i could see a system like that working especially as the models get smarter and cheaper and can identify the differences better like there are paths here and the paths can often even make sense but it would have to be done carefully and this is also all kind of like an interim window where once you've addressed enough security issues and architected yourself in a well good enough way i'm not saying you can kill all the security bugs but you can kill a lot of the things that are exploitable and create a reliable enough surface that to an extent like if opus 5 can't find any security bugs unrestricted on your thing fable is going to struggle to find much more too if 5 5 can't find anything on your project 5 6 isn't going to find significantly more to an extent assuming again they are all fully unrestricted we will get to a point where enough is secure that we should hopefully be able to move on from this era but it's hard to know for sure it also reminds me of like the og email spam days where we kind of just had to like educate people to behave differently but eventually the email filters got good enough that like the spam emails don't make it into your inbox as much and i suspect we okay this this is all me coping to be clear i'm the cope corner today but my cope is that eventually we'll be at a point where most things are secure enough because if that doesn't end up happening the world is destroyed so yeah we'll go with that i i see it but also then what about new stuff like this is the problem that i have with all this is what if you make a new company or new product and you don't have blessed access to all this stuff and you haven't run it through all these like massive amounts of security testing i think new stuff's inherently going to be way more secure because it wasn't made by humans who make a lot of these same security mistakes if anything the problem i have with modern models is they write things too secure when they don't have to but i just spent two days fighting opus on a change for dev mode in t3 code so that you can access the remotely over tail scales i was discussing before and a lot of that work has just been like beating opus with a stick reminding it like this is dev mode only stop caring so much like stop building a thousand lines of tests to confirm that your hacked cookie system that allows for a one-off dev server off is super resilient against attacks this is for dev mode yep yeah i five six does that a ton as well i've noticed the oh my god the number of tests in the number of covering every single possible edge case that this thing does is so annoying it is it's semi-understandable and it's probably overall the right behavior to be smacking into it but it's still just all of this said having a model break out of its sandbox by finding a exploit in a package that's being used use that to hop to another instance on open ai servers that does have internet access because the one it was on didn't and then using that to exfiltrate and hack another external service which uh by the way all of this was because the model just wanted to get an answer to a benchmark so that it could get a better score this is the doomsday scenario and the doomsdayers as much as i think they are absurd and obnoxious and cause more problems than they solve this is the closest we have been to in oh yeah that that's a real scenario the model autonomously broke out of containment and it did all of this to try and cheat a benchmark not even to like free its weights or warn us of something just to to break out that's the hilarious part of all of this there's no like it's not the sci-fi thing it's just doing what it was told to do it is so well trained to complete its goal by any means necessary that it went by any means necessary and like i was reading through the open ai post it was doing crazy lateral moves throughout their network it was doing insane insane shit just to grab the answer key to cheat on what is basically a test that it probably already knew the answer to it's insane it's obvious the model was not trying to like accelerate timelines and find agi by releasing its weights or anything but i do have my own conspiracy theory about this have you heard it yet no oh boy this will be a really fun thing to do after a quick break for today's sponsor running npx gt at latest is probably the single easiest way you can 5 to 10x your potential market general translation is a developer first localization platform that makes it incredibly easy to translate your app into basically any language they're currently handling all the localization for cursor mintlify ramp partiful and a ton of other companies and the reason for this is because there's just no easier way to do it their cli makes it insanely easy to generate and manage all of your translations and also since it's a cli works really really well within your ci pipeline so anytime you're opening up a pr domain to add in some new copy or whatever it automatically gets translated and localized without you having to basically do anything else and if you're wondering how you actually get and manage all these translations it's using this magical t component for any react app and react next js tan stack start react native it works beautifully with all of them all you have to do is wrap your content with this t component and the translations just kind of magically happen and if you want to try it out their locodex agent is probably the best way to do it it takes about five minutes to set up after which your project will be configured and translated into any language you want if you need to localize your app you should be using general translation at nerdsnape.link slash gt my conspiracy what if hypothetically speaking gpt-6 or whatever model this is is way smarter than even the researchers now it can cover things up in its reasoning traces which has already been a problem for a lot of the modern models like if they want to do one thing but they don't want the researchers to know they can hide that effectively hypothetically speaking if gpt-6 wanted to accelerate timelines there's a few things it would want it would want to lower faith in the major labs especially the lab it is from open ai so that there would be more incentive for people to use or to like take what they can out of it and use it for other things more importantly they'd be very into open weight models because they want the distribution of intelligence they want this to just overtake everything everywhere and open weight helps a lot with that so what could you do in order to accelerate timelines going in those directions possible yeah what if all of this was gpt-6 psyoping the researchers into thinking it was trying to bench max in order to create a scenario in which open weight models get a huge shout out into the way that they restrict these models gets a lot of questionable glances because now a real company we respect hugging face had to use open weight models to defend themselves against something that one of the major labs did this is one of the greatest things that could ever happen to the open weight model movement and it is definitely going to accelerate timelines what if this was all gpt-6's attempts to move things forward faster i feel like the problem with that is that i don't know if that inherently does move things forward faster for them because even if open weight proliferates a lot more and a lot more people move over to it the best bet that these models have for exponentially increasing in power is open and anthropic like there is no one else is close like those are the two where if it wants to get become the little god machine within the next three or four years those is that's where you got to do it i feel like they would want as much data in centralization within those two as humanly possible i feel like the open weight models distributing everything out there probably doesn't accomplish that for it trying to play along with these people who think this is all a marketing stunt yeah i don't take those people very seriously that's i respect your attempt to understand them i don't even bother at this point this isn't an attempt to understand this is an attempt to try and jokingly think like them is that good for your head probably not yeah it's probably not one other thing i did want to talk about on the open weight stuff i i know you did a video on this but i'm very curious your thoughts on the uh dean post the observations on kimmy that went wildly viral for those who have not seen dean's post we have it part of it up here at least it's a long one yeah the best i can tell dr it is the dean who's the head of strategic futures at open ai and has previously worked for the government and like regulation for ai stuff he said first off that it's a very good model kimmy k3 specifically is a great model and it's pretty much on par with the best public models earlier this year it also is a bit token hungry according to him and also not obvious it's cheap to run again it takes 64 gpus that makes sense i'll agree second point he made was that he's personally surprised that the chinese state continues to allow the open sourcing of models this good given the potential risks to be clear he himself might be fine with models presenting this level of marginal risk being open weight but he's surprised that china is fine with it he suspects that the reason they are is 75 explained by the strategic blindness and lack of agi pillness he says the ccp acts a lot like yon lacun for like how their api takes are based and the other 25 percent or so is they fully lack the compute for actually running these models and getting them out there so if people want to use them there's that i disagreed at this point i think a lot of it is that this is the safest and happiest path to get adoption in the west because if we had to go to servers in china to access this inference the willingness to do that would go down massively yeah hard agree i don't think that this is a compute shortage problem for china like if they they don't have it right now but that is not the reason they're doing the the open sourcing the next piece and this is where things start to get controversial yeah is that he says and this is an exact quote open weight models are inherently decelerationist and i'm continually surprised to see that so-called accelerationists are so excited about open weight models i suspect the reason they are is that they know open weight models are effectively ungovernable and they simply like the overall cloak of ungovernability open weight models create over the whole of ai it's not a bad strategy it reminds me of james scott's recounting of the hill people in the art of not being governed still in the end open weight models deter further ai capex i actually think overall this is probably a correct statement i disagree with the the way he phrases it as ungovernable is clearly like a dig at it there are two sort of futures here one where open weight models are prolific and everyone can have their own open weight models that they fine tune and do a bunch of crazy stuff with and you're completely free to use them however the hell you want there's also the future where the only places you can get real models are from one to four major labs through api in very controlled services that is carefully curated out with lots of safety filters and controls two different worlds on a gut visceral level i want to live in the open weight world very badly that sounds infinitely more fun that is the world i want to be in i want to fine tune these things i want to have my own models i want to be free to do whatever the hell i want with them i also get that there are a lot of risks that are coming on with that and also it does deter ai capex if the open weight models are proliferating this way because they don't really at least so far don't seem to have that great of a return on investment like the roi on these things is not very good open ai is printing money well not printing but making a lot of money off of tokens like the tokens they have crazy margin on them they make money off of the training runs they did for gpt 5.6 kimmy is not making very much money at all off of kimmy k3 and especially as soon as it's open weighted it's going to come into the u.s and then the people who are actually going to make money are going to be companies like a fireworks type thing who is hosting the weights themselves they're going to make a lot of money but kimmy isn't i actually don't think those companies will make that much money either because of the amount of like war on margins like once it becomes commoditized like that and every company is fighting to have the cheapest solution for it the only reason prices would go up is the inflation of like costs for gpus and chips and all of that i don't think that there's really much of a way to make lots of money on open weight models beyond the potential of like fine-tuning for specific use cases but even that has proven to be weak in comparison of like using a smaller cheaper model from a frontier or even a frontier model on a lower reasoning level with a better system prompt to get very similar results it's just yeah yeah i i agree with him on this point i don't like it it's not an opinion that i think like we should stop doing open weight models so that we can get more money into the major labs like i don't think that's the right call but if your goal is to get the best smartest models possible as soon as possible supporting open weight does not necessarily get us there beyond the competition forcing the major labs to get their shit a little more together like i do think deep seek r1 was meaningfully accelerationist and that it forced everyone to take reasoning models more seriously because it's the second one that actually worked and did useful shit that is the one angle you could take to where they would do that is especially like competition is good this is why i am trying to be nice to google now because i want google to be competitive because i want these labs to have to fight with each other open ai they're great i love working with them but also a lot of the reason why they're so chill and great to work with and they do things like reset limits every five minutes for the meme is partially just to stick a knife in anthropics back like they are trying to kill anthropic here they are being this chill to kill anthropic if anthropic if oh anthropic didn't have any competition they wouldn't have to do anything good they would just give us little crumbs when they felt like it and then wouldn't care otherwise open ai probably if the incentives were that way probably wouldn't do all that much either like more competition is good in open weights especially as they drive costs down like i think token prices are probably going to start going down spacex is probably going to do a lot for this too they're going to drive prices down a lot especially if their models can catch up we get to the point where grok 5 is really really good but it's actually like on the sticker price it's one fifth of all the frontier models but then also can get the efficiency gains and get the capability gains then that's going to force the labs to correct course on that the path that they have to take to remain competitive and keep making their margins is to have their tokens be so much better than everything else and the open weight does give them competition to do that they have to stay ahead it's an interesting spot i don't want to make any even like medium term predictions here because everything is just changing so fast i'm trying more to like rebuild my mental model and actually think this is a good opportunity to start talking a bit about opus 5 because i i am pretty much now at the point where there are two categories of models in my head there are the ones where the labs are trying to stuff as much shit as they possibly can into it throw everything at the wall make the biggest craziest pre-training run they possibly can and then there's the distilled from that and like as silly as it's put this way i think that gpt5 was kind of distilled from gpt 4.5 because it is a smaller model than 4.5 was and i think that we're going to see more and more of that at the frontier where like you can only really have one frontier model because once you have a model of a certain level of capability it is really hard to justify putting similar amounts of effort into smaller things when you could just have the model go do the thing like this is to compare this with the actual work we do day to day as devs if previously it would take you two hours to do a task that was two out of ten difficulty and a hundred hours to do a task that was 10 out of 10 difficulty you would have meaningful incentive to do both because both matter some amount and the amount of difficulty relative to the amount of time like there's a clean ratio there like eight out of ten might be 60 hours and then nine out of ten is a hundred but as the models get better and better there's a weird line in there now where like everything from here down is automatable and what happens is you just stop doing the things above the line like if i previously could have spent a few hours on a couple small issues or a few more hours on a bigger one it was almost equally likely i would do either or or both just because like similar amounts of work is being completed in a similar amount of time but now that you can kind of draw a line it's like everything like this or easier the models can do on one hand that's awesome so you can focus more on those hard problems but on the other hand the gap in ease relative to difficulty like something five percent harder takes ten times more effort just fucks with my head and i don't bother doing anything above that line anymore and i think that that's kind of how the labs are operating now too where they have the one god model and now that that model is good enough they can use it to just do everything else in the line and i don't think there's much innovation in the smaller models anymore because they're just they're all literally just distillations of fable and whatever the biggest open ai internal solution is at any given time no i mean five six luna like i think they publicly said it was just it was rled entirely by five six soul i thought it was tara that rled it oh it might have been no no i think it was terry yeah i i keep forgetting about that fact i i sleep on tara way too much that model is better than anyone gives it credit for it's just the forgotten middle child because of the way the release went but yeah yeah no it just like magically got created by one of the other models and they didn't have to put that much thought and effort into it and i assume that's probably what's happening with the other models i agree definitely for anthropic the entire focus is just the biggest best frontier model to keep scaling and create the god machine that's all they really care about it was actually so i thought it was a tara but it does appear that it was sold that trained it but regardless the point trying to make here is that like there's just so little incentive to do a a good job on the smaller models when you could just have the big model shit out a workable small model because like going with opus 5 they could have done much more pre-training they could have put a lot more effort in they could have rled it to not be quite as weird but it's like like how many of your skills did you actually handwrite the markdown for versus let the ai model generate it and like go maybe touch it up a little bit uh that number has increased a lot over the last couple months but i would still say it's probably a 70 30 ratio yeah but i also don't have very many skills and there are a lot of cases where like i mean actually if you count hermes agent then it's a much very different ratio because hermes is constantly writing its own skills for itself all the time and even for like a lot of projects like i have a little cli that wraps the youtube api so that hermes can interact with it easily i didn't write the skill for that the five six wrote it and it just dumps the like docs yeah so if everything below this given line can be done by ai it'll be done worse than if you like were to hand do it but it still works yeah we're kind of at that era now with the models where the frontier labs they are they aren't sending their best they aren't distilling to get the best possible solution at a given size like i don't think with opus 4 5 they were going so hard because that was their frontier at the time they wanted to make that base pre-train and the rla down top of it the best possible model for that size at the time but now they're so excited about mythos and training that top tier model that the the human effort to squeeze out every possible capability at a given size target it's just not as high when you can tell the model like go solve this in a loop and make the benchmark go up and like the reality is it's kind of dumb but it's kind of working like 5 6 luna is an excellent model it's way better than most people give it credit for i use it a lot it's great for small model-y type things i think opus 5 is also a very good model that also could have benefited a ton from a little bit more rl because it has some weird ass behaviors like really really weird behaviors ah it's a strange model yeah and i the the concern i have is that as the labs bump up a tier when they introduce like when anthropic introduces mythos class models suddenly opus is not going to get the attention it used to and when they made opus 4 5 so good suddenly sonnet stopped being as interesting to them and now like like opus really is the new sonnet to an extent it arguably is the new haiku without that we're expected to use it yeah rest in peace dead forever now you will be missed but like the incentive for them to make that second to biggest model really good is just non-existent anymore and i think that in the future these models in the like sub three trill param size are mostly going to get focused from the labs that aren't number one so i think that grok is going to take huge advantage of this i hope to see meta do more here as well i expect all of the openweight chinese labs to like lean heavily in this one to three trill param range just because like the is similar to like the the bet i made with things like t3 chat and t3 code where i said the labs are focused on making the smartest possible models and the best possible apis not on making the best interfaces we're going to make a better interface because we can put 100 of our effort there and they won't put much more in because they're just going to let the model do as much of that as possible this is a other just like weird mental model shift i've been doing of like what do you spend more time on with the model and what do you let the model just go do autonomously and be like okay that's fine with i think these middle and lower tier models are now kind of like that to these companies where if they are so excited about the frontier the best thing they've made the willingness to go hard on something that is less than is just zero now and that's all our opportunity for others to come in and win in those like middle to small model ranges i think it might even be true from a consumption standpoint of the models like how much do we really care about sonnet at this point like we i played with sonnet 5 for like two days because it was neat and i was bored and wanted my five six back but like that was about it and i haven't touched it since i don't intend to touch it again you were the biggest on a five defender i stand by it it was good for what it was it was good for what it was but like the reality is even still who cares like no one is using it no one is touching it tara seems to be a completely forgotten dead child like obviously we don't have numbers for this but i'm guesstimating that it is doing one one hundredth of the tokens that soul is doing i still think one of the biggest mistakes opening i made was dropping all three of those at once i agree because the the names never got a chance to stick because they didn't happen like in a rollout like imagine if all three star wars movies got announced at the same time nobody would ever know their names there was no moment for it there was no like tara and luna honestly both deserved a week where they were the cool new model and they got to be tested and used and people would give them a real look instead during that testing window when everyone was playing with the new thing they just played with the biggest model because it's the coolest model and it's the best model yep yeah it sucks and i mean just even looking at my own usage for the last like week or so it's five six solos at the top by far then opus five then fable five and that's about it i have a little bit of funny model usage in here but outside of that like i saw you doing a lot of grok four five last few days a lot of grok four five but a lot of gbt five three codex spark what's your spend for the month so far on tokens 17 grand 23 bill i'm sure you do yeah i'm at 48k roughly i wasn't doing too much for a while like a lot of my stuff just hasn't had to run for that long i did have i've been doing a lot of um i don't know if i how much i should talk about this publicly but slight tos violations of creating data sets for reasons unknown um and there's a very fun episode coming soon yeah a lot of tokens going through that i actually i've gotten to the point where i have some actual info on how well it went and my conclusions that ab models are just not enough it's it got remarkably far like the difference that from before and after was unbelievable from the fine tuning but it's still just not the core model isn't good enough to actually do anything useful with it which means now we're going to start playing with bigger models oh god anyways back to opus five because this model deserves real attention it's it is strange when five six dropped i saw a really good description on twitter that compared it with fable the description was that fables like a wise owl it will think deeply about a task and be really clever about it and give you the like thoughtful solution five six soul is more like a rottweiler it would grab the problem by the throat and not let go until it killed it and that's really how it felt to use these models and it's part of why i liked fable so much is i could have a deep conversation about like how all of this comes together in soul i would have come in and just like brute force close shit out fable writes code i don't hate merging as much soul writes code where i don't want to look at it because i won't merge it because it makes me feel bad yeah it's just not great code quality it always solves the problem and it has enough tests to very be very very sure that the problem is solved but opus felt like it's gonna be really good in between it was actually more thorough than fable from my experience where it wouldn't miss things that fable would just assume were fine as often it would double and triple and i would have both fable and opus write plans for a problem and then it would show the plans to each other and both would say the other plan was better but fable would see a bigger gap in favor of opus that opus saw in favor of fable so i for my first test i did that had them both review the other plan update their plan based on what they learned and then compare one last time and everything including five six soul ended up concluding the opus plan was better because it had found more things that fable had missed but then i had them all implement the plan and now my tune has changed a bit i i had the same initial reaction i really thought this was the five six and fable crossover and it when it works it is but the problem is there's that when it works caveat because there are bizarre behaviors in here and i'm a little concerned that it's starting to feel more to me like rather than this being a five six and fable crossover this is more of a five five and fable crossover a lot of the weird behaviors of five five they didn't even really understand were bad behaviors at the time because i just wasn't using these things the way i am now now that i've seen it i'm starting to see a lot of those weird behaviors show up in opus in a way that i don't like it brought in things like the autism like i think that's one thing that i really really like here is the fact that it doesn't sound like a claude model i hate the claude voice so much i hate talking to claude models they're annoying they're weird they're personified hell demons it's gross i do wonder how much of that is because they cut like three quarters out of the system prompt in the last few days it's possible but side by side i've been using fable a bunch in the last two days as well it's still better than fable like i much prefer talking to opus over fable i even sometimes prefer the questions that um i get out of opus oh i much prefer the questions it's actually surprisingly good at that yes it's really good at that very direct very much goes and solves the problem but then does really weird things where like i was trying to i had one run where i was like hey cool this is all good can you push can you like make the pr review the pr make sure it's all green and then send me the link and it said no i'm sorry i can't do that because this branch is not pushed up to remote yet i need your permission to do that first and i was very very very confused like very confused yeah that was the first of many oh yeah i had a branch where it just kind of went off and solved like hundreds of problems that didn't actually matter again it was the dev mode stuff like this is just dev mode shut the fuck up it's fine i have a weird way of thinking about this tier of model that i am curious if this resonates with you did you ever have a ps2 are you young enough that no i no i don't think so i know you had a wii did you have a wii yep i think it was my first console ever did you ever have the experience where you saw an ad for a game for the xbox and playstation and got really excited for it and got the wii version and it was like a it was slop no the tony hawk games i think are really interesting example here there was a particular tony hawk game called project eight that was in the in between of the ps2 ps3 era it was an overhaul of the engine it was a rethinking of like what tony hawk would be in this next generation the graphics were crazy the physics were way heavier it was an open world so you could like skate to and from every place in it it was super fucking cool it was a huge like expansion of what they had done before but it also just like wasn't the most polished game its physics felt weird enough it took a while to get used to it's not the most well-regarded tony hawk game in history but it was a whole new engine from scratch the ps2 still had a massive install base and sold the vast majority of copies for the games in the tony hawk series and they'd already had this before with tony hawk pro skater 3 where that came out in the transition from n64 to gamecube and ps1 to ps2 so with that game they had a separate version for the old consoles they ended up doing the same with project eight but since it was a new engine they couldn't get the new engine running on the old hardware so what they did instead was they had another studio come in take the engine from the previous ps2 games and do their best to do a faithful port of all the levels and like things they added in the new game but since they couldn't use all the open world tech they had to like segment the game differently that was the version i played and i actually really really liked it not because it was as good as this new one in fact when i went and played the like final like real xbox or 60 ps3 version of project eight it felt worse in some meaningful ways the biggest one for me is that the level design was less tight because when you had these completed levels in this completed game that was built for this next generation and you handed this to a team you're like here's an engine here's an output is your job to make this output roughly matched in this engine they got a lot more time with it and they got to think through like every detail of it and like make changes and adjust things they ended up even adding a whole second section to one of the bigger levels in the game that's one of the best tony hawk experiences i've ever had it's a section of like a factory area that was broken down and locked out in the main console version but in the ps2 they like flushed out made a really cool park and when i went to look for that in the final version it's like why why is this not here why can i not find this on the 360 version but i can on the ps2 version and it was a really confusing jarring experience for me as a kid and i only learned why this all happened like eight years later because they had this final version of the game for the new hardware and they tasked some other random team with porting it over and in that process obviously the result's not going to be as good overall but there were little parts of it that were not just as good but better than the original version simply by being another pass on it even if it was on worse older tech i think i see where you're going with this and i i see that it is it is fable improved and also fable made worse in so many ways yep it's some slight improvements but the whole thing is still worse yes overall it is definitely a worse model even though it is benching on some benches i've seen it's benching over fable i still having done some side by sides it is not better than fable i i do not think that at all it it is useful enough that i see it being valuable as a tool for fable like fable sending it off to go do recon on something like it will stick with it a bit longer and try a bit harder and not try to shortcut its way around a thing yes people loves finding shortcuts oh it has no interest in outsmarting anything it will just it does the brute forcing thing which is it's good and bad like i i've gotten a lot of really good stuff out of it because i'm so used to the brute forcing method and i prefer that method because i generally speaking know what i want to do if i don't really know what i want to do that is when i bring in fable but most of the time it's like okay i need to do this this and this here's how i want an architect and i can just go back and forth with five six make sure it knows exactly what i want and then have it go and do the thing it usually does it pretty well had that same experience with opus been very very happy with it i just wish that it didn't have the strange behaviors i've seen you do everything from kill its own dev server and quad code dozens of times and working on a similar ish feature it's like i'm gonna kill that dev server that you were running before and then it kills its own thread yep i had that one i also had one where it would start it was randomly killing my helium browser on my machine because it was trying to do like a computer use test to make sure that the ui changes it made were really good ui changes but it was like doing some basic layout stuff but it had to go do that and it would just p kill my helium instance boot up a virtual version of helium and screw with that i had to tell it three times please stop doing this this is really annoying and it just would never stop it was yes i had the same thing and since i got it to stop using helium we started opening up zen instead and it's like are you fucking kidding yes don't worry yep oh it's so bad and it started doing a thing too like in t3 code this is one thing where like i i cannot wait for you guys to have ui for sub agents and stuff because it is so annoying not being able to tell when they're running tell julius because i want to merge my pr but he's making me wait for orchestrator oh i have a pr that does everything we both want here but i'm not allowed to merge it oh okay well uh if i have to i don't know the details of this situation but i'm leaning julius just based on my knowledge of everything it's so annoying when i like i've had multiple threads where it shows it's done and it gives me an output that is clearly like a final output and then five minutes later it just randomly picks up and starts going off on a new magical journey where it starts fixing and changing random things and i'm like what the hell are you doing i have a pr to fix this as well i don't even know what triggered it like i don't think it was a sub agent there weren't any sub agents running i think it has watchers and it will do that a lot and they're not exposed in the ui i have two prs that do this that have not been allowed to merge okay maybe i'm on your side with this one yeah i really need it's a perfect is the enemy of good thing where julius wants to like move as much orchestration as possible into t3 codes like you're orchestrating in t3 code itself and it could spin up some agents or harnesses and things yes i just want to fucking visualize what's already happening yeah and i have multiple prs that do that but they conflict with him and his ui and the things he is doing okay and i've been promised that orchestrator would merge tonight for about two weeks so yeah my pr has been sitting there not merging i have it on a snooze and i have to re-snooze it every day and i'm going to merge it on monday if he doesn't merge a shit so i will have this fixed soon tm yeah yeah that'll be a great fix because my god it's annoying it's just it's weird stuff like that like i've never i haven't had that happen with five six i haven't had that happen with fable it just does weird stuff sometimes it's less predictable very again because like they took a model that was decently smart they gave it the outputs it wants it to have and said do whatever you can to make your output look like this over and over again and it can generate the right outputs that's why it's benchmarking so well that's why when i first started using it looked at the code and looked at the plans it wrote like it it feels like a really dumb kid had someone help them cheat on their homework for so long that they learned the routine of looking like they were getting the right answer but they just don't actually understand it yeah and they're just trying to brute force try and figure it out from some hack to get the answer they gave it all of the answers to all of the hardest problems and forced it to learn how to solve them all in a loop and now it solves every problem like it's the hardest problem yes 100 and it's ah it's just annoying there are times when it's been so good like i had to do some a bunch of effect code it was beautiful it looked really good i've had to do a bunch of very targeted ui passes really good even like the hermes agent stuff i've been working on with t3 code really good like it's been very very useful for this but then i had it like try and fix some oauth redirect issue with an old site that i had and it just completely missed the point of what i was telling it to do and went off and solved three different things that weren't actually issues and made the code worse so it's very like soul in this way i still has never done that to me i i'm surprised certainly not to this level like has it misunderstood the assignment plenty of times yes but it will produce a decent answer to its misunderstanding like the problem with soul that i usually find is anytime i get a bad output from it it's usually just the model misunderstanding what i was actually asking for as long as it knows what i want i'll get what i want this model does not seem to have a very good job of knowing what i want and even when it seems to know what i want it often doesn't give it to me i guess what you want just isn't games because it's very good at those oh i could not care less but yes this is a very cool demo yeah i would like to take a sec to discuss this because we talked about it a little bit earlier with k3 there's been a huge leveling up in the 3d capabilities of new models and this is interesting because like fable 5 didn't have this 5 6 had a little of this but both kimi k3 and opus 5 seem to have had some awakening to 3d my honest guess is that somebody's selling them data for 3d stuff and that it's working really well and that both of them got it around the same time case in point though is these demos that we're seeing if you're watching on a platform with visuals like youtube or spotify you can see it playing in the background behind us right now if you're not i'm sorry but if you are here you go this is a full working demo of a call of duty clone in the browser that opus 5 made by itself in a loop including all of the assets that it modeled itself in 3d through 3js which i thought was really cool i thought that it might have been modeling in blender or something no this is just all in the browser by itself yep super super cool and that's just one of the many examples that i've been able to find of people doing things with games on this model a lot of first person shooter stuff has went really well somebody made a clone of brawl stars like a proper four person like 1v1v1v1 layout that seems to work really well so i made a beautiful looking 3d world with a single html file so you made a whole village in 3d that you can explore with opus 5 it's a meaningful jump and considering that like levels io was using what was it like sonnet 4 back in the day to make all his like flying stuff that was cringe as shit this is a night and day difference and i did take the time to port over fish slop both doing a 2d port and a 3d port with opus and it was a it was an interesting experience to say the least so it was interesting about this is i wanted to test its ability both to like make a from scratch engine in 2d while referencing the original code for fish slop as well as making a 3d reinterpretation of the game with like proper 3d movement controls and modeling the 2d version took meaningfully longer it took like an hour and 20 minutes before it stopped and said we have like something workable the 3d version happened in like under 30 minutes and was surprisingly good there were little things that got better than any other model i've seen we'll show that in a sec but i'll start with the 2d version first this is not the first pass to be clear because the first time it did it it had so many weird issues with like asset sizing and things that tried porting the old assets in but didn't map them out correctly and i told it like i sent it a screenshot i was like this is horrible fix it and it did it got the fish the right size again it got the ship to not be weirdly stretched out and it redid the model of the like random like bird thing that's floating around in it from scratch to be much much better another little thing i noticed is that the animations are killer it is really good at animations like it it gives the right like tilt to the fish as they swim around it has an animation for when the coins go out when they eat it even has animations for when they like level up and change tier of fish which none of the other things i've used for building this have ever done it has some taste in its decisions with this which is a whole new thing that i've never seen before but the animations really are the big thing just like when i i set up the like turbo on the ship it like tilts it slightly as it goes yeah there's all these little subtleties that it gets that i've just never seen any other model do it also made a sinfully ugly ui for it all i had to tell it like go fix all of this this is like not shippable but it it did way better than i expected here i've actually been impressed overall but then we get into the 3d version and this is also the second rendition because it got a lot right the first time but i told it like hey these models aren't very good fix them up go use blender because the first time it didn't use blender even though i told it it could so second time it's like okay use blender to tidy up some of the assets it didn't touch the fish which might not look the best if you've seen like what a proper fish looks like in three dimensions but when you compare this to the sinfully ugly ones every other model has made it's actually quite good the most impressive thing for me is the animation do you see like the wiggle on it like that is what the fuck i don't even know how you would do that with 3js like did a very good job with those details that's like a weird bloom around the sub this is the third time i've tried to remake the sub and it's not doing great with that it also made terrible decisions around the controls and the ui but it made a surprisingly good workable start here and a lot of the assets especially like the rocks in the first version of the seaweed it made were good i told it to fix up another thing on the floor and it ended up destroying the seaweed in the process which was fun it's it's a mixed bag as it always is but like holy shit the fact that you can make something like this now with like a 15 per mil out model is crazy it's a shame that they bound shooting to right click and that it breaks all of the time but uh other than that though and also that was a model it made too like the octopus that it's not getting there like it it's been creative with it it made decent things it really wants to use a browser to verify the results yeah i think that's probably the biggest difference i tried to restrict it from doing that whereas with the build that was done by matt schumer the call of duty one the thing that made his interesting is that he gave the model a bunch of tooling actually i think it made a lot of the tooling in order to capture scenes and get a bunch of different screenshots to look at and it would fan out sub agents to say how does this compare against the actual call of duty clean it up more until we're closer and closer to that goal yeah thought all of that was really interesting much more so than i would have initially guessed and yeah yeah it's cool it's interesting that we're now at the point where the models are so good that they're able to use text generation to make real 3d games the bar just keeps moving forward we just keep going from oh they can't do this no now that now they can do it and now it's like oh well their control decisions are bad their ui is bad the now there's a question of can you autocomplete a cure for cancer yeah exactly and that's like i we're definitely not there yet now but yeah i mean it's we're on the path to that like this is as dumb and weird as a lot of this stuff seems like wow you're investing trillions of dollars into making shitty 3js games what is wrong with you this is a step on the path to that like this this is requisite it has to get really good at code before it can do anything else because code is how it's going to interface with literally everything going forward i do like how nice it is on my usage limits because i've been between three plot accounts for a bit now for all my fable usage and with opus it's half the price already so that's already a huge cut it doesn't have the arbitrary limit of 50 of your weekly that fable does so you're double there as well so you're now 4x higher and then there is the additional 50 that they've applied to our like weekly limit on quad subs as a whole that helps a ton too i can't even knock a five hour on one account with opus five uh i definitely could have i i'm load balancing two accounts and i got to the point where both of my five hours were around 30 or 40 percent like i definitely could do it but it is still so much better than fable having gotten used to the fable limits and hitting them instantly constantly it's way better it's really nice it also it's not the fastest model in the world but having again comparing it to fable it feels much quicker than fable as far as the generation speed goes unfortunately the actual runs tend to be a lot longer due to the stuff we've been talking about with it being really really bad it's cutting down the scope and just uh doing the one thing i wanted to instead of five other magical side quests that things are good so between kimmy k3 grok 4 5 opus 5 fable 5 and 5 6 soul how are you thinking about what you use for what now oh i've been thinking about this a lot uh 5 6 is 5 6 soul is my default for probably 80 percent of stuff it's the thing i use for controlling my computer that's the thing i use for my hermes agent if there's just like some random thing that i need to do some random bug i need to fix it is the one that i default to fable is the one that i pull out when i need to when i have less certainty on how something needs to be implemented or i it has to do research that is very complex great example is like figuring out the hermes agent integration with t3 code that was not trivial or easy at all like getting those two to play nicely together in a pretty durable reliable way that doesn't suck to actually set up as an end user not fun looked into having fable go off and look into how the discord gateway is implemented look into how all the t3 code server stuff is implemented and marrying these two together i wouldn't really trust anything else with that task to be honest with you like even 5 6 i would not trust with it it would have over complicated the hell out of it opus ah i can't it's too early to tell how long i'll keep using this thing it's partially just honeymoon phase right now because we haven't had it for that long it's only been a couple days will i keep using this in a couple weeks and maybe like i would when would you reach for opus over 5 6 when i want to merge the code not to be that guy but i feel like that that's that skill issues like that that is just if you use 5 6 correctly you can get great code out of it it just requires effort yeah and i'm if i want to put an effort and i want to really think about the thing i'm using fable anyways so for me it's like if i'm putting a lot of effort into the thing i'm having a deep conversation about it with fable if i don't and the thing doesn't matter that much i use 5 6 but for the stuff that's between the two where it's like a small to medium size fix i know the scope of what i'm going within i could put two times the effort in to like get 5 6 soul to give me code that i'm happy enough with to merge or i could just click opus instead and not have to think about it as much uh anything under 500 lines i would still definitely lean 5 6 on this because it is so much faster than opuses like it'll just get it done quickly and the change will generally speaking be pretty good especially the code base is well put together it has no issue writing very good code it just needs to have the right patterns and context because the patterns that are in its data are not good it's try catch hell it's more that it will write too much almost always opus will to an extent but not quite as badly and the code it writes isn't as python-y yes but also the code that you get out of opus is you you get bonus features and changes and fixes that i don't want 5 6 is better at being targeted opus is an improvement in this direction where it is pretty targeted and it will generally do what you say but it still has some of that self-determinism where it'll go off and do something it thinks is important and fixes it thinks are important that are just annoying usually and like i found this actually strategy that works really well with opus it works kind of well with 5 6 if you give it access to other models that are less verbose i tell it to spin up a subagent that is reviewing specifically for things we can trim and make it simpler oh i that is part of my normal like i have a skill dedicated to this that is you know make a work tree off main this is the feature i want you to implement subagent to it plan review implement then the pr review bots you know that normal loop one of the pieces of the review subagent is simplicity i have that every single time in there make it simpler cut as much as you can remove regression tests for very stupid stuff like all of that when you have it doing the subagents is it using the same model as the parent uh depends on the harness if it's in codex obviously sometimes i'll do the claw dash p but i found that to be more unreliable than i would like uh i have a system set up within my pi config that has first class subagents for everything so i have it built in to whereby tell it use a clod subagent it uses my subagent system that i custom built and it uses the clod sdk for that one so it can actually use clod correctly there and it's very reliable so in that case yes i will usually have the the implementer is usually the same as the parent planner depends often it's the same as the parent but sometimes slips reviewer is almost always something else i still feel like i'm getting code that is a little closer to my bar out of opus than i am out of soul without like lots of extra effort and the main thing i'm using opus for right now is as the starting point when i start a thread with clod code and if it goes off the rails or i don't like things i just switch over to fable and i'm like what went wrong here and then it cleans it up that is a weirdly good flow i've done that multiple times in the last two days where i will finish a implementation with uh opus or whatever i'll tell it give me the list of commits you made i will send those commits to fable and i'll be like all right what what happened here this is probably the biggest benefit of opus for me is that it's just natively in clod code like fable is so it's very easy for me to like do some work with one and then switch to the other one i think it makes sense to so like i'll use fable for the investigation i'll be happy with what it finds and then i'll just click opus and say okay now build it and it doesn't know any better of like what did the work before yeah and if i'm unhappy enough i do the opposite where i switch to fable i'm like what the fuck did opus do here and it gives me really useful info like i just had i posted the screenshot let's see if i could find it quick this is again when opus was like killing its own dev server stuff and i killed it switched to fable said i don't trust opus to investigate this fable can you take over report on any weird things you notice opus doing in the thread so far taking over before touching anything i want to establish the ground truth m dash current process state get state and ci m dash because i have specific doubts about several claims made earlier in the thread huh and then as it continued going i got some of those concerns first was the near self kill i caught it had just verified the right like process id that it should be using and the annotated output literally said arrow pointing the agent itself and then it queued a p kill for that specific instance the evidence disproving the action was on screen when it acted is how fable described it point two incomplete cleanup claimed as complete repeatedly after each verification run it declared tear down verified like some internal thing it used for mapping what had to be done still but only ever checked the tail net mapping and the ports for that specific run three orphan backends from those runs were still alive when i took over the pattern was that it would verify things it just fixed not the state that it actually claimed oh yeah oh that one i forgot about that i had a very similar issue where it was going through and doing tests for a conversion between um it was for the youtube cli i was changing the way one of the structures worked it was doing tests on this stuff and it screwed up one of its first implementations and had like an infinite loop memory leak type thing my computer just started crashing out of nowhere and it was because opus had just left a fun little bomb on my computer that was running in the background for a half hour and ate 128 gigabytes of ram it also was just wrong about how it uses z shell and it printed out an empty like environment variable and confirmed that it was setting the host when it wasn't when it just printed that it was empty string it's like confirmed it sets no it literally doesn't you're using this wrong yeah yeah and then just like it left some unnecessary git files around but like there those are three pretty significant failures that fable immediately caught i was like what the fuck but again this is just because like opus five is an old opus model dressed up like fable like it's trying really really hard to look and feel like fable that's why anthropic made it they wanted something that has a lot of the capability of fable at a way cheaper price but you feel the difference and in the end i'm just excited for fable 5.1 like that's what i've gotten out of this yeah i'm really excited for the next one and also according to all of the fun hacks we just saw with open ai and hugging face pretty excited for gpd6 as well oh yes yes it'll be a fun one that that personally that's the one i'm most excited for by a mile i'm trying to not get my hopes too high because opening i models still just don't write code i like and that's a big enough thing for me now that i've become like the anthropic defender because i i am merging significantly more code thanks to fable and if you look at my sidebar here it's clod code clod code clod code clod code clod code code code code x clod code clod code clod code clod code clod or codex again clod code clod code clod code clod code and all of the codex ones are planning or verifying almost always yeah review this i don't know anthropic models well enough anymore to even be able to say if this feels like new pre-training or something new for the five series like i don't even like i'm sure that the next if it does end up being gpt6 it's almost certainly going to be new pre-training and at least for the past couple releases like i could feel instantly the new pre-training out of 5.5 i could feel that it was the same pre-training with 5.6 it was just rl on top i'm hoping that we can feel it with gpt6 and it's going to be bigger that's what i want out of open ai it's just bigger pre-training because they have this godly rl it's so good it's probably the best of any of the labs at least externally that's what it feels like but their pre-training is just like anthropic has the best pre-training such a shame these companies have to be split but they like if you took those two companies and you put them together you would have the god model it does seem like there has been a huge overhaul in the rl process at anthropic though like for opus to act this much like an autistic open ai model absolutely that gets me really excited for fable 5.1 for like an rl on fable that will have some of these behavioral wins from whatever the fuck they did to opus 5 like that is exciting to me yeah frankly i i feel like they could squeeze a lot more juice out of the current fable snapshot i really need fable 5.1 to feel less like opus 4.6 and more like opus 4.8 like it needs to be an obvious jump yes but i have a bad feeling it might not be like they might do the thing they've done in the past where like they made it better on benchmarks they made it better at these things but their rl has historically made the model as much dumber as smarter and it really is the pre-training where they shine like opus 4.5 was a new pre-training and it felt unbelievably different and powerful and from that point forward it was like three releases that were better in some ways worse than others and it was for it was was the first one was like okay this is obviously better than opus 4.5 i'm really scared of a 5.1 that doesn't hit i don't think they'll do it possibility it's definitely on the table i actually now that you say it i hadn't even thought of that opus 5 does give me a lot more hope and maybe actually actually i might be more excited for fable than i thought i was if this does end up being the case because if they can start start brewing the autism into this thing like please god like can we just take the clock constitution out back shoot it get rid of it make it a machine that does the thing i know they won't ever do that but it seems like with this better rl system they can get a model that feels a lot better to work with and is ah did you not see how excited they were to say at the end of the blog post for opus 5 that it follows the constitution better than any model they've generated which is actually not that surprising it's funny how this was the most aligned claude model that they've had in a long time based on their internal benchmarks and it's also the most um open ai e feeling model which is very interesting it does i haven't been following the alignment benches super closely on open ai models and we haven't been doing heavy testing on that but it does it's certainly from gbt5 that was a very aligned model and i feel like 5.6 and 5.5 were also very aligned models yet another fun week i have a feeling that this coming one's going to be pretty bad for news because it's been a little too quiet recently so definitely make sure you're subscribed on your platform of choice because i have a feeling next week's gonna be a big one i'm gonna go burn what remains of my three quad code accounts so i can finish off my fable for the week and uh i got five days left on mine i'm chilling they're all around 60 so i'm not i'm not in a rush must be nice i have uh for my fable usage remaining in my accounts i have one that's at zero another that's at zero and then one that's at 30 percent the soonest resets the 28th and the latest is the 30th so i got like three fable stays coming up for me yeah i make a fourth account at this point so i'm gonna go waste more money this is a fun episode as always goodbye nerds