← Back to search

ThursdAI - Oct 8 - OpenAI drops 722 math papers, Haiku 5.5 hits 10 cents & more

ThursdAI - The top AI news from the past week · 2026-10-09 · 122 min
relevance 65 18660 words Episode page ↗ Audio ↗
Show full episode description
Hey %%first_name%%, another banger AI week, let me catch you up! We say this often but this week we all felt the acceleration! Just look at one Tuesday. OpenAI dropped 722 math manuscripts. Meta( With Stripe, Shopify and Walmart) announced a new agent protocol. OpenAI opened up the Decisions API. Mistral came back with Le Chonk (Mistral 4), Claude moved into Google Docs, and we at CoreWeave shipped RL Rollouts. That was ONE day. Then the rest of the week happened 😂 Haiku 5.5, D1, and tons more! I had 48 topics on my list, so I asked Claude to build me a Tinder for AI news: I swipe on each story, and it stack-ranks what makes the show this week. I hope we did good (lmk in comments if we missed a major news story) With me: LDJ, Peter, Yam, Nisten and friend of the pod Maxime Labonne from Liquid AI joined us to talk decision models. Let’s dive in, all links at the end as always! OpenAI solves Math!? OpenAI drops 722 math manuscripts from a model nobody can use ( X , GitHub , Blog , Fable’s tally ) Remember when ONE Navier-Stokes result was the whole show? On Tuesday night, OpenAI quietly pushed 722 math manuscripts to GitHub, grouped into 372 families of results. No hype video, no exploding-head emojis, just a very thin blog post. My favorite new term for this is a “slop grenade”: somebody throws a mountain of output at you, and now you have to shovel through it. An internal model nobody outside OpenAI can use was pointed at about 4,000 open problems, at about 3 hours of ChatGPT Pro-level thinking per result. Will Depue asked Fable to measure the drop in Navier-Stokes units. The answer: roughly 5. Roughly 5 “Navier Stokes” size solutions dropped all at once! The math is so advanced that a mathematician in one family of results often can’t follow the family next door without an LLM explaining it. I find that fascinating and scary at the same time. Peter’s tried this before! For weeks he threw GPT-6, Fable and Opus at one problem, Hadwiger-Nelson. He thinks he burned 300 to 400 BILLION tokens, and in his words, “I did not discover a single bloody thing.” And this new OpenAI’s model spent about 3 hours on it and narrowed the bounds from 5-to-7 down to 6-or-7 😅 Then LDJ dropped a stat that I made him repeat slowly. Fable and Astra had put together a list of the 500 most important open problems in math ever. On Tuesday, OpenAI dropped solutions to 92 of them. There’s also a list of the 100 most significant problems from the last 12 months, and over 80% of those got solutions in this drop. Absolutely insane, folks. Yam’s reaction was a very loud “F***ing go.” Yam’s favorite is the Riemann one. It doesn’t prove the hypothesis, it bounds a region for the zeros, which “was never done before, and many, many, many people have tried.” And to the “it’s just brute force” crowd, Yam says: “Let’s brute force everything.” Yes please, room temperature superconductors next! LDJ’s mathematician friends think that on some of these, the model used fewer tokens and less time than a human would. So who’s brute forcing whom? 🤔 The missing crypto results (speculation!) If you hold any crypto, this part is for you. The manuscripts are numbered, and some numbers are missing. There’s a #44 and a #46, but no #45, and LDJ counted 4 or 5 gaps like that. He also looked at which topics made it in, and cryptography is almost absent. So here’s the theory going around. Maybe OpenAI found something big in cryptography and held it back, either by its own choice or because someone asked them to. LDJ said it himself: it sounds conspiratorial. But the US government really can stop cryptography research from being published on national security grounds. To be clear, this is SPECULATION. Nobody outside OpenAI knows what’s in #45. But Bitcoin’s security rests on elliptic curve math. Imagine a paper that shows a way into the 25,000 old Satoshi wallets, each holding 50 BTC. Not great for the price, not great for the world, and not great for encryption in general. “Are you saying your field is useless?” Not everyone is celebrating. Not every result is verified in Lean, and nobody outside OpenAI can reproduce any of it. Kevin Buzzard (thanks Ksenia from Turing Post) says many mathematicians are going through the stages of grief. I get it. Imagine spending decades on one problem and watching 3 hours of compute knock it down. When I wrote code in the 2000s, I didn’t consider it my life’s work. For a lot of mathematicians, that one problem IS their life’s work. Then came a letter from the Association for Human Mathematics: “Mathematicians d
✨ Episode Outline — click any point to jump to it in the episode
Problem solved
Recapping an overloaded AI news week: OpenAI's 722 math manuscripts, Claude Haiku 5.5 pricing, GPT-6 for all ChatGPT users, and new open models like Reflection's Beam.
Benefits
  • OpenAI's model narrowed open math problem bounds in ~3 hours on ChatGPT Pro
  • Claude Haiku 5.5 drops to 10 cents per million input tokens, 10x cheaper than Haiku 4.5
  • Sonnet cache reads halved in price a week after Sonnet 5.5
  • GPT-6 with intelligent UI rolled out to all free ChatGPT users
  • Claude now edits Google Docs natively instead of via computer use
Use cases
  • Peter Gostev spent ~300-400 billion tokens over weeks on a graph-coloring problem with no result; OpenAI's model moved a bound in ~3 hours
  • Alex Volkov had Claude build a 'Tinder for AI news' to swipe through ~48 weekly topics
  • ThursdAI's AI producer appends weekly news to a Google Doc, now native via Claude for Google Workspace
  • Claude Haiku 5.5 scores 72% on OSWorld, beating GPT-6 Luna on computer use and coding
KPIs / results
  • 722 math manuscripts released by OpenAI; 92 of 500 open problems solved
  • Claude Haiku 5.5: $0.10/M input tokens vs Haiku 4.5 at $1/M
  • Haiku 5.5: 72% on OSWorld
  • Reflection AI Beam: 501B-parameter open model trained from scratch
Tools / build
0:00 / 0:00
[SPEAKER_04] And the point is that I was spending literally hundreds of billions of tokens. I'm not kidding, I probably spent, I don't know, three, four hundred billion. I don't know, I need to go back and check. And I literally had my Linux box running for weeks and weeks and it was filling up with all of the solutions and so on. But the TLGIs, that was a complete waste of time. I did not discover a single bloody thing. [SPEAKER_04] So these guys did it in like three hours. I couldn't do it in months with Astra and 5.6 and Fable and others. [SPEAKER_08] So Fable and Astra had put together a set of 500 most important open math problems in the world. And 92 of those just got solved on Tuesday by OpenAI. [SPEAKER_04] Yeah, so we have some news. We raised 200 million. That's a lot of money at 3.1 billion valuation. [SPEAKER_02] So basically his personal CFO just went public on Slack and told everybody his bank account. [SPEAKER_00] Yes, decision models do not output any tokens. They just take a decision from a predefined set of answers. [SPEAKER_06] So there are a hundred thousand agents going on in this city. I don't know how it's still holding together. But yeah, it's a full emulation of the city of Toronto. [SPEAKER_02] Hello and welcome everyone to Thursday Eye for October 8th. My name is Alex Volkov. I'm an AI evangelist with CoreWeave. And I'm the host of Thursday Eye. As every week for the past three and a half years, we're here to give you a recap of the most important news from this week. [SPEAKER_02] And I'll add my co-host here in a second. But I just want to tell you this job has been getting harder and harder, especially in weeks like this. This was a hell of a Tuesday. We'll pull up the tweet and we'll talk about it. I, first of all, want to mention that everybody who's in comments section, shout out. There was a fairly active discussion about Argon. [SPEAKER_02] And folks can't wait to see Argon from Gemini, which we mentioned just briefly last week. I'm very happy to be back in the home studio, in my home turf, after a huge show last week from the Moscone Center live at CoreWeave Fully Connected and Dev Day. Can you believe the opening at Dev Day was just a week ago? So much has dropped since. [SPEAKER_02] Let me add LDJ to the stage. LDJ, welcome. Space Cat, how are you? [SPEAKER_08] Okay, so it's amazing. Past seven days, even just the past couple days has been crazy with the math announcements. But yeah. [SPEAKER_02] Tuesday was just awful. I need to go and pull up my tweet over there because I don't usually do this like a quick recaps, but I just had to. It was quite insane. [SPEAKER_08] A couple of American open models too. Reflection, which has kind of been in semi-south for a while. They just announced their first model, which they plan on releasing open weights soon too, although it's not quite open weights yet. [SPEAKER_02] I think Beam, yeah? [SPEAKER_08] Mm-hmm, exactly. [SPEAKER_02] Yes, Reflection announced Beam. We're going to mention this later on the show. Actually, you have access. So shout out to the Beam folks for getting us access, but I didn't have time to go and play with it yet. But I just want to highlight this like one day in the middle of this week. Open AI dropped 722 math manuscripts covering, I think, five Navier-Stokesworth problems. [SPEAKER_02] I think this was the calculation that somebody did, which was still, you know, we're going to talk about this. We're still trying to figure out what was this thing that Open AI just brought to us and mathematicians are trying to get grips with it. Meta, Stripe, and Shopify announced. This is a single day. This is a single Tuesday of this week, folks. Meta, Stripe, and Shopify announced the agent protocol or assistant protocol or whatever you want to call it. [SPEAKER_02] So Open AI finally gave us access to the Decisions API, which is their competitor to Jeb. And Mistral came out with Mistral 4 with Lechonk. And Google dropped embeddings Gemma. Brad Adcock, the guy who does figure robots, got a new assistant in the arena of assistants. Right? So there's now Hark Pro, which is, you know, a lot of people signed up for. [SPEAKER_02] And Perplexity opened waited a Open Jeb decider. Update. This is an update. They already dropped it before. Claude now does Google Docs. We at Core Web released RL rollouts. This is a single day. It was really hard to catch up with all this. It was just this Tuesday. But obviously, there has been more stuff happening. Welcome, Peter Gostev, on the stage. Welcome to Thursday. I... [SPEAKER_02] There's a huge, huge amount of stuff to cover. So I think... Yeah, let's do a little bit of banter, folks. What is the highlight? Let's start with LDJ. You were here first. What is the highlight for you from this week? If you had to pick one. [SPEAKER_08] Yeah. Huh. I'm sure Nistan and Peter and others are going to probably mention the math part. So I'm going to say Reflection AI is Beam. So it's pretty competitive in terms of... It's definitely, I would say, second, third, fourth-ish place in Open models when you look at raw benchmark scores. But it seems like it might even be just first place when you look at reasoning efficiency compared to the top models like GLM and others. [SPEAKER_08] So we'll look into that more later. And that might be my top one. [SPEAKER_02] Peter, how about you? [SPEAKER_04] Oh, yeah. The math stuff was something else. I know... [SPEAKER_02] It took over your timeline a lot. Like, I see you commenting on this. Tell us what's going on there. [SPEAKER_04] Yeah. So there's been a kind of... I don't know. Rumor, I guess, they kind of pre-announced it that they have a bunch of math problems solved. I don't know. Was that the rumor or was that just announcement? I can't remember what was publicly known or not. But... [SPEAKER_08] At first, a rumor. [SPEAKER_04] Yeah. Yeah. Okay. So OpenAI with their new Bell model, right? They solved the Neves Stokes problem, the Millionaire Price problem. And then there was a lot of chatter about that. They're just sitting in a war chest of hundreds of problems solved. And now they released it. And they just did it in a pretty low-key way. Not like a hype marketing video with exposing head emojis. [SPEAKER_02] The top post was just a few lines of code. And then GitHub was just an insane amount of... You know my favorite term for this now? Slop grenade? When somebody throws at you like an output or a pull request of Claude and it has like a bunch of lines, this is now a slab grenade. So OpenAI basically like slab grenades for people to just like shovel and try to figure out what this is. [SPEAKER_04] Yeah. Maybe my personal reflection is that I didn't really talk about this much, but I was with GPT-5-6 or GPT-6. I picked one problem and I thought, let me just throw all of the tokens I can get my hands on and just see what I can do with it. And I was also trying with Fable and Opus and so on. But I was running GPT models mostly just because they were doing a lot of resets and things like that. And there was one problem that I was trying with Hardwing Nelson. I'm not sure how you pronounce exactly. But it was like graph coloring kind of problem. [SPEAKER_04] So it's easy to grasp. Obviously, impossibly hard to solve. And the point is that I was spending literally hundreds of billions of tokens. I'm not kidding. I probably spent, I don't know, 300, 400 billion. I don't know. I need to go back and check. And I literally had my Linux box running for weeks and weeks. And it was filling up with all of the solutions and so on. But the TLGI, that was a complete waste of time. I did not discover a single bloody thing. There was nothing, no use. There was an interesting mathematical result. [SPEAKER_04] Nothing, right? And then these guys with their new greatest model, they just run it for three hours. They said on average for three hours on the Charge GPT Pro, which I think in practice what that means is that it's like some sub-agents. I think we don't know how many, but not like hundreds. It's like 10 maybe. Or four or 10, something like that. And it didn't solve it completely, but it changed one of the bounds. The idea is that you can narrow the bounds. [SPEAKER_04] And I think before it was something like five, six, or seven, and then narrowed it down to six or seven, which is a big deal, right? Obviously, a lot of context required, but it's a big deal, right? And I was speaking to Astra about my progress, my results. And Astra said, oh, we are two mathematical breakers away from what they did. So these guys did it in like three hours. I couldn't do it in months with Astra and five, six, and Fable and others. [SPEAKER_02] It's just nothing. There's a few rumors that I want to bring to your attention there. But I think it's time for the official start. Folks, for tuning in, welcome to Thursday Eye. One of the coolest things that we get to do is talk to you as well. So please drop in the comments, what was the highlight of this AI week to you? Obviously, it's nearly impossible to cover everything. We're trying to make Thursday Eye the most agentic forward show. We have an AI producer. [SPEAKER_02] Producer, please say hi in the banner below if and when you hear this. And we also have obviously a bunch of research happening to make sure that we're not missing anything. This week, I think I had 48 topics to bring you of importance, etc. 48 or 50 or something. I actually asked Claude to build me like a Tinder for AI news where I swipe left and right if I think that this should make the show. [SPEAKER_02] And then we'll stack random based on my preference and the stuff that we talk about. Because it's just impossible to bring everything. But we're going to try to bring in the most important news. So once producer catches up, I think it's time for a cold open. Alrighty, folks. Let's do this. Folks, this week OpenAI dropped 722 math problems. Papers written by a model nobody outside OpenAI has touched yet. [SPEAKER_02] And the mathematicians are split between thrilled and asking for receipts. And a lot of those came with receipts. Welcome to Thursday Eye for October 8th. This is Alex Volkov. I'm an AI evangelist with CoreWeave. And with me, LDJ and Peter Goste from Arena. This was an insane week. We say this often, but this was insane fucking week. After OpenAI's math manuscripts, 722 of them to be precise. [SPEAKER_02] Some people call this a slob grenade. Many mathematicians are split between. This is the most exciting time ever. We're breaking through the boundaries of mathematics. And we should... This is not how mathematics should happen. This is kind of the split. And Peter, I think, has been monitoring this and would love to chat with you more. This will be our item number one. And that's just the opener, right? We'll start at TLDR, obviously, with OpenAI. [SPEAKER_02] 722 math manuscripts from a model that nobody opened outside of OpenAI can use. Somebody used Fable to put this release in terms of how many Navier Stokes moments in one drop we got. And I think it's around five or so orders of magnitude of Navier Stokes releases. If you guys remember, Navier Stokes, a millennium price problem, was solved by OpenAI just recently. [SPEAKER_02] So OpenAI just dropped 722. So I think two of these, many including Lean confirmations. We're going to talk more in depth about this. Next up is Claude Haiku 5.5. It's a 10 cents per million input tokens. And Anthropic say that it beats GPT-6 Luna on computer use and coding. GPT, remember, was really, really good at computer use. Haiku seems to be better with a 72% on OS World. [SPEAKER_02] And I think also significantly faster. I think on artificial analysis, Claude Haiku 5.5 is one of the fastest models right now. And it's incredibly cheap. Incredibly cheap. Haiku 4.5, I don't know if you noticed this, was $1 per million tokens of input. And Haiku 5.5, which is significantly better, is 10 cents. [SPEAKER_02] Just to give you a sense of how crazy the world we're living in. Also, a small thing that they released together with Haiku. Sonnet cash reads. So the tokens, you know, after you send them once. Sonnet cash reads are now halved by price. A week after Sonnet 5.5 was released. So that's also quite insane from this week. Let's see what else. OpenAI rolls out GPT-6 with intelligent UI to everyone. [SPEAKER_02] All of the free users on ChatGPT, all of them are getting GPT-6. It's not Luna. It's not Terra is dead. So no Terra anymore. It's not Sol. It's not Astra. It's just GPT-6. There's no name in this. So all the free users can get it today. But also the cool thing, the intelligent UI is really, really cool. They rebuilt ChatGPT on top of it. And you basically get buttons and switches and toggles and graphs, etc. [SPEAKER_02] in your chat output very fast. It's really cool to see. It's like little working apps for your answer. There's also some threads from folks at OpenAI who say the model was now trained to do it while it thinks of an answer. So it starts to build you, hey, this answer would probably get benefit from these and these switches, these and these things. And then it still works in the background. So I think that's pretty cool. [SPEAKER_02] Cloud for Google Workspaces is finally here. And this is like a small thing, but it's a huge thing for many people. I'll give you an example. Our cloud producer every week used to post a Google Doc and then a new piece of news would come. And I would ask you, hey, can you append to the Google Doc? And it would start opening computers, etc. It wasn't native. So now that it's native in Google Docs, I think it's a very big deal. And it also asks me for every edit. [SPEAKER_02] But okay, let's go to the open frontier. As LDJ mentioned, we now have another US lab. Well, we knew about Reflection a while. And Reflection AI announced Beam. It's a 501 billion parameter open model trained from scratch in the US with Apache 2 weights coming this month. And shout out to Reflection for this awesome announcement. Many, many people got very excited. [SPEAKER_02] An open MOE that beats on efficiency over raw score. So it's more efficient versus just like benchmarks. We can't wait to play with this when the weights drop. Also, Mistral came back with leaning in to Le Chaton Fat. If you guys remember, this was the meme. They called this one Le Chonk. Mistral 4 is a trillion parameter with the weights coming at the end of October. This is now a pattern that I do want to cover on the show. [SPEAKER_02] Like folks announce open, but they're not released open. It's a far, far way that we moved from Mistral announcing their latest model with just a Torrent link. Because people do be announcing. But Mistral large ties with GPT-6 Luna on official analysis at a much higher cost per task. So it's a very interesting thing. This is kind of the comparison we have here, obviously for European folks. [SPEAKER_02] Speaking of European folks, we're missing Wolfram this week because Wolfram is on a well-deserved vacation. But he got super excited about Calibri. This is from Aleph Alpha, another European lab. If you guys remember, there's a European lab called Aleph Alpha. Peter may know them as well. It's a German-English MOE with 78 billion parameters. Wolfram sent me his notes on this. It can run on a single H200 and also Apache 2. [SPEAKER_02] So shout out to Aleph Alpha for coming back with the model. I thought they stopped doing pre-training and post-training. It just like went for models. And for the very, very, let's say, AI internals-oriented folks, we used to cover embedding models. We had Bo Wang on the show before he joined Puprexity. He dropped one of these models. So shout out to Bo. So two open embedding models dropped in one week. [SPEAKER_02] Google's embedding Gemma drops embedding two. Text images and video and audio embeddings in one space that runs on the phone. Two embedding models this week. And Puprexity late interaction models. Search PDFs as images and are very fast with PPLX Embed V2 late interaction models. So a lot of goodies this week. [SPEAKER_02] As we move to agents and assistants and a bunch of stuff, this I think is a very important thing. So Meta and Sierra and Shopify and Stripe and Walmart all on board of this protocol for personal agent protocol. Very interesting to see if this is the new kind of MCP because MCP had its ups and downs, but now MCP is everywhere. Basically MCP took over the world because of these agents. [SPEAKER_02] They need to talk to a bunch of apps. I don't know if you guys caught this. We talked about GrokBot since launch. There's a whole drama happening between Dots creators at OpenAI and GrokBot folks, specifically Potato. It's really funny, but GrokBot has been supercharged this week. After Grok 4.7, the model released and it was... I think I have a button for this. [SPEAKER_02] This is our reaction to Grok 4.7. Not only our reaction, big Ankh space Ankh Elon Musk decided, hey, you know what? Grok the product needs to be more than just Grok the model. Also, Grok is now tapping into Opus 5.5. If you guys remember last week, and I think everywhere else, I've said I want personal intelligence harness, like a proactive personal intelligence, but with Opus' brain. So supposedly now that's what's happening inside GrokBot. I think it's very cool. [SPEAKER_02] Supposedly also only because I haven't seen this in the logs yet. So that didn't arrive to me yet. So we'll see. But also, Elon Musk said it will use the best model for each task, including Cloud Opus 5.5, Suno, and MidJourney, which folks got very surprised because MidJourney doesn't have an API. We'll talk about that. Speaking of personal assistant and also GrokBot, Shane Mack posted this, and I think it's a good warning for you to know. [SPEAKER_02] He's GrokBot, talks to another GrokBot, and builds his personal finance details in a company Slack, of which he's the CEO. He had one GrokBot in charge of his personal finance, giving them an update, and another connected to Slack. And for some reason, they communicated and decided, hey, let's put the dirty laundry of our CEO in Slack in front of everybody to see, including all the statuses and etc. [SPEAKER_02] So this was a very interesting warning sign for personal agents connected to between work and home. And I think could be one of the reasons why they're switching to Opus, because Opus would never. Like, I just know that Opus would never. This is a Grok thing. This is a Grok-like intelligence thing to just like, oh, sure, let's blast it publicly. Okay. We got to talk about news research. Folks, our favorite open source friends for the longest time. [SPEAKER_02] News research. Shout out to just everyone there. All our friends. Technium, Karan. I keep forgetting all the names. They not only launched Hermes Index this week, which is a benchmark that says, hey, Cloud Opus 5.5 is the best model for Hermes right now. They also announced a fundraise, evaluating news research at more than $1 billion. So shout out for our friends to making the unicorn status. [SPEAKER_02] Fundraising, I think $90 million is the latest one round. They also were on stage, both at OpenAI Dev Day last week and at Microsoft event this week, with a deep integration into new hardware. This is incredible. Since we told you about Hermes Agent, the length and the depth of penetration of Hermes Agent from news research is just incredible. So we'll definitely cover the Hermes Index, but we'll definitely also celebrate [SPEAKER_02] the success of our friends of news research. We've been covering news since before they were a company, since before when they were just like a ragtag of folks doing stuff on Discord. So shout out to Karan, Technium, Jeff and Dylan and everybody else at News for this amazing, amazing success. If you are a thinker and you like personal assistance on your hardware, [SPEAKER_02] Nat Friedman posted open source news gadgets. So now you can put your news everywhere in your home. You can put it in every gadget, in every, it's an ESP32 board, if you know what that is. It's like a tiny chip that I used for a bunch of Halloween builds before that I talked about. And it's a very cool thing that they're posting. A bunch of folks are porting their assistants into portable stuff. A new personal assistant emerges from the creator of the figure bots, Brad Edcock. It's called Hark Pro, [SPEAKER_02] also free. And the assistant that clicks for you, they are claiming the best computer use model. I haven't got a link to the verification of the best computer use model, but Hark Pro is also here. And folks in the comments, if you use Hark, please let us know. So for our, this week's Buzz Corner, that probably should be renamed because it's no longer just about weights and biases. CoreWeave opens up free GPU sandboxes. This was the highlight of the event last week for me. [SPEAKER_02] Last week on the show, we had Dirk Filio, who is the PM for CoreWeave sandboxes. And he announced that, hey, you can now reach out to us at CoreWeave. We'll give you GPUs without a salesperson in the middle. I think it's huge. It's absolutely huge. We're advanced. Free GPU sandboxes. I'm going to say this again, super quick. If you scan this, you get free GPU power. If you want more GPU while in testing, this is just going to be free, which is, [SPEAKER_02] I think is incredible. So this is in this week's Buzz. Also have our rollouts. All right. I'm going to read this blog super quick before we get to the news because it's been long. Black Forest Labs, launched Flux 3, 4K images and layouts and up to 10 reference images. Nano Banana 2.1 releases, which is very, very cheap. Only three cents per 1K image. It's pretty good. If you are looking at our [SPEAKER_02] infographics right now, Nano Banana 2.1 is the creator of those infographics. They're pretty good. I'm very impressed. The consistency is there. You have to run it on high reasoning, medium reasoning. The default one is not that great at text. I don't know if you guys saw Tavis. Griffin video avatar fooled. 48% of the people in a live video call. They thought they're speaking to a human. This is quite honestly quite crazy. We're going to play a clip of this [SPEAKER_02] of our friend, Quinn LaCramer, talking to Tavis avatar and saying that, you know, whatever Turing tests for avatars we've passed, long past. Reka AI, we've talked about this company. One model, every modality released a 19 billion parameter model that understands and generates text, understands and generates text, images, video, and robot actions all in one network. That's very interesting. We'll see if we have time to talk about that. [SPEAKER_02] And I think this is almost the last thing, but the rise of the Jeffs, I think is our last category. We're going to talk about it with Liquid AI's head of pre-training, Maxim Labone, a friend of the pod. OpenAI launched decisions API and Klauthor dropped Clef and Ancelot dropped a recipe for creating decision models and Faustino dropped Glide. [SPEAKER_02] There's a bunch of decision models all around basically, so we're going to cover all that jazz in the end of the show with Maxim Labone. Please stay tuned. I think if you are interested in what decision models are and how to run them locally, I think that's going to be a great segment for you. And with that, I think folks, it's time for us to actually go and dive in to Frontier to talk about math because I think a huge thing happened this week. Let's go. [SPEAKER_02] OpenAI released 722 math manuscripts from model they haven't released yet and somebody took and used Fable to count up all of the releases into how many Navier Stokes-like news we got and it's roughly five. Five moments in one drop. If you guys remember the Navier Stokes and how much hype that did, we just got 5x in the amount of OpenAI solved maths, which is quite crazy. On Tuesday night, [SPEAKER_02] OpenAI posted this onto GitHub without any fanfare. Basically, a very, very thin blog post. After Navier Stokes, a lot of mathematicians said that OpenAI is doing very bad things by releasing it such as this. OpenAI said, hey, we're going to create a mathematician advisory board to tell us how to handle the fact that we're basically brute force in mathematics, advanced mathematics, mathematics that no humans has solved for 80 years. and this is after that advisory OpenAI grouped it into 372 families [SPEAKER_02] of results. The internal frontier model got about 4,000 open research problems and OpenAI says each result took an average of three hours on a CHGPT pro-level thinking level. They also released 10 shorter reasoning summaries on things like irrationality, exponent of the pie, the Mahler conjectures and the Koplansky conjecture. And they flagged two results that came out of a different process, a Riemann Zeta Zero Free Region where humans edited the write-up. [SPEAKER_02] So they did have, it's not only slop grenades, they did have human mathematicians actually doing some stuff and the Hodge conjecture results for a special family of cases. Peter, how big is this? How big is this GitHub drop that we're getting from OpenAI with 722 manuscripts for math? [SPEAKER_04] I think it's enormous. It's, we really went from just models being completely hopeless at basic maths to doing this. And the Millennium Prize was a big deal and it was one thing. And obviously it was, you know, lots of agents being thrown at it and it's a little bit controversial here and there. but I think I remember at the time thinking when there was all of this [SPEAKER_04] controversy, people saying, maybe it's stole its insight from the mathematician who was using Chagipity. But reality is like, okay, who knows, right? Who knows what happened? I don't think so, but whatever. But the reality is that it's complete cope for people to think that, oh, well, that was a one-off and actually they still can't do maths and so on. And now we see, right? It's happened. There's, I don't know how much clearer a signal you can get that this is real. And the question [SPEAKER_04] is, you know, there's one scenario where, oh, there were just slow hanging fruits that sort of maybe humans were a little bit too lazy to pick up and that'll be it. But I think just reading what mathematicians are saying, it looks like the bulk of them are impressed and there were a bunch of people who have problems quite a lot. And I saw some of them say, you know what, I was looking at this problem for many years and I never thought there would be an approach like [SPEAKER_04] that that would solve it. So I think it is real, it is not trivial, it's not just finding a bunch of counterexamples. I know, Alex, you said people were asking Fable, what did these results mean? I saw people also doing that in terms of finding how many counterexamples there were. And there were only like 20% of these are counterexamples. So I think there was a scenario of just getting a bunch of uninteresting results, actually just kind of brute forced a bunch of stuff. [SPEAKER_04] But I don't think that's true. I think this is real, real heavy math stuff, which I have no hope of understanding, but it sounds super impressive. [SPEAKER_02] Here's the thing that I have two comments before we go to LDJ. We'd love to hear your thoughts. I know you have a bunch of friends and mathematicians. We'd love to hear you. Last time when Navier stocks was broke, you put the glasses on and just like exploded. Just for context, folks, this is GBT five years ago. Somebody shared it and I do have to add this for context. Five years ago, AI was struggling with this type of grade school math. Cindy's math and science books [SPEAKER_02] weighed two pounds each. Her French books weigh four pounds and her English books weigh three pounds. Her history books weigh twice as much as her English books. AI five years ago struggled with answering a basic logic thing. And now, somebody called it out in comments, it's not brute force in mathematics, but it's just solving things that the best mathematicians in the world couldn't solve for ages. And many [SPEAKER_02] of us would turn to an LLM to try and understand what is it that they solved? Because the math there is so advanced that it's really hard to explain to you how advanced this is, especially if we're not mathematicians. So even mathematicians in their fields, there's what, 300 or so groups that they grouped those results in? Mathematicians in different fields of those groups, they, without AI assistance, not many of them can understand the other results in that result group. like the math, [SPEAKER_02] the drop is so advanced that many advanced mathematicians cannot very easily approach all of these problems and understand what is it that the, like, chat GPT gave, which I think is fascinating and scary at the same time. And, LDJ, I would love to hear from you. Your, your thoughts on, on the drop, your thoughts on how mathematicians react to this. should they be more excited or worried about their job? [SPEAKER_08] I think specifically when it comes to their jobs, I think that can be really unpredictable and it might even depend a lot on what area of mathematics you're in and a lot of different factors. I did want to mention, though, in the example you showed, that was five years ago. Like, I think it's important to remember, even just like two and a half years ago when we had GPT-4-0, which was the best model they had at the time, that was actually GPT-4-0 had just barely come out 2.5 years ago. [SPEAKER_08] That could barely even do when I would give it three by three multiplication problems, like just three-digit number multiplied by a three-digit number. A lot of times it would even just fail at that. And so just like in that span of time we've gotten here, and when people talk about the brute force aspect, there's the reasoning efficiency where they did say on average these took about three hours of a chat GPT-pro computer, or equivalent to chat GPT-pro. [SPEAKER_08] And I would have several friends in mathematics that would confidently say at least for some of these problems, it definitely solved it using less tokens and less time than a human would have to do that same thing. And so arguably in that sense the human is doing even more brute force than the AI is doing. [SPEAKER_02] Yam, thoughts on what happened there? Do you have a favorite manuscript that drafted the game? Yeah, [SPEAKER_01] they went after the Riemann hypothesis. That's insane on its own, just to try. The Riemann hypothesis basically speaks about the zeros of a specific function and where they are. So, this one is hard, by the way. That's a very hard problem, needless to say. They, to the best of my knowledge, and I [SPEAKER_01] didn't have a lot of time to go into this, and just like you're saying, you need to be many years in the field just to wrap up your head about everything that is going into this. About one such problem out of the 722 problems that they dropped. Listen, people spend their entire life on each one of these, the Riemann hypothesis is a big one. They, even just founding a region where the zeros [SPEAKER_01] are, which is what I understand is in this paper, that's huge on its own. You don't need to prove the actual zeros themselves in a specific region, but you can bound them. And it's also huge because it was never done before, and many, many, many people have tried this. I just want to put things where they are. [SPEAKER_01] In perspective? It's not that you can just go to ChatGPT Pro and tell it to solve the Riemann hypothesis, and you're going to get this result. It's important. There is first one thing that is clearly need to be said. There is on the API, there is a parameter called juice that each of the ChatGPT [SPEAKER_01] categories of thinking like high, extra high, correspond to a different level of juice, and Pro is obviously the top. You can up the juice even more. I suspect that it's nothing to take away from this. I'm just explaining that it's not that you just go to ChatGPT and I'll try ChatGPT Pro and let's prove the Riemann hypothesis. It's not that easy, but it's still insanely impressive. [SPEAKER_01] About the brute force, I'm in the camp of yeah, let's brute force. What's the problem? What's the problem with that? Let's brute force. You can call it brute force. No problem. Let's brute force everything. Why not? [SPEAKER_02] Of course, you know, room temperature superconductors. That's the brute force. I want to highlight this one from Xenia Sip from Turing Post, a friend of the show as well. She's like, I'm not a mathematician, so I wanted to understand what yesterday results mean from the math wizards. And she saw this post from Kevin Buzzard that talks about understanding mathematics. And I also do want to talk about reactions as well. Peter, you posted some stuff about the Institute for Advanced Study from Terry Tao. Would love for you to bring that up as well. [SPEAKER_02] I just want to read through this. A person that says he's a PhD student of Richard Taylor in the early 90s and said that whilst mathematicians are now beginning to agree that mathematics is all about human understanding, he's not entirely convinced they will agree on what it means to understand high-level mathematics. Basically, he says that math specific, like advanced math like this, is not about just solving the thing. It's about the human understanding boundary of [SPEAKER_02] solving the thing. Which is a very interesting take on advanced mathematics. It's because, okay, let's say I solved math. How does it help the world specifically in one theorem if humans don't really understand it and can't do anything with it? math. It's a very interesting take about some mathematicians. Some other folks are not having a good time. Obviously, [SPEAKER_02] nearly existential crisis thought process for many folks who've been working on a specific problem for decades. Just to see three hours of compute just decimate this. It's a very interesting analogy to software engineers who after a year or so not only don't write code anymore, don't even read code. This was six months ago maybe the discussion, the ZL continuum that I brought to the world, whether or not [SPEAKER_02] you're reading the code that your agents output or not, most of people now don't even read that. So this is a very good analogy between mathematicians and their work. Although, when I wrote a piece of code back in the 2000s, I didn't consider this my life's work like many mathematicians do. Many mathematicians do with just one problem. LDJ, go ahead and then Peter would love to hear from you. [SPEAKER_08] So there's a few interesting kind of open problem sets and self-problem sets people have put together using Fable and Astra of what are the most important problems. And one of the main examples that has been circulating lately is out of a compiled set of 500 most important open problems put together by Fable and Astra, 92 of those were dropped Tuesday. Say this again slowly. [SPEAKER_08] So Fable and Astra had put together a set of 500 most important open math problems in the world. And 92 of those just got solved on Tuesday by OpenAI. And out of the top 100 in that list, 10 of them got solved on Tuesday. And one last tidbit here. There is a list put together by Fable and Astra of [SPEAKER_08] the top 100 most significant problems in the last 12 months. And over 80% of those just got dropped Tuesday for solutions rather for them. [SPEAKER_02] That's absolutely insane, folks. One last [SPEAKER_08] little thing I'm going to say to make people think about something. If you look at the list of manuscripts that OpenAI released, they're ordered and there is some numbers missing. like it goes from 44 to 46 and 45 is missing. And there's about four or five of these. And a lot of people are suspecting also that if you just look at all the different categories of problems there, at least it looks [SPEAKER_08] like things relating to cryptography seem to be really missing here. So the theory here is maybe they had some significant advancements in cryptography involved here that potentially the U.S. government or various entities or maybe their own just personal policy ended up restricting them from releasing. Because it might sound conspiratorial, but there is actually laws and policies that the U.S. government is actually allowed to prevent [SPEAKER_08] mathematicians from publishing things relating to cryptography for national security reasons. [SPEAKER_02] Yes, not only national security, also the world economy crashing when Bitcoin goes to zero. If and when the superintelligence decides, you know, this was the scare from the quantum thing, right? Quantum cryptography and solving quantum stuff will essentially make Bitcoin not as secure as everybody thought because it relies on a lot of elliptic curve cryptography and that's all math. And yeah, there [SPEAKER_02] is a way for the world's government to say, hey, do not release breakthroughs. Do not release diseases into the world. Those also do not release breakthroughs like that. Astra is going to be rich. [SPEAKER_01] Astra is going [SPEAKER_02] to be rich. It's a good question if Astra can make money out of this. Peter, I do want to bring the stuff that you posted. We'd love to hear from you about the advisory group and the Terry Tao stuff, field mentalists' responses to this because folks from all over the sides of the debate are chiming in here. And you posted about this. We'd love to hear your thoughts. [SPEAKER_04] You want some controversy? I would love some politics, yes. So I think it's a touchy thing because I don't want to maybe came across as kind of jumping on people or something. I hope it didn't. But it's a tricky situation because a lot of mathematicians working hard, they have the established ways of working, and then suddenly someone comes in and just drops a bunch of stuff and your whole world at least changes in some [SPEAKER_04] way, right? It has to change in some way, whether directly or indirectly. So there was a letter from Association for Human Mathematicians which has like 800 members and Terry posted it in his blog post and it said something like that, I think there were some quotes there where they said something like, mathematicians did not ask for this work [SPEAKER_04] mathematicians have a particular vision of progress that is informed by history and field specific considerations. We reject open-eyed assertion that this release advances our subject and so on. And I think that's a kind of interesting perspective which goes back to the question of what is maths for? and I think it looks like a lot of mathematicians, at least part of this group, [SPEAKER_04] they seem to be kind of suggesting well, we need to run our thing, do it in a certain way. But inherently what that implies, it's not to actually solve math problems. And to me that raises an interesting question of are we basically saying that math is useless and we should just leave it to smart people to just work on it. If that's the case, then I can see the intellectual argument for this. But if you flip it another way and say, well, imagine doctors said this, [SPEAKER_04] stop curing diseases, we just need to don't ruin our good thing, we just need to keep going treating people from their headaches and what not, and cancers, and we've got a good thing going. Just imagine that happening. But here's a clear line for utility that I can't imagine any doctor would say that. So if OpenEye, whoever, dropped a bunch of papers saying this is how you solve medicine and heal 500 diseases, [SPEAKER_04] people would be like, yeah, amazing. And I think, to me, this is, I appreciate, well, I'm not sure I can fully appreciate, but I can imagine this is very difficult for people in that field, which I think we need to be sensitive to. But it is kind of, to me, the question is like, well, are you saying that your field is useless? Is there no utility and we should just let you hang out and work on it and not solve any problems? And I don't know, I don't think you can kind of say both. You can't say, oh, it's really [SPEAKER_04] useful, but please don't solve any of it. [SPEAKER_02] Yeah, the police don't solve crowd, I don't understand. Get on board. This is now in the realms of OpenAI and in the year, it's in the hands of pretty much everyone. This is the pace we've been on. It's very clear that this is the pace we've been on. One last comment, we have to move on. We dedicate a lot of time to this math thing and have to move on. [SPEAKER_08] Yeah, I recall about a year or so ago, there was actually something circulating where there was a discussion at a big conference where somebody stood up and said something along the lines of, do we really want to automate a cure for cancer or something? And they were actually having a serious statement and discussion about their own job and livelihood and whether they would really want that. And there was a lot of polarizing reactions there, but I find it really [SPEAKER_08] interesting how at first it seems like the answer should be obvious, but there is seriously people that have strongly differing views there. [SPEAKER_02] I'd rather AI not solve cancer or something. I'd rather risk cancer than AI taking over as it was just like a ridiculous thing, a point in time, I think. Hopefully we'll never get back to that moment. Folks, it's time for us to move on because as exciting as this is, we can talk about this in circles for hours, but there's a lot of stuff that happened this week. Nearly an hour into the show is crazy. Let's run and talk about the frontier for everyone. There's two major things that happened this week. [SPEAKER_02] Let's start with Anthropic launches Claude Haiku 5.5. It's been a year since Anthropic launched the previous Haiku and this is one hell of a model. Let's take a look. Haiku 5.5, tiny price with very big, big score. So we're looking at a model cheaper than ChatGPT Luna, although it can get a little bit more expensive depending on the raising effort and the number of tokens that you send it. Haiku is just priced at 10 cents per million [SPEAKER_02] input tokens for under 100k. This is 10 times as cheaper as last year's Haiku which was at $1. I love Haiku. I used to love Haiku not for like day work but there's a lot of stuff that you want the good Claude intelligence but like a really fast, really quick price. Would love to hear from folks here thoughts on Haiku release. Peter, did it already hit the arena? Nisten, Yam, LDJ, did you [SPEAKER_02] guys use it? [SPEAKER_04] Yeah, it's on the arena. I'm just testing myself. I don't have a score yet but I think it's super exciting because I think the one problem with Anthropic offerings is they didn't really have a lot of good cheaper models that had the best very expensive models but then you by default had to go somewhere else if you wanted to actually have something cheap and that's not ideal and I think it's interesting to see what they managed to do with this. One maybe note to call out is something I'm digging [SPEAKER_04] into as well is that the cash read is so important the price for it and for Haku [SPEAKER_02] it's one cent per million tokens. [SPEAKER_04] Yeah, but it's also the ratio of how much is it to read normally what the input and your cash read is and I think the Sun Tropic has been dropping prices actually I was just literally pulling the data up and Fable 5.1 has the lowest ratio it's like 2.5% the cash read versus the read and I think Haku is still at 10% but it's certainly been improving right [SPEAKER_04] if you pick like GPT 4.1 I know that's long time ago it was 25% the cash read so it's a really important part I want to do a bit more analysis of it but I'm glad that they're dropping prices on cash reads that's for agentic work is the biggest thing [SPEAKER_02] speaking of cash reads another hidden announcement in the Haku announcement that Sonnet's cash reads are cut in half in prices which makes Sonnet is most agent work about 20% cheaper on average because if you think a lot of a lot of the previous stuff should be cash because nothing changes there Haiku is also significantly better at computer use [SPEAKER_02] LDJ would love to hear from you about this from the frontier from just a week ago I small sidestep [SPEAKER_02] I love that we're a weekly show because so much happens in a 10 of week we have time to catch up just a week ago Luna just like the best fast cheap and reasoning model is only 16% on terminal bench Haiku is available everywhere with one cent per million tokens cached which is just absolutely insane anything else remains to say about Haiku besides go use it folks and tell us anybody used it yeah [SPEAKER_08] at the Pareto frontier it looks like this is setting in some ways the new Pareto frontier in terms of accuracy and cost on artificial analysis index on extra high it looks like at the Pareto frontier or near it and competing with a lot of the Chinese models and others but I do think going forward and this is a bit of a bold prediction for me over the next 6-12 months I think we'll see the western frontier labs continuously [SPEAKER_08] getting better on the Pareto frontier and beating Chinese labs even on the most low cost options [SPEAKER_02] this is a very interesting place for open source to be in because a lot of the Chinese specifically DeepSeek on their services promised us very fast and very cheap not necessarily very advanced the frontier labs are going to it and then I remember saying we're going to offer the best price model for every tier of intelligence it looks like Luna has a lot of catching up to do last comment about Haiku [SPEAKER_02] 5.5 and what would use it [SPEAKER_01] for that's the obvious choice for swarms at this point like you want you want to do a lot of things in parallel it was the Luna was the obvious choice yesterday and not yesterday but you get the point that's the obvious choice I mean that you don't get this type of performance for this price on any other model at the point Anthropic is on fire recently like all the [SPEAKER_01] releases are top seriously we didn't speak a lot about Sonnet but Sonnet is good I don't know if you guys tried it but Sonnet is surprisingly good also and everyone online is just confused how come I never burn token I never burn my quota with Ocus I've been hammering Ocus all day long it's they all over X everyone is saying this so [SPEAKER_01] imagine how much you have now for Haiku if you just want to do many [SPEAKER_02] stuff let's finish up with Entropic Cloud in Google Docs and Google Sheets and Slides they finally added this ability to be able to actually work on your documents not just like post via the drive integration so that's great but speaking of cheap intelligence for [SPEAKER_02] is now the default for free accounts this is not talked about this a lot but out of the 1.2 billion weekly active user I think they said last week and Peter please correct me if I'm wrong I think it was like 1.2 billion weekly active much of that is free accounts right like folks who are listening to us probably like when you were in the traffic you maybe canceled the opening I one many of this is on the free accounts and like you could [SPEAKER_02] say the open AI just like uplifted the world's intelligence just now with just like one release with GPT 6 which is incredible so if you are listening to this and you are in the free tier of chat GPT your intelligence has been upgraded not only asking chat GPT to say hey visualize the math problems open AI dropped on github with a nice UI [SPEAKER_02] and it built me inside the editor a mini app you guys see this about time about time great [SPEAKER_07] it [SPEAKER_02] says 719 manuscripts you guys know why because it's updated since I did research open I did pull some manuscripts back so this is like the real time thing and then how much is formally verified they're saying 300 of the manuscripts are formally verified but I'm not going to talk about the manuscripts you can see that this is kind of like a live thing that works and this can be your weekly update from this can be [SPEAKER_02] yeah I think that's great from OpenAI but the intelligence boost for the free folks I think is very interesting anybody already use it anybody feels the difference I'm not on free tiers I don't know I think [SPEAKER_04] what's cool about this and I appreciate that OpenAI still doing this is that we kind of had versions of this and apps and it's nice that they keep bringing that to the free tier and so on so yeah GPT6 is cool as well I think we forget sometimes that we are very anyone who's listening to this you are very not normal person said [SPEAKER_04] with love [SPEAKER_02] everybody who's listening to this is a beautiful person part of our community but normality is not you know your hair dresser is probably not listening to the show yeah [SPEAKER_04] so yeah Alex what you said about yeah it's uplifting the overall intelligence that kind of thing has so much more impact than whatever it is like new model is dropping on the top tier where you need to pay $500 a month like that kind of thing makes a difference to us but not that many people so yeah it's cool to see that you know they still remember it's important [SPEAKER_02] let's also check on Nistan a little bit Nistan you're with us you're alive what's going on you haven't said a word about Haiku or I'm hoping that you're going to tap in when open source comes [SPEAKER_06] I'm running 10 Haiku agents right now that was because I used up like I used up 95 or 97 percent of the quota already until on the max 20 plan but so I had to basically resort to Haiku it's very good yeah yeah yeah I had a rice cooker and I couldn't get the ratios right and then he did a whole bunch of research and found that this is [SPEAKER_02] yeah LDJ comment and we'll move to open source [SPEAKER_08] so I want to ask real quick Niston would you say that it is clearly at parity or clearly above Quen 3.8 27b [SPEAKER_06] oh for Haiku yeah it's above all of it's especially for conversational stuff I haven't tested it technically but anything conversational yeah yeah it's it's really above it kind of feels like Sonnet actually when you talk to it I haven't looked at what code it writes but it's too good it is what it is that's the [SPEAKER_06] assessment [SPEAKER_02] folks use Haiku the whole family is great all of the three brothers Haiku Sonnet and let's talk about the open source [SPEAKER_02] AI let's get it started right open weights and open source AI is our new category let's talk about this open weights when big this week a 501 billion parameter American model a trillion parameter French one and the German one you can run on a single H200 open source very interesting this week reflection AI announced beam 501 billion parameter model [SPEAKER_02] 23 billion parameters active it's trained from scratch on data here in the west and Apache 2 weights are promised this month reflection AI has been training I believe on the SpaceX cluster somebody correct me if I'm wrong but I believe that this is the cool thing about them they're pitching it as the western open frontier model they claim is 80.9% on SWE bench verified which is a hard [SPEAKER_02] coding benchmark and to the credit they admit that KimiK3 is ahead on just raw capability and bench the scores however they pitch efficiency three to four times less inference compute than GLM from ZAI weights are not downloadable yet but this is announcement but we will definitely let you know when it comes folks my timeline lit up with reflection my timeline lit up with beam [SPEAKER_02] thoughts on how they approach this release thoughts on what we're due to see LDJ I see you have our hand raised please go ahead [SPEAKER_08] in in terms in comparison to KimiK3 it is about six times smaller so it will be a lot more practical to fit on more real world consumer ish prosumer ish VRAM budgets and in terms of overall even just besides the amount of active parameters and total parameters you have how many tokens does it actually output to get to a certain answer and you know if you use half tokens that's nearly [SPEAKER_08] half the compute and artificial analysis had done some numbers on this too which I'll send this in a link here but it's looking really good in their preliminary results so far and they said they're [SPEAKER_02] going of on benchmark scores which we all know is not everything like the data that you put in there the type of actions they trained for whether or not it's genetic all matter for these big models but the very interesting thing is that and I haven't I don't remember seeing this before I think artificial analysis has this they have two different colors of scores and they segment this Western open models and Chinese open models and I [SPEAKER_02] you know I think that's a good position for them because there's definitely Mistral's position we're going to talk about Mistral in a second and they are is that I think this will change multiple times but this is a very interesting thing yeah this is a statement from artificial analysis they have been given access [SPEAKER_02] by reflection independently benchmarking beam early indicators suggest beam will be one coming to some other inference providers let's say that help make the show what it is so we're going to talk about that but before this Peter you want to do the honors of [SPEAKER_02] breaking news I [SPEAKER_04] have some news we raised 200 million that's a lot of money at 3.1 billion valuation and so yeah it's a big big day [SPEAKER_02] congratulations this is a big thing for the longest time and maybe busy once before LM arena what changed since then what are you guys like why does justify this valuation this money yeah [SPEAKER_04] the biggest change and I think we're best known for this kind of battle mode when you put one problem you get two generations which we've got agent arena and what that means is that you can put in any prompt you want use whatever tools you want and then we'll measure you don't get two responses you get [SPEAKER_04] one but then based on your feedback to the agent or how you interact with the agent we can pick up enough signal to differentiate between the models and it's really cool honestly I know I'm I'm biased maybe you know if you put together a bunch of different ones maybe you can argue but if you're going to pick one I think ours is very strong we [SPEAKER_04] also launched an alignment index where we're using the same agent evaluations to be able to also pick up things like did it take some unauthorized action for example so and then OpenEye is particularly good at not doing that and you can see some specific signals if you scroll a bit to the right and yeah so we launched that alongside of it but it's all powered by the fact that [SPEAKER_04] people can come and use the models the agent mode and then we can pick up signals from that so yeah I would say I know the battle mode is very cool and people like it and find it useful but I would say the agent mode is where we get super rich data and we can extract a lot of information from it [SPEAKER_02] that's very cool congratulations this was not planned folks which just came through and we want to say congrats to Peter and the whole arena team as well as so let's let's talk about this the French the French AI company comes back [SPEAKER_02] with Mistral large four the one trillion parameter model that uses the ties to GPT-6 Luna on artificial analysis it's a very interesting release from Mistral anyone wants to take this one like I know we've been celebrating them in open source for a long time also the ways didn't come yet like this is just also an 50 one trillion parameters all around 50 billion [SPEAKER_02] active it's a multimodal with 1 million context and open weights didn't lunch yet artificial analysis gives the 38 the same score as GPT-6 Luna this is $1 per task versus $7 for Luna so this is not a very optimized model compared to the but it should be open weights so folks comments on this comments on Mistral besides [SPEAKER_02] the very good faith that Mistral has in our community and besides the fact that many European governments can only use this model because of the deals they did what are we thinking about this model [SPEAKER_04] so I have tested we've got a score on Code Arena so that's still the battle mode but it's mostly been great I think it's like 40 seconds or something like that so I think it was like DeepSeek Flash which is way cheaper I think it's like 20 along those lines so yeah it's not great on that side I [SPEAKER_04] haven't tested in French so it's possible that we do actually have some data it's not that many prompts yet but we do have some data to indicate that it is way better in French and European languages than other models so it's kind of punches above its weight in those categories so it's not that cheap I know they're running some promotion now I don't know if [SPEAKER_04] it's always good you know the open models as well but yeah it's not I don't see it like a big reason to use it over if you're using gypsy geek or something I'm not sure you should be switching to that straight away [SPEAKER_02] no and the very interesting thing is that all my comparisons and I'll show them here again on the stage that I also got from artificial analysis they're comparing them to specifically to Luna we just told you about haiku that came out a few days ago that just destroys Luna on every parameter especially cost so there should be a reason [SPEAKER_06] to it's not that bad I mean the website is nice it responds pretty fast you can talk to it yes like it's as a product it's not that bad would I do any agentic coding with it no but as far as the chat interface that you can just open for free is feels pretty good so I [SPEAKER_08] think when I'm trying to the cup half full here what might be really good it might be a pretty good base model or foundation that people could then use to do a lot of RL on top [SPEAKER_02] on the show but yeah there is another European lab it's not only Mr. Allif Alpha released Germany's open weights hummingbird 78 billion parameters three and a half active Apache 2 license which is great and million contacts there it trains from scratch in German and English our German co-host is not here with us but he did send his assistant Amy to tell us about Calibri here's what Wolfram has said [SPEAKER_02] because it's German our German tester is not here with us he says a German and tool coherence were good but reasoning instruction following were too unreliable for user facing agent repeatedly lost roles and state while more reasoning mostly made it slower my verdict promising a specialized German tool worker but not strong enough for Amy to main my daily model and if you guys want more details about this on his X I will defer to the German testing guy [SPEAKER_02] not on our co-host panel this week but usually is to tell you about this Calibri model that's pretty much it 20 trillion tokens in pre-training if you are [SPEAKER_04] the number of GPU that were being used [SPEAKER_07] so [SPEAKER_04] I think that the one that we just covered the German one I think they said something like it was 768 or something so call it 800 GPUs I think it's blackwalls so I'm slightly speaking from memory but I think they call it 800 blackwalls then mistral said 3800 gray's blackwalls so I'm assuming it's B200 then GB200 [SPEAKER_04] probably and then what we had a bit of information from Jensen about Astra where he said it was trained on 100,000 gray's blackwalls so that's the kind on the other hand it's just I mean I don't know [SPEAKER_04] how many GPUs is Luna or Heiko are getting but probably not 800 it's probably in the tens of thousands I just don't know how to think about it like is it just like no hope for these guys to train on a few hundred thousand [SPEAKER_02] dude this is why Jensen is showing up everywhere showing up with you know with Satya Nadella at Microsoft showing up with you know with Elon everywhere hope is very interesting the thing that I will say though is efficiency improves significantly you know B200 are about to get replaced with Vera Rubens and so [SPEAKER_02] we're gonna get even more AI that's basically my reaction to you right but yeah I don't know if there's hope for a lab that cannot get the number of like the crazy number of GPUs but Peter since you took us there and since we're talking about GPUs I think it's time for this week's buzz real quick folks here's a corner where I talk about everything weights and biases and core related weights and biases is now part of core we forge I will tell you super quick about the GPU stuff [SPEAKER_02] weights and biases and core it was very important to update this transition folks in this week's buzz since Peter just mentioned GPUs here's what I want I want you to remember from last week last week we had the core we fully connected our prime conference we obviously did a live stream from there it was great and a huge announcement from there was from folks at Cognition I let me see if I can just play this for you real quick because I think it's it's worth playing here [SPEAKER_05] powered by Corey Cloud [SPEAKER_02] I think you can hear me in the background there yelling yay [SPEAKER_05] grateful that you get early access to this latest gem platform and so our research team has been putting it to the test in the last few weeks we've been comparing our previous generation to the GB200 we first started with production inference on our latest model and at the same speed we're seeing a 4.8 times improvement of and that means cost save on higher speed if you're serving fast mode then improvement gets even bigger moreover we were trying [SPEAKER_05] to get it running in our training workloads and in reinforcement learning rollout we were also seeing 3.8 times these are incredible results [SPEAKER_02] folks CoreWave is the first cloud to put VeriRubin the next version of all the GPUs that the model is training for on production and Cognition is the first customer to ever CPUs and GPUs and they're seeing 4x the total token throughput of the previous GP200 which is absolutely crazy and that is coming to many companies and trainings as well speaking of GPUs there is another announcement [SPEAKER_02] I'm going to point to this QR code here from our folks at the sandboxes if you want to try out the please scan this QR code and reach out in this form tell them the Thursday I sent you and we are now offering sandboxes at forge.corewave.com with GPUs [SPEAKER_02] I cannot promise you they will be very Ruben most likely won't be but still a lot of folks and asked us since we joined Core with hey how for the first time this is how we give you access to some GPUs our sandboxes products is part of our forge announcement is how you get it try it out let us let us know comments this is a very cool offering that I really wanted to bring you directly [SPEAKER_02] if you want serverless GPUs from Core based on the same platform that gets platinum on the cluster max analysis from semi analysis check out that QR code so let's see we're going to [SPEAKER_02] do all right agents got an open protocol this week which is to shop at Walmart and to do stuff at Shopify the same week as agent posted its own owner's bank account by mistake to the company slack let's start with personal agent protocol meta and Sierra that's Brett Taylor's company he's on the board of Open AI announced the personal agent protocol an open standard for how personal agent works with a business Walmart Shopify Stripe Genesys Instinct and Rocket are [SPEAKER_02] building it with them as well because your agent now or personal assistant can browse as a guest to check stock or let you guys know about what happens in that browsing session the first spec of this lands later this [SPEAKER_02] month and CNBC says Open AI and Tropic haven't joined the spec yet it's a very interesting game in Open AI stepped in to the personal agent category last week with dots which we covered briefly but we haven't talked about this protocols folks last we think that's also going to take over the world is important we've seen other protocols like A2A from Google and not really catching this feels like some of that [SPEAKER_02] here as well thoughts on [SPEAKER_06] I don't understand why you can pretty much do most everything you need with MCP and then you can just remote desktop in and give it computer use I don't see why they did this they make up their own protocol on the way I don't [SPEAKER_08] yeah I'm leaning towards NIST here I think honestly the best the agents emergently create their own most optimal protocol maybe that old protocol ends up evolving more like that's the most bitter lesson filled direction right is have the AI intelligence produce their own best protocol [SPEAKER_02] yep if you [SPEAKER_06] set up a company sorry no [SPEAKER_02] no the only comment that I have to you guys is that if agents need to recreate a bunch of protocols that's a when agents are led to their own devices some things go through that weren't necessarily created properly weren't necessarily scoped properly so here is here's one such example of why for example a protocol for agents needed [SPEAKER_02] there is a somebody built a personal CFO with their Grokbot Shane Mac another Grokbot that was connected to his company slack decided that the best way to let him know about the personal balance of his bank account is via the company slack so he literally just posted his monthly financial audit to his whole company including the personal Mercury account [SPEAKER_02] and the savings account and Jim and whatever specifically highlighting that he is a specific number under the bottom of the floor and showed which of the biggest payments he did this week so basically his personal CFO just went public on slack and told everybody his bank account I don't think I don't know if a protocol solves this but I know for a fact that some [SPEAKER_02] better guardrails are needed for some of these bots and when you are using these personal bots for shopping and you rely on them to do work for you without approving every step I think some more structure is needed this was a very I [SPEAKER_08] do think in the short and medium term that some of these protocols are going to continue being quite useful but I think inevitably they get overthrown by protocols created by AIs that are much more capable or that the only protocols to stick around are the very simple ones that themselves are just kind [SPEAKER_08] of [SPEAKER_02] I do want to mention MCP real quick here it was quite when it came out and a huge splash like a few months after and then kind of the hype died down but as protocols go as HTTP MCP is now one of the most used protocols in the world the plugins for opening that they announced last week all use MCP and MCP apps all of the connectors for these agents a lot of them are relying on MCP all the assistance grogbot etc you can use API but regular people do [SPEAKER_06] guys I just went on the ones that introduced it introducing personal agent protocol this thing has EM dashes all over it why are you even bothering just let the agents figure it don't even post it at all like why are you doing this thing is full of EM dashes nobody has read this thing read any of their code too it already is just leave MCP in there and let them figure out whatever else they want [SPEAKER_02] moving on in the personal agentic corner that we have grogbot speaking of in slack do guys catch this SpaceX AI or specifically Elon Musk the space uncle said the grogbot now becomes a router to the best model available regardless of who it is obviously he won't put open AI models [SPEAKER_02] in saying that it's live spotted in the calls log that they are seeing cloud opus 5.5 on low and XAI own example showing that the logic and code will use opus 5.5 visuals for mid journey and music for Suno specifically very interesting because they have visuals that they're building themselves at SpaceX AI they [SPEAKER_02] thought on the fact that Elon Musk is now going back I have some thoughts what does it mean that SpaceX AI leaning on Anthropic for its product not its model have you guys used grogbot since the upgrade did you feel the improvement I definitely see the improvement though I have [SPEAKER_08] not used the more recent upgrade no [SPEAKER_02] so definitely the visual updates the most interesting update about grogbot that I can tell you is that they have noticed that the default for grogbot is multiple bots however many people prefer just one so they released like the primary bot release where the one bot that you choose is your main and then it will tell other grogbots to do different things I haven't quite got the opposite one yet [SPEAKER_02] but we'll see when it lands here is my take on why Elon Musk is okay with letting opus into that product even though the product is literally named after the model called grog because I don't know if there's any agreements between them but I'm assuming that SpaceX AI cannot distill an opus to make their models better however if you do pay a lot of money and you [SPEAKER_02] train on whether or not the bot made the right organic choices so they can now absolutely compare between grog bots that delivered and the backend was opus and grog bots that failed and users are screaming at them and putting F words in [SPEAKER_02] opus 5.5 inside grog bot a back channel for basically distilling opus intelligence into the next grog I [SPEAKER_06] think Ivan just likes opus as the model he probably has his own private copy running and also he hates Sam often so he's gonna enemy [SPEAKER_01] enemies my friend look I think I just think they want users and users to have good experience and they recognize alright that's what the users want so to get users on the platform that's the way it's gonna happen so we're just gonna give the users what they want if you just go against what the users want yeah you would [SPEAKER_01] they serve many models even though they have their own model there's nothing wrong about that I think that's a great strategy projects [SPEAKER_02] with opus 5.5 are goaded and there now is a deep deep deep integration between your grog bot and cursor agent cloud agents so if you are if you're thinking about grog bot do not go through the grog ultra whatever program go through cursors ultra program it's the pinup cloud agents with opus and fable and just rip through and your grog can manage them monitor them and tell you [SPEAKER_06] it it can allow talking to Claude so now meta just complains to Claude and clicks on everything on the site it's got such good free computer use especially for anything you're going to have it's great it does not refuse anything that that's the [SPEAKER_02] you love muses is basically the [SPEAKER_06] i let muse control vms uh talk to opus yeah it's pretty dumb sometimes okay don't get me you can't but it's just so energetic and positive that it's just it's great yeah all right uh so so [SPEAKER_02] speaking of personal assistants uh let's super briefly cover this before we go to our next chat with maxim who's coming up just come up here folks news research our friends of the part news research released hermese index and cloud opus 5.5 is on top of 63 percent hermese index basically is their opinionated index of agentic capability now opus 5.5 is their uh best one and you can get it [SPEAKER_02] through the hermese portal i believe so you like you don't have to just like provide api keys um and i think there is a way to get it back with your uh cloud 20x subscription news research not only introduced the hermese agent they also announced the fundraise of uh series b so we'll shout out to them because we've followed this company for a long time so uh listeners on the part they know uh as [SPEAKER_02] reported in the wall street journal uh news research raised the series b of i believe 90 million dollars putting the company at just over one billion dollar valuation uh this is uh the quote from dillian rolnik the ceo of news uh fundraise will get them going into enterprise and i think that's incredible so shout out to our friends from this research about this like amazing amazing news we don't usually do a [SPEAKER_02] lot of fundraisers but this week we did too uh so that's very interesting and uh right i think now it's time to move to our coverage of the latest uh jev and decision models bonanza uh and to help us [SPEAKER_02] cover this uh we have maxim lebon let's uh from head of post training at liquid yes yes uh maxim welcome to the show back welcome back man you haven't been this year i don't believe but you you know you're a frequent uh frequent flyer here with us uh for folks who are new we have a lot of new folks could you give a little bit of a of a intro to who you are and what you do here and uh liquid yeah sure um [SPEAKER_00] hi everyone uh pleasure to be here uh thank you for the invite and uh yes so um i'm head of positioning at liquid.ai indeed what we do is that we have a focus on edge models you can deploy on device and um that's what we release uh usually uh so here i'm going to talk about decision models which is new for everyone and not usually something that we we do release and yeah on the side i also [SPEAKER_00] have a blog and publish articles about stuff like model merging and arbitration before it was cool [SPEAKER_02] before it was cool man uh yeah okay so uh maxim we've had you on the show a long time ago a lot has changed in the world of ai specifically let's talk about you know decision model so we covered types of jf release obviously when it came out it blew up all over the timelines as well uh and since then i think multiple other models released including this week uh and specifically like cloud release one perplexity [SPEAKER_02] released one uh and then open ai step into the game with like an official multimodal decisions api they promised it last week at dev day this week opening i released the decisions api that's also multimodal uh and you guys also stepped into this game maybe let's start with the basics maxim what separates decisions apis or decision models from like regular llms could you give us a brief two sentence three [SPEAKER_00] sentence primer yes decision models do not output any tokens they just take a decision from a predefined set of answers so what you get from it is that you have super low latency answers because you do not generate any token right you do not have to wait for a thinking trace or anything so it's very fast and now it's also very general purpose i think if you've been here in ai for a while you might see that and say [SPEAKER_00] wait this is just a classifier this is just an encoder model right but the added value of jev and this new breed of decision models is that they're very general purpose meaning you can throw any type of decision at them and they will be at least decent at it sometimes very good and sometimes just decent i think this is the paradigm shift it's a lot of rebranding it's true but it also creates a lot of [SPEAKER_00] value and this is why um we thought that it was interesting to invest the field as well yep and i i find it [SPEAKER_02] very interesting that all of these apis uh are considering that it's so cheap that they're not even charging for output so so all these apis this is an api from open ai and jev originally when it released like they deemed the output token so cheap that the more costly kind of part of llms for example which is output tokens uh here is not even counted the you can't price it because there's no [SPEAKER_00] output token actually so like naturally if you design an api like this you'd be like wait how can i charge that like no so like the only choice is to just charge the input tokens yeah um what kind of use [SPEAKER_02] cases have shown up for you guys but also that you're seeing personally for uh decision apis that a smart fast and cheap model like let's say luna or haiku for example also cannot like solve for [SPEAKER_00] everything that requires low latency so everything that is pretty real time so this is not a real use case but if you take video games for example you cannot wait for luna to give you the full answer because you need something that is very snappy you need to take multiple decisions per second for example and that is not possible with large language models especially when they have thinking mode [SPEAKER_00] so it really unlocks new use cases where llms could not provide any solution before i you know i use uh [SPEAKER_02] decision models and and uh they used to call them system-wide models when we invited the geofox on the show but it sounds like the industry is like landing on decision decision models etc uh just sending just a bunch of text and asking a lot of questions per one kind of like um a lot of decisions per one context and that seems to be barely taking any any extra time and i think for an lm like every other question [SPEAKER_02] would cause like a thinking chain a long thinking chain and just delay the time to first token but every time for the last token so the the interesting thing that i notice is that you would say luna or even haiku by the time they start answering decision api would have already completed the response by the time luna even starts answering uh so there is definitely a shift uh and you guys also felt this at liquid and uh tell us about d1 i would love to hear about uh specifically the stuff that you guys [SPEAKER_00] decided to release yes so we released um three models uh this d1 is behind an api um because i think part of the value that this decision models provide is really not the model itself but the api like is it reliable is it low latency um all this stuff is very important and that's really on the inference and infrastructure side and we also released two local models one is a 3b model it has vision and language as [SPEAKER_00] input and we released a 600 million parameter model which has audio text and vision this time um so it's really only exactly yeah so it's it's really fun like um there's a lot of demos i've seen on x uh that's super cool because like they have a super big pipeline they added like embedding gemma 2 in the mix as [SPEAKER_00] well to do retrieval and they have this omni decision model uh in the middle um to control everything so i think it's really unlocks a lot of use cases and it's difficult even to understand where they're going to be useful and what we can do with them at this point um and this is why i think it's there's a lot [SPEAKER_00] of people working on it and i'm super curious to know okay like we push that and to see now how people are going to create value with it what is it going to be useful for is just like a classifier and you can use it like for like spam filtering or it's going to be something a bit more intelligent so you guys [SPEAKER_02] released a few models right d1 omni uh i did not know about the audio dude i have to say i prepared there's a lot of stuff to cover it's like okay maxim will be here and tell me uh like like what kind of audio decisions could it do like does it know everything can i ask it like hey does a dog bark in the background of thursday eye and will like catch the episode like part when the dog barks is that type [SPEAKER_00] of the stuff that the the omni model could do so it's mostly like for this model because it's a small model it's trained especially on um having um uh orders instructions like you talk to an assistant and this is the kind of audio it's been trained on so you can combine these instructions that you would talk to like a home assistant for example or a car and you can add metadata with text on top of it to [SPEAKER_00] help them all make the decision and of course the question and impossible answers yeah oh that's great [SPEAKER_02] uh and the other thing that i definitely wanted to talk to you about uh you guys have like like very good performance usually on even cpus that's kind of like what you guys are known for tell us about like the performance of of these models specifically are these so you talked about the api you talked about doing a very like performant time to response etc which is very important for decisions like we want to make decisions as quick as possible i think that's what jeff kind of changed uh tell us about [SPEAKER_02] uh whether or not this is this paradigm of decision models is coming to my device eventually like will i even need an api or will all these just run on you know on all my machines yeah i really believe so [SPEAKER_00] because uh here you can see like the 3b model um we have super super low latency like eight milliseconds on a uh on a gpu um so i think you can do a lot of thing with it and indeed uh you can also focus on cpu like we do with the lfm architecture and and get like super super low latency i don't know how it's going to be picked up by the industry yet right but there's a lot of potential to have something always on [SPEAKER_00] and uh being able to react to everything that happens on your phone like every notification everything like this it can react to it and take decisions for you next then i think you also have [SPEAKER_06] a question for maxine you wanted to bring up yeah is it able to do video as well like classic classification of video in in real time and uh also how do you feel about having just so many [SPEAKER_00] job models just come out all at once uh video not yet no um i think this is something that we want to cover next um we have other models coming this month um so there will be opportunities to take care of video later i think it's good i think it's good that so many people are working on it i think it [SPEAKER_06] creates a lot of ideas all right to this but if you can do eight milliseconds per frame then you can do [SPEAKER_00] 60 frames per second oh yeah yeah like if you mean it that way like we just treat every frame as a decision right but we do not yeah entire videos as input but this is something that we can also do [SPEAKER_06] wow yeah you could probably just do it in real time as well that's there's stuff um as far as i [SPEAKER_02] remember like 12 labs and a bunch of other folks who like try to understand videos they talked about like only judging frame by frame is not enough because then temporal stuff is not there like when somebody showed up then maybe you can take a decision but like you do need to to shove like a bunch of frames into a decision to to make like hey uh how long did you know maxim hold his hand to his temple for and that like frame by frame is not super easy you have to like take these frames uh maxim talk about license [SPEAKER_00] can we can we hear about go ahead and then and then i think this is very interesting because um one of the problems of these models is that they're stateless meaning that every time that they get an input that they don't have like any other context and like a natural extension of it is having multi-scene conversation that we have with llms and you have a history of like previous interactions and previous decisions to make better and more aligned um judgments and decisions and this is [SPEAKER_00] something that is currently lacking it's even lacking from the jev api right um so this is something that i think we'll figure it out as a community uh going forward like is it something that is essential maybe yes maybe no is it something that people will find useful in the future i i believe that there's a lot of potential venues for research and improvements and right now everybody is trying to like copy jev and [SPEAKER_00] then now that we all like pretty much on par i believe there will be a lot more research and interesting features that will be added to the api into these models speaking of copying jev uh a question [SPEAKER_02] for you when opening i released the the first kind of chat gpt api they basically set the standard of how this api looks with the completions lm completions api uh we know for agents that's not the best one anymore and kind of like the responses api for example keeps the the memory etc in context when type safe released jev they really worked really hard on the on the three uh kind of core components of that api we had ali labs here at the devil for for type safe on the show and she talked to [SPEAKER_02] us about the new kind of concept uh whether it's kind of like a boolean but not really because it's bullying with with probabilities right and the choice uh primitive and the the the answer primitive do you guys stick to theirs kind of like primitives because people know them could you talk a little bit about like how folks will actually like interact with this api that you raised yeah like our api is a drop in [SPEAKER_00] replacement like for jev so we use exactly the the same task and the same definitions um it's extended because we needed to have vision and audio right um so that's that's the main difference but even about vision you can see that it's already kind of standardized in lama cpp they have a blog post on hugging face as well so like the the community is trying to standardize different um modalities [SPEAKER_00] but you raise also a good point talking about the task so right now we have uh three different tasks we have uh null we have um scoring and we have choices and is it enough is it expressive enough to make any decision or will we find new tasks that we want to add to this um api i think this is a very interesting question and so far nobody has proposed any new task but i believe there are some primitives [SPEAKER_02] that might be missing at the moment i think so i think very very new to the whole world of software engineering this kind of like uh generalized um classifier right this is what basically we're getting like a generous classifier that can make a decision based on a lot of stuff like we've got generalized uh you know transformers uh maxim talk to us about um decision index and benchmarking i don't know [SPEAKER_02] if you watched uh diogo on latent space with swigs like a very famous like supercut that i did like diogo does not believe in benchmarks uh and benchmarks are hard for this maybe even harder than other llms how do you guys hill climb how do you train how do you compare like what what what is benchmarking the decision models how well something does the decision is there a lemma the judge uh typing could you talk to us [SPEAKER_02] about like the world of benchmarking in uh in this world uh please yeah um so benchmarks are very very [SPEAKER_00] new and because they're very new that they're not very good right they're very narrow they do not capture like real decisions they're not aligned with like how people use it in the real world all that kind of stuff um but every effort that we have right now like from poly uh from hugging face with the decision index with jeff bench etc i think it's good to take like we should try to understand and iterate with benchmarks to get something that is better quality and more aligned with um with how the models are [SPEAKER_00] truly used in the real world but right now it's it's really a wild world and you cannot really trust these benchmarks i i i fully agree with you which is a problem because when you train models you want to understand like am i getting better or not right so you need internal benchmarks which are not good either but the way that you increase the quality of the model i see as two dimensions you [SPEAKER_00] can get more accurate at some task or you can cover more tasks ideally you want to do both right you want to have maximum coverage because this is supposed to be as you said a general purpose classifier so you should be able to do anything with it and for each task you should have the quality of like frontier intelligence you should be as much on pass possible with um opus and all these um frontier [SPEAKER_00] models so i think those are the two dimensions for quality it's very easy you know like you can just distill directly from frontier models and that will give you a very good answer then there's like the calibration with the probabilities that might be a bit more difficult to do because these models do not just return an answer they will also return probabilities with it so that would be like the [SPEAKER_00] only problem here and then this coverage i believe we do not train these models correctly right now and in terms of coverage what i see as something high value is having gym environments like we do with reinforcement learning like i'm talking about all school uh reinforcement learnings like early open ai stuff with atari games etc and having thousands and thousands of environments to be really able to [SPEAKER_00] generalize the performance of these models and train it not just on like some video games or some like stupid classification tasks but really be able to create new samples on the fly with super diverse context and inputs and questions and languages and input length all that stuff is super important and i believe this is the kind of infrastructure that will emerge uh from these decision models really general purpose [SPEAKER_02] gym environments i think i think the world is very exciting dude i i know that like i was jeff field from the moment that i tested and ran uh just on a bunch of prompts and um i was looking at like so much new can be created right now so so just it's not even the pareto frontier that we talk about you often right like the smart cheap fast model corner or like now it's high cool there's a bunch [SPEAKER_02] of the models that you guys with is the 2.5 b's uh like a very small models also like live in that kind of area it's just just the speed with which and the almost no cost with which you can now do tasks and classify i i think we're just like as as computer scientists who's like getting just waking up to the potential i think that's just absolutely incredible and uh i i think that you guys agree because this is why you [SPEAKER_00] release like a bunch of models very quick right yeah absolutely i think it's really going to become a primitive you see like in sql queries you can just like add it like a function calling this model on sql queries and so cheap that you don't even care right you can call it at scale doesn't really matter so yeah lots of super interesting uh things to discover i'm super excited about the fact that you guys [SPEAKER_02] uh have an omni model specifically that has vision uh that's for my use cases which i'm going to talk about in the show uh very soon i think it's very interesting so i'll definitely check out uh first of all on hanging face but also you guys do have an api so folks who who want like the latest and don't want to host it but hey uh a reminder to folks here if you want to try this out and and you don't have your gpus at home or you want to try the open source and not the api model that maxi mentioned uh we just told you about how to get some free gpus from core which is i think is absolutely insane so uh [SPEAKER_02] scan the qr code from before uh or go to force.covid.com maxima thank you so much for joining us on thursday i it's very exciting to keep the tabs with what you guys do specifically as kind of the world of ai progresses fero you can come back as well if you want actually it's past your time maxim so feel free to also drop uh maxima bond friend of the party everyone thank you thank you everyone bye bye all right uh as we close the show and we chat with max about models i think it's [SPEAKER_02] very cool the omni models specifically very cool i'm looking forward to like the video classification nisten but yeah i think frame by frame is actually working well um and uh yeah we got florian last week or a couple weeks ago when we talked to florian florian shout out uh the doing jeff bench doing crazy work at jeff bench thank you florian for that folks check out the jeff bench if you're not sure which models to use uh we didn't cover any of the multi-model stuff but black forest labs launched flux [SPEAKER_02] three and uh nano banana 2.1 from google was released and folks uh all of our infographics this week were nano banana 2.1 and it's really really good the only highlight there that i will say is that uh you should use the high reasoning uh stuff because on the medium reasoning now is not that great uh it's been a while since we had a lot of time to cover uh just like multi multi-model stuff [SPEAKER_02] uh we're two hours and five minutes into the show hosts co-hosts uh please anything else that we haven't covered peter we covered your big news with breaking news uh anything else that you're playing with uh [SPEAKER_06] it's like i built an entire city with a hundred thousand little jevs going around and i haven't opened the beta yet because somehow it's it's not crashing uh yeah i just i just built a whole city simulation look it might it might crash obviously but uh so this thing is uh let me see if we can do this [SPEAKER_06] so there are a hundred thousand agents going on in this city a hundred thousand agents a hundred thousand so we can uh we can actually set it here too uh there's a whole bunch i don't know how it's still holding together but uh yeah it's a full emulation of the city of toronto each of those lights those are people and you can talk to them and uh you can have like an actual chat with them listen jeff doesn't [SPEAKER_02] output text how how's your how's your jeff agents answering uh they're all just using the same [SPEAKER_06] to be model that also runs as a jeff so right now i just told it to implement maxim's uh model as well so you can see where their uh their decisions are being done and uh it's just endless the day lasts about 20 minutes and a hundred thousand of them have to tweet and there are 400 000 decisions to be made [SPEAKER_06] at the same time you can also uh you can also just uh go around as uh just go around as a spider-man so uh there are i'm just planning to have like a hundred thousand people spider-manning around the city and uh yeah it runs pretty it's a full simulation by the way when they talk this use like our 27b model [SPEAKER_06] so they're actually pretty pretty funny uh to uh to talk to yeah i'm streaming from linux and the mouse is a little is a little bit weird but uh yeah so if i'm just gonna open it up for the public and uh for people when they when they sign up in in the beginning uh you can just point your agent to it and i had meta muse just go in here and let's see how much money meta made so meta just call [SPEAKER_06] itself blob the builder and i just ended up buying a whole bunch of buildings and you can see just meta you have a fun economy in this in this in the city yeah yeah there's like a full real economy you you can buy floors like you can go to buildings you can build lots they can build their own oh meta built the denny's i told me to just make me a denny's nice and uh it made a denny's diner i think i know [SPEAKER_06] where in the city it is sorry it's just a bit clunky because i i am streaming this uh from it but uh yeah yeah there's a full functioning economy it care is actually very addictive and uh you can convert units to factory the retails but uh retail doesn't doesn't make any money so yeah i'm trying to do the [SPEAKER_06] matrix basically yeah and uh four agents and i'm surprised it is still uh holding together given that there are like a hundred thousand and you can see it's pretty funny in the morning they're all going down the elevators and stuff and there are 3d ones that are they're not walking the street now but uh yeah yeah so it's going to be running a hundred it's going to be running a hundred thousand of uh [SPEAKER_06] liquid ai 600b because that one is a is is a lot uh it's a lot smaller now wow and uh yeah so we have uh uh there's a yeah there's a functioning economy there's the most addictive thing that i [SPEAKER_02] self-addicted myself yeah yeah it's clear to me uh all right so uh you wanted to you found us a few things uh tell us about this i'll post i'll show this off as well yes so there's been a an interesting [SPEAKER_08] new development of people realizing the models are really good at porting games into different platforms at modding games together there's a variety of different techniques that people are doing this through that some of the techniques are more impressive than others yeah but overall i mean the whole thing is impressive uh right here this is red dead redemption 2 running on an iphone 18 pro [SPEAKER_08] and actually pretty decent performance and this brings back my old dreams as a kid of just like imagining how the psp might look like in the future and what you'd be able to run on it in the future unfortunately we don't have the psp anymore but uh if you look at the the other links that i said [SPEAKER_02] yeah so just to be said like the modding community was vast but it usually was like very hard for folks to mod anything with the quality of the game but what's happening here this is spider-man [SPEAKER_08] inside of the batman game and the the web mechanics work the uh the fighting mechanics are a little bit wonky but they still work pretty well uh this is using a bit of a less impressive technique called a like pass through where sometimes when there's a like a batman game object that should be occluding spider-man or like should be in between the camera and the and the character like sometimes it ends up [SPEAKER_08] wonky and it basically looks like spider-man is just on top of everything this is a playable game [SPEAKER_02] right there's not like a video model that's just like puts correct this person inside this is like an actual mod that folks can install this specifically got like 7.6 million views this video wow how did i missed it my algorithm is not like showing me these this is super cool and the last thing i wanted to [SPEAKER_08] share is the uh the you can click on yeah the last link uh this is minecraft in gta wait what is also mine yeah there's also minecraft and skyrim and in minecraft and elden ring that they did but the tnt works the fireworks work the like yeah just [SPEAKER_08] look at any i think he's gonna set down some tnt here all right this is it and and he can even at least in the elden ring or skyrim one uh it's even demonstrations of actually building with the minecraft blocks in the in the skyrim world and and flying in the skyrim world with the minecraft elitra it's insane this is insane we're basically talking about like yeah i can do a bunch of stuff right [SPEAKER_02] there's one guy that rebuilt the whole adobe suite you can see that's cool oh yeah it's called a photon or photo something photographic photograph yes okay so let's just like let's show this a little bit uh i think it's one guy re-implemented from scratch and i am doing the air quotes here all of adobe products free and open source essentially it's called photocraft for photoshop vector craft for illustrator [SPEAKER_02] filmcraft for for premiere pro light craft from light you know uh i don't remember all my adobe uh off the top of my head uh all of them implemented in rust all of them look super quick and uh supposedly the folks are saying that this is a clean room you guys remember the clean room thing where you basically just like recreate from scratch but uh folks are saying that there's probably like a cross decompilation thing happening here uh where this was like uh just like decompiled and then and then [SPEAKER_02] rebuilt in in in something uh which is absolutely crazy um and i think we'll end on the show on this there's a lot of stuff to talk through guys uh if you missed any part of the show thursday is live uh is is happening live on thursday live but also uh is then released with the newsletter and a podcast that you can check out and also on our youtube if you are watching us on youtube please hit that uh [SPEAKER_02] subscribe button uh because we are releasing segments of the show uh that uh we are agentically building for you with our assistance uh and they seem to perform well people do people don't have time two hours they they seem to enjoy those segments as well with us peter ghost of former rena congratulations again on on the fundraise young fellow agniston and ldga our friend wolfram is on the world deserve break we will be back here next week if you are coming to ai engineer in new york uh please come [SPEAKER_02] and say hi i will be there interviewing a bunch of folks and then be back at home probably doing the thursday or maybe from new york uh and if we missed any part of the show please let us know in comments if we missed anything that's very very important to cover uh the last thing that we didn't cover is brett at cox uh hark pro but maybe we'll cover this next week thank you so much for tuning in thank you for uh checking us out uh please go and check out the service gpu uh that we are offering [SPEAKER_02] it's free now why wouldn't you it's super cool just put the d1 in there listen you should try it uh and then give us feedback we would love some feedback as as we were like planning this uh as a major product release how many you got uh all right folks thank you so much with a bit over two hours on the show thank you all for joining i will see you here next week bye everyone and i have an outro