← Back to search
Claude now leads 26% of Anthropic’s R&D, Meta drops an 816B open model & Gemini 4 Argon
ThursdAI - The top AI news from the past week · 2026-10-16 · 124 min
Show full episode description
Hey, it’s Alex 👋 What a freaking week! I spent 3 days at AI Engineer in New York, and we did the show from a real studio in Manhattan (the internet did NOT cooperate at first, but we got there 😅). And while I was running around the Sheraton, the labs kept shipping. Meta became a frontier lab, for real this time. Muse Wonder is #2 on Arena, and then, in the last minutes of the show, Meta ALSO dropped Muse Spark Open, an 816B open weights model. Google’s big Gemini is finally back with Gemini 4 Argon, and Omar Sanseviero and Geneviève Huskens from Google DeepMind came on to show it off. Anthropic gave us 4 stories plus a rumor, and Nathan Benaich came by to talk State of AI 2026. My theme this week: AI engineers are sleeping on personal agents. More on that below, including how my Muse found this studio and got me coffee on the way here. With me: Wolfram Ravenwolf, Peter Gostev (Arena), Nisten, Yam Peleg and LDJ (who brought the breaking news, as always). Let’s dive in (all links at the end as always!) Meta is BACK at the frontier Muse Wonder - Meta’s first frontier model is #2 on Arena, and it smells BIG ( X , X , Arena , Blog ) Folks, we told you after Llama 4 was a disaster, we told you when Zuck started hiring people with soup: do not, under any circumstances, bet against Zuck (and now Alex Wang). Muse Wonder is #2 on Arena Text at 1513, 12 points behind Argon, and it definitely smells like a big model. There are 2 of them. Muse Wonder is in the API at $4 in / $20 out per million tokens, basically Opus 5.5 pricing. Muse Wonder Full, the better one, is only for Muse Code subscribers at $50 or $200 a month. Did you see what they did there? Everybody calls their top setting “Max”, Meta called it Full, so it’s... Wonder Full. Wonderful 😂 No open weights for either, Meta says they’re too big and maybe too risky. Meta’s own table beats Opus, Astra and Fable on a bunch of evals... but Personal Superintelligence Index, GDP.pdf, JobBench, MemoryBench (which Meta built, and then won)? I’ve never seen most of these. Every launch now is the iPhone 18 moment: numbers bigger, best one yet. On evals we know, Artificial Analysis has Wonder Full at 56 and Wonder at 52, Opus 5.5 on top at 58. And on OSWorld, the Wonder we can use sits below Fable and Opus, and computer use is the number one thing I need from an assistant. Peter’s Arena view: #2 on text means people like talking to it, but it’s 12th on Agent Arena, and he tried building websites with it, “it was kind of awful.” His theory: Theo loved Wonder Full and thought the API Wonder was bad, so maybe Meta is talking about a different model than the one we got. Not ideal. He also found the half-life of a benchmark quoted in a launch is about ONE release: quote it once because the number looks good, never see it again. Wolfram says it’s excellent in German. Nisten joined not knowing what Muse Wonder was (”No idea”), sometimes our co-hosts come to get their news from the show. My Muse got upgraded with a click Here’s where I disagree with the timeline. “It can’t do 3D scenes, so it sucks” misses the point. Meta is tuning Wonder for Muse, the personal agent. Officially, Wonder will “soon” power more of Muse, subscribers first. I’m on the paid plan, and my Muse (Wilfred) flipped to Wonder the moment I subscribed. Matrix moment: “I know kung fu.” Then I kept shoving tasks at it. Muse found the podcast studio I recorded from in NY and emailed Ezra at Virgo Studios with my requirements, got me coffee on the way here, and designed and ordered the wristbands for our AIE sauna meetup. The failures weren’t logic, they were the real world: a site that’s blank because Cloudflare doesn’t like agents, a Stripe link that wouldn’t connect, and businesses that hung up because they didn’t want to talk to a personal assistant 😂 Peter’s read: that’s the harness, not the model, and it’s exactly why this stuff is so hard to benchmark. Muse Spark Open - Llama is SO back ( X , X , Blog , HF ) In the last minutes of the show, LDJ hit us with breaking news: Meta released Muse Spark Open 20 minutes earlier. 816B total, 42B active, 1M context, multimodal in, text out. Meta calls it the most capable US open-weight model to date. I wish I had my air horn but
✨ Episode Outline — click any point to jump to it in the episode
Problem solved
Catching AI engineers up on a frontier-model week: Meta's Muse Wonder,
Gemini 4 Argon,
Anthropic's investor-day numbers, and the personal-agent wave they're sleeping on.
Benefits
- Arena leaderboard read on Muse Wonder vs Gemini 4 Argon (text vs agent splits)
- Claude Code now emits OpenTelemetry traces viewable in CoreWeave Agent Lens
- Anthropic IPO/investor-day numbers instead of rumors
- Live DeepMind guests (Omar Sanseviro, Genevieve) on Gemini 4 Argon
- Nathan Benaich on the State of AI report
Use cases
- Meta Muse Wonder ranks #2 on Arena text leaderboard, 12th on agent arena (Max version)
- Gemini 4 Argon ranks #1 on Arena text and visual evals, 4th on agent arena
- AI Engineer NYC hackathon: 72 SaaS clones built using ~100 billion tokens in a weekend
- Claude Code OpenTelemetry traces piped into CoreWeave Agent Lens for observability
- Capital One talk on when code quality still matters for banks ('toxic access' to loan-book records)
KPIs / results
- Muse Wonder: #2 Arena text, #12 agent arena
- Gemini 4 Argon: #1 Arena text, #4 agent arena
- 72 SaaS clones, ~100B tokens spent at AI Engineer hackathon
- Claude leads 26% of Anthropic's R&D; Meta open model is 816B parameters (per title)
Tools / build
- Meta Muse Wonder / Muse Wonderful (in Muse and Meta Code)
- Gemini 4 Argon
- Claude Code with OpenTelemetry + Claude Agent SDK
- CoreWeave Agent Lens
- Fish Audio Drama 3 (open-weights)
[SPEAKER_07] We're live from New York. Welcome everybody to Thursday. I right after AI engineer New York another another very exciting one we are There's a big week to talk about Meta finally shipped the frontier model. Meta literally became a frontier lab Google's Argon is out for everyone Though I personally haven't been able to test it. However, we have two great deep mind guests today Omar Sansevro and Genevieve who's on the developer relations team [SPEAKER_07] at DeepMind are going to join us in the next hour to talk about their release and Anthropics showed investors the numbers of IPO and there's a bunch of other stuff that happened as I bring up try to hear Wolfram. Welcome everyone. Peter, welcome Wolfram, welcome. We're live from New York. Well, at least I'm live from New York. Wolfram's live from Germany. Peter, London? [SPEAKER_05] London today. [SPEAKER_07] London today. Folks, this is an international podcast coming to you live. Welcome to Thursday. I'm October 15th. I'm live from New York. My name is Alex Volkov on AI Evangelist with CoreWeave and with me Wolfram, also AI Evangelist, also from CoreWeave and Peter Gostev evals and model behavior at Arena. [SPEAKER_07] And I just spent three days at AI Engineer here in New York. I can't wait to tell you guys all about this. And while I was there, it was an incredible week just blowing up. So we had to try and catch up. MetaShip Muse Wonder, which was rumored by them as this big watermelon model. And it smells big. It definitely smells like a big model. And you can use it right now in Muse and we're going to talk about this. [SPEAKER_07] It's number two on Arena, which is puts MetaSquare in the Frontier Lab race. As we told you before multiple times, do not under any circumstances, do not bet against Zach. We told you. We told you this two years ago. We told you this after Llama 4 dropped. It was a disaster. And we told you this when he started hiring people with soup. And so now we're going to feel it. Peter, I think you tested it out. I think you have some thoughts. Would love to hear from you. [SPEAKER_05] Yeah, maybe to differentiate. We do have a few different leaderboards and they are second on the text arena and actually 12th on the agent arena. [SPEAKER_07] Interesting. [SPEAKER_05] Yeah. And it's kind of interesting to interpret. I think we do have this kind of divergence now going on. So for example, Argon from Google as well did really well on text. So it's actually first. So you can roughly interpret it as how much people like to chat to the model. So it's interesting that there is a divergence between who people want to talk to versus agent capabilities and coding and all of that. [SPEAKER_05] And it looks like it's different. So yeah, Argon is fourth on agent arena and Muse Wanda Max. So it's not the full version. It's on its rank 12th. [SPEAKER_07] Very interesting. We also have to talk about the whole thing where I saw the timeline react negatively because Muse Wanda wasn't able to recreate the 3D scenes. And people were like, oh no, this is a bad model because it can't compete with Astra on like the 3GS thing. We have to address this because I don't know if my assistant every day needs 3GS. [SPEAKER_07] So I don't know if this is like a personal thing that I need my assistant to do. Okay. What else? Okay. So folks later today, Omar Sanseviro from Google DeepMind joins us to talk about it about Argon specifically. We also have thanks to Peter. We have Nathan, Nathan Benyash from the State of AI report joining us in an hour to talk about the State of AI. They've been running this for a decade, Peter. Is that correct? It's like a crazy thing. I'm really excited to hear from him. [SPEAKER_05] Yeah. Yeah. He's a great guy. Yeah. He's been doing this for ages. Yeah. I actually haven't checked how long, but yeah, I've been reading it for many years. [SPEAKER_07] I think it's a decade, but Slopos prepared the news while I was running at the engineer. So hopefully Slopos did a good job. Good job, Slopos. And then what else? Oh, Entropic had an investor day. So before we only had rumors and leaks, whatever. Entropic is about to go IPO. We have some numbers and a specific thing for kind of the only sponsor for this podcast core with our Agent Lens product. [SPEAKER_07] Entropic cloud code specifically now supports open telemetry. So you literally can start running cloud code and then see your results in Forge Agent Lens, which is super cool. I think I've seen some examples already. So we're definitely going to talk about this at least a little bit. And then, so after Nathan Benyash, we'll talk about state of the art, but I think since I was a little bit of out of the loop, I do have a whole prep thing here. [SPEAKER_07] Would love to hear from you guys about the highlight of the week. I think we'll start with Peter. [SPEAKER_05] Well, my highlight was, I was at the NVIDIA event yesterday. [SPEAKER_07] Oh, you met Jensen, right? [SPEAKER_05] Yeah. Yeah. Tell us. Tell us. Yeah, he was lashing on the show. Yeah, they had a kind of, it was very confusing afternoon because during the day, during the afternoon, they were building, they had this kind of builders event. So it was all like, you know, people you meet at AI engineer conference kind of thing. So that was really fun. And yeah, met a few people from Twitter who I have remember before. [SPEAKER_05] They had like, it was an office in London of a company that does like humanoid robots, which I had no idea. Like it was central London, you know, so that was awesome. Yeah. And they were kind of, it was all about open claw kind of thing. So they were kind of showing how they also launched the laptop, right? The one with the kind of spark or rather did it start shipping or something. I think they announced it a few months ago. So, yeah, a few cool things like that. [SPEAKER_05] Yeah. And then in the evening, it was very different. It was in a fancy kind of reception lounge of, I know, there's a weird converted church building in London. And there was AI minister and all of the fancy people from, I don't know, universities and industry and a very different, very different vibe. [SPEAKER_07] That's so cool, man. It's kind of like besides the AI engineer London that we both went to and we recorded from, this is the first time that you're saying something about like somewhere else outside of the States that gets me FOMO. I'm like, oh, that sounds super cool. I wish I would have been there. So that's your highlight, obviously. Folks are welcome to check out Peter's timeline for this. You posted very well from there. And you can tell us more about this. And then, Wolfram, what's your highlight? What's your one highlight from this week? [SPEAKER_01] There's so much stuff. But if I have to pick one, just one that I'm currently most excited about, it's an open-waves release of Fish Audio. Fish Audio Drama 3. Yeah, that is currently... I mean, there's so much. I could name multiple things, but... [SPEAKER_07] Since we're going to cover Gemini 4 Argon and you are our Google guy with the Pixel phone and whatever, and you didn't choose Argon, I think that I'm very interested to hear what you think about Argon, since this wasn't your highlight. [SPEAKER_01] Yeah, I can say something about this. [SPEAKER_07] Okay, yeah, we'll definitely get to that. Highlight of the week for me. Okay, this is a highlight and a hot take. I've spent the last three days among the top AI engineers in New York, and also this AI engineer conference was adjacent to, obviously, Wall Street. So this was finance and AI engineers together. So there's a lot of finance bros, a lot of big trading companies, a lot of banks. The highlight for me is I realized something that I kind of... [SPEAKER_07] Wolfram and I talked about on the show, Peter, maybe with you as well. AI engineers are sleeping on the personal agents' renaissance explosions happening in the real world. It's kind of crazy. So folks who are listening to Thursday, I know you guys are AI engineers. Hopefully we do a good job to tell you about the stuff that's happening outside of our bubble as well. It is kind of crazy. It feels to me that because everybody's building their own harness, or they're so used to cloud code doing stuff for them, [SPEAKER_07] they're trying to force this behavior into tools that don't need to do these tools. Meanwhile, the world is changing for normies because of Glock bot and Muse, and to some extent bots, or dots, to some extent dots. And this is like the summary for me because I'm like a very personal assistant pilled, or a personal agent pilled. And I see that I talk to folks and they don't have one running around 24-7 doing stuff for them. This is my highlight of... [SPEAKER_07] This is not the AI news, but this is news to me. That even the engineers that we talk to every week for the past two years, that love to stay at the forefront of what's happening, are missing a huge, huge thing that's happening. So hopefully we're here to tell you about this huge new thing, and we're here to make sure that you guys don't miss it. Because I think it is huge. Because most of the normies that I talk to did hear about some of these personal agents. Wolfram, go ahead. [SPEAKER_01] What really excited me this week, it's another thing. It's not an AI release or something, but it is something someone I very much respect said. Because Linus Torvalds really said that not using AI to create code is like, it's just a tool. So if you don't use it and you are a smart person, you should be using it. He said if anyone is submitting, as an example, it's as crazy to say you can't use AI for doing something. It's like saying you can't use a compiler to create code. [SPEAKER_01] Because that is also a machine tool that you are using and you are not using a sampler to create the code. And that was an example. And it's great to see the mindset change. I mean, he already had an open mindset. And he says, as long as you understand the code, I don't care how you create it. That is the important thing. And I think more people should be thinking like that. It doesn't matter what tool has been used, but whatever you created, do you understand it? And does it work? That is the important thing. Not how you create it. [SPEAKER_07] The do you understand the part is interesting. We should definitely talk about this. Because I definitely slop. I turn my slop cannon towards some projects that, Peter, I think you can resonate. I have no idea what's going on under the hood. As long as I see it and it works and it's dope. But I do agree. There's some stuff, especially in critical infrastructure. So the folks at OpenAI, let me scratch that. The folks at AI engineer from banks, for example, there's a lot of stuff that needs to be human-readable. There's a lot of stuff that they need to understand everything. [SPEAKER_07] Somebody taught me the concept of toxic access, where the loan book, the master loan book record or whatever, something inside the bank that all the money lives in for everybody. If there's people with access, they can change some stuff. That's like toxic access or something like that. So that obviously people need to understand what's going on there and be accountable as well. Folks, I think let's go to TLDR and then we'll start with the show. Comments from the audience as well. Feel free to tell us [SPEAKER_07] what is the most exciting stuff that happened that we may have missed. But let me run to TLDR, which is a section of Thursday AI where we tell you briefly about the most interesting, exciting parts of what happened, curated to our interests. I think I'll start saying this soon, but we used to be able to cover all the news. I no longer with fair conscience can say this is everything that happened in the world of AI. But here is the most important thing that happened in the world of AI. Let's go. [SPEAKER_07] All right. Here's everything on Thursday AI that we're covering on October 15th in about five minutes. Let's get into it. So we're live from New York after AI engineer where I think the highlight was Capital One Max Canut Alexander made the best case I've heard when code quality still matters. This is probably the MVP talk and I would recommend folks to tune into the live show [SPEAKER_07] and wait for the release. SWIX also came on stage at his own conference and talked about how to kill your SaaS hackathon. There was a hackathon this weekend and 72 clones of big sases spent 100 billion tokens over this weekend to build the clones. And then Anthropic shows dynamic workflows as well. Let's see what else we have with the TLDR. Yeah. In big companies and APIs Meta's first big and heavy [SPEAKER_07] and frontier-ish model or maybe frontier model. No, it's actually a frontier model. Meta Muse Wonder is number two on Arena and this is the best version. And it is now available in different tools. So it's available in Metacode and Muse only for subscribers. Some builders are speaking about this. Also, there's two versions there. There's Muse Wonder and Muse Wonderful. [SPEAKER_07] Did you see what I did there? Muse Wonder and Muse Wonderful. That's, I think, what they did. Because everybody calls their models Max on the Max reasoning. They call it Full and it's wonderful and it's pretty cool. Gemini 4 finally launched. Welcome, Gemini 4. Again, as a reminder, we'll have the folks from Gemini, Omar Sancivero and Genevieve joining us in about an hour to talk about Gemini 4 Argon. It's out for everyone. It's number one on Arena and Visual Evals and we can't wait to talk to these folks and hear all about the use cases about Gemini. [SPEAKER_07] And then we talked about Entropic opening open telemetry traces and talking about their IPO. And Agent SDK still runs on your cloud subscription. Wolfram, hopefully you caught that. There was an update and now there is an Agent SDK and if you guys remember the whole maybe Entropic will block you from the subscription service if you use it for like Hermes Agent or whatever. Now, we're back at a place where Agent SDK supposedly runs [SPEAKER_07] on your cloud subscription. Entropic also showed investors a $16.8 billion quarter ahead of its reported November IPO. and there's a rumor about Fable 5.5 showing up in Azure but it's unconfirmed just a rumor. Folks from TypeSafe this was just this morning this is like nearly breaking news I think it's in the breaking news category Jev 1.14 which can now see images. So Image Jev [SPEAKER_07] we heard that it's coming and Image Jev is now here so it's now in preview and I think it's wonderful I really want to try it I didn't have time running around but like Image Jev is something I've waited for a long time. Microsoft shipped its inference Maya 300 and its own decision model Microsoft Decision 1 so decision models are taking over and I'm very very interested to see how Decision 1 stacks up with Image Jev. Open Source LLM [SPEAKER_07] in Open Source LLM four open decision models was released Amos 1 from 0.5b to 27b they also answer with bounding boxes so it's not only Jev Jev answers with bounding boxes so do these models I haven't I haven't had the time to test them Mini CPM returns with 4.7 and runs a 35 billion per arm vision model on iPhone in about 3 gigs Liquid [SPEAKER_07] shout out to our friends from Liquid AI there is the biggest Liquid foundation model 24 billion parameters built for local agents and we constantly talk about running agents locally and then Quinn Geyer RL framework for Long Horizon agents came out I think it's a paper in this week's buzz we told you about Forge Corvi Forge the rebrand for our Agente Cloud offerings and SaaS [SPEAKER_07] Forge model distillation is now in preview and I think that's super cool and also Agent Lens in preview and Agent Lens is just hot this week because now you can connect Cloud via open telemetry and see the most useful and not useful traces from your cloud run directly in Agent Lens I think that's super cool apparently XCI dropped Grok Voice 3 do you guys here see this speech to speech leaderboard [SPEAKER_07] it tops the speech to speech leaderboards and apparently all of the Starlink support calls are now used by Grok Voice 3 and Wolfram you also mentioned Phish Audio correct Drama Voice just summarize for us [SPEAKER_01] yeah it's a new a model a speech model that you can direct where you can give instructions for how the voice should be sounding I think this is best if we when we talk about it we show some examples [SPEAKER_07] yeah it's open source [SPEAKER_01] as well so you can run it locally and it's also at the top of what is possible with voice right now [SPEAKER_07] that's super cool alright let's go to industry industry where we talk about stuff that are not releases but I think it's important from this week Andre Carpathi says 6 billion people have barely touched NLLM he did some back of the napkin math and don't we love a good Carpathi tweet that says the industry lays and it feels like he's been tweeting less since he's joined on Tropic so he literally had this [SPEAKER_07] funnel of saying 6 billion people barely touched NLLM and AI and the gap he says is getting wider and then he kind of does calculate so hopefully everybody who tunes into Thursday AI folks you're on the top of Andre Carpathi's funnel but he does have this thing where like only a handful of people touch the models at the Frontier Labs I'm like okay fuck you open up the weights and then let us touch them sorry I love Andre Senpai is incredible I just like way to be [SPEAKER_07] guardrails and not let people in with this thing alright and oh and last week State of AI report came out Nathan Beniesch the author the guy who's been running this for a while is going to join us was that 10 a.m. Pacific which is going to be in an hour to talk about it and I'm excited Peter you connected us to Nathan I'm very very excited he's been running this State of AI report for a while now [SPEAKER_05] Alex I think did you mention Nemetron I don't think so right it just launched about half an hour ago [SPEAKER_07] Nemetron [SPEAKER_05] 3.5 super VL 120 billion total parameters and 12 billion active parameters it looks like they're focusing on more it's multimodal so the idea is that it's good at video understanding image understanding long running agents and roughly where they're kind of pitching it at deep seek v4 flash including small that kind of level [SPEAKER_05] but it's meant to be quite fast and they say it's performance at that sort of level so yeah [SPEAKER_07] shout out to our friends from Nemetron team green and our friend who works on that team and always join us and talk about this shout out to NVIDIA there's also rumors that NVIDIA you guys see this and NVIDIA is in talks to buy reflection I think it was reflection the lab behind beam that we talked about beam last week I'm pretty sure there's rumors that Jensen is collecting the infinity stones of open source also [SPEAKER_07] rumored also friends of the pod the Israeli company rumored that NVIDIA is talking about purchasing them as well all right I think it's time for us to go to Frontier Labs and talk about Metamuse Wonder all right folks let's talk about do not bet against Mark Zuckerberg [SPEAKER_07] and now Alex Wang so in addition to having great meme taste Meta is now back at the frontier Meta's frontier model is number two on arena and just 12 points behind Argon which we're going to talk about but some builders are calling this benchmarks and I think several things can be true Meta Super Intelligence Labs released Muse Wonder and Muse Wonder Fool is kind of the over-the-top model that we don't get in the API [SPEAKER_07] it only exists in meta subscription tiers Alex Quang called it their largest and most capable model yet I numbers are bigger and they claim it's their best yet and it does we're kind of there right we're kind of [SPEAKER_07] they actually show that they beat Opus on multiple interesting benchmarks and those benchmarks are not your standard benchmarks besides the SWE bench one I'm pretty sure that those benchmarks are a little bit different Muse Wonder in parentheses full is the better model that's the one that they say beat Opus and beat Astra and beat Fable only for Muse Code subscribers and 50 or 200 dollars a month this is the tier that you need to pay either [SPEAKER_07] 50 dollars a month or Muse Spark the model that was there before Muse Wonder they told us about open source and Muse Spark is basically the new llama they're going to open source Muse Spark I hope this is the latest version 1.3 but I'm very excited Meta says the Meta Wonder models are too big and maybe even too risky for [SPEAKER_07] open source that's an interesting call out let's just dive in there Peter I know you guys tested Muse Wonder would love to hear from you [SPEAKER_05] you are right to say that it was second on the text arena which means that people like talking to it it is 12th on the agent arena and agent arena is when people like you and I go on arena and use it in various ways which could include coding could include work so that doesn't that's not amazing I must say 12th is not that great which is kind of interesting like people say [SPEAKER_05] or is it benchmarks or like Alex Wang is fighting back and saying oh you know who needs 3D and all of that but you know 3D aside I have tried also building websites with it and I must say it was kind of awful at websites I don't I didn't spend many hours with it but I think that's kind of important as well I've never seen a model that's really bad [SPEAKER_05] but it's good at 3D and it's good at websites but it's fundamentally bad I [SPEAKER_07] I so they're optimizing for that thing we should probably mention the two innovation type things that they're adding there they're not evaluated which is dynamic memory and dynamic contact stuff however Peter to your point while asking my assistant to create 3D castles and 3D cities or whatever Nissan created and they showed us last week a virtual copy of Toronto with 100 thousand agents running that's not what I'm going to ask my assistant to do [SPEAKER_07] building websites is kind of what I think many people would like their agents to help them with building websites one-off websites for like a use case or like more stuff as well so I think that this is definitely in the realm of something that people will ask their system to do Wolfram you have any thoughts on meta wonder on [SPEAKER_01] one hand that is exactly this what I've seen less technical people who are not as much into AI they use AI assistants to create websites or update their websites so that is very it has the colloquial listens and stuff so this is great yeah it's a good model and [SPEAKER_01] I think that they have their own agents that they are offering personal agents means that it [SPEAKER_07] we're to get there but cloud SDK looks to be backup what's [SPEAKER_01] always possible is to have your agent just use another agent I'm using all the agents like that that on my own system as my main and connecting to crockmod and the other ones and just taking control of them I've been using this for a long time even before the agent stuff came out but I was calling it [SPEAKER_01] minions Amy has her minions and she a website or crockbot the app can be controlled with computer use and [SPEAKER_01] it's just another agent for a sub agent for the main agent so very easy to set up okay [SPEAKER_04] I do it all the time in the terminal cloud calling codex codex calling cloud the variety gives you a lot for 100% I'm just wondering because these are these are different systems they're extremely powerful connected to many different services just the use case what are you using it for one [SPEAKER_01] use case is a council like GBT Pro which you can't just use the API and so on we are using it that way if we are creating a concept for something it can make cool [SPEAKER_07] we definitely should talk about use cases though so we talk about models and numbers etc but I think as we started to say in the beginning we're approaching the level of capability where for most of us the stuff that we need to do we can probably already do so go ahead [SPEAKER_05] one thing to add as well I think about the release and I imagine they probably just don't have enough compute but it sounds like I think they don't yeah I don't think so because why is the full one not available via API I don't think they're just being stringy right maybe they want to cross sell the subscriptions but that feels I [SPEAKER_05] came out and said well actually the full one is great the regular one that we got he thought was bad so I haven't tested the full one so my impression is very meh about the model but maybe the full one is amazing but we don't have access to it so I wonder if there's also that kind of mismatch that they think their model is amazing then the reception is like or is it benchmarks or what but actually maybe it is amazing but they mean a different model it's not ideal [SPEAKER_07] this is kind of on the same topic of folks let's say inside on Tropic they're probably not using opus they're probably using opus plus whatever model and so when they talk about the stuff that works for them in the public I'm not sure that they mean kind of the models that are available to us that's a very interesting point I think Tio also gets early access to the models that are not available in the API you can get Muse wonder within the product so I got to wonder if any of you guys used it within the product itself [SPEAKER_07] when I realized that I can only get access to this new intelligence essentially what happens is you know about this you have been doing this for a while essentially I realized that I [SPEAKER_07] Muse Wolfred my Muse to wonder and I gave it a lot of tasks here's use cases time let's talk about use cases a design and [SPEAKER_07] order it and shipped to the hotel and all of this was on Muse Wonder I was just kept shoving tasks to it and it was executing them mostly perfectly the failure modes of this model Peter is really hard to benchmark so we need to talk about how we benchmark those use cases but the failure modes of this model were not the logic parts the failure modes of this model was the website doesn't work the website is blocking us because cloud doesn't like agents or the stripe link doesn't quite connect or something so the failure were real world failures [SPEAKER_07] including a very funny ones where Muse called some businesses and business hung up because they don't want to talk to a personal assistant it was really funny so I think again I think we're approaching level of capability stuff for them and then the question is does it do stuff for them so I think when Meta released the models they had some evals in there I think we should pull them up that [SPEAKER_07] are not the standard ones there are some personal systems ones and Peter the question back to you is you are saying on the agent arena muse wonder is kind of suffering I gotta wonder if my experience with the personal agent stuff versus the agent arena where are those tasks falling where's the daylight between those tasks [SPEAKER_05] yeah it's definitely hard to say and I think it could the kind of failures that you are talking about they do sound more like the harness things like that so it's definitely hard to say one thing to say about the benchmarks and you mentioned different ones this is kind of a pattern which is [SPEAKER_05] where so many benchmarks just get quoted once because they get a good number and never seen again so I wonder I'm not accusing them of this in this case maybe they're meaningful benchmarks I don't know but that's also something to look out for is that quite often I would open the release and I'm looking at it I don't recognize even the titles of half of the benchmarks I'm in the industry so I don't really know how reliable are they going to stick to those benchmarks or not but [SPEAKER_05] if they're inventing new ones because it has a good score I think that's also not necessarily indicative of the performance [SPEAKER_07] that's true and I think to some extent this makes sense because capability let me try to find the exact thing you're talking about but I think the capability moves and different benchmarks need to switch as well right like we've seen that open EA decides not to do sweep bench anymore because sweep bench was in full and sweep bench verified pro and a bunch of updates to this paradigm but I definitely want [SPEAKER_09] to [SPEAKER_07] I love how co-host of Thursday sometimes joins as co-host with deep insights and knowledge and building things in engineering and sometimes they join as guests and just chill here and get their news from the live show is incredible Muse Wonder is their new model this is their opus the beats opus on multiple things so I think it's time for us to show some numbers how [SPEAKER_09] do that compare to Argon how they shoot [SPEAKER_07] yeah so this is the thing they did not do any Argon comparisons the Muse Wonder full and Muse Wonder on max are compared only to cloud fable 5.1 Astra 6 and cloud opus 5.5 and I think Peter maybe this is quite sure how that works out [SPEAKER_07] and on that index Muse Wonder the one that we don't get access to on Muse Wonder full is 59 and Muse Wonder is 54 beating pretty much everybody besides opus 5.5 max which other ones here Peter would you say [SPEAKER_09] wait [SPEAKER_07] hold on just one second Peter which [SPEAKER_05] a little bit so we've got GPT PDF Job Bench [SPEAKER_07] yeah GDP PDF I have never seen before Job Bench I have never seen before I think we're missing LTG here he knows about some of these I will say on the artificial analysis index which I think a new one is OS World for example we know and love right OS World is a benchmark that [SPEAKER_07] measures computer use and browser use and on this benchmark let's see Muse Wonder the one that we do have access to matches Fable and is significantly below opus 5.5 on computer use and even below cloud so computer use this model is not the best which okay so going back to what assistant need computer is like the number one thing that I need for my assistant it needs to go and click the right [SPEAKER_07] buttons otherwise it's not useful to me a very interesting set of releases here and you mentioned Argon we're going to talk with folks from Gemini about Argon in about an hour we have Omar back and Genuive so we're going to talk about that with them I think we're good on the news coverage or anything else left there Wolfram go ahead [SPEAKER_01] yeah since we were looking at the benchmarks as I'm creating Wolfbench 2.0 I have a crisis moment and I realized something it's so hard to benchmark agents because they are complete systems they are not just the model you have the skills and memories and all this and everybody using an agent has a different one even if it's the same system because of all the different personalizations connectors and so on even the internet access and the search providers they all affect the results and the thing is nowadays we are now at [SPEAKER_01] a point where you can have your agent create a benchmark for your own purposes and use the same system I'm doing it with Hermes so I can put the Hermes in the sandbox with my soul and skill and so on and use a model and do some tests specifically for users I have and I think everybody can now do this just have your agent build it you can build all the software in the now you can also build an evaluation I mean as someone building evaluations professionally that is a strange moment but it is really here now where you [SPEAKER_01] can have a customized evaluation for your own purposes with the agents you are using and test new models test new agents all of this is now possible which wasn't possible just a lot of work now now it's possible [SPEAKER_07] there was a talk at AIE that calls out this paradigm where you talk to your slab generator you're like hey how can I know that you're right and the slab generator says well I can do an eval for you here's a few things to test myself on and then you say okay do it yourself and so the slab generator generates the evaluations for itself and they're also slab and where are we if nobody reads those traces we're going to talk about reading traces in a bit in the cloud section this is muse wonder [SPEAKER_07] and the call out there for our open source folks is that the new spark model is going to get open source this is a new llama and I think that's one that's great I need to start using the word wonderful because it's confusing but I think that's great let's now or I guess again we're so back you can run the agent SDK on your [SPEAKER_07] subscription so this means that you can plug it into Hermes I see Wolfram shaking he said very strongly because folks I have told you before so all of this personal agent type revolution what I hold on so what I mostly need from this is that personal agent ability with opus 5.5 capability and now it seems that while I was able to pay API to do this with my cloud subscription so I'm definitely excited to try this out I've seen this [SPEAKER_07] in multiple coding harnesses and agents what else do we have and also rumored fable 5.5 in Azure coding which is a very interesting thing because opus 5.5 is kind of beating fable 5.5 for most stuff they're not like super super strong so let's talk about some of the anthropic stuff how is this [SPEAKER_07] one more [SPEAKER_01] there's even one more they posted an article about strengthening their defenses against distillation attacks so after clamoring for we have to regulate and ban this stuff now they basically did it by themselves increasing their defense they reduced the amount of distillation that happened to the reasoning traces and so on by 90 to 99 percent so it will be much harder to distill clot reasoning now [SPEAKER_01] and we'll see what effect that has on other models [SPEAKER_07] I know the effect that it has on me there's not a day goes by that my clods are disconnected I need to re-log in again and then the remote control thing fails and so from New York I can control my Mac I don't know if this is the effect of their security on the distillation but it affects me personally I don't want to call this out please let me use my clods wherever I am this is getting old this is really getting old all right let's talk about this let's talk about open telemetry which is kind of a nerdy [SPEAKER_07] subject but it's an important subject open telemetry is this open standard for how systems communicate their traces to each other basically the logs open telemetry has been around for a long long while and then folks like the we folks from weights and biases and now the agent lens folks which we're now launching in deploy preview this is how systems communicate with those platforms and now cloud itself the cloud harness is now supporting open telemetry what I'm saying is before you had to run something like [SPEAKER_07] a 1B hive mind which we told you about that shows you all the cloud conversations now cloud itself can send its traces to whatever tool of choice that you have obviously on this pod we strongly ship the core with agent lens product we just launched but anything that supports open telemetry cloud code now supports I think it's incredible and I think it's very interesting how what you just said Wolfram where they try to prevent distillation while also simultaneously allowing cloud [SPEAKER_07] cloud to send his trace in open telemetry though this was never a problem cloud was always writing the conversations to local right so you always able to get it on disk there were not encrypted the reasoning traces are encrypted but the conversations themselves aren't but I think it's a cool tool you can send your traces to observability stacks and this includes everything agent turns and tool calls and MCP calls and latency and token usage so you can properly measure how much cloud is costing you [SPEAKER_07] and if you're a company you can do this at scale oh there's one stat here from ramp they said this helped them to cut token use to 10 to 20 percent on some agents because they learned that some of their agents are running through loops will you guys be are you guys following your agents in any way are you evaluating them based on this way will you be using this from cloud code niston and jan both shook heads definitely going to pick it up [SPEAKER_09] yeah this sounds good especially for stopping loops and getting overall agent stuff i have noticed that cloud code does tend to stop often if you have anything to do with the reasoning now it just triggers it as a word like if you're working with any type of LLM like I've been working on web GPU the moment I even mention anything about reasoning now it just triggers it right away which is like for a small model but yeah [SPEAKER_07] still [SPEAKER_09] good though it's you can't beat Opus [SPEAKER_07] inside the cloud code harness I think it's still the best tool for overall tasks and doing some stuff Opus also helped prepare the show [SPEAKER_09] just Opus low if you need to do something just try all in Opus low guys it is just straight up vibing [SPEAKER_07] so I think because of this reason Antropic is also going to do an explosive IPO they reported an insane quarter 16.8 billion dollars about 45% from the second quarter increase adjusted profitable to the second quarter in a row and cash flow positive Antropic is profitable and cash flow positive and gets billions of dollars and going 45% from the previous quarter [SPEAKER_07] still it's absolutely insane this show definitely has no financial services things don't take anything we say as financial advice but if this IPO is coming are you guys putting some money in Antropic finally on the public market [SPEAKER_09] I think it's a buy as much as you know what I've said I've called Dario a communist but I think as far as a stock that will grow in my opinion this is just vibing advice it's just I think it's a buy I think they're way better opinionated coding than [SPEAKER_09] Astra but the actual quality of the code it's not that good I mean it's pretty good but it's not as good as Opus and when it comes to DevOps even Opus is bad and Astra is even below that I think they still have a lot of tricks and it's mainly because of Haiku honestly Haiku is the most disruptive model I [SPEAKER_09] that's not going to run out at 100 K whatever app you're going to serve to people if you keep it under 100 K context for Haiku which most conversational UX stuff is good enough that is more than enough and on the max plan that $200 that's probably going to support like a thousand users pretty easily [SPEAKER_06] 10 [SPEAKER_09] cents in one cent cash that's yeah it's just way too cheap that's why I think it's not the worst idea to get someone tropic [SPEAKER_07] get you someone tropic [SPEAKER_09] and it pains me to say that so [SPEAKER_07] let's talk about the subscription saga I'm going to put you [SPEAKER_01] now possible to do so I canceled my subscription with opos when or with entropic when you weren't able to use it without getting banned some people apparently got banned for using it and so that was when I decided I don't want the subscription if I can't use it with the agent I'm using I pay for this and if it's cloud code or it shouldn't matter in my opinion and now it's possible to use your personal agent whatever you use and connect the SDK [SPEAKER_01] and use opos as the main model in your own agent that way again in a legal way without being banned so now it's possible again I have to reconsider research So [SPEAKER_07] integration called sign with your chat GPT and then other products could use your tokens from your subscription right so Devin is one example from cognition it supports that use case from the get go Hermes agent was called out on stage shout out to our good friends from there and congrats on their round again and Hermes agent is now going to support both and this was always use case with OpenAI it looks like Anthropic needs to follow suit so now it's going to be possible Quinn Slack also a friend from AMP the coding agent [SPEAKER_07] said this is [SPEAKER_05] it's kind of annoying I don't know aren't they making enough money to just let people be I feel like there should be more freedom but yeah I don't personally I just don't because I've got enough stuff going on I use other models via Pi for example if I need to I find that more convenient and I'll tell you why is that I think with Anthropic with OpenAI they [SPEAKER_05] and to coordinate everything and I think it's just easier to use other models in other ways but yeah I prefer to just match the harness to the model basically and simply put [SPEAKER_07] harness to the model I less so in other places and that's why it's for agents like Hermes it's sometime to catch up IPOs were probably in November that's the thing that I missed so watch out for that and there's a rumor and we don't love rumors here but we will say it [SPEAKER_07] that Fable 5.5 showed up and I think I'm like I'm getting tired of rumors so I'm going to do this Anthropic is going to release Fable 6 also and then Opus 6 is going to come out and then Opus 6.1 or 6.5 is going to come out after that it's going to [SPEAKER_07] trying to snipe the news of something that we all know is on the continuous improvement loop all the time just tell me if that's interesting to you when you see somebody and just like oh I've seen this leak or whatever we know the models are going to get better we [SPEAKER_07] other than just knowing the news but given said all that somebody did spot fable 5.5 so I'm very excited to see how that's going to go comments did you guys hear my do you pushback do you feel like it's important to snipe and do breaking news etc at this age or are we getting a little bit of tired of model getting bitter numbers got better I [SPEAKER_09] want to see them run a full factory or something that's that rumor I want to see a whole bunch of humanoids running whatever model that yeah yeah the models are going to get better of course there's going to be all opus 6 I think Anthropic is [SPEAKER_09] actual humanoids running a factory running a hospital or something we gotta leave the rumors for more actual physical stuff yeah I agree that's not sure like everybody else it's common sense I don't know [SPEAKER_07] yeah Wolfram from an evaluation standpoint do you think that sniping the news and telling you this model is coming is still important or are we like past that [SPEAKER_01] I mean right now the most important thing for most people using agents is to have a way to use a subscription so if the new model comes out and it's API only or it's so expensive that you run out of your tokens as quickly and you [SPEAKER_01] agent are not need need to book me a flight or plan my vacation and stuff like that I think there's a limit to what we really need and everything beyond that is on a company level and not on a personal level anymore [SPEAKER_05] I completely disagree I think we're going to look back at this a year later we were so stupid and naive I think it's going to be amazing I get your point I model from [SPEAKER_05] Google for 8-10 months I haven't counted so they were only shipping flash models so I find it perfectly reasonable for people not to pay attention to rumors but I think there is information in it when companies actually ship so let's say if there is a scenario that Anthropic doesn't ship 5.5 Fable for another two months I [SPEAKER_05] I think it can go both ways which I don't think you have to have rumors for but I still get a little bit of information out of it so we'll see I leave 6.1 solar side because [SPEAKER_05] I think that was out of cycle a little bit but the next Astra you'd think that's probably another month away but if it's two months away then we'll get some information out of [SPEAKER_07] I talked to our fans in IRL they love this when we disagree we give multiple sides of the story and I think they actually do love this so that's great Peter and I do think that you're giving an additional insight there where now the Anthropic is going to be public in IPO whether or not they're going to release a new model actually will move the news move markets given how big Anthropic is that's a very interesting kind of thing so maybe we're after doing this for quite a while maybe we need to find the passion again to say oh this model is coming let's [SPEAKER_07] go but yeah we'll definitely get there so I yeah okay this week AI Engineer in New York this is the second New York AI Engineer conference for those of you [SPEAKER_07] has been curating incredible folks throughout these conferences so we went live earlier this year twice one time from Mosconi West with Wolfram during the in the middle of AI Engineer and then Peter actually interviewed you when we chatted in London as well so we went live from London Wolfram as well and this time for some reason they decided not to do it on Thursday but we're here in New York after AI what interests me [SPEAKER_07] about this specific conference is the zeitgeist is incredible because a lot of folks come there first of all to learn so that you [SPEAKER_07] Raw what it says now that the big companies are actually running agents and when I say now that the company is running agents to contrast initially out Tra deutsche was the year of agents in quotes and the zeitgeist was agents don't work this was before opus 4.5 [SPEAKER_07] this was the for the kind of the november january revolution just scroll back just a year ago agents did not do work and now all these folks are saying oh shit agents do so much work that we need to like put guardrails on them and do like how do we how do we deal with autonomy and models crossing the thresholds from non-coding work to like making decisions there's now agents inside banks there's golden data sets there's evals there's a bunch of stuff that the industry now contends with that [SPEAKER_07] they didn't have to a year ago because a year ago agents did not work most of the ai in companies were software engineers trying to get lines of code out now there's like agentic things happening inside [SPEAKER_09] most of these ones but two years ago i think i posted something about i've never seen an agent actually do anything useful other than plenty is buying crypto and that was uh that was only two years ago so just remember everyone's opinion is always wrong at some at some point yeah there was a tweet from yesterday i don't know what people are doing with agents no but also we're changing [SPEAKER_09] so fast completely flip one one yeah and this is what we're talking about when we talk about acceleration [SPEAKER_07] you can't be looking at the current state of the world right now and say hey ai is this you you have to start this is why we're talking about acceleration this is why i'm trying to bring in the news in the use cases because we kind of feel that this is happening so all the banks are now thinking about this very strongly not only ai engineers are spending tokens banks are actually using agentic processes and the favorite of many people i did like i said before uh we did like a little corrive event you can see [SPEAKER_07] this on my on my uh hand there's a liquid cooling irl event i found it really really funny muse helped me come up with it we went to a cold plunge and we called it liquid cooling uh because core with data centers um and then i did like a little survey uh shout out to the folks who came with me to our event and then many people said that max cannot alexander he is i believe he's a distinguished engineer in [SPEAKER_07] at capital one he was the code simplicity guy uh and he did a bunch of kubernetes stuff at google he's a fascinating dude to talk to his talk was the highlight uh he said like he talked about does code quality even matter uh given the background of this person i think it's very very important principal software engineer and open source back in google he led a bunch of the bugs stuff in google uh he basically says the code quality matters in proportion to how durable your software is [SPEAKER_07] uh because ai is about to make a whole new kind of software way more common disposable software just write once use it and then just throw it away so when you have disposable software does quality matter if it works if you build a calculator or an app for yourself party planner like who does it matter so this is one great uh show and i recommend folks to tune into the ai engineer live stream on youtube which you [SPEAKER_07] can watch if you want to the talks will get released the other kind of like hallway conversation was first of all about evals and whether or not agents are working um but as i said in the beginning of the show the thing that was highlight for me which i do want to ask you guys about as well the disconnect between how advanced all those ai engineers are in use cases and harnesses and personal [SPEAKER_07] agents with the revolution of personal agents for normies and how when i talk to some of these engineers they're like hey do you use instinct muse grogbot or whatever and i don't get an answer yes from them and that's really really something because many of the engineers just like build their own harnesses or use cloud code or codecs and why not use codecs for some of these stuff and so there's like a big disconnect which is very interesting to me specifically because how big and wide open claw and [SPEAKER_07] the hermes agent got and those are harnesses you can tweak and play with um so i did i did notice this in the halls as well and specifically this is highlighted for me because i did talk like wolfram does to every person in the world there is not an engineer wolfram talks to stewardess on planes to try to ai build them uh so i did the same i took a page out of your book and i talked with there was the token billionaires lounge we told you about before this a engine had this again so if you are spending more [SPEAKER_07] than one billion tokens a week or something you get this golden card i think i have it hold on wolfram there's a new card that you need to get the token billionaires card kind of looks like this and then this gives you access to this like lounge that this lounge is kind of like four billionaires lounge there were a lot of service providers in that lounge that were not part of the conference and i tried to talk with them here's the thing that i got from them not from the engineers all of them [SPEAKER_07] know about news because muse is getting pushed down their throats in every instagram post zak is really going out of their way so they all know about news many of them didn't install it or didn't get why it's why it's good for them didn't yet catch that uh so those conversations were fascinating um let's see what else so a engineer if you haven't been there the highlight for me and i tell it to [SPEAKER_07] everyone also if you're listening to the show you're good enough to submit a talk proposal to a engineer if you submit a proposal to the engineer you get access and wolfram and peter both can comment that this is true you get access to this where am i oh there we go uh you get access to this amazing speakers lounge where all the speakers that are hand selected are top of the level of people and you get also access to speakers dinner and to me the speakers dinner and the speakers lounge hangout is the highest [SPEAKER_07] tier of folks who understand whatever word i say you don't need to explain what harnesses you actually learn a lot one example that i can shout out is i just were walking by and somebody somebody's tag said linear i was like oh who's this dude oh he's the you know senior like i don't remember the full i think he's like manager software for linear all of linear linear is a agentic forward task management software [SPEAKER_07] and i had a bunch of feedback about linear we just sat down just right there and talked about how i use linear with my agents and what they can do to improve this is one of those conversations so if you are listening to the spot if you have anything to say i strongly strongly recommend to apply for a conversation talk at aie um wolfram peter any thoughts on that space it's the highest level of [SPEAKER_01] networking you can have in the eye space because you are not talking to some other attendees of a conference you are talking to the the big shots who are creating the air news we are reporting about well this is super super amazing and even if you don't become a speaker i mean just attending and going to the booth and talking to the people you meet there there are all the the people from google and entropic and open my eye they're all there so it's the best way to connect to the makers and shakers of the [SPEAKER_05] air space definitely best conference i know 100 um peter go ahead yeah now i would say also everyone can apply but do take time to make a good application because i can't remember what the stats were but i think swix was saying not for this one but for the one in london i think it was like the the acceptance rate [SPEAKER_05] was very low so i would say put effort in is it's yeah the the reason why it's a good networking uh opportunities because it is a bunch of good people so yeah it's it it's worth you know don't just lllm generate uh your your summary and proposal yeah take take your time here's an additional insight [SPEAKER_07] that i got and i think we've talked about this on the show but obviously this is on our mindset right so a engineer in london the zeitgeist were talking about should we still read code we don't write code anymore should we still read code i talked about the zl continuum on my way back uh i think it's clear now that barely barely anybody's reading code now we're like stressed how to even test it to make it valuable [SPEAKER_07] the zeitgeist here was the end of attention and how do we scale from managing one or two agents directly by steering them and seeing outputs to running multiple multiple tens or hundreds even agents and how do we have the attention to attend to all of that and make decisions or whether or not we just like let this swarm run and deal with this and i think it's a very interesting i talked to many many [SPEAKER_07] people there's not a lot of solutions ryan carson our by the way shout out ryan we hung out during this whole engineer our previous our co-host um me and ryan we were talking like a bunch of people ryan gave a great talk about this as well and he's missing a control plane he's missing like a one central place where he can like see all of the stuff that's working another thing that was in the zeitgeist is the [SPEAKER_07] cloud cloud cloud you can now hand off tasks to the cloud as well and they even give you like i think you said they give you some budgets like a hundred dollars to try the cloud thing uh obviously devin and cursor they have cloud agents um and folks are getting to the limit of their environment not only because it doesn't run 24 7 you need to run caffeinate to keep it running but also because the local environments are running out of memory and peter i think you mentioned this as well you're like coding so much in your things that the agents just like killed your computer so you got a [SPEAKER_07] different box uh and when you spin those up in sandboxes on the cloud you kind of don't have this restriction you can't go wide so like we're getting to a point where not only are we using more billions tokens more mike is definitely peter's case definitely wolfram's case all of us here probably like 10x for the past few months and as i guess in the engineer as well we also need the place for that all thing to happen all the tools to happen and that's not happening on local environment [SPEAKER_07] i heard um the engineer um hi folks next up we have omar sanseviro and genevive husk i believe uh they're joining us now uh just before that as always given the show is sponsored by one sports only core weave i would love to tell you about some of the stuff that we have and then we're going to talk with genevive and omar about finally we got the new big gemini back and it's so exciting to [SPEAKER_07] hear about this as well let's go to this week's buzz and just right after that we'll have our guests [SPEAKER_06] for we from core weave i think that it's time to sunset that transition and come up with a new [SPEAKER_07] transition wolfram because as of two weeks ago we forge is now the home for the weights and biases as a tool and also a bunch of other tools specifically agent lens agent lens launches this week and if the highlight here for me in agent lens will from tell me if you're also on this is that cloud code now supports open telemetry exports and then now you can use our tools to actually look at the agent [SPEAKER_07] runtimes that you have and analyze if it did the work correctly so um would would love to just like let you know that you can now use that uh you can sign up on the forge at coreweave.com the highlight for me from fully connected two weeks ago was that given the previous life at core weave where you had to get you had to go talk to a salesperson register for long contracts and sign up forge for the first [SPEAKER_07] time is our register as a regular sas offering and then you get access to a bunch of tools from our core agentic cloud tools like distillation as well we talked about model distillation a lot on the show but now via the open pipe folks which are great folks friends of the show you can get a model distillation for yourself you don't have to set that all up it's all part of forge.com um wolfram did you set up your cloud code with the agent lens yet are you sending hotel [SPEAKER_01] uh at the moment i i don't it's all local right now but it makes definitely sense to integrate all of this and the integration is what forge is all about to have integrations between all the different tools and the whole ai loop from the evaluations to the inference it's running on to the curation of the data that's it's all connected and that is a new thing i think it's not just weave the product or the individual product where you have to export import stuff you can jump between them [SPEAKER_07] and all um on account i want to call out yesterday yesterday we said at the event that we did we sat down and we talked about some stuff and obviously i was like okay this is a core we sponsored you know let me at least tell you about our offerings and folks who use weights and bias they didn't even know we have aria and somebody's like what is aria so aria is an author agent assistant agent research assistant basically auto research from corpathy personified in the [SPEAKER_07] product so if you are using which advice is for anything people didn't even know that we have this offering so we definitely need to work a bit about externalizing our offerings but if you have seen auto research as a concept from under corpathy in his suite it's now in ga you can just use it inside forge on your training runs all right folks i think this is enough about forge and now we're going to move on uh because we have not quite breaking news but definitely news from the frontier of ai [SPEAKER_07] with two great guests here and let's go talk to them hello hi omar and genevieve are joining us from deep mind from google deep mind and you guys have news to share and it's okay can i just say it's about [SPEAKER_07] time it's about time we talked on the show before and folks it's about time for a big model so you guys launched gemini for argon uh but i would love for you to tell us about the launch about what's new about what are the use cases i'm all ears it's out for everyone it's number one on arena and uh it helped make gemma three times faster or that's kind of your corner as well right i would love yeah so please give us a highlight and then we're gonna dive into use cases and generally great to meet you [SPEAKER_02] thank you so much for likewise thanks for having me okay yeah let's go i can go first so yeah i mean [SPEAKER_00] a gemini for argon is our latest model uh the ldr it's a great model you have probably heard many things about it my only recommendation is to go try it out and i think many of you will be pleasantly surprised uh yeah the model can be used for many things we've been using it internally uh as you said we've been doing optimizations for gemma we have been able to achieve over 100 a thousand tokens per [SPEAKER_00] second with the e2b so our smallest model uh the model has a one million unified uh context uh so pretty much you can pass very long videos but you can also have very long output so that is quite exciting uh and yeah we have been very excited to see what the community has been building with [SPEAKER_02] the models what do you think jenegu yeah we've had so much fun building with aragon and it had been a while since we launched a new model and so most people will know that this is the first model that came out in our gemini 4 series by far our largest train run ever and so argon i think is the first actually quite early checkpoint that already is doing really well but we're also super excited about everything that we can build next now and so argon is already training our future generations of models um i think we've really enjoyed the fact that argon can do many more long-running tests [SPEAKER_02] and so now everyone is shipping a bunch of stuff overnight and you wake up and you just wake up to really fun things that argon has been cooking in the background and so it's much better at 3d um much better at creativity personally the writing is really pleasant and so in general it's just been super fun to build things with model and again we think there's many more great things to come from us uh and in actually weeks yeah yeah go ahead or more please [SPEAKER_00] uh yeah just as usual we have been working with all of the open source ecosystem to make sure that you can use the model regardless of what's your favorite tool so we have been working to make sure that this model works well with open club with hermes with all of the tools uh uh sorry a blank chain uh versal and so on so super excited to also collaborate with all of this ecosystem to to get [SPEAKER_07] the model out there i want to say the release definitely is a white surface release right it's available pretty much everywhere and also in your own api in the ai studio shout out to us to you folks for obviously getting us in to the fold uh gemini up only for pro and ultra is that correct so folks who are subscribing to gemini app on the pro and ultra tier are giving this and onto gravity as well two dollars in and ten dollars out uh this is introductory pricing exactly yeah uh and then what happens then [SPEAKER_07] extension new model another interactive pricing um breaking news breaking news for for next month all right and then based on what it says now then it doubles one million token context and i think let's talk about multimodality uh google and gemini model specifically were always kind of like the front leading in multimodal including like watching the video that you send to it one million [SPEAKER_00] token is a great thing for that yeah you'll see that the model is very good at understanding visuals um the space and it has a very good understanding of the world and physics so many people kind of forget that models are not just for coding like models are to understand the world and the world is highly multimodal uh multilingual as well and in general for gemini we make sure that this model works well across all of these kind of use cases that's incredible let's [SPEAKER_07] talk about vision uh roboflow called this model is like number one in vision and detect ocr stuff as well very very very strong um any specific demos around vision that you guys want to show us any cool [SPEAKER_02] things that'd be though uh let's start with the ones that i have ready and then oh no you have anything else vision in the background uh the ones that i pick for today are mostly very cool looking but yeah it's great to see the vision capabilities always if you're building with a model and it can check its own work if it can record videos and kind of see what's going on then that is super super helpful so even for some hard 3d stuff that we're now doing it's just so pleasant that gemini can watch its own videos back even things like motion graphics is very helpful because [SPEAKER_02] it can look at a video understand the style and so i made a trailer about argon earlier this week that took me just maybe 30 minutes and it came out really good um but yeah in terms of the demos that we've been cooking the first one that i made was a desk racer um and so first of all if you go into google ai studio and you click on the gallery you can find a bunch of stuff that we've made with argon um so that's racer came from our multi-modality where i wanted to build a racing game on top of my [SPEAKER_02] actual desk and so the microphone that you see here and then this like cute little cat um just gave gemini some reference images and said okay just built me a racetrack um and what's fun about this is i mean it's a long running task to understand where to go around the objects to make it look good uh to sort of plan the lighting and everything um and so now if you can play this um and then argon even i can switch camera view and turn the sound off argon even prompted our omni video model [SPEAKER_02] um to extract sounds like engine sounds from different race cards listen to all the sounds found the sort of uh audio that was able to loop and then stitch all of that together and so like yes it looks good but it's also cool to think about the rest that's happening in the background to in the end uh come up with a very uh shiny uh result and so this one i really like because it's where i spend all day every day uh but it's it's fun to see argon uh do such a good job and so i built all these [SPEAKER_02] little other camera views in it uh and so this you can play in ai studio there's a timed mode where you just drive three laps and so that's pretty fun um and then this one just before you switch could i [SPEAKER_07] ask you for audience how do you approach building this demo is this all in the ai studio are using any other harnesses are using like the internal tools could you tell us how to get building this where argon can download the files then like using omni to generating things and like extracting where [SPEAKER_02] where's all the tool happening yeah that's a that's a great question and actually i wrote a guide about that earlier this week and so in essence started with the reference images um use grill me so like ask me questions about how i should plan uh this project and then just use a big goal command and said you know i just want you to spawn a bunch of sub agents work on it however long you need some things that helped were a um using adversary uh sub agents and so there was always one sub agent [SPEAKER_02] whose job it was to look at the outputs grade them for accuracy for lighting come back with feedback and then um give a rating out of 10 and then keep going until the rating was uh very very high uh so that first of all uh was really helpful and then secondly we've all really been enjoying using nano banana to make 3d even better and so in this example for instance i'll turn off the audio but this is a demo that looks really really good it's like a pagoda demo and so if you have a poorer render and you [SPEAKER_02] give it to nano banana and you say make this look really good and realistic and then you pass that image back to argon and say do whatever you need to do to make it look like this um i'm now using that in all of my uh 3d projects and so the lighting that you saw in desk racer uh the neon lighting was designed by nano banana because at first it looked very bland and so this i think is a really nice way to interact uh with any model just that image inspiration is a really good uh visual reference [SPEAKER_02] oh yeah i did write a guide about this so sorry if you look no no it's all good there's [SPEAKER_07] like a definitely great tip for our audience as well use visual references because the model is very good at multi-modality from other models for this that was incredible amazing show us another [SPEAKER_02] one so that's great yeah so this one is it's pretty cool because like yes it's fun to do things with blender and tree.js but in this case my colleague actually asked argon to create this 3d scene completely from scratch so zero library zero uh external things being pulled in um and what's cool about this is if you think about how a model needs to do that is it needs to think about all the light all the ways that objects might interact with each other and so [SPEAKER_02] the light bouncing off the wooden floor here needs to be refractored in a certain way the light shining through the glass needs to exist in a certain way and so if you think about the amount of real world understanding and then also the quality of code generation that you need to make happen to make a project like this uh that we also thought was very impressive um and then the other thing that was fun about this video i'll spare you the audio but uh argon also narrated it by itself um uh picked [SPEAKER_02] the the music with lyria and so again the idea that you can build something cool give it back to argon and then say now make this into a cool video that all just ends up working and i think in our previous models we probably would have had to babysit that a lot more and and give much more guidance on how things should look um and so another one that i think is just a cool example of what you can do [SPEAKER_07] um i want to i want to get back to here with omar as well and maybe both of you i mean uh the token output changed as well like argon is like generates outputs significantly larger could you tell us how this affects like demos of like this on other stuff omar would love to hear from you as well how does that improve people's kind of like out results specifically because if you think about kind of like evaluating models [SPEAKER_07] a lot of the times the folks are focusing on the pareto frontier corner where reasoning used to take a long long time and just like with more reasoning you get better results and then there was a little bit of a movement towards improving that so that the model will give great results but still not reason as much not waste as many tokens but this is not about that this is like about the model being able to generate like long long things so could you tell us about the outputs a little bit i mean this is mostly [SPEAKER_00] for long horizon or scientific projects where you do want the model to do lots and lots of reasoning and that's pretty much the kind of task where you would need that kind of context in the day-to-day most likely you do not need that level of output for input maybe right like if you want to pass a very long video or lots of pdfs you may need to do that but this is mostly for scientific or long horizon [SPEAKER_07] that's awesome and generally did it affect any of the work that you did um no i i agree with omar [SPEAKER_02] that in most cases like yes you could write one or several books in one prompt but that doesn't end up being super useful so to us it's mostly on the on the very long reasoning one project that i have been working on in the background but it needs a bit more work is one 3d project where i'm modeling 55 cities in one go and so whenever it sends off to sub-agents then they all start building many many cities at the same time but again in most engineering applications it's still much better to just do [SPEAKER_02] things in slightly smaller batches but if you have to think about a very long uh difficult math group for a long time then you don't want the output window to be constricted by the fact that the model already thought for a long time and so that's where we see most application um i think it is a very new thing also for us and so there's also some complexity with how this works even in things like external harnesses but in general the idea that the scientific community can use this is is very [SPEAKER_07] interesting to us yeah that is very exciting for sure uh so we talked about the visual kind of understanding and number one uh omar and generally i think you mentioned this that now argon is working on next versions of argon that suspiciously sounds like rsi and omar you're talking about argon training gemma models and that's like parent teacher not necessarily distillation but it's also in the process of training them give us a little bit more of that i think people do want to know whether or not this model can help train other models and how that works uh specifically around gemma work how much of [SPEAKER_07] this was autonomous or how much of this is like intervening uh could you guys talk to us about [SPEAKER_00] model improving our models in the future yeah i think just as we think about ai for science ai for math ai for biology ai for quantum computing we can think as well about ai for ai right and i do find this a very exciting area this is not this is something that we as an ecosystem have been doing for some time it's just that as these models are getting significantly better it's quite exciting to think about ideas so [SPEAKER_00] for example in the context of gemma what argon did was optimize bllm which is one of the most popular open source libraries for server side inference and it did a bunch of blm optimizations it found a bunch of areas where we could improve things similarly you could see how models could be held on co-designing training code or co-designing architectures uh so yeah uh do you have anything any last comments [SPEAKER_02] before we let you guys go no i mean i think yeah the sentiment in the market about gemini has has varied i would say please give argon a shot it to us feels like a very different model from uh the models that we've launched up to now and so i hope everyone gives it a creative shot because there's so much i think that you you can do with it um for us i think it's the start of a new era without sounding to commercial but yeah i think this is a very exciting first step to what hopefully is that it's going to be a couple of fun weeks and months we've been waiting and we're now very [SPEAKER_07] excited to have it back and like to see that google is back so folks thank you so much for joining us generally nice to have you on the podcast uh feel free to come back with awesome demos which we always love omar as always from the pod for a long long time thank you guys for joining us and congrats on the release and we know that you have a lot of stuff to catch up on during the release week uh we'll see you here next time cheers thank you all right folks from one conversation to another [SPEAKER_07] now we're going to chat uh now we're moving forward and i want to introduce you to nathan benyash and let me bring peter back on i i followed you for the longest time but i think we've never talked like this on video and i followed the state of the art report so let me just let me just do this reading uh folks the state of the eye report turns nine this year is that real nine years i feel like i've been doing [SPEAKER_07] this for three and a half years and this is a long time uh nine years uh the word ai was not familiar to any like many folks back like nine years ago yeah it was uh it was the era of like ai dash ml and data science if you remember that and uh this year's 20 to the uh 244 slides a scorecard on last year's predictions uh would love to hear about last year's predictions and whether or not we can hit [SPEAKER_07] them and nine new predictions including agi 27 nathan benyash the founder of air street capital is with us the author of state of the art report um nathan welcome thanks for having me uh peter also joined us for this interview peter thanks so much for making this connection uh i want to talk about some like the highlights some favorite numbers i have cloud led 27 of measured model r d and anthropic in august [SPEAKER_07] we just talked with folks from deep mind about argon and like rsi whatever um would love to hear from you how this kind of like shapes takes shape between last year and this year people do talk about rsi being a thing people do talk about models training other models whether or not we're going to hit this like i think uh what are you seeing from there what are the what are the reports says about rsi and [SPEAKER_08] continued improvement by the model yeah yeah yeah i would say like the the major story in the last 12 months has been has been rsi and you know super intelligence to move from agi to super intelligence uh and then you know increasing evidence uh you know at the frontier that shows like we're kind of making progress towards this idea of like ai building ai i think um what if like the two the two experiments i found most interesting there one of them you cited which is just the general stat of like how um [SPEAKER_08] anthropics like r d team when they're doing core uh r d work on the model uh so 26 of uh of like the directions in research are being guided by claude and that's up from basically zero a couple of months [SPEAKER_07] ago as not to oppose this but i think the code generation stuff that i remember maybe you have an updated one but as far as i remember like 80 or 85 percent of code is generated by cloud maybe not [SPEAKER_08] directed so yeah yeah so now yeah that was like a few a few months earlier you could see like a chart of like code commits that were that were basically accepted by engineers uh as a function of like how models have been changing and it was sort of like flat for a while and then like the mythos class models came and then now it's like 8x the the average and you see a similar stat at open ai um and then the other uh the other data point that i like that we made a slide on is this this like um experimental setup where a [SPEAKER_08] human researcher engages on a on a research path um kind of goes through like several leaves down like the node of that research path and at some point the path forks and then one of the fork is a model that continues the research and the other fork is a human that continues to research and at the end of like both of those leaves uh human rates uh which one was better and um and you saw that like the old old school models were definitely not better like i think 10 of the time humans preferred them but then [SPEAKER_08] the mythos class model it was like 60 of the time humans said the mythos uh suggestions were better than humans so i think that's like also um it's also interesting and it gets to this point about scientific taste um which i think is like on the on the tip of everybody's tongue like no plan intended it's scientific taste yeah yeah just just the idea for for for those who are like less familiar [SPEAKER_08] it's like um i did a phd in biology and like what i remember were like the great papers were the ones where you would the grad student or the postdoc would uh would be uh running a couple of experiments and then every week going to their supervisor and showing the experiments the supervisor who would look at the work and you know kind of mull it over and say like okay the next the next best thing you should do for the story is this and i think the best pi's would be the ones who would suggest experiments [SPEAKER_08] or ways to interpret the experiments that you did and the results you got in a way that you thought was like novel and interesting and elegant and then the bad pi's like do that very poorly and then you see this in the result of a paper where like the story kind of reads beautifully it's very logical like each experiment adds additional signal that the previous one didn't have and i think that's like what people are trying to [SPEAKER_08] encompass in uh what they call taste so can a model like approximate that [SPEAKER_05] yeah i think there was one more from prime intellect i think where it was the non-gpt right where yeah what what sort of did it almost get to the human level right in yeah but i think there was something that it was still narrow the the scope of the work that it was there yeah yeah there's a like to get [SPEAKER_08] into this more what i there was um a study uh from open ai in the report where they looked at trying to categorize like the the token usage of their uh of their engineers and researchers um and um and what they did was like identify some tokens that were about uh helping decide the research direction what experiments should you run to test that research and then other kind of category of tokens which is [SPEAKER_08] like implementing those ideas like designing the infrastructure engineering the infrastructure during the data pipelines and doing the data analysis and just writing the results of the study and the vast majority of the token usage was on just like the raw engineering or just like get the thing working and then writing about it and very very small amount of tokens was on what research should i do in the first place and like how should i test it and um and so mine i think there's different [SPEAKER_08] ways you can interpret this but but one of the interpretations to me is there's still like so much more to go on improving uh you know the use of models for like ideation in brand new ideas brand new areas [SPEAKER_07] um you also gave yourself a scorecard can we talk about that from last year's predictions and misses uh what stuff did you hit correctly what's that or like the report hit correctly and i'm assuming that's you as well uh what stuff yeah you miss yeah i'd love to hear um yeah like the stuff i got right was um [SPEAKER_08] let it be a push back from big labs and neo labs into open source and this is and um in the us in the west and the idea with that is it was very clearly part of the national ai strategy from the trump administration to sort of regain supremacy of um of american ai and like open source as a way of doing that and very soon after we published that report like reflection ai sort of went from the coding agent work that they were doing towards saying we're going to publish an open yeah open weights model and [SPEAKER_07] then nvidia supercharged all that um by by basically i think it's rumored i'm not sure if this is quite there yet but by saying that nvidia is going to maybe like get reflection in there after getting hug and face in there also this year so yeah you're kind of and full side and full side and other [SPEAKER_08] potential ones and uh so you're bang on on that as well so that one and then the second one was um on nimbyism for data centers idea like not in my backyard like i don't want to have a data center next in my garden um that this would be a real problem in the yeah you know hampering like the build out data centers but also influencing part of the midterms which we'll see in the next couple of weeks um we substantiated this stat with like a survey that we found where americans say that they're much [SPEAKER_08] they did much prefer living near a nuclear power plant than a data center which is uh quite wild wild [SPEAKER_07] yeah we covered this on the show as well and we still can't believe it it's it's just the amount of news that are going on and just people on the street talking about just ai boiling [SPEAKER_08] the planet uh yeah yeah yeah and it's not just like in the us it's certainly happening in europe and in korea and other places where you know there's been like a small proportion of the population that's benefited like enormously from ai and many people who haven't and um you know there's some social issues with that uh i think the last the last one that i got right was that um you know chinese labs would overtake us ones uh in some major leaderboards and you know i think it depends on you know which [SPEAKER_08] which timeline you actually look at and peter probably knows this by heart but um but certainly you know kimmy and glm and others have done really really well yeah what about sound the misses yeah yeah yeah well um there's been so much talk about like ai sovereignty which could be interpreted in different ways but broadly speaking the one of i i as a country want to have independence [SPEAKER_08] or control over my own destiny when it comes to uh building models um training them serving them having the infrastructure for it and the energy etc and um and my bet was there would be this concept of like you know i'm ai neutrality in that kind of mirrors like uh uh sort of mutual defense um agreements where a country is like too small to be able to afford uh you know its own military or its [SPEAKER_08] own defenses in the same way it's like too small to be able to build its own data centers and have people build models etc so it just forms like a strategic defense uh you know agreement uh or ai um provision agreement with a bigger neighbor or an ally that says you know i will i will i'll promise you to give you all all the ai in exchange for something else um i haven't quite seen something like that yet there's spent a bit of a move with a like u.s pack silica um policy that um [SPEAKER_08] that tries to like bring um a collection of countries that have uh access to core um parts of the chip supply chain whether it's like minerals or fabs or lithography or whatever um but but overall the ai neutrality thing hasn't really taken on just yet no no yeah um and overall i'd say the predictions are like directionally well like there's a guy on linkedin who like looked at the last like nine [SPEAKER_08] years uh and then tried to see like over that nine years which report what proportion of all predictions eventually happened it was i think like over two-thirds and so sometimes it's just a question of timing more than just like this is ludicrous well that's what i like about your [SPEAKER_05] prediction is that i think it's quite easy to make predictions that uh you're gonna nail like 90 percent correct you just kind of predict roughly what just happened oh yeah tropics is gonna make more money next year you know it's like okay that's not yeah it's not very interesting so yeah i find it interesting you've got the hit rate of like 50. yeah yeah so is it like how is it difficult to to come up with the predictions are there some that you you kind of decided against i don't know too wild that [SPEAKER_08] that you didn't end up making this year yeah um there's one i couldn't quite formulate around um around uh one is like cyber security and like can ai somehow disrupt um encryption on the internet as it might be uh as that might be accelerated by the fact that ai solves math so i don't quite know how to yeah basically i just ran out of time for like thinking about how to formulate it properly [SPEAKER_07] to be able to defend it but i feel like let me ask you let me ask you there uh maybe the the question will formulate this for you last week we talked about uh the the math manuscripts dropped and there i don't know if you noticed this or saw the conversation about this but the the numbering there is not sequential between 44 and 46 there's a missing 45 and there's a few of those and the rumor mill [SPEAKER_07] says that hey those are the manuscripts that talk about um elliptic curve cryptography and that being a lot of meth and there's a supposedly they're they're not able to release some of this because the government kind of steps in and says hey anything regarding that do not uh what's your take on that [SPEAKER_08] i think it's believable yeah you know why in the same way in the same way that you know like that spy thing went two days ago like [SPEAKER_07] that's probably just the tip of the iceberg can we just talk about this for a second could you say like what do you mean because many folks have like maybe missed this piece of news yeah it's become a bit of an [SPEAKER_08] internet meme mostly because of the photograph of the guy peering behind the tv but um but like at least from what i from what i read there was uh you know an individual an individual was employed at open ai who um uh was basically found to have planted like a listening device um behind a tv in one of the meeting rooms in the open ai headquarters that he tried to deduce was ones where like key executives would have meetings because like he had somebody like poll pa to like ask which meeting room would be relevant [SPEAKER_08] um and then there was like a device that was like back to the tv like and it was wrapped in toilet paper uh and apparently he's like he left some like trails because he was a bit amateurish about it and then they found him and reported to the fbi and um and so i know he's even in custody and that's that's the one they caught basically yeah that's that's that's what they disclose um so that i guess is like directionally towards the prediction i had of like someone's gonna like exfiltrate model weights [SPEAKER_08] i said it would be like ai driven but turns out like humans could be human assisted when when you [SPEAKER_07] predicted that were you talking about someone someone being human or someone a model being like you know copying its own weights out where where were you there yeah i thought it was more going [SPEAKER_08] to be the model like more sci-fi than yeah actually just the prosaic like a guy walking in and [SPEAKER_07] downloading it yeah gotta be very interesting if if we were even be able to tell if the model is smart enough to do that will it be not smart enough to not hide its listening devices in a bunch of newspaper [SPEAKER_05] behind the tv yeah can i can ask you so you speak to all kinds of people and we're a little bit in the ai engineer kind of bubble like what sort of things that people don't ask you about because you've got different parts of the report about even going into politics and things like that what do you what do [SPEAKER_08] you hear people care caring about um there's a segment of people who really care about what we discussed like the the rsi like rapidly kind of accelerating progress like what does that mean for the rest of us basically um there's there's some who um you know are just like increasingly like [SPEAKER_08] uh pessimistic over like software companies still because um you know now like if you're going to believe that rsi is coming and basically the the quality of the software you can produce or the work you can produce is like really limited by what model you have and how much money you're willing to spend on it then um there's certain people that will just like organizations that will accelerate way faster than others and if you're not part of that like what do you do um there's always like chatter [SPEAKER_08] you know every two weeks or something i think now we're in one of those negative cycles around like is there too much compute circular financing like how is it financed like is the debt going to be able to be serviced and you know there's a bloodbath in your clouds in the last like two three days because we're just in that part of the cycle um um you know it's always always debate around like will the labs monetize how will they monetize or these like funky financing structures with multiple [SPEAKER_08] stages at different prices all within a couple minutes of each other is that like a sign of i don't know just like rabid demand and not very much thinking um how about normies i have to ask [SPEAKER_07] you nathan how about like just regular folks on the street and how they view ai do you track that instead of yeah do you talk to them how do you see that opinion shapes so would you talk about the change in the public opinion about water use etc uh what about the kind of the use case for yeah what do you see there what do you track there the rise of personal agents is now upon us crockbots and mules and all of those do you consider those in the report yeah but that's still like in [SPEAKER_08] the small minority of tech people who know who know what it is um you know i i would i would still bet like 90 of people are using like a model to maybe at best like read their email or answer to their email or you know ask questions in the same way they would ask google but they probably never click the work button like what does that even mean like do i want to open a consumer app and like work yeah not really you know and most people don't even know you can swipe left open codex when it [SPEAKER_08] on your machine and like talk to your machine and make it do work for you so like i think that's so far away which is one of the reasons why like i've just started saying like i think these big labs should open like a genius bar or like an actual physical store and just like start walking you know regular people through the magical things you can do with ai that this is the kind of also the geek squad [SPEAKER_07] genius bar comment right like exactly they need hand holding and imagination unlock um yeah that's what we're trying to do on the show though the show does cater to ai engineers and so if you're still listening to this uh you are in a very few i don't know if you saw the carpatic tweet as well about the he kind of talks about this as well yeah exactly when he segments the world exactly it's [SPEAKER_08] it's they were still very much very much like that um and there's an argument that like that gap that gap expands even further if like capabilities get better and rsi happens and all that stuff you can see it a bit in like the um you know the ramp data that we show with the 70 000 us customers that they have and tracking how much money they spend on models it's like you know the average spends like 12 bucks per employee per month and the top one percent is like seven thousand five hundred roughly [SPEAKER_07] speaking so it's like 580 times difference between the potential permanent underclass if this continues yes right and and the top spenders the billion token ones nathan thank you so much for joining the thing that i do want to ask you because you've been doing this for a long time again from ai ml days to then generative ai to very small context windows like huge ones we've been doing this for three [SPEAKER_07] years and it feels like incredible time you've been doing this for for for much much longer um your personal beliefs and whether or not your pdoom stuff i don't like the pdoom economic culture about the bubble stuff i would love to just hear from you how do you see your views change throughout these years [SPEAKER_08] if you can yeah about this for a little bit briefly yeah i i think the major also my pdoom is very small i'm not like a big believer in x risk um i i do think along that angle that cyber security is very precarious um and should be taken like insanely seriously but i think that's different than like existential risk of like destroying humanity um i think the the major change i've had in the last like [SPEAKER_08] two three years is i've generally believed that you use the right model for the task in the same way that you know you don't drive your ferrari to go like the coffee shop or maybe you do to show off or something but it's like not the most reasonable thing to do just like walk um but i think there's like a weird behavior with like with like frontier intelligence if you if you have like 200 bucks a month to spend and you value the quality of your work do you really want to like take the risk of [SPEAKER_08] putting the slider down to low and like using a cheaper model on the chance that like the quality of the work is worse so while i could switch for like emails or whatever i just don't i'm just like [SPEAKER_07] on astra high like who cares you're answering emails on astra high yeah then yes right [SPEAKER_08] not all my emails but like the ones we have to think about it like yeah you know why would you take the risk using a shittier a shittier system um or potentially shittier system so um yeah so there's that and then like you know will we ever make money on frontier intelligence and i think it started to show that like we will because the value of the tasks that they're doing is increasingly high and just the more people from all sorts of different parts of the technology world who come into working in ai [SPEAKER_08] bring all of their knowledge and optimizations to to the fore and just you know human ingenuity plus lots of ai equals more progress so i'm like quite um quite bullish on like margins improving and you know i don't know what anthropic is adjusting when they report their adjusted numbers but um it's you know it's probably not like community adjusted ebitda like we work like i think it's a serious company so um um so i'm i've changed my mind on on that like i think you can really make money on frontier ai and [SPEAKER_08] i'm increasingly convinced that it's the the open source versus like frontier labs like um ios versus android it's sort of cool for a while to play around with your android and mod everything and some people want to do that but yeah at some point you just want product that auto updates is like reliable connects [SPEAKER_07] with everything and it looks nice so all right those are great stuff uh folks nathan benius thank you so much for joining us on thursday i will always welcome you back after the next one as well [SPEAKER_08] are you planning to keep going yeah yeah yeah yeah well i'll keep going i mean you know if agi 27 happens and i'll be on the 10th year anniversary and then i might be able to stop but uh all right yeah in the meantime you can find everything online it's all free at the state of that it's a [SPEAKER_07] state of dot ai yes it was a great you're all nice and thank you so much for joining us yeah thanks guys bye bye um that was super cool i don't we don't often we don't often get somebody here that does this longer than we do uh and ldj join us as well uh this is great nathan uh peter thank you so much [SPEAKER_05] what's your take on i the the most the the very cool thing about this is that there's obviously i i find this really helpful to keep up with this stuff every week but there's something really valuable i think that to just look back at the year and you're just like oh i completely forgotten about this or i didn't actually realize there were papers there that maybe didn't quite make the splash this week so we skipped it because you know there's too much to cover but actually they turn out to be quite important so [SPEAKER_05] i really value his work and i've been reading his stuff for years now i can remember was the first one that i read but it's just so useful to just look at it and and just read the stuff so i really recommend just book i don't know an hour two hours in your diary and just scroll for this you don't have to read every single thing but just scroll for this i know it's like 250 slides you can do it in an hour quite easily don't don't don't ask your agent to read this don't ask your agent to read this [SPEAKER_05] just read it read it put some effort in no it's worth it uh so i will just ask other jared thank you [SPEAKER_07] for joining us niston how's your 100k city going are you still ai build and peter what new and interesting did you start using or or building this week just whoever wants to jump in let's just chat [SPEAKER_09] hell yeah there's uh i i got i got some some crazy stuff coming up for uh release um it's uh yeah all the entire decompilation of games and stuff is enabling fallout simulations can we [SPEAKER_07] talk about this decomposition so ldj we mentioned this like last time that people are putting things in other games and now there's a whole decomplation thing happening somebody told me somebody decompiled or reverse engineered the airdrop protocol what the guys it is crazy it's like a renaissance happening [SPEAKER_09] right now right yes all right here's here's what's happened the entire internet the last two months has just gone completely nuts by using like the nsa hydra protocol which allows you to like reverse engineer any piece of software and then there was also the law in 1996 from the phoenix bios or abc i might be wrong on that that allows you to actually provide like visually functional uh pieces of software [SPEAKER_09] as long as it does not violate any trademark so you see a lot of these things called photocraft grid craft and stuff which are like photoshop or excel builds that are done in room builds yes yeah so those are mainly done in rust there were also crazier builds on the on the game side so if you think that is cool and it is pretty cool on the game side people were able to reverse grand theft out [SPEAKER_09] of five and they were able to run it on web gpu and i'm not sure if i can show it and stuff because that one i don't know it has all their brand logos yeah and stuff but i tried it on the phone i tried it like on the oldest m1 machine that uh that i have and it ran what used to be a 100 gig game that you [SPEAKER_09] had to download uh it just ran it perfectly at 2k resolutions via web gpu the models have just gotten crazy good at the optimizations that come with web gpu because now you can do instead of games being full 32 you can do float 16. so you have fallout a tier games like playstation 4 or playstation 4 playstation 5 games just running fully playable very smooth very visually nice directly on your gpu [SPEAKER_09] and uh yeah so they're taking advantage of all of all of these new optimizations but they're also using the game assets that that were published before so you shouldn't be running this if you don't have a an actual copy for example of gta stuff but yeah it's a complete renaissance in uh in apps it's going to be a very interesting next three months as entire countries and economies figure out what kind of office [SPEAKER_09] suite do they need to run uh and the very interesting thing is that actual professionals and creatives tend to like the vibe coded versions more because they're so much more portable and they're fun yeah and they're fun again they're not perfect for example in photoshop the reviews were that the magic one tool is not very good because it used like an older machine learning model to remove the background that's going to just be fixed in the next like two weeks the second he turns his slop cannon towards [SPEAKER_09] that issue that issue is solved yes that issue is solved yeah they're just they're throwing entire gauntlet prompts and they're i've seen entire apps the entire adobe suit in design after effects uh lightroom uh all just running in the browser very performance too and you can run them without any telemetry or [SPEAKER_07] crap or spyware without adobe paying also a bunch of money to adobe listen i i do have to pause you because ldj just bars breaking news uh let's go ai breaking news [SPEAKER_07] coming at you only on thursday i all right folks we have quite exciting breaking news l did you want to take [SPEAKER_03] this one sure i can so uh yeah i i do feel like overall these past few months we've had a bit of a renaissance of of western american open models and meta just announced uh literally within the past 20 minutes that mu spark open is released which i presume is maybe like partly censored version of mu spark [SPEAKER_03] it's 816 billion total parameters 42 billion active parameters so this is around three times smaller in total parameter size than uh than kimi k3 while being approximately half of the active parameters of kimi k3 so you could expect it to be a lot more practical to run a lot faster i'd say probably in terms of [SPEAKER_03] parameter count of active and totals probably most fair to compare to maybe the full glm 5.3 and i think it is maybe beating glm 5.3 though or at least trading blows with it in the benchmarks so i guess we'll have to scroll down or kind of wait for people to benchmark things yeah let's take a look let's take [SPEAKER_07] a look they're comparing themselves a muse park open to inkling to nematron to beam to mistrial and only then to quen and glm right so like they're starting with what is it inkling thinking machines oh yeah uh so look they're literally saying hey we're comparing ourselves to american counterparts first then to european conference only then quen and glm and kimmy yeah i i think what's interesting if you [SPEAKER_05] scroll down into the coding area compared to the chinese models so we've got the terminal bench 2.1 so the muse park has 76.4 then qan 3.8 has 82 and glm 5.3 has uh 83.9 kimi 85 so it's like 10 points behind [SPEAKER_05] no well not quite 10 points but like seven and eight points behind them deep three as well but basically behind glm and kimmy k3 which by the way been out for what a month two something like that yeah so yeah definitely better than the us models but not necessarily at coding this is the the top [SPEAKER_07] jet chinese models yet yep um it doesn't look like they were going to break soda one oh somebody says yeah but they're like three exercise right so glm and all those are like in the billions of tokens [SPEAKER_05] i think glm not so much right isn't it isn't it like yeah it's quite large here let me find the exact [SPEAKER_03] parameter count real quick here uh because flash is around 300 billion parameters uh yeah okay so total parameters for glm 5.3 the not flash version which they're showing the not flash version in that chart it's 753 billion total parameters so that's comparable yeah and 40 billion active parameters so almost [SPEAKER_05] the yeah very very similar very similar very similar and gets beaten well it's uh glm is better at coding like quite a bit yeah quite a few points so so yeah i think it's well what do you guys think is this what was the relationship to spark uh muse spark through uh 1.123 [SPEAKER_07] oh that's i'm assuming that this is like a cleanup version to some extent yeah yeah but they're not talking about this but yeah muse park from 1.1 to 1.3 did like quite heavy jumps um and this one is a multi-modal text image video and audio output is only text with one million at the inference context window and a great vocabulary size i think a comparison could be made between kind of the just the abilities right and then they see like whether or not this [SPEAKER_07] is like one of the muse parks uh this is super cool llama is back essentially is what we're saying right this is like the first release from meta for a long time that's in an llm space nisten [SPEAKER_09] thoughts excitement just i'm just going to i just need to try it [SPEAKER_07] the weights are available to download uh the model is available in bf16 and f pretty big and vfp4 to run on your system the model requires a gpu with at least 700 gigs of vram and vfp4 and two terabytes for bf16 it also requires an insurance deployment framework supported frameworks sg lang vlm hug and face and llama cpp we're definitely gonna chat with our inference folks to try and to provide this inference for you because we do believe this is a good model [SPEAKER_09] yeah yeah yeah yeah it's especially for computer use i think this will be pretty key right now i'm just reading through the license it just asks that anything you do with it it should say built with muse spark open in the interface of each product or service that uses the software or any derivatives that's the main license and everything else is like an open usage policies [SPEAKER_09] and no violence and drugs and stuff um or not impersonating and this info yeah otherwise it looks close to just like an open license there's no restrictions on like how much money or anything like that uh this is yeah oh no there is a restriction where the third-party outputs of such models models of service and the aggregate gross revenue of the licensee and its affiliates exceed 20 million us [SPEAKER_09] dollars equivalent during any consecutive 12 month period so as long as you make less than 20 million dollars a year you can just offer it uh yeah and then probably a license yeah those are the things but this is amazing because it's uh it it will be great for generating data and stuff yeah i'm pretty sure companies will offer it it's probably one of the best computer use ones that is so cool let's go i [SPEAKER_07] wish i had my my my arrow button folks llama is so back and the promises of uh space ong where they release a new grog the old grog will be open source that doesn't often come to fruition i don't remember the last time grog was open source uh zach and alex wing did say that the the open source is back after the manifesto and everything and so now they're after the releasing muse so two two huge news from [SPEAKER_07] meta this week right they released news wonder and wonderful and that's folks are split on that a little bit and now they release the open source version of spark in new spark openness 816 billion parameter 42 active billion parameters we don't have the club buttons here also the applause but like every lab that releases stuff with the license that's open we applause uh and this was our attempt to adjust the conversation the breaking news game uh there was a bunch of other stuff this week that we didn't get [SPEAKER_07] to including the voice so grog voice released and then tiny fish as well um let's see what else is interesting here um you know i think it's pretty much it we've covered most of it besides the oh and multimodal jev didn't get to i really wanted to jev is now multimodal the original jev not the 700 open source or so clones including the one from microsoft that's called this decision one including the one from open ai it is called open ai decision api uh jev the original jev from type [SPEAKER_07] safe ai that we told you about what a month ago that's it that's only a month since jeff came out yeah pretty much literally a month ago um jev is now multimodal so you can look at images and return bounding boxes uh we also told you about gemini argon which is gemini 4 and we had both folks here genevive and omar sanciro from the developer experience team at deep mind to tell you about these models as well coming to you live from new york ai engineer the conference had a lot of folks [SPEAKER_07] who listen to thursday and a lot of talk about the type of stuff that we struggle with including attention including how to run multiple agents including when to sleep and how to sleep when you're just about to send something to an agent and it can reply and wait for you the whole night this is also a part of the conversation as well uh signing off from here uh if you missed any part of the show [SPEAKER_07] the show comes to any podcast app that you use apple podcast spotify etc and on youtube including shorts uh if you haven't subscribed please like and subscribe um if you have help us on other platforms as well it really really does help for exposure uh wolfram go ahead okay i just wanted to say the [SPEAKER_01] news are coming so fast but it is now the time where it is the easiest to actually implement implement the stuff when we started with the show and i saw we will be talking about croc voice i told my agent put this in my my voice app and it's now running on my phone already i can talk to amy with the croc voice and it would be cool if the audience would also um tell us next time what of the last show did you [SPEAKER_01] implement yourself if you have your agents what is the stuff you are actually using what what about the things we reported on are you using now this is super interesting to us uh not just reporting on the news but actually the uses of the news items what our audience is doing that would be very interesting to hear [SPEAKER_07] yeah please send and comment and tag us on social media it's gonna be great uh with that we're signing off folks thank you so much peter gosteff wolfram ramell of nistan tahira ldj young bellag was here before and our three guests nathan benyash genuiv and omar sanseverro from deep mind and all of you for tuning in for listening for sharing for liking for sending subscribes and sharing with your friends it really means a lot to us especially if you come up and tell us that you're listening thank you so much and we'll see you here next week bye bye everyone