Inside Shopify's Plan for Agentic Commerce and AI Shoppers | Andrew McNamara, VP of Applied ML, Shopify

What are you noticing in terms of agents and how that intersects with commerce? One of the or a couple of the core shifts we're seeing is that shopping is just fundamentally changed. Agents are a huge part of that. It's like the new front door to commerce almost. On the AI commerce side of things, it's growing like nine times faster than social commerce and nearly three times faster than mobile did at this stage. We're in another generational shift in how people are shopping. What are you excited about the future of what's to come? This is unique to me, but I love... Andrew, welcome to the podcast. Thanks for having me back. When did we record? Like three, four months ago? Something like that? I think so, yeah. Yeah, four months ago probably. And it seems like a lot has changed in that time. We were having a bit of a preamble before this. There's a lot of new features, specifically within Shopify. And then we were talking about the larger kind of macro ecosystem of what agents are now enabling as well, right? Yeah. Yeah. I think... I mean, even in general, the last six months since December when like, you know, the significantly updated Opus models changed. And, you know, we just had like a huge, huge shift in how people were using these models in terms of coding and development. And now we're seeing that same shift kind of hit, you know, shopping and merchants. So it's pretty... It's, yeah, totally different world than last time we got it. Yeah. I want to know more about that from your perspective, right? Like for me, the kind of word of the moment for the past six months as agents. I was actually just down in Florida doing a presentation for a HR conference on everything that's possible with agents, right? And people have many different experiences. You could have an OpenClaw or a Hermes agent. And I know that you've got some like personal anecdotes that you can talk about with this. And also like you can have agentic experiences within Claw or OpenAI and like different tools that you're using. But I'm wondering from your perspective, like on the Shopify side of things, what are you noticing in terms of agents and how that intersects with commerce? Yeah, I think one of the... Or a couple of the core shifts we're seeing is that shopping is just fundamentally changing. I think, you know, we're going from like retail stores to desktops. And, you know, what is the surface of tomorrow that people are shopping on? And I think the people, the way that people are finding ways to buy are just multiplying. And I think agents are a huge part of that. It's like the new front door to commerce almost. Like it's just, it's been crazy. I think one way we think about it is that like anytime there's like a big platform shift or a new way of doing things, it seems to, it seems to really grow, you know, the pie, so to speak. We don't see like, you know, other shopping services go away or anything like this. I think we really see that now shopping is available in agents and everyone is using agents. And a lot of shopping is happening in that surface and more shopping is happening. So, yeah, we just continue to keep building for everyone. We want to be everywhere so that, you know, when our merchants want to sell on any kind of platform in any kind of way, we're right there for them, you know, kind of on the head of that curve. Yeah, it's interesting. Yesterday I had a conversation with David McIntosh of Instacart. And they've obviously got like, they work with a bunch of retail stores, but they also now have like a physical cart that's enabled with AI that like gives you a running total and like automatically counter shopping and charges your cart at home. And they're now integrating a bunch of agentic features as well, right? It's like you can walk into a store and say, I've got $30 to spend. I want to make this dinner, but my first child is allergic to this. And my second child has this problem. And through like these agentic experiences, you can have this level of convenience and ease that wasn't possible before. And I really like, I kind of want to know from your side of things, being on Shopify side of things, like how are you enabling this agentic functions for your customers, for your users? What does that look like? Yeah, I think there's a couple different audience here. I think there's the buyer, which you're talking about. There's a merchant. How do we do the best thing for them and get them in all these services, which has with as little as work as possible on their end? Or how do we help them optimize for this as well and capture the data from it? And then there's just, you know, buyers that are making their own agents and this kind of thing. One product that we shipped recently inside the shop app, so the Purple Shopping app, is that a completely, you know, very personalized shopping experience and agent directly inside that app. So there's always search functionality inside this app. But a very, very personalized shopping assistant is now available to use, which knows you, learns about you as you shop, knows, you know, what type of products you're interested in, what your style of purchasing, your budgeting, etc. And it provides a pretty incredible experience in an agent with directly integrating to, you know, the catalog and showing products right there on the screen. And you can select them, ask more about it. So it's a great experience. And it also learn about different personas. Or like, you know, let's say you shop for different family members. It's going to learn from that and be like, okay, you're shopping for your dog or, you know, you have a cat even or you have, you know, children. As you search through, it's going to learn and optimize for who you're shopping for as well, because not everyone just shops for one person. What have you noticed in terms of customer feedback from this? We're having higher conversion rate on this than traditional search. So we're seeing that this is a surface that people are using more and more. We're also seeing people's search queries shift from traditional keyword type queries. Once they learn that there's the potential of an agent there as well, their behavior kind of shifts into more natural language queries. I think we saw this anyways, like on search engines, you know, over a period of time. But it's happening in shopping too. And people's behaviors are shifting to heavy research when shopping, looking at many products quickly. And yeah, just in general, doing a lot more research and therefore it leads to higher conversions. So, you know, the feedback is very positive on how accessible that makes it and the improved relevance via the, you know, the personalization and really getting to know you as a shopper. Because the more you use it, the more it adapts to you and how you search. And therefore, you know, they're coming back and using it more and more. Because everyone searches differently, right? For sure. It just becomes like more personalized over time. I think where my brain goes to with this is I'm really curious about the data side of things, right? So, for example, you're integrating this feature that personalizes shopping, understands shopping behaviors, customer behaviors, customer insights. Like, and data in general is like one of the biggest moats you can possibly have in 2026. So, I'm curious how you, the Shopify team, think about this. You're building this incredibly huge data set of customer buying behavior, like potentially one of the largest. So, how does that work internally? What do the conversations in these executive rooms look like when you're talking about the data that you're collecting? Yeah, I think our, the play here on personalization for us is that we're using the data that you, that you are using with the agent to personalize your shopping experience for you. So, like some people, some people when they're browsing, you know, they're looking for shoes and they just want tons of, they want tons and tons of shoes. They want to scroll through very quickly. And we can learn this behavior from their, from them using it. And there's other people, maybe they're buying like a motherboard or something. And they have like a very specific need. You're not going to scroll through a whole bunch of motherboards based on, on pictures. And there's just different kinds of shopping shoppers. Or like when I'm looking for a boot or a shoe, just show me like one or two options. I want the best ones for, you know, like hiking, you know, being up North fishing, et cetera, being out in the bush. And I just want the best waterproof hiking shoe. I don't want to flip through many pictures. So, you know, where, where the agent is learning for me to be heavy research based, you know, find the exact product where someone else, it might be like show them tons and let them scroll through. This is how that, that data is coming back and being like hyper personalized for each individual. Give me 30 seconds and I will give you the secret to effective AI implementation. If you like the AI report, you already know what we're about. Cutting through the hype and helping leaders actually use AI. At Upscale, we're more than a newsletter or even this podcast. We work with global organizations like the Society for Human Resource Management and Fisher Phillips Law Partners to train entire departments, boosting efficiency and productivity by an average of 41% in just weeks. If you want your company ahead of the curve, instead of chasing it, book a free strategy call at Upscale.com. That's U-P-S-C-A-I-L-E.com. Schedule today and future-proof your tomorrow. Agent, Agent Commerce, talk to me about UCP. I'd heard of it before. We had a bit of a preamble before. So I'm kind of curious if you can break that down for us and we can really decipher what's going on here. Yeah, I think on the AI commerce side of things, it's growing like nine times faster than social commerce and nearly three times faster than mobile did at this stage. And these two things, like mobile and social, both changed the way a lot of people do shopping, kind of reshaped commerce. And I think we're in another generational shift in how people are shopping, you know, and that's agents. So the UCP is a unified commerce protocol, which is a protocol made by a whole bunch of e-commerce companies like Meta, Etsy, Shopify, MasterCard. I think there's a whole bunch of companies here that agreed that this was a great protocol for exposing catalog APIs for the different shopping platforms. So, you know, in our data, we see that like catalog powered AI search is converting like two times better than general AI search. And orders coming to Shopify stores from AI are actually up 13x. Sorry, to break that down from a user experience perspective, right? Like that's someone typing in chat GPT or Claude or Plexity. I want to find boots for fishing in Northern Canada in November. And then it links through to Shopify specific store. That's right. Yeah. Right. And the increase in this has been enabled because of universal commerce protocol. And can you break down what catalog APIs are? Yeah. Yeah. So catalog API is an API that follows the universal commerce protocol, which exposes the Shopify catalog basically. So as a merchant, you have the ability to enable this sales channel. And if you've enabled the sales channel, then you will, your products maybe start to be exposed in agents that have integrated the UCP. Okay. And I mean, like the logic behind this is like, as a frontier model that you're using, because it's built on UCP and because it's cataloged there, there's just more inherent trust built. So when that AI goes through and it's ranking all these search terms and what is the most trustworthy, because it's built on this and because it's attached to Shopify, it's kind of more likely to recommend it, right? Yeah, that's right. And then we have, we've built dashboards and tooling inside, inside Shopify. So that as a merchant, you know, you're able to view this data. You can even use Sidekick to help optimize your exposure here. So yeah, there's a lot of, there's a lot of tooling on the merchant side of things to help understand what's happening. All right. So like, for example, it's understanding where do I rank for these specific search terms within this specific search engine, right? Yeah. Or not search engine, but AI model for example. Yeah, exactly. Yeah. Okay. So they can see orders, sale conversions from ChachiBT, you know, Google AI mode, Gemini, you know, Shop, et cetera. They can see the top AI queries that, you know, are leading to, to sales, what they're ranking for, for those queries. You know, when products show up in AI conversations, but are converting, you know, we have maybe some suggestions of what to fix, et cetera. So lots and lots of tooling inside the, inside Shopify admin for this. Cool. We were talking before, and you mentioned someone at Shopify, like building their own, Hermes agent to get free stuff about, like set to their home or something. Like explain that for me. Yeah. I think a lot of, a lot of Shopify employees are using, you know, OpenClaw or, or I think maybe Hermes more recently. And, you know, Toby, I think he tweeted, last week or a couple of days ago that he, he gave his own personal agents. I guess one of them is running on Hermes. And he gave it access to the, to UCP. So that I guess it can access the catalog from Shopify and other places. And he gave it a budget and I guess created a skill around it. And now he says that he just receives gifts in the mail from his Hermes agents periodically. So it's, it's pretty, it's pretty cool. What all of this enables, especially as more and more people are having their own shopping agents. It's like your personal AI shopper. I know we all wish we could have a personal shopper, but I think agents makes that very accessible for us. So gifts in the mail, is that from like a free trial that the agent will sign up for and then cancel? Or like how, how exactly is he getting free gifts in the mail? Oh, I mean gifts to him, but he's technically still paying for them. Yeah. The agent is paying for them with a budget that is the agent has. So he'd use it as a gift for himself, but it's just, you know, the agents, his agents and all of our agents, they learn about us. You know, everything we do, not just shopping, but how we run our lives. They, they help us across the board. So now you can very easily just, you know, give it access to Shopify's catalog. And then it's up to you. If you want to give it a budget, if you want to just have it be proactive and suggest products to you that you might like, and then you, you can build your own experience on, on how you want to make those purchase. But, you know, we proactivity is, you know, just such an important thing as we go into the future that, you know, proactive shopping is, is a, is a very cool thing, I think. And this enables that. Yeah. Yeah. I mean, there's obviously like a, a kind of, a lot of opportunities here, right? You have like Samsung smart fridges, for example, right? Like you can kind of be proactive with these agentic functions as well about like ordering food in when you're running low on olive oil, right? Can recognize that. Like there's, there's a bunch of things that could be done here. Um, maybe it's not something you've thought of before, but I'm really curious. So we're talking more of like a business to consumer standpoint, right? Like you have a merchant that's on Shopify, they're selling their products. They want to rank higher so the consumers can find them through AI. But I'm wondering if you guys have thought about like the B2B functionality of something like universal commerce protocol. I wouldn't even know where to start with that, but there, there has to be kind of some functionality and use cases or anything that you've seen so far. Uh, not that I've seen, but I know there are like app developers, um, um, who may be developing like niche or the niche apps that want to create a very specific shopping experience in a certain industry or something like that. Like instead of maybe a global, you know, instead of a global agent, inserting everything, there may be this niche fashion shopping app that somebody creates and now they have the ability to build very specifically for, for fashion. And use the catalog API just within that industry so that they can, they can really build a niche product theirs. And I think this is, you know, this is one area where we see, because this is just a tool available to developers that people who are not even those businesses or merchants are creating these new apps that use shopping. And, you know, merchants are benefiting because they've chosen the sales channel. You know, these developers are benefiting because they're able to create these niche apps that have like instant access to different catalogs. So it is certainly, you know, seeing uptick in this kind of niche applications being built. What are you most excited about personally? We spoke about things beforehand. We talked about, and I want to talk more about like the sidekick app extensions, essentially functions as like an NCP functionality. We're talking about UCP. We're talking about agent-to-agent marketplaces and functionalities here. The agent-to-agent functions and shop. Like, you're super hands-on with this. So like, and you already seem to be ahead of the curve. But what are you excited about if your trip wants to come? Yeah, one. I mean, I don't want to give, maybe I'm going to give an underwhelming answer here. But this is unique to me. But I love, one of my workflows for like buying stuff in general is like I take pictures of things or like I see something, whether it's on TV or in person or I'm, you know, in someone else's shop, for example. Like shop, like maybe workshop, I mean. And I like can take a picture of like a cool tool or something that I find useful. I take a picture. And then I'll like throughout my day or week, I'll take a bunch of pictures. And a lot of my shopping is actually driven by like referring to pictures that I've taken. And one of the cool things about the catalog API that I like find very exciting is that you can also search by images. So not only can, you know, you can, not only can you search catalog, you see, you know, the catalogs with image search, but as a developer and someone who has a Hermes agent and this kind of thing. And I use it, you know, kind of like a bullet journal a lot, as well as automated tasks and reminders and whatnot. So the ability to just dump in all the pictures I've taken related to this. I mean, like, hey, can you go research these things, find the best place to buy it, you know, get back to me tomorrow morning with all of this. And that's one of the huge unlocks that I've, that personally I'm very excited about. But this, I know whether it's just me that shops like that, I imagine, you know, maybe people go to stores and take pictures of dresses and whatnot kind of too. But this is, that's just really, really an interesting scenario that seems so maybe not interesting, but I think is just so, so powerful. But yeah, I don't know whether, you know, that just fits into like the way that I like, I like to shop and use agents specifically. Yeah, it's not something that I do personally, but maybe it's kind of due to the nature. Like you said, you're in workshops quite a lot, right? Mm-hmm. Right. So like, maybe it's just like, I don't know how tools work. Like I've used tools before, but there's probably some very niche tools and you're like, I have no idea what that thing is. So let me take a picture and figure that out later. Yeah. Yeah. I don't think I'm quite ready for the, you know, the Toby, give it a budget and, you know, send me, send me gifts all the time scenario. But definitely it's, it's been great to, to plug into UCP and have it, you know, send me recommendations. And then, you know, I, I, I kind of do the final mile, but I know, you know, some people will just have it, ship it automatically, which is, it's fantastic that you can do this kind of thing. And, and this world that we live in. Yeah. Is there any pushback whatsoever from your merchants on any of the agentic functions or universal commerce protocol? Like from my standpoint, it seems like a no brainer, right? Like Google is changing, search is changing. It's all very difficult. And it seems as if Shopify has got people's backs who are merchants. Like, let's, let's help your stuff rank because the better your business does, the better our business does. But do you see any pushback? Because there, I just want to try steel man the case for a second. Like, is there any negative feedback or like, what are the common kind of complaints, if anything? Yeah, I haven't heard any. And I think this is probably because, you know, merchants can, you know, they don't have to be on this sales channel. So if they, if they see the benefits of it, then they enable it. And if they don't, then they, you know, they just don't. So there's no, you know, I don't hear much complaints because I think the people who, you know, don't want to enable this, it's just, it's just not enabled. So I think just giving merchants that flexibility has kind of, you know, reduced, released this. Another thing that we kind of mentioned briefly before, but is in a similar vein, right? So we've got universal commerce protocol, right? It's enabling people to access these catalog APIs. Again, just like reminds me of MCP. And we spoke before about Sidekick app extensions as well. Which from what I understand isn't MCP, but like functions in a very kind of similar hat. So I was wondering if you can kind of explain what's happening there. Yeah. So back in our December editions, we had talked about Sidekick getting into enabling app extensions, which allows app developers to plug into Sidekick, both providing it like fetching data that the users are asking for in Sidekick, as well as, you know, making, you know, updating certain fields. And basically using that application via Sidekick and then, you know, filling forms automatically kicking into that app, et cetera. So I think in March we enabled it for a select number of app developers. And I think we had like 20-ish partners at launch, which launched at editions a couple weeks ago. So, and the feedback on it has been very positive. I think like JudgeMe, Klaviyo, Loop, Matrixify, all these partners have integrated into Sidekick. And on Twitter, I see every day more and more apps doing this. It's one of those things that since Sidekick launched, this was a very heavily requested feature from app developers and merchants, actually. Because they wanted to, you know, use the apps that they've installed and have them tightly integrated with Sidekick because they're seeing all the value and using Sidekick in general. And they're like, why can't, you know, Sidekick know about this app and the data there? So we've provided the app developers a way to plug directly into Sidekick. And it's, yeah, like you mentioned, it's like MCP shaped or skill shaped. It's basically, you know, Sidekick is searching for relevant apps for any given user query. And if there's a relevant app, then if there's multiple, then it's going to disambiguate. If there's just one, then it's, you know, it's going to go ahead and use that app, you know, and then present it back to the user. And users can also turn off and on these integrations similar to MCP. So it's one of those things we think we, you know, is a great ask from the community. And it took us some time to figure out how to do it exactly right. But we think we really nailed this one. A lot of this conversation you have spoken about, like, feedback that you see on Twitter, for example, or feedback that you see from the community. And I'm wondering for, like, you as an individual and your team, how you choose what to build, right? Like, you're mentioning this feedback that you're kind of, it sounds as if you've got your finger on the pulse. And you're obviously ahead of things in terms of enabling these functions for people. But, like, how much do you build out of what you just believe to be right? And how much do you build out of we've had enough requests for this, thus people build it? I think there's no, like, straightforward answer on this thing. You know, certainly we're, like, trying to be ahead of the curve. Like, what do people want before they want it kind of thing? Or what do we see happening? And how do we get ahead of it and build something really cool? Like, how do we build something that people aren't even asking for? And I think this is a great way that we think about it. At the same time, Twitter and the community and these other things is a great way to be like, hey, what did we get wrong? Or, you know, what did we not think of? You know, what could we doing that we're not doing? What could we be doing better, et cetera? So this is an important signal as well. And, you know, things going on elsewhere at Shopify and new features being built. And how do we plug into those and make sure, you know, Sidekick has full coverage of the new features across the board. So there's no straightforward answer. It's kind of a little bit of everything. So Sidekick, I mean, we mentioned it a lot there. And we spoke about it last time, but for people on a pointy with it, tell me a little bit more about what exactly Sidekick is. Sidekick is basically an AI co-founder there for you. You know, right from the beginning when you start your store, it's useful for, you know, small merchants. We're finding our new merchants. We're finding them use it to help create their themes, help, you know, define their product vision, their business vision. You know, people come in all different stages. Some people just go to, you know, start a Shopify store without any idea. And they can use Sidekick for like brainstorming and discovering, you know, what kind of store they want to build or what kind of business they want to start. We see, you know, very large merchants using it as well, heavily on like analytics because you can get very deep insights from your store through natural language, which we were seeing is like a huge unlock. But basically you can do many actions on your store through natural language. And it's just like a more natural interface for people to do things. And as Sidekick increases its capabilities more and more, we're seeing, you know, people run their business a lot through conversation. And Sidekick is there to enable that for you. I think one interesting story I have on like the beginning of Sidekick is I think we did a survey at one point where we looked at merchants who, you know, had an entrepreneur in their life who they could ask questions for and entrepreneurs who didn't. And then we surveyed them when they first started their store and, you know, a year down the road. And the merchants who started their store with having an entrepreneur that they could ask questions had a much higher success rate of their stores than the ones that didn't. So, you know, one of your vision for Sidekick was how can we provide that entrepreneurial expert for merchants so they can ask these questions and as if it's a friend or, you know, there's no judgment when you ask like Sidekick things, right? Whereas, you know, it's in some ways easier to have this like entrepreneurial expert there, you know, right there for you if you don't have that in your life. So, break that down a little more for me. We're talking about a co-founder or an expert that is made of artificial intelligence, right? So, I'm curious, like what does its brain consist of? Like what LLM, if any, does it run on? Where is it data coming from? Like is it gleaning insights from Shopify? Is it tied to larger data sets? Like where does this advice come from? I was wondering if you could break down more of like the functionality of what makes a great AI co-founder. Yeah. So, Sidekick, its architecture is basically an LLM driving it and this LLM is, you can think of it like as a loop, looping through, you know, different tool calls or having different skills. This LLM in particular, right now we're running on the Sonnet model from Anthropic, hosted on Google's infrastructure on GCP. And so that's the, you know, the main model driving Sidekick right now. Will that change? Sorry, like can that be model agnostic? Yeah, yeah, this can be model agnostic. We find Sonnet to be quite good for this use case at Argentic and calling tools and being conversational. Sure, but that's not like a user choice that's from your side of things. Yeah, that's right. Okay, cool. And then talk to me about, I want to know where these insights come from. Like where does this knowledge base come from? How is that updated over time? Yeah, we have knowledge bases internally, like from, like basically a knowledge base of how to use Shopify. And then, and this is where like it, and then, so that's one tool. I know I'll come back to that tool. And then there's tools for like natural language to, you know, GraphQL, for example, which is like the underlying language of Shopify. So this, this is what helps give Sidekick a lot of ability to work with your store, understand your store. There's also one for, you know, analytics for looking into, into analytics and, and this kind of thing. There's many, many different tools. The knowledge base for Shopify, it's available publicly as well on help.shopify.com and these documents. And, and Sidekick is able to look through these documents. And as well as like, you know, Shopify blog posts that we've written and et cetera. Like Shopify has a lot of, a lot of public content that they put out there. And, and Sidekick has access to, to draw on these things in order to provide, to provide its answers. Here's where my brain goes with this. So I don't think we've touched on it yet. You touched on a pre-conversation. You have this new feature. It's available campaign autopilot, right? And obviously with all these merchants that are now running on Shopify, you know what merchants are successful and how and why and what ones aren't and how and why. So is that data, that information being used to help kind of feed Sidekick, provide the right decisions, right? Because if they've seen someone that's in your vertical and spend their campaign budget in this way and use this specific color and hex code in a specific format and that works and that's generating a lot of sales. Does that information feed? Like, are you taking what's right and wrong from other stores and then feeding that into the brain of Sidekick? Or is that siloed to each individual merchant? Yeah. Yeah. There's a couple points to touch on here, I think. So it's not being used in that way for that campaign autopilot. For example, this would be you can give it a budget. You can give it specific guardrails. And you can think of like a Carpathie auto research loop where it's trying certain things and it's like hyper specific to you. And, you know, it's seeing what's working and it's, you know, the coming back and it just keeps iterating, iterating. So it's like a research loop hyper for your store and the exact scenario and the exact campaigns that you're trying to run. And there's a product called Simgym where we are training a basically an AI shopper so that it can. Have you heard of Simgym before? Have we talked about this? I think we did speak about it briefly. Basically, basically, from what I remember is you can emulate buyer behavior based on the current conditions of your store along the right lines. Yeah. So we have. We we're training basically these AI shoppers so that if you show them a version of your store and another version of your store. That you can have these shoppers like let's say we have 300, you know, try to complete a task on your store and 300 try to do it on the other version of your store. You know, you made me think of this like the hex color button thing. So if you want to do something like move the button around or do this or enable reviews or disable reviews, then you will. It's basically like running an A-B test, but without. Without actually having to run a real A-B test against real merchants. So, you know, if you have many, many ideas of what to try, but you're not sure, you know, it's expensive to run it on live traffic and you may be like losing business for the bad ones and whatnot. You can actually run all your ideas with SimGym, maybe take the top two and then do a live A-B test. And I think, you know, we the way we built this is that, you know, people have made changes to their to their store. So when we create these AI shoppers, you know, we're testing that in the past when there's changes to stores that we're we're saying the our AI shoppers are saying the same results as what actually happened. So on the store. So these these AI shoppers or simulators are basically learning to identify, you know, real issues that then then you can use for A-B testing. Hmm. I mean, one one interesting thing here that I actually ran on my store, which is like a maple syrup store, is that not only for A-B testing, but you can actually just have it run as a research loop on your store. So even your current theme. In the same way, the campaign, you know, auto research car path, these stuff is like all the buzz right now in the in the agenda. Can you can you explain that for the audience more as well? Like in detail, if possible? Yeah. So the idea behind the auto research loop, let's say, is that there's there's some metric. That you have. So let's say let's let's talk about model training for a second. And so I want to optimize my model on some metric why. And so I can instead of me, you know. Analyzing outputs from the model and trying to figure out as a human, you know, what the best improvements I need to make to the data. Like, let's say I'm training a cat or a dog classifier or something. And and I have some metric and I want to optimize for that metric so you can have an LLM run tests. Test that metric and then look at the results of the test. And make suggestions for the next round of the model training, maybe identified a whole bunch of, you know, bad labels on your training data or, you know, cast our dogs, et cetera. Or like there's some other objects in there. So the LLM is able to make those recommendations, make the changes on its own, run the scripts to retrain the model. When the model is done training, it'll run the evaluations again, look at the evaluations, find all the errors as an LLM and then make suggestions as an LLM and then make ideas on on how to retrain the model. So it's this loop where a model, an LLM is continuously running, trying to optimize a specific metric. I think it's used in almost anything, not just model training. Okay. Is there a lot of parallels between what's been described here and how a lot of these frontier models were developed in the first place through reinforcement learning? There can be, yes. Yeah. What's the difference? If you like go to a super high level, then yeah, it could be fairly similar. Like reinforcement learning is a, some state space or some like environment and you can carry out an action and look at the, there's a reward function. Like what did this action do? And is this good or bad? And then give that signal back to the main model? I guess with a, with an LLM driving this loop, you can, you can do like the LLM itself can make some of the suggestions. Whereas in, in reinforcement learning, maybe you're using like, you know, human feedback loops or maybe you're doing preference optimization between, you know, two different outcomes. But yeah, at a high level, it's, it's pretty similar. Okay. Yeah. Let's flip things back to Shopify just to finish off. So we've got a lot here. We've got like the psychic app extensions, psychic app extensions. We've got UCP. We talked about agitation commerce, campaign autopilot and the energetic assistant with shop. And for a user, for a merchant, for someone that uses like the Shopify store for someone who uses the website, what should we be excited about coming in the future? Is there anything that you can tell us that's maybe just recently been publicly revealed or will be publicly revealed in like eight weeks when this episode releases? Yeah, I'm not sure I can, you know, say too much on, on upcoming, upcoming launches, but I can say that it's, it's a very exciting time. I think, you know, agents, you know, since December have been going through this transformational stage and how people use them. And, you know, for both shopping, I think we talk a lot about that on the sidekick side of things. I think there's just, you know, some big improvements that we can make. And I think we're, we're excited to, to share some of those changes, you know, over the next, I don't know the timeline, but, you know, I think we're building some really cool stuff. And there's, there's reason for, you know, the 250 million verified shoppers on, on Shopify and, you know, the millions of merchants to be extremely excited about, you know, the future of what we're building. Sure. Andrew, two things. One, thank you for your time. Two, if people want to find out more about who you are and what you do, how do they go about doing that? Uh, reach out to me on Twitter or tag me on Twitter. My Twitter is at Drooch, D-R-E-W-C-H. Always happy to talk, uh, agents, Shopify and, uh, commerce. Awesome. Thank you so much. All right. Thanks for having me. From the history of Hershey to the business behind the hit podcast acquired, Cold Call, the chart topping management podcast from Harvard Business School, gives you a seat in the classroom where these legendary cases are brought to life. I'm Brian Kenney, the chief marketing and communications officer at Harvard Business School and the host of Cold Call, which explores the real world case studies taught in our classrooms, featuring the professors who wrote them and often the executives, entrepreneurs and innovators who lived them. In each episode, we break down the decisions, trade-offs and leadership lessons behind some of the most fascinating organizations. Not to mention, we just dropped a new video series examining AI case studies like the Atlantic's partnership with OpenAI, Mass General Brigham's use of AI to streamline clinical documentation and Harvard Art Museum's experimentation with generative AI. So don't miss out. Follow or watch Cold Call on Apple Podcasts, Spotify, YouTube or wherever you get your podcasts.