This $30 Hardware Hack Replaced My Smart Display With a Local AI Agent

Think about the smart display sitting on your kitchen counter right now, or you know maybe the one on your office desk. Oh man, the one that basically just shows the time. Exactly. Under the hood, that device is actually a remarkably powerful computer, but as the user you're completely locked out of that power. Yeah, it's totally restricted. You get the clock, you get the weather, and whatever rigid unchangeable widgets a corporate firmware team decided you deserve like five years ago, you are forever stuck in their walled garden. Completely boxed in. But today we're looking at what happens when you strip all of that away, bypass the corporate limitations entirely, and build a screen that literally designs itself. It's fascinating because it completely shatters our traditional understanding of how we interface with our environment. I mean instead of the user navigating a rigid operating system to find information. Like we're used to doing. Right. Instead of that, the information dynamically builds its own custom operating system for the user in real time based purely on context. Which brings us to the actual mission for this deep dive. We are exploring this incredibly ambitious project by Adam Conway, the lead technical editor at XDA. Yeah, Adam's project is just brilliant. He essentially took a $30 totally bare bones piece of hobbyist hardware and tethered it to a local AI on his home network. The result is a dynamic self-designing interface that answers whatever question you throw at it. And it is an absolute master class in hardware hacking. Mm-hmm. Like leveraging local large language models and navigating some brutally clever user interface constraints. Oh, for sure. But before we get deep into the technical architecture of how this actually works, I just want to ask you a quick favor. Take a second to hit the follow button on whatever podcast platform you're using to listen right now. It really makes a difference. It does. Following ensures you never miss a deep dive and it genuinely helps the algorithm recommend deep dive AI with Robin and Howard to new listeners who love this kind of breakdown. We massively appreciate it. Okay, so let's unpack the baseline here. Set the stage for us. What is the actual infrastructure Adam is running? Well, the crucial thing to understand is that the brain of this operation is completely decoupled from the screen itself. Oh, okay. So the screen is essentially a dumb terminal. The actual intelligence lives on Adam's own Proxmox home server where he's running an AI agent. A local one? Yes. Specifically a Hermes agent powered by a local large language model. Meaning all the computational thinking, all the data processing is happening securely on his own physical hardware sitting right there in his house. Exactly. He's not sending his data out to a massive server farm halfway across the country. Which is huge for privacy. Privacy and speed are the primary benefits there, yes. And his method of communicating with this local AI is, well, it's as frictionless as possible. He's not writing code to talk to it. No, not at all. He doesn't use a specialized coding terminal. He just uses Telegram, you know, the standard messaging app on his phone. Oh, wow. Just regular texting. He texts his server exactly the same way he would text a friend. So he can send a message saying, check my network status or give me the latest news. And the Hermes agent just gets it. Right. It interprets that natural language, processes the request, and then has to figure out how to display the answer. And to understand how the AI figures out that display, we really need to look closely at the physical canvas it's being forced to work with here. Oh yeah, the hardware constraints are wild. Because this isn't some beautiful OLED tablet. We're talking about an Electro Crow panel. It's a 1.46 inch rotary display. That is an absurdly tiny piece of glass. We're talking about a screen roughly the size of a large coin. It retails for about 30 bucks. And the silicon driving it is an ESP32 S3 microcontroller. Which is pretty standard for hobbyists, right? Yeah, for those familiar with microcontrollers, the ESP32 is this legendary, highly capable chip for Internet of Things projects. But I mean, it is a far cry from a smartphone processor. Definitely. We're looking at a system with just 8 megabytes of PS RAM. That's the memory used for active tasks. And 16 megabytes of flash storage for the firmware. That is microscopic by today's standards. I mean, my phone has gigabytes of RAM. It's extremely limited. And on top of that, the display itself is a 360 by 360 pixel IPS touch panel. But here's the most unique hardware feature. The entire perimeter of the screen contains a ring of addressable LEDs. Okay, neat. And the screen housing itself acts as a tactile clickable rotary encoder. So you can literally grab the edge of the screen and turn it like a volume dial. And you can press the whole screen down until it clicks. Okay, I have to pause on that because that dial is a massive deal. It really is. Whenever I hear the phrase tiny touch screen, my user experience alarms just go off. Trying to scroll on a 1.4 inch screen with a human finger is universally terrible. It's the classic fat finger problem. Exactly. If you look at an early smartwatch and try to scroll through a notification, the second your thumb touches the glass, it completely blocks the text you're trying to read. You're basically navigating blind. Yeah, the physical interface usually just ruins the digital experience on these micro devices. But that rotary knob bypasses the fat finger problem entirely. Because your hand is on the outside. Right. By offloading the scrolling action to a physical dial on the perimeter, your hand never actually crosses the visual plane of the text. Oh, that's smart. You twist the bezel, the content scrolls fluidly, and your line of sight remains unobstructed the entire time. It turns this frustrating touch experience into a precise mechanical one. It's a brilliant hardware choice. But, I mean, it doesn't excuse the AI from having to deal with an even bigger issue, which is the sheer geometry of the thing. The screen is perfectly round. Yes. The circle problem. Designing any kind of user interface for a circle sounds like a mathematical nightmare. It is, because computing is fundamentally a rectangular medium. Monitors, televisions, smartphone screens, code structures, image files, they all assume a rectangular matrix of pixels. Everything is a box. Everything. So when you introduce a 360 by 360 pixel circle, a significant portion of the traditional square canvas simply vanishes. It's physically masked by the plastic bezel of the device. The corners just don't exist. They don't. And if you try to fit the largest possible perfect square inside a 360 pixel circle without any of his corners being clipped off by the edges, that square is only 254 by 254 pixels. Wow. That's a lot of lost space. It means nearly half of your theoretical digital canvas is completely unusable for displaying crucial information like text. So if Adam doesn't explicitly warn the AI about this bizarre physical limitation, what happens when it tries to render a screen? Well, left to its own devices, an AI will default to standard web practices. It'll try to justify text right up against the top left corner of the 360 by 360 canvas. Right. Like any normal web page. Exactly. Yeah. But on a physical circular display, that text would just disappear behind the plastic bezel. You wouldn't be able to read the first few words of every line. Right. So to fix this, Adam had to act as an architect. He had to define a strict safe zone. He configured a central visible area of 272 by 236 pixels. Okay. So a specific rectangle inside the circle. Yes. And he hard baked this specific geometric constraint directly into the AI's CSS style sheet and its skill file. He established this ironclad rule for the local model. Basically, under no circumstances can you render critical information outside of this invisible box because the human eye will literally never see it. Got it. So we have a $30 underpowered ESP32 chip connected to a tiny round screen with a strict safe zone. How on earth does a local AI take all of that and dynamically draw a complex, beautiful interface? It's tricky, right? Because as we established, this chip does not have the processing power of an iPhone. It doesn't have an operating system with a robust UI framework just waiting to render drop shadows and anti-aliased fonts. No, it doesn't. And that lack of local processing power forced Adam to design an architecture that is incredibly ingenious, even if it seems a bit unorthodox at first glance. We can refer to this as the long way pipeline. Okay, break down this long way for us. Walk me through the exact journey of a piece of data. Let's say Adam opens Telegram on his phone and types, show me the latest tech headlines. That message travels to his home server where the Hermes agent processes a request. The AI decides what information is relevant and then literally writes the user interface as a chunk of raw HTML and CSS code. But formatted for that specific safe zone. Exactly. Perfectly tailored to the 272 by 236 pixel safe zone. So it's just writing a standard webpage from scratch. Precisely like a standard webpage. Yeah. But we know the tiny ESP32 chip on his desk cannot natively read or render HTML. Right, it would just choke on it. So instead of sending that code to the screen, Adam's powerful home server spins up a headless Chromium web browser. Headless meaning there's no actual window popping up on his monitor. Right, it's essentially a full version of Google Chrome running invisibly in the server's background. That hidden browser opens the HTML the AI just wrote, renders the page perfectly at 360 by 360 pixels, and then takes a digital screenshot of the result. Wait, it takes a literal picture of the webpage? It creates a static PNG image file. And that resulting PNG is what actually gets transmitted over the local Wi-Fi network to the display on his desk. Oh wow. So the display isn't doing any complex rendering. It isn't parsing code. It is merely acting as a digital picture frame, showing the snapshot it was just handed. Okay, I have to challenge this architecture for a second because taking a screenshot of a hidden web browser on a high-powered server, just to show a few lines of text on a desk clock. That sounds wildly convoluted. It sounds like a lot of extra steps, I know. Why wouldn't he just send the raw text data to the screen and have the microcontroller render it natively? Well, the convolution is actually the secret to its brilliance here. Mm-hmm. Consider the computational cost of what you're suggesting. Okay. If you send raw code to that display, you suddenly have to teach an underpowered ESP32 chip how to parse HTML tags, how to calculate CSS flexbox layouts, how to rasterize custom fonts. Oh, right. All the rendering math. Yeah. And how to mathematically smooth the curved edges of those fonts so they don't look pixelated. Those operations require heavy floating point math that would completely overwhelm the hardware. The screen would just stutter and freeze. So the strategy here is fundamentally about shifting the burden of work. It is the ultimate expression of moving the heavy lifting from the edge device to the core server. Let the big computer do the hard part. Exactly. By doing it this long way, the ESP32 never has to calculate a single CSS gradient or parse a single tag. Its only responsibility is to light up pixels according to the image file it receives. Which it can do super easily. Right. But the true genius of this architecture is that it makes the physical hardware infinitely adaptable. The complexity of the user interface is gated only by the AI's ability to write CSS. Because it's just a picture. Yes. If Adam decides tomorrow that he wants a radically different layout, custom graphics or a dark mode, he never has to plug the screen into his computer and flash new firmware. The AI just dreams up new CSS on the server. Chromium snaps a new photo and the hardware updates instantly. It makes a totally dumb terminal future-proof. Okay, that makes perfect sense when you frame it around computation, but it introduces a pretty glaring bottleneck with the memory. Ah yes, memory is the catch. You mentioned earlier that this ESP32 chip only has 8 megabytes of active PS RAM. If the server is constantly blasting high-resolution PNG images over the local Wi-Fi, that tiny memory bank is going to overflow almost instantly, right? Memory management was a critical hurdle for sure. At its absolute limit, the device's PS RAM can only cache about 10 of these image cards simultaneously. Just 10 screens. Yeah. So to prevent the system from constantly overwriting itself and saturating the slow 2.4 GHz Wi-Fi connection, Adam implemented this highly efficient caching system using SHA-256 hashing. For those trying to visualize this, how does hashing act as a traffic cop for the memory? Well, you can think of a hash as a unique mathematically generated digital fingerprint for a specific file. Okay. When the headless browser on the server creates a new PNG image, it instantly calculates the file's SHA-256 fingerprint. And instead of blindly sending a heavy 100 kilobyte image file over the network, the server just pings the display with that tiny alphanumeric fingerprint. And it asks, hey, do you currently have an image with this exact fingerprint stored in your active RAM? And if the display checks its local memory and recognizes the fingerprint, it simply replies, I already have it, don't bother sending the file. You got it. The server skips the transfer entirely. That saves massive amounts of network bandwidth and prevents the ESP32 from duplicating data it already possesses. Which is huge when you only have space for 10 cards. Right. It guarantees that the limited 8 megabytes of RAM are only ever populated by genuinely new information. That is a remarkably elegant workaround. But, you know, rendering the images and managing the network traffic is just the mechanical layer of this project. Where this deep dive gets truly fascinating is examining how the AI model behaves when it's forced to act as an autonomous editor. The autonomy is really the defining feature here. The AI isn't simply executing a rigid script. It is making active, contextual decisions about how to present information. Which is where it gets kind of spooky. A little bit. The system was designed to give the AI multiple options. It doesn't always have to trigger that heavy HTML to PNG pipeline. Oh, it isn't. No. Atom programmed two native card types directly into the ESP32's firmware. These are incredibly basic hard-coded layouts for rendering simple ASCII text or a single massive number. Like a temperature reading. Oh, I see. So it has some lightweight options. Right. And the AI's core directive was simply evaluate the request and pick the cheapest, most efficient rendering method to answer it. So it actually has to evaluate the weight of its own answers without any human intervention. It does. And observing this led to some incredible emergent behaviors. The first major example is the AI acting as a real-time news editor. Let's hear it. Adam requested the latest headlines from his own tech publication, XDA. The AI parsed the request, realized that a news feed requires thumbnails, distinct typography, and structural formatting, and decided, okay, the simple native text card won't work here. I need to utilize the HTML screenshot pipeline. Which is the logical choice for a rich media feed, obviously. But then it immediately slams into the physical limitation of the 1.4-inch circular screen. And that is where the magic happens. The AI pulled the RSS feed, but it recognized that standard digital article headlines, something like, Samsung Galaxy S24 Ultra leaks reveal brand new camera specifications were far too verbose. Oh yeah, that would never fit. They would spill completely outside that 272-pixel safe zone. So, autonomously, the AI began truncating and rewriting the actual titles of the articles. Wait, it rewrote published journalism on the fly? Yes, and it didn't just lazily chop off the end of a sentence and slap a dot dot dot on it, which is how a standard line of Python code would handle text overflow. Right, the typical stream truncate. Exactly. Instead, the large language model analyzed the linguistic syntax of the headline, identified the filler words, extracted the core semantic meaning, and generated a brand new, punchier headline that perfectly fit the physical dimensions of the hardware. That is wild! It preserved the context of the journalism while respecting the limits of the silicon. The idea that it comprehends physical hardware constraints and applies linguistic editing in real time to solve them is just wild. And the second emergent behavior Adam documented takes that comprehension even further. We can look at this as the AI acting as a curator. Okay, a curator. Adam asked the Hermes agent to perform a network scan, essentially asking it to ping his home internet and report back on every single active device. The scan returns 75 live hosts on the network. 75 separate devices, but we know the physical memory on the display can only cache 10 image cards before it overflows. Right, if you feed an array of 75 items into a standard, non-intelligent program running on limited memory, you generally get one of two outcomes. The buffer overflows and the device crashes. Always fun. Or it blindly renders the first 10 random devices it happens to see. Maybe a couple of smart plugs, a printer, and a thermostat. And then it completely discards the rest of the list. Neither of which is actually helpful if you want to know the health of your server network. Right. But the local AI model understood the broader context of what Adam was actually asking for. It looked at the raw data dump of 75 IP addresses and host names. And what did it do? Instead of attempting to force all of them into the tiny screen, the AI evaluated the host names and identified the eight most critical pieces of infrastructure on his network. It filtered them. Yes. It pinpointed his OPNsense router, his TrueNAS storage array, his frigate security camera recorder, and his Jellyfin media server. However, it dynamically generated rich HTML cards solely for those eight marquee services to display on the desk. And then it quietly packaged the remaining 67 trivial devices into a simple text file and dropped it into the Telegram chat on his phone. Oh my God. The best way I can visualize that is comparing it to an elite executive assistant. That's a great comparison. If you ask your assistant for a status report right before you walk into a crucial board meeting, they aren't going to hand you a raw 75-page spreadsheet. They synthesize the chaos. Make a sack? They place a one-page executive summary on your podium, highlighting the specific metrics that could tank the company. And they email you the dense appendix to review later. To see a local AI performing that level of contextual synthesis for a $30 plastic screen is remarkable. It really bridges the gap between raw data and actual human insight. But, you know, as beautiful and seamless as that software intelligence was, we do have to talk about the physical reality of hardware hacking. Oh, right. The gremlins. Yeah. The AI did its job flawlessly, but initially the screen itself was completely dead. We have to address those hardware gremlins. This is always the hidden friction of open source DIY electronics. It sounds like a sci-fi utopia when we talk about large language models writing CSS, but the actual silicon is messy. From what I read in Adam's notes, he compiled the firmware, the Wi-Fi connected, the server authenticated, but the display remained entirely black. Yeah, he was staring at a dark piece of glass. And this is where the detective work comes in. Adam had built his firmware using the exact vended code provided by Elecro, the manufacturer of the screen. So it should have worked perfectly. It should have. When he queried the ESP32 chip over his terminal, the software reported zero errors. The memory allocated perfectly. The network packets were arriving. From the microprocessor's perspective, it was flawlessly painting a masterpiece. Wait. If he used the manufacturer's official, provided code, and the board's diagnostics reported absolute perfection, why is the screen black? Does that mean the actual LCD panel was dead on arrival? Like a busted ribbon cable or something? That would be the logical assumption, yeah. But the truth was far more frustrating. The problem was a fundamental lie in the manufacturer's documentation. From I. Adam dug into the source code of the graphics library they were using, called Luvian GFX. The official spec sheet claimed the screen was driven by a specific display controller, the ST77961. Okay, a standard chip. Right. But when Adam started dissecting the actual initialization sequence, the specific hex codes sent to wake up the screen's registers, he realized the manufacturer had quietly gutted the library. What do you mean gutted? They ripped out the instructions for the ST77961 and replaced them with the boot sequence for a completely different piece of silicon, a JD9855 controller. Why would a manufacturer secretly swap the brains of the display without updating the basic documentation? Honestly, it's just supply chain realities. In the world of cheap electronics, manufacturers will constantly swap components to save a few cents per unit or to bypass a temporary chip shortage. That is so annoying. It really is. They update the code just enough to make their own test models work, but they rarely bothered to update the public data sheets. The ESP32 was sending perfectly valid instructions, but the screen was expecting a completely different dialect. So what did Adam have to do? He had to manually extract this undocumented mutant boot sequence from the vendor's files and hard code it into his own custom firmware just to shock the pixels into waking up. And even after he cracked the initialization code, the hardware fought back again, right? Once the screen finally turned on, there was a massive issue with stale pixels ghosting across the display. Oh yeah. Huge vertical columns of old pixels were lingering and tearing across the screen every time a new image loaded. Resolving this required stepping away from web development and diving into the literal physics of the hardware. The first issue was a collision within the DMA or direct memory access. How does a DMA collision actually manifest on the screen? Like what is happening? Well, the ESP32 uses direct memory access to shove image data directly to the screen at blistering speeds, bypassing the central processor to save time. But the software wasn't properly terminating the right transaction before initiating the next one. Oh, so it's like trying to print page two of a document on a piece of paper that is still halfway through printing page one. The new data is literally writing over the old data mid-draw, creating a garbled mess. That is the perfect analogy. Huh? The frames were crashing into each other. So he had to rewrite the transfer logic to guarantee the first frame was fully pushed to the glass before the second one was allowed to begin. That makes sense. But even after fixing the software overlap, the image was still artifacting. That's when Adam realized he was pushing the electrical tolerances of the cheap hardware too far. The vendor's code was attempting to run the SPI data bus at 80 megahertz. 80 million cycles per second sounds incredibly fast for a $30 board. Too fast for the physical copper traces on that specific circuit board. At 80 megahertz, the electrical signals were degrading. The digital bits were literally arriving late to the screen's controller, missing their designated clock edges. Oh, wow. The literal speed of the electricity was the problem. Exactly. Adam had to step in and manually throttle the bus speed down by 50 percent, dropping it to 40 megahertz. By slowing down the transmission, he gave the cheap silicon enough time to actually interpret the electrical pulses correctly. Only then did the image become crystal clear. It is such a stark reality check. You know, you can deploy the most futuristic, cutting-edge local LLM capable of dynamically generating perfectly truncated HTML on a server. Yeah. But ultimately, a human engineer still has to sit at a desk, read voltage tolerances, and debug the clock speeds of a circuit board just to make the lights turn on. It proves that while software is rapidly advancing into an era of autonomous generation, the physical layer, the actual hardware we touch, remains bound by old-school engineering constraints. The friction of the physical world hasn't disappeared. So, after battling the geometry, the memory limits, and the hardware gremlins, what does the final interaction actually feel like? How fluid is this hybrid system? The integration is stunningly cohesive. Let's go back to that physical rotary knob on the device. When Adam pushes the screen down, it triggers a 90 millisecond LED flash. Just a quick blink? Yeah, just a brief pulse of light to give him instant tactile feedback that his click was registered. But simultaneously, the ESP32 fires off a WebSocket event back over the network to the server. Okay, so if he's reading one of those truncated news headlines on the screen and clicks the knob… The screen instantly alerts the server. The Hermes agent registers the click and can autonomously open the full article on Adam's main PC monitor. Or it can generate a three bullet point summary and drop it into his telegram chat. That's amazing. It transforms an isolated piece of plastic into a physical bridge that interacts with his entire digital ecosystem. The potential there is staggering. What began as a static, cheap hobbyist panel evolved into a hyper-contextual interface? And because the AI handles all the design, Adam doesn't have to spend hours pre-planning dashboards. Not at all. I know he mentioned the ultimate goal is setting this up as an ambient display driven by a morning cron job, an automated schedule. Which would be incredible. Imagine waking up and before you even ask, your server has already parsed your calendar, checked the traffic, analyzed the overnight server logs, written the custom HTML, and painted a bespoke morning briefing on that tiny circle by the time you reached the coffee machine. It really redefines what we consider a smart device. Yeah. We're moving away from the paradigm of hardware dictating the software. The screen is no longer a device that houses apps. It's a blank transient canvas that dedicated intelligence temporarily paints on based entirely on the user's immediate context. And that transition leads perfectly into a final thought I want to leave you with today. We kicked off this conversation by examining the rigid cookie cutter smart displays we all have in our homes right now. Devices that are inextricably linked to apps that a developer hard coded years ago in a corporate office. Right, the static world. But if an AI model can instantly comprehend physical hardware constraints, write custom code, render a bespoke user interface, and curate complex data for your exact situation in a matter of seconds, well what does that mean for the long term survival of the app? It's a massive question. Are we accelerating toward a post-app reality? A world where user interfaces don't sit dormant on a hard drive waiting to be launched, but instead only manifest for the exact moment you ask a question, perfectly shaped to your needs, and then a moment later simply dissolve back into nothing. It's a profound shift to consider. We may be witnessing the end of static software and the dawn of the purely transient interface. Something to seriously maul over the next time you mindlessly swipe through the grid of icons on your phone. That is all for today's deep dive. We will catch you on the next one.