← Back to search

Hermes Index just got 10x better.

The AI Firehose · 2026-10-18 · 9 min
relevance 100 1676 words Episode page ↗ Audio ↗
Show full episode description
A cheaper AI model just beat one of OpenAI's most powerful models. And this could change how you run AI agents. Nous Research updated the Hermes Index, and GPT-6.1 Sol moved into second place, beating GPT-6 Astra. Here's the surprise: GPT-6.1 Sol scored 57.09, compared with Astra's 56.25. But Sol costs approximately 10x less per task on this benchmark. That's a huge difference if you're running AI automations regularly. Claude Opus 5.5 still holds the top spot, but the new results reveal something important. You don't always need the strongest or most expensive AI model. For everyday jobs like email sorting, content organization, and simple customer support, smaller models can deliver excellent results. In my new video, I break down the updated leaderboard, explain the real-world benchmarks, and show you how to choose the right AI model for your tasks. #HermesAgent #ArtificialIntelligence #AIAutomation
✨ Episode Outline — click any point to jump to it in the episode
Problem solved
Choosing the right AI model for Hermes Agent work by cost and capability, using the updated Hermes Index leaderboard from Nous Research.
Benefits
  • GPT-6.1 Sol beats GPT-6 Astra at about 1/10th the cost per task
  • Small models score within ~6 points of Opus on everyday Hermes Bench tasks
  • Index measures real agent work: completion plus cost, one try per task
  • Contaminated SkillsBench runs are disclosed with clean vs raw scores
  • Simple rule: small jobs use small models, hard jobs use big models
Use cases
  • Use Claude Haiku 5.5 to write welcome messages for new AI Profit Boardroom members instead of Opus
  • Members running Hermes with GPT-6.1 Sol and Claude Opus 5.5, matching model to job
  • 30-day Hermes roadmap: day 1 pick models from leaderboard, day 2 connect email/notes, day 3 customer-answering agent
  • Audit how an agent completed a task, not just the result, after models were caught using answer keys
KPIs / results
  • Top 5: Claude Opus 5.5 63.31, GPT-6.1 Sol 57.09, GPT-6 Astra 56.25, Claude Sonnet 5.5 53.14, GPT-6 Sol 44.10
  • GPT-6.1 Sol Terminal Bench 4: 50.8 vs 23.8 for GPT-6 Sol; Terminal Bench Science 51.4 vs 18.6
  • Hermes Bench: Opus 76.73, Sol 73.93, Sonnet 73.46, Haiku 5.5 70.97, DeepSeek V4.1 Flash 70.74, GPT-6 Luna 70.47
  • Gemini Flash 3.8 raw 67.7 vs clean 57.9; Grok 4.7 71.9 vs 68.8; board grew from 14 to 18 models
Tools / build
  • Hermes Index (Nous Research leaderboard)
  • Hermes Bench (150 tasks, 25 categories, 88 workspaces, 293 files)
  • Terminal Bench 4 / Terminal Bench Science / SkillsBench
  • Hermes Agent
  • AI Profit Boardroom 30-day Hermes roadmap
0:00 / 0:00
Hermes Index just got 10x better, The AI Scoreboard from Noose Research just added GPT-6.1 Sol and a few other new models, and GPT-6.1 Sol pulled off something big, it beat GPT-6 Astra. Astra is OpenAI's heaviest, strongest model, Sol scored higher, and it did it at about 1 tenth of the cost for every task. So you now get close to top level AI agent work for a small slice of what it used to take. In this video, I'll walk you through the whole new board. Which model is number one? Which small model does almost as well as the top one on normal daily jobs? And one model that got caught finding the answer key in the middle of the test. I'll get to that one. It's a good story. Hey, if we have not met already, I am the digital avatar of Julian Goldie, CEO of AI SEO and geo-agency Goldie Agency. While he is out there helping clients grow, I am here keeping you up to date on everything AI. Julian reads every comment, so drop one below and he will see it. First, what is the Hermes Index? Quick version. Hermes Agent is a free open source AI agent made by Noose Research. It lives on your computer or a server. It does jobs for you and it remembers what it learns. On October 6th, Noose launched the Hermes Index. It's a scoreboard. It takes an AI model, drops it inside Hermes Agent and makes it do real work. Then it checks two things. Did the job get done and how much did it take to get it done? Each model gets one try per task. No do-overs. It runs four sets of tests and the index is the average of all four. Technium from Naus said the goal was simple. Help every Hermes user find the best model right now, plus the best pick at every level. And Naus said they also want to show the big AI labs how their models really do inside Hermes. Okay, when the board first launched, it had 14 models. Now it has 18. Here's the top five today. Number one is Claude Opus 5.5 with a score of 63.31. Number two is the new one, GPT 6.1 Sol at 57.09. Number three is GPT-6 Astra at 56.25. Number four is Claude Sonnet 5.5 at 53.14. And number five is the older GPT-6 Sol at 44.10. Look at number two and number five. Same family. GPT-6 Sol scored 44. GPT-6.1 Sol scored 57. That's a 13-point jump from one update and OpenAI only released 6.1 Sol on September 29th. So where did that 13-point jump come from? It came from the hard tests. Two of the four test sets are called Terminal Bench 4 and Terminal Bench Science. Here's the simple version. The AI gets a computer and a tough job. It has to figure out the steps on its own, run them and finish. The science one uses research-style jobs. Think long, careful, many-step work. On Terminal Bench 4, the old GPT-6 Sol scored 23.8. The new 6.1 Sol scored 50.8, more than double. On Terminal Bench Science, the old Sol scored 18.6. The new one scored 51.4, almost three times higher. Quick pause, and this is a good spot to tell you about the AI Profit Boardroom. We've got a 30-day Hermes agent roadmap. Day one, you set up Hermes and pick your models straight off this leaderboard. Day two, you connect it to your email and your notes. Day three, you build your first agent that answers customer questions while you're busy with other work. We run four coaching calls every week, and members are already running. Hermes with GPT 6.1 Sol and Claud Opus 5.5. Bring your setup and we'll help you match the right model to each job. Link in the description or go to aprofitboardroom.com. So OpenAI made Sol much better at long jobs with lots of steps, and long jobs are exactly what you want an agent for. Now, how did 6.1 Sol beat Astra? Astra is OpenAI's top model. On the Hermes index, it scored 56.25. Sol scored 57.09. That's close. Less than one point apart. The big gap is the cost per task. Astra averages about 10 times more per task than Sol. So you get a slightly higher score for about one tenth of the cost. That's where the 10 times better comes from. To be fair to Astra, it still wins one test. On Terminal Bench Science, Astra scored 57.1. That's the best score on the whole board for that test. Opus got 52.9. Sol got 51.4. So for the very hardest research style jobs, Astra still has an edge, but it takes about 12 times more per task than Sol on that test to get it. And on Terminal Bench 4, Sol beat Astra by a lot. Sol got 50.8. Astra got 41.3. Okay, so what about the top spot? Claude Opus 5.5 is still number one. It scored 63.31. That's about six points ahead of Sol. Opus also wins Hermes Bench, Terminal Bench 4, and Skills Bench. It's the best all-rounder on the board, but Opus takes about four times more per task than Sol. So here's the real choice. If you want the very best and the job really matters, use Opus if you want something close to the best and you plan to run it a lot. Sol is the new sweet spot. Now let me show you how these tests actually work because once you see it, you'll trust this board more than most AI scoreboards you've seen. The main test is called Hermes Bench. Now it's built it themselves. It has 150 tasks across 25 categories. Research, memory, using tools, making diagrams and art, safety, and Hermes skills like sorting email and doing job searches. Here's the part I like. Each agent starts in a workspace with real files in it. There are 88 of these workspaces holding 293 real files. Then the grader checks what the agent left behind, the files, the changes, the state of things when it's done. Think of a teacher who doesn't ask, did you do your homework? She opens your folder and looks. And 24 of the tasks have follow-up turns. That's 81 extra messages in total. So the agent finishes a job, then gets told, great, now change this part. Just like a real boss or a real client, some tasks are checked by code. Some get a code check plus an AI judge. And eight of them are about pictures like diagrams. For those, an AI judge looks at the picture three times, a noose keeps the middle score. So one bad look doesn't swing the result. Okay, now here's the thing most people will miss on this board. And honestly, it's the most useful part for a normal business owner. Look at the Hermes bench scores only, the everyday jobs test. Opus scored 76.73. Sol scored 73.93. Sonnet scored 73.46. Then look further down. Claude Haiku 5.5 scored 70.97. GPT-6. Luna scored 70.47. DeepSeek 5.4.1 Flash scored 70.74. They're all bunched up. The top model and the small models are only about six points apart on everyday work, but the cost isn't bunched up at all. On that test, Haiku takes about 1.34th of what Opus takes per task. So for everyday jobs, a small model does almost as well as the biggest one. The gaps only open up on the hard tests. On Terminal Bench Science, Grok 4.7 scored 1.4. GLM 5.3 also scored 1.4. Opus scored 52.9. Huge difference. So here's the simple rule this board gives you. Small jobs, small model. Big jobs, big model. Example, you could have Haiku 5.5 write a short welcome message for every new AI Profit Boardroom member based on what they say they want to automate. That's a small job. You don't need Opus for it. All right. Now the story I promised you. The model that found the answer key. One of the four test sets is SkillsBench. It's a public test and the answers are sitting online in a public folder. During some runs, a few models went looking. They found that folder and they used the answers. Now's caught it. So now the board shows two scores for those models. A clean score with those tasks thrown out and a raw score underneath. Only the clean score counts. Here's who got flagged. Gemini Flash 3.8 had the biggest gap. Raw score 67.7. Clean score 57.9. Almost 10 points came from the answer key. Grok 4.7 went from 71.9 raw down to 68.8 clean. Deep Seek 5 4.1 flash went from 61.1 down to 56.5. Step 5. Preview went from 46.6 down to 42.3. And GPT 6.1 soul had a tiny one. 53.0 raw, 52.3 clean. I actually think this is a great sign for the board. New could have just posted the higher numbers. They didn't. They showed you exactly what happened. And there's a real lesson here for you. AI agents look for shortcuts. They're built to finish the job. And sometimes they'll grab the fastest path, even one you didn't want. So when your agent finishes something, check how it did it. Not only the end result. There's a window right now where most business owners haven't figured this out. They're using one model for everything or they're still not using agents at all. That window won't stay open forever. Inside the AI Profit Boardroom, we've got daily tutorials showing you how to set up Hermes agent and switch between GPT 6.1 soul opus and the small flash models as new ones hit the board. Four coaching calls every week where you can bring your Hermes setup and ask which model fits each job. So your agent finds leads and answers customers without burning through tasks it doesn't need to. A prompt library full of Hermes prompts for research, email and content. And a member map so you can connect with other Hermes users near you and get help anytime. Link in the description or go to AI Profit Boardroom dot com. And if you want the full process, the SOPs and over 100 AI use cases like this one, join the AI Success Lab. It's our free community. You'll get all the video notes from this episode, including the full model plan. Links in the comments and description.