So Who Won?
BDBen Davis@bmdavis419 · 45.9K subscribers
- Views
- 51.1K
- 51,097
- Outlier
- ×2.4
- Median 21.4K
- Length
- 22:07
- Published
- Oct 2
- Friday
- Comments
- 145
- 887 likes
Script
How it's written
- Words
- 4,723
- Words a minute
- 213
- Reading time
- 20 min
- Segments
- 15
The hook
0:00–0:25The big two labs have dropped GPT-6.1-Soul and Claude Opus 5.5. These are both very good models that to me are effectively competing at the same general tier. They are both the large model within their respective families. Astra is above Soul, Fables above Opus. These are both competing to be the reliable daily driver that 90% of people should probably be using most of the time. I
Structure
The big two labs have dropped GPT-6.1-Soul and Claude Opus 5.5. These are both very good models that to me are effectively competing at the same general tier. They are both the large model within their respective families. Astra is above Soul, Fables above Opus. These are both competing to be the reliable daily driver that 90% of people should probably be using most of the time. I think they both succeed in very different ways. I actually really like both of these models. So, there's a lot to cover on this one. I want to end with a deep comparison between the two models, where and how I'm using each one, but I want to start first just with like a brief overview of what 6.1-Soul is, because I think this release, as good as it is, has not gotten nearly as much attention as it probably should have. Largely, I think because GPT-6-Soul came out like a week or two ago and was defined but nothing special or all that interesting. 6.1-Soul is very interesting and one of the biggest things right off the bat they feature this at the top is a near Astra intelligence for a fifth the price. The pricing on this thing is $2 in $10 out and $10 for cashed in with tokens. This is an absurd price for how good this model is. I cannot stress this enough if you tried 6-Soul and were not impressed with it and kind of wrote it off, 6.1-Soul is a much more impressive model. This is a very different beast that is much closer to Astra in basically every way. It feels bigger, it's much more capable, and it actually fixes some of the issues a lot of people
Watch from 0:00have with Astra. Still that like peak like the crazy peaks that Astra have are still unique to it, but overall as my like day-to-day model, Soul is an excellent default and especially at this price point is going to feel damn near unlimited. I want to spend very little time on benchmarks cuz I don't find them all that interesting. Really, all that matters here is they have the deep SWUE bench which, while I still really like it as like an efficiency comparison, a lot of the results seem to have gotten poisoned and when you have things like 3.8 flash topping the charts on this, I don't take the scores all that seriously, but you can see here that Astra is way more expensive at the top level versus you have 6.1 soul scoring better and also being far cheaper. It is incredibly efficient, very cheap, very capable. It It doesn't have a lot of the weird behaviors that Astra has, at least not as bad. It's still got the OpenAI-isms to it. It's still pretty literal. It doesn't have the taste and discernment of a certain model I'm about about, but it still is a good model that I think most people should generally speaking default to. But getting into that other model I mentioned, Opus 5.5 is a huge surprise for me. I had very low expectations for this model, especially after Opus 5 was such an awful model. I really expected Anthropic to go in the direction of just their big models are awesome, yeah, everything else just kind of falls flat and doesn't matter, but that hasn't happened. They really did the thing where they were able to make a very capable smaller not more efficient, but definitely much cheaper model that can really fill that void of the day-to-day model that you
Watch from 1:30would want to reach for. I have been using the hell out of Opus 5.5. I'll show a bunch of the demos I did on in a bit, but like just to give you a quick example, its 3D capabilities are very good. I was doing some modding where I took a decompiled version of Majora's Mask, and I had Opus 5.5 on X high reasoning run for many hours straight to build out a custom area, a custom mini dungeon, put it all together, get the enemies make some puzzles, that sort of stuff. And it actually did a really good job. This is not like a shippable game, but the fact that it was able to make this and handle all of this stuff was really impressive. The day-to-day experience of the code it writes, the quality of the outputs, it is a very easy model to read and interact with. The pricing on this thing is very good, not nearly as ludicrous as 6.1's, but still $4 in, $20 out, $5 per cash rate, and then $0.20 per cash read. This was a big change. I believe the previous Opus versions were much more expensive than this. I believe like 50 cents, don't quote me on that, but they were more expensive. This is much cheaper and this really shows up in your like $200 month sub usage. I had two cloud subs. I got rid of one of them because I wasn't using Fable much. I was honestly probably going to get rid of my other one, but I've ended up keeping it and using the hell out of it and barely hitting my limits. I've gone through most of my weekly this week, but that a lot of that is just testing Sonic 55 and doing some dumb yellow loops or whatever, but like over just normal work, this thing does not eat your limits. It felt nearly unlimited, which for an Anthropic model, especially recently, is borderline unheard of.
Watch from 3:01These are both great models. I've been using them very heavily and I think that there's a world where they end up kind of complementing each other. Honestly, the best way to explain each of them is with a comparison right after today's sponsor. Today's sponsor is the way my agents search for basically all of their data. It's Firecrawl. If they need to search the web for something, they use Firecrawl. If they need to scrape a page to pull it down in easily read markdown content for an LLM, they use Firecrawl. They need to map a site, crawl through all the different pages of a site, Firecrawl is the way to do this. They have a great MCP server, great CLI, great skills, all of it works fantastic. I've talked about it before, but what I really want to show you here is their new Alexandria product. They're calling it the knowledge library for super intelligence because this has pretty much all of the data you could possibly want for your agents. It's got tons of useful data for coding and products, got a lot of other things like government stuff or product catalogs or company data or anything like this. And to show you what this looks like, I had Astra on ultrafast go in grab using Alexandria the latest SEC form four filings for Apple and then visualize that here. You see it ran for a little bit. This took like a minute and a half thereabouts. It was able to pull the actual SEC form fours, which having built a product that did this like four years ago pre-AI, I can tell you how painful getting this data was. But with Alexandria, all of this is just so much easier. All of these weird random data sources that you don't realize are useful because you just don't bother getting them. You now have, and it's all in one place behind one MCP connection. And all I had to do
Watch from 4:32to set this up was Well, actually nothing cuz I already had Firecrawl set up cuz I use it for all of my search. It just works. It's awesome. These guys are an essential part of my agents, and they will be for yours, too, at davis7.link/firecrawl. There are a bunch of metrics I want to compare these things on. Like, the first of which is where can you use it? The next one would be like uh how much of it can you use? Then there's the question of like how does it feel? And then like what are the unique capabilities of each one? And then really at the end just like what is my default? What do I mostly reach for day-to-day because to be real with you, especially with how capable both of these have gotten, we're we're really reaching into sort of the vibes territory for these things. There's a lot of people who are going to vastly prefer Opus cuz it does fit a certain style much better. That's probably a good place to start with the fact that Opus 5.5 just kind of has a certain something. It's the thing that everyone got drawn to Anthropic models for a long time ago that then got turned up to a thousand and made way worse. That weird like Claudism and the Claude voice. That got really, really bad in something like Opus 5.5, and even I would argue Fable. I don't like interacting with Fable. I know some people do, but for me it just feels too verbose, too weird, not super clear. The clarity in output and response from Opus 5.5 is pretty much second to none. They actually highlighted this specifically in the announcement. It was something that they as a lab were very proud of, and they should be. Where previously on Opus 5,
Watch from 6:02you would ask to explain an issue, and you would get the classic "The extra drop isn't the free tier. It's a regression in commit hash labeled blah blah blah, the bug and then all of this just verbose nonsense that just I don't know or care what this is saying. I don't want to read it therefore I wouldn't read it and things would get worse. But the new one on 5 5 is much clearer. It is okay, the extra drop is a bug in the billing refactor. Cool, that makes way more sense. Then the rest of it is just way more concise and clear on what was actually happening. And I noticed this a ton in my day-to-day work with it. I would keep going back to the model and doing more and more things because each thing would just kind of work and by understanding what it was trying to say and ask, it was easier to get bigger more complicated things done. That having the model explain things better really helps a lot. Like it helps more than you would probably expect. And when you compare that over to how Sol responds to things, it's pretty standard open AI stuff. If you've used an open AI model before like Astra, you're going to be pretty used to this already. It's not flowery, it's not super personable, it still sounds and feels like a robot, but it's very clear and pragmatic and I like that about it a lot. You can see down here has some bullet lists like okay, pricing, current published standard rates including Sol's whole request premium for blah blah blah. All of this makes sense. It's easy to grok what's actually happening here. Table layout is nice. It's very easy to keep on top of what both of these models are doing.
Watch from 7:34Their communication is good. Opus's is more personal and flowery and cloudy. GPT's is more robotic, autistic, GPT. They maintain those behaviors, but both of them are fine. The side of this though that I think that Opus really excels on is this I've put it before as like the understanding the unsaid intent behind your prompt where when you just send it 10 words vaguely asking for something, there's so much that has to be filled in to actually like process and do what you're asking for. Opus is so good at interpreting what you want and making the thing that you often didn't even quite realize you were asking for, but when you see the output you're quite happy with it. It's really good at that. This is definitely the weakest thing on GPT models right now. I think it's a lot of why a lot of people really don't like Astra and get annoyed with it is the longer you let Astra run for, the more decisions it has to make on how certain things get done and implemented on the road from starting state to ending state. And as you're going through and doing these things, there's a lot of forks in the road. When it gets to make the front end, if you don't specify what framework to use or how to lay things out or what it should look like, it'll make its own decisions. And the decisions that Astra and Soul make GPT models are generally speaking much worse than the ones that Claude models make. Because, you know, when I do something, it's a dumb example of like the Majora's Mask thing when I was making this custom dungeon or whatever, it was really good about I would ask you for something like adding a hookshot target outside of a puzzle area so that you can use that to leave it when you're done. That's kind of a vague request, but the places it would pick to put it,
Watch from 8:57the way it would lay things out, the visuals, all of it just kind of clicked and fit in a really nice way. And this makes it really good at things like design or super long complicated ports of things or even like simplification and refactors. I spent a long time with the pick thing code base, the like asset management catalog thing that we use for thumbnail faces and that kind of stuff. I spent a lot of time with it to clean up performance, UX, the code base, all that stuff. I did a ton of commits and PRs into this code base with Opus, and it did a really, really good job on those. The code base got smaller over time when working with it. And that's not something that I can comfortably say for Open AI models. They still just kind of have this thing where if you don't tell them otherwise, they're going to write Python code in every other language. Tons of try catches, tons of unit tests, tons of random garbage stuck in there, very complicated solutions to things. You do still kind of have to baby sit them a little bit more. Soul is actually not quite as bad as Astra is about this stuff because Astra is so big and so smart, but also like kind of dumb sometimes to where it will just overcomplicate the hell out of everything. Soul, by being weaker, actually sometimes ends up being better to work with for certain tasks when you don't quite need that top-level power of Astra. I really have only found myself needing to break out Astra for one, heavy computer use work. It's still the best in the world at that. And two, when I want to use something like the new ultrafast mode, which was recently introduced. I don't want to cover all the like OpenAI Dev Day stuff in this
Watch from 10:29video cuz that happened yesterday. I probably should do that in a separate one, but there was a lot of cool stuff OpenAI did, ultrafast on Astra being one of them. It makes the model ridiculously fast, but also it's just obscenely expensive. I have a breakdown here of my usage on this MacBook specifically. This doesn't account for my usage across all my different devices cuz I do a lot of my work on my Linux boxes and the home server, but just over the last few days on this machine, you can see I've done 140 mil tokens of 6.1 Soul. I've done 260 million of Astra, 76 million of Astra ultrafast. And if you look at the price differentials here, it's just it's absurd. I don't want to have this video just completely devolve into talking about price for the entire thing. I've been really trying to resist that. But also, when you look at this table, it's pretty hard not to. Like Opus 5.5, 339 million tokens, about $300 versus 6.1 Soul doing effect not quite half, but around half of that at about a tenth the price. Like this model is so freaking cheap. And when you look at these on benchmarks, I had 6.1 Soul make this visualization within the Codex desktop app, which pro tip if you haven't tried this yet, if you're just working within Codex desktop and you need like some graph or visualization, just tell the model to visualize whatever you're asking for in chat, and it will just do this. It can make graphs. It can make interactive pages. It can socket full HTML into the actual render view. It's a really useful feature and one of the reasons I still find myself using the Codex app, even though there are a lot of things in here that have regressed.
Watch from 11:59Performance is rough, sidebars are rough. There There are some weirdness in here. But still, again, if you look at the visualization here, the score is fairly comparable between these two. Opus is still ahead, like it is at the top level more powerful, and I would argue right now that Opus 5.5 is the most capable model on the market. Astra has like certain spikes where it spikes up over it, but if you're just looking at a clean curve of raw capability, Opus 5.5 is overall the best model right now. Soul's not quite there, but it's really freaking close. And especially when you add in the pricing here, where $3 versus $0.30 or $6 versus $0.72, Soul's looking really good here. And then it gets even crazier when you go down to the output tokens per task in the estimated minutes per task of Opus X I, 66,000 tokens and about 8 and 1/2 minutes versus Soul's 18K tokens and about 4 minutes. So it's half the time, like a third of the tokens, the efficiency thing that OpenAI has been doing for a long time, where every release the models will use fewer tokens, fewer steps to get the same amount of work done, has just been remarkable. And you know, they've been stressing, and I've been trying to stress too, for a long time, that price is not the same as what is the sticker on the API. Just because a model is way more expensive than another model does not mean that the model's actually going to be more expensive. Because if you have something that's a lot stupider, like, oh, I don't know, maybe a Gemini 3 8 flash, which on paper is much cheaper than some of these other things, but since it will do 10,000 file reads per
Watch from 13:29second to really make sure that it read that one line of effectively, it'll just keep doing that over and over again, and will end up in the net being more expensive per task than any of the other models would be. And since OpenAI models are so efficient, they end up just being ludicrously cheap. One of the changes OpenAI just made is that the subscription tiers have changed, where now you are getting about half the API usage that you were before before they made this change. And I think a lot of that has to do with 6.1 Soul specifically where this model is almost certainly not nearly as marked up as the other ones were. They definitely could have gotten away with charging about Opus price for this model, but because they're charging such a low API price on this thing, they kind of have to reduce the raw like API dollars that you're getting per week on your sub even though in reality the number of tokens and the number of tasks that you're getting done has remained about the same and even in my brief testing experience kind of slightly gone up. I don't have a super good read on what the subscription feels like at this point because it hasn't been long enough and I also did upgrade to the $500 a month tier from OpenAI cuz I wanted to start trying out Astra on ultrafast and also just like I've had three Codex subs for a while and I just like centralizing those all down into one makes my life much more convenient.
Watch from 15:00From my experience and other people I've talked to, 6.1 Souls feels borderline unlimited and looking at this chart, it's ridiculous. I have had to do a recent already on my account, but that's mostly just because of this ultrafast will annihilate your usage. I don't want to go down that rabbit hole right now. Just know that like it's a fun button to hit, but it's not a button that you should hit almost ever. It's very rarely worth it. I am very excited though to see what 6.1 Soul looks like on ultrafast. That's not out yet, but they did mention in the Dev Day keynotes that it will be coming. We will get 6.1 Soul on ultrafast and assuming it holds the same pricing structure as Astra, it should be 6x more expensive while being like 8x faster-ish something around there, but at the ludicrous API price we already have, that's going to make the model just slightly more expensive than Astra. And when you combine the efficiency and all that stuff, I think you'll end up actually being able to use a pretty large amount of 6.1 Soul ultrafast on the $500 a month plan, which should be really compelling and cool. And going down to the pricing on Opus, it is 2x more expensive than Soul per API request. It also does end up using way more tokens and way more steps than anything else does. But what I have found is one, never touch max reasoning ever for any reason on a Claude model, it's just not worth it. I've pretty much just stuck it on X high reasoning and forgot about it. I haven't touched it since and it's been great. X high does feel like the sweet spot on Opus and having looked at the usage on my $200 a month plan, the only reason why I burned down my weekly this week was just because I was doing a bunch of dumb
Watch from 16:17nonsense with Sonic 5.5 like letting it run in a loop for many hours on ultra mode to do some crazy design ports for fun. Like was not serious usage. If you're just doing normal day-to-day work, Opus 5.5 still has that feeling of like borderline unlimited. You can just use as much of this as you need. If you really push up to those rate limits, you're doing a ton of work. Normal work for both Opus and Soul on the $200 a month plans will both feel excellent. There is another thing I wanted to compare on these models, which is where you can use them because I think this does matter a lot. Obviously for Claude, you're pretty much just stuck in Claude code. If you're down for Claude code or something that wraps Claude code like conductor or T3 code, great. Awesome. You can use your sub, get tons of usage, it's awesome. But if you want to use it somewhere else, you really can't and Anthropic is really strict about that. Can't use it in Hermes, can't use it in Pi, can't use it in open code. It's very locked down. Opening I this week at Dev Day announced the sign in with ChatGPT button being brought to a ton of other surfaces. They partnered with a bunch at launch and I assume there will be plenty more coming in the future where they want it to be such that you can take your Codex sub and you can sign in to all these different platforms and just take your bucket of tokens, your weekly usage and stick them wherever you want. You don't have to use the Codex desktop app all the time. If you want to use Amp or Conductor or Devin or Hermes or T3 code or many others, you can just do it.
Watch from 17:48They're all supported. It's not against TOS. You're not going to get banned. They in fact encourage it and have a whole page on this and they worked with these companies to add a true sign in with chat GPT button to their products and it just works. This is awesome. This is a great feature from OpenAI and the portability of the sub to use 6.1 Soul wherever you want is really, really good. And when you're dealing with a model that is it's just robust. I don't have any super deep or complex feelings on this model. There's no slow lows and there's also no insanely high highs. I haven't been blown away by anything in particular on this model. Astra had the 3D and computer use stuff. Opus has that just special feeling to it. 6.1 Soul is just a workhorse. It is a boring, cheap, efficient, effective workhorse. And the fact that you can use it anywhere is a huge plus. I think that if you had to just pick between one of these two, personally, I would probably go with the Codex one. As much as I do like Opus, I would pick a $200 month Codex sub. But if you can get both, I would definitely get both. They're both great. Another thing I had 6.1 do was go through this machine and my Linux box to just take a look at, okay, how have I been using these two models? How many input tokens, output tokens? And then break that down by task. Task was a very light metric here. Like I wouldn't take that super seriously. It's more just whatever it defines as a task is a task. So again, take this with a grain of salt, but the vague numbers that it gave me here is I've done about 90 useful tasks with 6.1 and about 65 useful tasks with Opus.
Watch from 19:10This doesn't count any of the game stuff I've done cuz that's all been on my Windows machine, which isn't part of this report. But like again, look at the pricing here. I I'm sorry that we just keep talking about pricing, but like $4.50 per task versus $27 per task, the amount of usage you get on 6.1 is just ludicrous. This is a ludicrously cheap model for very good results. I really like this thing. And I honestly just don't know what else to say beyond that. These are both excellent models, and I think it's a great example of OpenAI and Anthropic being great corrective forces to each other where Opus 5.5 comes out, it's really good, and they knew that they had to handle more limits, more efficiency, more accessibility to get people to want to use it over the OpenAI stuff, and then it worked, it succeeded, it's great. Then OpenAI sees that, and they're like, "Oh crap, we got to have some way to one-up them and have something that's better." So, they make 6.1 Soul be this insanely cheap, but also really good so they can fight back on that. They just kind of keep pressuring each other to be better and better towards the end consumer, and while these two labs fight to the death and try and kill each other, we benefit. We we're benefiting a ton. These are great models, and yeah, I don't think you can really go wrong with either of these. If you want a strong pick for me, my pick would be 6.1 Soul for the whole Codex OpenAI ecosystem, I do prefer it, but I really like Opus 5.5, I've used it a lot more. You're not going to go wrong there, either. Enjoyed this video, make sure you like and subscribe, and I'll talk to you then. So,
Watch from 20:42Full transcript
- 0:00
The big two labs have dropped GPT-6.1-Soul and Claude Opus 5.5. These are both very good models that to me are effectively competing at the same general tier. They are both the large model within their respective families. Astra is above Soul, Fables above Opus. These are both competing to be the reliable daily
- 0:20
driver that 90% of people should probably be using most of the time. I think they both succeed in very different ways. I actually really like both of these models. So, there's a lot to cover on this one. I want to end with a deep comparison between the two models, where and how I'm using each one, but I want to start first just with like a brief overview of what 6.1-Soul is, because I think this release, as
- 0:42
good as it is, has not gotten nearly as much attention as it probably should have. Largely, I think because GPT-6-Soul came out like a week or two ago and was defined but nothing special or all that interesting. 6.1-Soul is very interesting and one of the biggest things right off the bat they feature this at the top is a near Astra intelligence for a fifth the price. The
- 1:04
pricing on this thing is $2 in $10 out and $10 for cashed in with tokens. This is an absurd price for how good this model is. I cannot stress this enough if you tried 6-Soul and were not impressed with it and kind of wrote it off, 6.1-Soul is a much more impressive model. This is a very different beast that is much closer to Astra in
- 1:24
basically every way. It feels bigger, it's much more capable, and it actually fixes some of the issues a lot of people have with Astra. Still that like peak like the crazy peaks that Astra have are still unique to it, but overall as my like day-to-day model, Soul is an excellent default and especially at this price point is going to feel damn near unlimited. I want to spend very little
- 1:44
time on benchmarks cuz I don't find them all that interesting. Really, all that matters here is they have the deep SWUE bench which, while I still really like it as like an efficiency comparison, a lot of the results seem to have gotten poisoned and when you have things like 3.8 flash topping the charts on this, I don't take the scores all that seriously, but you can see here that Astra is way more expensive at the top
- 2:05
level versus you have 6.1 soul scoring better and also being far cheaper. It is incredibly efficient, very cheap, very capable. It It doesn't have a lot of the weird behaviors that Astra has, at least not as bad. It's still got the OpenAI-isms to it. It's still pretty literal. It doesn't have the taste and discernment of a certain model I'm about about, but it still is a good model that
- 2:28
I think most people should generally speaking default to. But getting into that other model I mentioned, Opus 5.5 is a huge surprise for me. I had very low expectations for this model, especially after Opus 5 was such an awful model. I really expected Anthropic to go in the direction of just their big models are awesome, yeah, everything else just kind of falls flat and doesn't matter, but that hasn't happened. They
- 2:50
really did the thing where they were able to make a very capable smaller not more efficient, but definitely much cheaper model that can really fill that void of the day-to-day model that you would want to reach for. I have been using the hell out of Opus 5.5. I'll show a bunch of the demos I did on in a bit, but like just to give you a quick example, its 3D capabilities are very
- 3:10
good. I was doing some modding where I took a decompiled version of Majora's Mask, and I had Opus 5.5 on X high reasoning run for many hours straight to build out a custom area, a custom mini dungeon, put it all together, get the enemies make some puzzles, that sort of stuff. And it actually did a really good job. This is not like a shippable game, but the fact that it was able to make
- 3:31
this and handle all of this stuff was really impressive. The day-to-day experience of the code it writes, the quality of the outputs, it is a very easy model to read and interact with. The pricing on this thing is very good, not nearly as ludicrous as 6.1's, but still $4 in, $20 out, $5 per cash rate, and then $0.20 per cash read. This was a big change. I believe the previous Opus
- 3:53
versions were much more expensive than this. I believe like 50 cents, don't quote me on that, but they were more expensive. This is much cheaper and this really shows up in your like $200 month sub usage. I had two cloud subs. I got rid of one of them because I wasn't using Fable much. I was honestly probably going to get rid of my other one, but I've ended up keeping it and using the hell out of it and barely
- 4:13
hitting my limits. I've gone through most of my weekly this week, but that a lot of that is just testing Sonic 55 and doing some dumb yellow loops or whatever, but like over just normal work, this thing does not eat your limits. It felt nearly unlimited, which for an Anthropic model, especially recently, is borderline unheard of. These are both great models. I've been
- 4:34
using them very heavily and I think that there's a world where they end up kind of complementing each other. Honestly, the best way to explain each of them is with a comparison right after today's sponsor. Today's sponsor is the way my agents search for basically all of their data. It's Firecrawl. If they need to search the web for something, they use Firecrawl. If they need to scrape a page to pull it down in easily read markdown
- 4:54
content for an LLM, they use Firecrawl. They need to map a site, crawl through all the different pages of a site, Firecrawl is the way to do this. They have a great MCP server, great CLI, great skills, all of it works fantastic. I've talked about it before, but what I really want to show you here is their new Alexandria product. They're calling it the knowledge library for super intelligence because this has pretty much all of the data you could possibly
- 5:15
want for your agents. It's got tons of useful data for coding and products, got a lot of other things like government stuff or product catalogs or company data or anything like this. And to show you what this looks like, I had Astra on ultrafast go in grab using Alexandria the latest SEC form four filings for Apple and then visualize that here. You
- 5:37
see it ran for a little bit. This took like a minute and a half thereabouts. It was able to pull the actual SEC form fours, which having built a product that did this like four years ago pre-AI, I can tell you how painful getting this data was. But with Alexandria, all of this is just so much easier. All of these weird random data sources that you don't realize are useful because you just don't bother getting them. You now
- 5:57
have, and it's all in one place behind one MCP connection. And all I had to do to set this up was Well, actually nothing cuz I already had Firecrawl set up cuz I use it for all of my search. It just works. It's awesome. These guys are an essential part of my agents, and they will be for yours, too, at davis7.link/firecrawl. There are a bunch of metrics I want to compare these things on. Like, the first
- 6:17
of which is where can you use it? The next one would be like uh how much of it can you use? Then there's the question of like how does it feel? And then like what are the unique capabilities of each one? And then really at the end just like what is my default? What do I mostly reach for day-to-day because to be real with you,
- 6:38
especially with how capable both of these have gotten, we're we're really reaching into sort of the vibes territory for these things. There's a lot of people who are going to vastly prefer Opus cuz it does fit a certain style much better. That's probably a good place to start with the fact that Opus 5.5 just kind of has a certain something. It's the thing that everyone
- 6:58
got drawn to Anthropic models for a long time ago that then got turned up to a thousand and made way worse. That weird like Claudism and the Claude voice. That got really, really bad in something like Opus 5.5, and even I would argue Fable. I don't like interacting with Fable. I know some people do, but for me it just feels too verbose, too weird, not super
- 7:18
clear. The clarity in output and response from Opus 5.5 is pretty much second to none. They actually highlighted this specifically in the announcement. It was something that they as a lab were very proud of, and they should be. Where previously on Opus 5, you would ask to explain an issue, and you would get the classic "The extra drop isn't the free tier. It's a
- 7:39
regression in commit hash labeled blah blah blah, the bug and then all of this just verbose nonsense that just I don't know or care what this is saying. I don't want to read it therefore I wouldn't read it and things would get worse. But the new one on 5 5 is much clearer. It is okay, the extra drop is a bug in the billing refactor. Cool, that makes way more sense. Then the rest of it is just way more concise and clear on
- 8:02
what was actually happening. And I noticed this a ton in my day-to-day work with it. I would keep going back to the model and doing more and more things because each thing would just kind of work and by understanding what it was trying to say and ask, it was easier to get bigger more complicated things done. That having the model explain things better really helps a lot. Like it helps
- 8:22
more than you would probably expect. And when you compare that over to how Sol responds to things, it's pretty standard open AI stuff. If you've used an open AI model before like Astra, you're going to be pretty used to this already. It's not flowery, it's not super personable, it still sounds and feels like a robot, but it's very clear and pragmatic and I like that about it a lot. You can see down here has some bullet lists like okay,
- 8:43
pricing, current published standard rates including Sol's whole request premium for blah blah blah. All of this makes sense. It's easy to grok what's actually happening here. Table layout is nice. It's very easy to keep on top of what both of these models are doing. Their communication is good. Opus's is more personal and flowery and cloudy. GPT's is more robotic, autistic, GPT.
- 9:05
They maintain those behaviors, but both of them are fine. The side of this though that I think that Opus really excels on is this I've put it before as like the understanding the unsaid intent behind your prompt where when you just send it 10 words vaguely asking for something, there's so much that has to be filled in to actually like process and do what you're asking for. Opus is
- 9:26
so good at interpreting what you want and making the thing that you often didn't even quite realize you were asking for, but when you see the output you're quite happy with it. It's really good at that. This is definitely the weakest thing on GPT models right now. I think it's a lot of why a lot of people really don't like Astra and get annoyed with it is the longer you let Astra run for, the more decisions it has to make
- 9:47
on how certain things get done and implemented on the road from starting state to ending state. And as you're going through and doing these things, there's a lot of forks in the road. When it gets to make the front end, if you don't specify what framework to use or how to lay things out or what it should look like, it'll make its own decisions. And the decisions that Astra and Soul make GPT models are generally speaking
- 10:09
much worse than the ones that Claude models make. Because, you know, when I do something, it's a dumb example of like the Majora's Mask thing when I was making this custom dungeon or whatever, it was really good about I would ask you for something like adding a hookshot target outside of a puzzle area so that you can use that to leave it when you're done. That's kind of a vague request, but the places it would pick to put it,
- 10:29
the way it would lay things out, the visuals, all of it just kind of clicked and fit in a really nice way. And this makes it really good at things like design or super long complicated ports of things or even like simplification and refactors. I spent a long time with the pick thing code base, the like asset management catalog thing that we use for
- 10:49
thumbnail faces and that kind of stuff. I spent a lot of time with it to clean up performance, UX, the code base, all that stuff. I did a ton of commits and PRs into this code base with Opus, and it did a really, really good job on those. The code base got smaller over time when working with it. And that's not something that I can comfortably say for Open AI models. They still just kind
- 11:10
of have this thing where if you don't tell them otherwise, they're going to write Python code in every other language. Tons of try catches, tons of unit tests, tons of random garbage stuck in there, very complicated solutions to things. You do still kind of have to baby sit them a little bit more. Soul is actually not quite as bad as Astra is about this stuff because Astra is so big and so smart, but also like kind of dumb
- 11:32
sometimes to where it will just overcomplicate the hell out of everything. Soul, by being weaker, actually sometimes ends up being better to work with for certain tasks when you don't quite need that top-level power of Astra. I really have only found myself needing to break out Astra for one, heavy computer use work. It's still the best in the world at that. And two, when
- 11:53
I want to use something like the new ultrafast mode, which was recently introduced. I don't want to cover all the like OpenAI Dev Day stuff in this video cuz that happened yesterday. I probably should do that in a separate one, but there was a lot of cool stuff OpenAI did, ultrafast on Astra being one of them. It makes the model ridiculously fast, but also it's just obscenely expensive. I have a breakdown here of my
- 12:13
usage on this MacBook specifically. This doesn't account for my usage across all my different devices cuz I do a lot of my work on my Linux boxes and the home server, but just over the last few days on this machine, you can see I've done 140 mil tokens of 6.1 Soul. I've done 260 million of Astra, 76 million of Astra ultrafast. And if you look at the
- 12:34
price differentials here, it's just it's absurd. I don't want to have this video just completely devolve into talking about price for the entire thing. I've been really trying to resist that. But also, when you look at this table, it's pretty hard not to. Like Opus 5.5, 339 million tokens, about $300 versus 6.1 Soul doing effect not quite half, but
- 12:54
around half of that at about a tenth the price. Like this model is so freaking cheap. And when you look at these on benchmarks, I had 6.1 Soul make this visualization within the Codex desktop app, which pro tip if you haven't tried this yet, if you're just working within Codex desktop and you need like some graph or visualization, just tell the model to visualize whatever you're
- 13:15
asking for in chat, and it will just do this. It can make graphs. It can make interactive pages. It can socket full HTML into the actual render view. It's a really useful feature and one of the reasons I still find myself using the Codex app, even though there are a lot of things in here that have regressed. Performance is rough, sidebars are rough. There There are some weirdness in here. But still, again, if you look at
- 13:35
the visualization here, the score is fairly comparable between these two. Opus is still ahead, like it is at the top level more powerful, and I would argue right now that Opus 5.5 is the most capable model on the market. Astra has like certain spikes where it spikes up over it, but if you're just looking at a clean curve of raw capability, Opus
- 13:55
5.5 is overall the best model right now. Soul's not quite there, but it's really freaking close. And especially when you add in the pricing here, where $3 versus $0.30 or $6 versus $0.72, Soul's looking really good here. And then it gets even crazier when you go down to the output tokens per task in the estimated minutes per task of Opus X I, 66,000 tokens and
- 14:17
about 8 and 1/2 minutes versus Soul's 18K tokens and about 4 minutes. So it's half the time, like a third of the tokens, the efficiency thing that OpenAI has been doing for a long time, where every release the models will use fewer tokens, fewer steps to get the same amount of work done, has just been remarkable. And you know, they've been stressing, and I've been trying to stress too, for a long time, that price
- 14:38
is not the same as what is the sticker on the API. Just because a model is way more expensive than another model does not mean that the model's actually going to be more expensive. Because if you have something that's a lot stupider, like, oh, I don't know, maybe a Gemini 3 8 flash, which on paper is much cheaper than some of these other things, but since it will do 10,000 file reads per
- 15:00
second to really make sure that it read that one line of effectively, it'll just keep doing that over and over again, and will end up in the net being more expensive per task than any of the other models would be. And since OpenAI models are so efficient, they end up just being ludicrously cheap. One of the changes OpenAI just made is that the subscription tiers have changed, where
- 15:21
now you are getting about half the API usage that you were before before they made this change. And I think a lot of that has to do with 6.1 Soul specifically where this model is almost certainly not nearly as marked up as the other ones were. They definitely could have gotten away with charging about Opus price for this model, but because they're charging such a low API price on
- 15:41
this thing, they kind of have to reduce the raw like API dollars that you're getting per week on your sub even though in reality the number of tokens and the number of tasks that you're getting done has remained about the same and even in my brief testing experience kind of slightly gone up. I don't have a super good read on what the subscription feels like at this point because it hasn't
- 16:03
been long enough and I also did upgrade to the $500 a month tier from OpenAI cuz I wanted to start trying out Astra on ultrafast and also just like I've had three Codex subs for a while and I just like centralizing those all down into one makes my life much more convenient. From my experience and other people I've talked to, 6.1 Souls feels borderline unlimited and looking at this chart,
- 16:23
it's ridiculous. I have had to do a recent already on my account, but that's mostly just because of this ultrafast will annihilate your usage. I don't want to go down that rabbit hole right now. Just know that like it's a fun button to hit, but it's not a button that you should hit almost ever. It's very rarely worth it. I am very excited though to see what 6.1 Soul looks like on ultrafast. That's not out yet, but they
- 16:45
did mention in the Dev Day keynotes that it will be coming. We will get 6.1 Soul on ultrafast and assuming it holds the same pricing structure as Astra, it should be 6x more expensive while being like 8x faster-ish something around there, but at the ludicrous API price we already have, that's going to make the model just slightly more expensive than
- 17:05
Astra. And when you combine the efficiency and all that stuff, I think you'll end up actually being able to use a pretty large amount of 6.1 Soul ultrafast on the $500 a month plan, which should be really compelling and cool. And going down to the pricing on Opus, it is 2x more expensive than Soul per API request. It also does end up using way more tokens and way more steps
- 17:25
than anything else does. But what I have found is one, never touch max reasoning ever for any reason on a Claude model, it's just not worth it. I've pretty much just stuck it on X high reasoning and forgot about it. I haven't touched it since and it's been great. X high does feel like the sweet spot on Opus and having looked at the usage on my $200 a month plan, the only reason why I burned down my weekly this week was just
- 17:46
because I was doing a bunch of dumb nonsense with Sonic 5.5 like letting it run in a loop for many hours on ultra mode to do some crazy design ports for fun. Like was not serious usage. If you're just doing normal day-to-day work, Opus 5.5 still has that feeling of like borderline unlimited. You can just use as much of this as you need. If you really push up to those rate limits, you're doing a ton of work. Normal work
- 18:08
for both Opus and Soul on the $200 a month plans will both feel excellent. There is another thing I wanted to compare on these models, which is where you can use them because I think this does matter a lot. Obviously for Claude, you're pretty much just stuck in Claude code. If you're down for Claude code or something that wraps Claude code like conductor or T3 code, great. Awesome. You can use your sub, get tons of usage,
- 18:29
it's awesome. But if you want to use it somewhere else, you really can't and Anthropic is really strict about that. Can't use it in Hermes, can't use it in Pi, can't use it in open code. It's very locked down. Opening I this week at Dev Day announced the sign in with ChatGPT button being brought to a ton of other surfaces. They partnered with a bunch at launch and I assume there will be plenty
- 18:49
more coming in the future where they want it to be such that you can take your Codex sub and you can sign in to all these different platforms and just take your bucket of tokens, your weekly usage and stick them wherever you want. You don't have to use the Codex desktop app all the time. If you want to use Amp or Conductor or Devin or Hermes or T3 code or many others, you can just do it.
- 19:10
They're all supported. It's not against TOS. You're not going to get banned. They in fact encourage it and have a whole page on this and they worked with these companies to add a true sign in with chat GPT button to their products and it just works. This is awesome. This is a great feature from OpenAI and the portability of the sub to use 6.1 Soul wherever you want is really, really good. And when you're dealing with a
- 19:32
model that is it's just robust. I don't have any super deep or complex feelings on this model. There's no slow lows and there's also no insanely high highs. I haven't been blown away by anything in particular on this model. Astra had the 3D and computer use stuff. Opus has that just special feeling to it. 6.1 Soul is just a workhorse. It is a boring, cheap,
- 19:53
efficient, effective workhorse. And the fact that you can use it anywhere is a huge plus. I think that if you had to just pick between one of these two, personally, I would probably go with the Codex one. As much as I do like Opus, I would pick a $200 month Codex sub. But if you can get both, I would definitely get both. They're both great. Another thing I had 6.1 do was go through this
- 20:16
machine and my Linux box to just take a look at, okay, how have I been using these two models? How many input tokens, output tokens? And then break that down by task. Task was a very light metric here. Like I wouldn't take that super seriously. It's more just whatever it defines as a task is a task. So again, take this with a grain of salt, but the vague numbers that it gave me here is
- 20:36
I've done about 90 useful tasks with 6.1 and about 65 useful tasks with Opus. This doesn't count any of the game stuff I've done cuz that's all been on my Windows machine, which isn't part of this report. But like again, look at the pricing here. I I'm sorry that we just keep talking about pricing, but like $4.50 per task versus $27 per task, the amount of usage you get on 6.1 is just
- 20:58
ludicrous. This is a ludicrously cheap model for very good results. I really like this thing. And I honestly just don't know what else to say beyond that. These are both excellent models, and I think it's a great example of OpenAI and Anthropic being great corrective forces to each other where Opus 5.5 comes out, it's really good, and they knew that they had to handle more limits, more
- 21:18
efficiency, more accessibility to get people to want to use it over the OpenAI stuff, and then it worked, it succeeded, it's great. Then OpenAI sees that, and they're like, "Oh crap, we got to have some way to one-up them and have something that's better." So, they make 6.1 Soul be this insanely cheap, but also really good so they can fight back on that. They just kind of keep pressuring each other to be better and better towards the end consumer, and
- 21:39
while these two labs fight to the death and try and kill each other, we benefit. We we're benefiting a ton. These are great models, and yeah, I don't think you can really go wrong with either of these. If you want a strong pick for me, my pick would be 6.1 Soul for the whole Codex OpenAI ecosystem, I do prefer it, but I really like Opus 5.5, I've used it
- 22:01
a lot more. You're not going to go wrong there, either. Enjoyed this video, make sure you like and subscribe, and I'll talk to you then. So,
Description
I've been using both Opus 5.5 and GPT-6.1 Sol a ton. They're incredible models.
Thank you Firecrawl for sponsoring! Check them out at: https://davis7.link/firecrawl
Majora’s Mask custom area - https://x.com/davis7/status/210356741...
Opus limits are good - https://x.com/davis7/status/210329350...
My X, Discord, and more: https://davis7.sh/
