It's Pretty Disappointing...
BDBen Davis@bmdavis419 · 45.9K subscribers
- Views
- 29.3K
- 29,334
- Outlier
- ×1.4
- Median 21.4K
- Length
- 17:48
- Published
- Sep 24
- Thursday
- Comments
- 34
- 347 likes
Script
How it's written
- Words
- 3,858
- Words a minute
- 216
- Reading time
- 17 min
- Segments
- 12
The hook
0:00–0:25Grok 4.7 is here and it is weird. Like very strange and not what I was expecting at all. I remember earlier this summer when Elon was vague posting about this thing, talking about all the things where it would improve, that this would be a better model than 4.6 in every way, that it would be more efficient, just a little bit more expensive to serve, that this would be their big answer to Opus and Saul and really get them competitive and be a
Structure
Grok 4.7 is here and it is weird. Like very strange and not what I was expecting at all. I remember earlier this summer when Elon was vague posting about this thing, talking about all the things where it would improve, that this would be a better model than 4.6 in every way, that it would be more efficient, just a little bit more expensive to serve, that this would be their big answer to Opus and Saul and really get them competitive and be a step up from Grok 4.6. That didn't happen. It It does a couple things quite well, but overall it's mostly pretty native. There There are a lot of regressions here that are very unfortunate and I think it's worth breaking down, you know, stuff like the efficiency, the price, the performance. To me, SpaceX is a very interesting case, especially compared to any of the other major labs right now. I do want to be crystal clear with the bias disclosures on this one up front that I have worked with Cursor, which is now part of SpaceX AI in the past and I will more in the future. I do really like the products they're building, but yeah, I want to be crystal clear. They have nothing to do with this video. They haven't seen it. They have not paid for it. They have no control over what I'm about to say and with how negative this is unfortunately going to be, that should be very obvious. I do want to start with what I think this model got right because I think that will help explain where things went wrong and why the places where it failed mattered so much. Like you can look right here, I just have a Grok build session open on Grok 4.7 fast X high. And when I run this, it's does a good job of like
Watch from 0:00actually doing the thing and also their new fast mode, quote unquote, that they added for this model is ridiculously fast. It feels great to use. They've always leaned into that option of like being the fast model or whatever and this is no exception. It is really good about like the modern frontier behaviors of handling lots of sub agents, running for long periods of time, checking its own work, doing lots of research. It's a good model that is capable of a lot of things. This is the type of model where if you're doing some very heavy pair programming-y type things where you're very in the loop, you're giving it very specific instructions on what to do when, where, how, and why, it's going to excel at that. And it's also a very pleasant model to work with. The way it writes is quite nice. I like reading its outputs like this one, which is a little test prompt I did. The outputs you can see here are very clear and readable. It gives me the information I wanted, nothing more, nothing less. It's pretty good about asking decent questions, and it's understanding of what you want is eh. Whenever there's some ambiguity in the prompt and it's not super clear on what exactly you're asking for, it'll come up with something to fill in those blanks. Sometimes it's really good, sometimes it's really bad, and well, I think that kind of leads me into just getting into the big issues with this model, which I want to start with the performance side of things. Because this model is pitched as the frontier performance from SpaceX AI. This model is clearly positioned as the cheaper, faster, not quite as powerful, but still really useful model that would fit into a curve like this. Like this is the cursor bench where, you know, you have
Watch from 1:30Fable 5.1 for the big task that's really powerful, really good at what it does, but it's more expensive. Then if you need something cheaper, but still pretty capable, you kind of cross over into that Grok 4 7 line. And like this trend line right here of capability to price, Grok 4 7 clearly fits here on paper, and I think that makes sense for how it's positioned. Seems reasonable, all good. Problem is that's not reality at all. Because oh well, you'll notice is I unchecked a box on Opus 5.5, which came out as of recording this yesterday. That completely demolishes everything on this chart. Now, this is also just a very, very good model. I probably will talk about this again in the future, but Opus 5.5 is excellent, and GPT-6 Soul, which came out today, is also excellent. And these are two models that fit into that same bucket on paper as Grok 4 7, and Grok 4 7 does not hold a candle to either of them, unfortunately. I try to not focus too hard on stuff like, you know, one-shot design benches or 3D visualization benches or anything like that, because while they are good sniff tests of how capable a model is, but often they don't fully capture the reality of working with one of these models day-to-day to do more normal actual work, but I did have one example here that was a side-by-side of both Grok 47 and Opus 5.5. It's a little thing, but what I did is on one of the sites I'm working on, I had this top bar, and this top bar, the like dark blue part would cut off right here, and I wanted it to slide back behind this sidebar, cuz I was experimenting with like the Apple sidebar thing. This was Opus's version, which looks really good.
Watch from 3:02It nicely fades back there. This is centered, it's all lined up. This looks good. I'm happy with this iteration of it. This was like three prompts to get it. This is the Grok 47 version, and this was after about five prompts bullying it repeatedly and cursor, like, "Please, just do this the way I want you to, and do it correctly." It looks awful. Like, all of this this top bar is just a disaster. I told it to add in some more padding so that this wouldn't be up at the top of the screen, and then it moved this down more, so now this is awkwardly floating here. Like, this isn't good. I would not ship this. This feels awful. And again, as much as this is a very small, singular example, I think it does a very good job of showcasing what my general experience with this model has been. It's good. It can kind of get there, but it's not at that same level as these other models. Before, that wouldn't be nearly as much of an issue, especially with the previous Grok models, because they were so much cheaper and had you could use so much more of them. Now, that's not the case. The the pricing here is unfortunately what I'm going to have to rant a lot about right after today's sponsor. Today's sponsor, Firecrawl, is one of the most essential pieces of my stack. It is the way my agents actually search the web. They have a great search endpoint, scrape endpoint, and many more for letting your agents search the web and then get responses down in markdown that is easy for them to parse and work with. It's also open source, by the way.
Watch from 4:30And they have what I would consider to be the gold standard for modern services of interacting with their API. They have a really good MCP server, a really good set of skills, a great CLI, really good API, really good SDKs. All of it is right there. It's easy to interact with it from wherever you need to because often times I need maybe MCP for one of my normal like Codex agents, but then for a product I'm building, I need to stick web search into it, then I would use the SDK. Anyway you need to use it, you can use it. But I want to talk about one more thing, which is their new developer index that is very impressive. Pretty much what they did here is they looked through the way people were using Firecrawl and realized that, you know, heavily agentic products that are doing a lot of dev work often end up searching the same things over and over again and generic web search isn't the best way to do that. So what they did is they made the developer index, which has over 70 million artifacts, which are readmes, PRs, issues, OpenAI specs, skills, and much more for the tools we use every single day to improve the speed and quality of the search results your agents get while working in the background. I actually have a little example here that I had Astra find for me. A few days ago, it needed to do a web search to get a specific error on ESLint within my Z config. There was a misconfiguration in one of my projects. And when they did the search for it on the normal search index, it ranked fourth. On the developer search index, it ranked second. So it's much quicker and easier to get the high signal thing that you actually want using the developer index. This is just one of many ways they are making search better.
Watch from 5:48If you're using or building agents, you need to use these guys at david7.link/firecrawl. If you go back to the post where Elon teased Grok-47 a while back, you'll notice that he said that Grok-47 would have even better token efficiency. And that is No. That that I'm I'm sorry, but that is just not true having used this thing. This is not an efficient model. It uses a lot of tokens, a lot more steps. Something like an Astra is substantially more token efficient than it. This has always been OpenAI's edge. OpenAI has just been the kings of efficiency for really since the start of this year. They've been investing super heavily in that. It does seem like none of the other labs have quite caught up there. Even Anthropic with their new Opus 5.5 is not nearly as token efficient as something like Astra. Astra on some of these benchmarks is benching at the very top while doing like probably a tenth of the tool calls as anything else. It's very impressive what they've done there. And this model on paper, you know, when they released it they were talking a ton about how hey, this is a new model. It is better than Grok-4.6 and it's the same price. So, originally Grok-4.6 was $2 in $6 out. Grok-4.7 is also $2 in $6 out. Great, right? It's the same price, but pricing models is not the same as just the raw token price. If one model is 10x more expensive than another, but that one that's 10x more expensive does 1/10 of the tool calls and tokens as the other one, they're going to even out to be the same exact price. And oftentimes these more efficient models are going to be higher quality. So, on a true cost per task metric, Grok-4.7 does not win out
Watch from 7:18on that. But unfortunately, the price issues do go even deeper than this. When I was testing this out, I was looking through my usage limits and all this stuff, the pricing on this thing this Grok-4.7 made this table and chart and did this research. You look at this price table here where the advertised of two in six out, yep, that's what it is. But you'll notice it says under 200k. So, under a 200k context window, you will get this price. But then, if you either turn on fast mode, which I showed at the start is really fast and great, or you go above that 200k token threshold, your price will double. So, you turn on either of those settings, your price doubles to four in 12 out. And then, if you stack both of those in, you put on fast mode and over 200k, which is generally speaking what you want, you want the model to be fast and you want it to be working with longer context cuz longer context is quite useful to have. But now, once you're up at this 6 in 18 out, you are almost at the same price as something like Opus 5.5, which is 4 in 20 out. And then if you go to GPT-6 Soul, which is their new like Soul model, which is very clearly in the same tier as something like an Opus or Grok 4.7, that is $2 in $10 out. So, on raw API price, these are going to be cheaper, and the OpenAI one is going to be much more efficient. So, this one will come out to be much cheaper than any of the other options. Opus will be similar in price, a little bit less efficient on the higher reasoning levels. But if you're running on like high reasoning or whatever, it's really not that bad. And on CursorBench, their own benchmark, high reasoning, it completely demolishes Grok 4.7 in capability, and it is cheaper than high
Watch from 8:49reasoning and X high reasoning, which are like the two reasoning levels that I would actually use for this thing. There are very few tasks where I would whip out like a Grok 4.7 low reasoning. Like that would just be, "Hey, can you read this file and tell me what's in it?" This is not for real tasks in work these days. This is where I would want to be. I'd probably be high reasoning Opus 5.5. It ends up being cheaper for much higher quality outputs. And I think these pricing issues, the reason why I'm talking about this so much, is that I think that this hurts a Grok model far more than it hurts any other model from the other big two labs. Like Fable and Astra are still very expensive and will burn your limits very quickly. They're not cheap or efficient models. Well, I Astra's pretty efficient, but it's not cheap. The point is, those are incredibly capable models that are a tier above everything else. They're genuinely novel in what they can do. Therefore, the premium on those tokens is worth it. It's worth dealing with the hassle of burning down your sub faster or having to pay more on API. People are willing to do that to get that really powerful good token. Once you go down below that tier, and now you're into this sort of mid-pack that we've got right now, of all these models that are now competing on this line of like the useful day-to-day model that is not the smartest, most powerful, but also not dumb, well-priced, good speed, good efficiency, like the stuff that most people will be using day-to-day for coding work or knowledge work or whatever. Once you're down into that category, Grok has historically had a big edge there, where Opus would burn limits much faster, Soul would burn limits much faster. Grok 4 6 and Grok 4 5, they felt borderline unlimited. Like
Watch from 10:20in the Super Grok heavy sub, which is their $300 a month sub, I've been using that, and previously the models would not burn my limits all that much at all. Now, just with Grok 4 7 having done some not an insane amount of work to be honest with you. Like I've done a lot with it, made a bunch of random stuff, and made some fixes to actual code bases, but nothing crazy. It's already burned 30% of my weekly limits after like a day of just pretty middling use. If I was going off and doing long-running big coding agent sessions, like the kind of stuff I do with Fable or Astra or now Opus 5.5, it would incinerate it instantly. Like the limits here, it no longer feels unlimited, and that really hurts them. I think it hurts them more than anyone else, because once you're into that, you know, it's more of a constrained resource, I'm now looking at this and I'm like, "Okay, I can probably get similar amounts of usage on Opus 5.5 as I can Grok 4 7." From what I've seen and what I've tested, Opus 5.5 is really easy on your limits, so you probably even get more of that. And Opus 5.5 is a much higher quality model than Grok 4 7, why would I pick this? There's no real compelling answers to that outside of the biggest advantage that I think SpaceX AI has right now, which is just that their engineering is the best of the labs. Between Grokbot, they're like evolution of Hermes agents, it's really, really good. I love it, I use it all the time. Their Grok build CLI, which is, I would argue, one of the best CLIs that any of the labs have. Claude code's gotten really good, but Grok build is also excellent. I really enjoy
Watch from 11:51working with it. Their all the cursor stuff is really good right now. Their cloud agents are great. Their new project system is great. They have good surfaces to use Grok in. If you really want those services, Grok 4:7 is the way to use them. That's one angle and that's kind of fair. You can do that. But I think what's going to be interesting to see is how defensible that position is. I made this very crude and kind of silly diagram to illustrate my thoughts on the three big US labs right now. Anthropic is the best at research of all the labs and they are not horrible at engineering, but they're not great at it. I mean, go look at the cloud desktop app. It's not in a phenomenal state. Open AI is kind of a hybrid where their research is pretty extraordinary in a lot of places and I would argue that in a lot of ways Astra is still true frontier in a lot of capabilities, especially anything involving computer use. They are at the very top. They're at the true frontier of efficiency. They have really, really good research, but they also have much better engineering than Anthropic has. Like the computer use stuff, all their product services, the Codex desktop app has a lot of problems. It's not perfect by any stretch of the imagination. Their sidebar makes me want to die. Some of the performance stuff isn't great. It's not all there, but they do have more of a balance than something like Anthropic does. Then on the other extreme, you end up with SpaceX AI, which is I would definitely the best engineering lab, but they're also the weakest engineering lab. Like Grok 4:7 would not come out of these two labs right now. These two labs are far beyond this point. Who knows where things are going to end up going from here. Elon actually posted this in September, so way after the post I
Watch from 13:21showed earlier and this was before the actual release of the model. It's a much more fair assessment of it where Grok 4:7 should roughly be on par with Opus 5, which I think is about right. Not 5.1, which ended up being 5.5 and yes, Opus 5.5 is much better. Better in some ways? Yeah, it has definitely improved in some areas and I wouldn't go back to Grok 4:6. I still do prefer Grok 4:7. Worse in others? Already covered all those. We need to fix multimodal performance. I don't know if this is exactly the thing that I would zero in on as like the biggest thing that needs to be fixed. Uh computer uses awesome and they I hope that they do fix that, but like this is not their biggest issue right now. According to him and yeah, I mean we saw what he said about Grok-4.7, take this with a grain of salt. But the idea here is that Grok-4.8 will be a noticeable improvement. Grok-4.9 is probably after Fable class. Grok-5 maybe better than anything, we shall see. And I think that's pretty correct, we shall see. I am very skeptical of SpaceX AI fully out-researching OpenAI or Anthropic. I think the only place where they end up with a substantially better model than those two have is if they have a novel data source that the other two don't have that they can RL on that makes the model really good. If RL ends up just really mattering and being way more important than something like pre-training, cuz that's the other part that I didn't bring up earlier. Anthropic, they're unbelievably good at pre-training. They have such good base models and they're able to make such strong foundations that they get that taste and discernment and whatever you want to call it that makes the Claude models feel so special. Their RL isn't always amazing. They do seem to be
Watch from 14:52improving, especially with Opus-5.5, which seems to be smaller than Opus-5, but also way better than Opus-5. Probably a slight distillation on whatever this rumored Fable-5.5 is. But really, the best path I could see SpaceX AI having on the model front would just be like constant iteration, really strong RL, like kind of what GLM is doing. Where they have this really impressive RL system that makes it very easy for them to rapidly iterate on new models and get them out fast with really good capability and squeeze those capabilities into smaller, more efficient models. I think that is a big market share that is quite useful, but clearly they are targeting the big boy models. Time will tell if they end up actually getting there. I'm skeptical, but I'm hopeful. We we we will see. So really, what I think I'm trying to say is that Grok-4.7 is one step forward, two steps back. SpaceX AI has a long way to go and while they have gotten a good model with good product services, they're not quite at the frontier yet. I hope they can fix these things and get to the point where they can truly compete with the other labs, but at least for right now, I don't see myself using Grok-1.7 as a heavy daily driver. It's really just going to live in the like it's a useful CLI to pull out every once in a while like the Grok build CLI for system management stuff. Yeah, I really like Grok bot, that'll still keep getting used and Cursor is quite useful in a lot of cases, but for my main day-to-day work, Opus 5.5, Astra, these are still the kings. Like these are the top two models. I don't think I have anything outside of that. If you enjoyed the video, make sure you like and subscribe and I'll talk to you all soon.
Watch from 16:22Full transcript
- 0:00
Grok 4.7 is here and it is weird. Like very strange and not what I was expecting at all. I remember earlier this summer when Elon was vague posting about this thing, talking about all the things where it would improve, that this would be a better model than 4.6 in every way, that it would be more efficient, just a little bit more expensive to serve, that this would be
- 0:21
their big answer to Opus and Saul and really get them competitive and be a step up from Grok 4.6. That didn't happen. It It does a couple things quite well, but overall it's mostly pretty native. There There are a lot of regressions here that are very unfortunate and I think it's worth breaking down, you know, stuff like the
- 0:41
efficiency, the price, the performance. To me, SpaceX is a very interesting case, especially compared to any of the other major labs right now. I do want to be crystal clear with the bias disclosures on this one up front that I have worked with Cursor, which is now part of SpaceX AI in the past and I will more in the future. I do really like the products they're building, but yeah, I
- 1:02
want to be crystal clear. They have nothing to do with this video. They haven't seen it. They have not paid for it. They have no control over what I'm about to say and with how negative this is unfortunately going to be, that should be very obvious. I do want to start with what I think this model got right because I think that will help explain where things went wrong and why the places where it failed mattered so much. Like you can look right here, I
- 1:22
just have a Grok build session open on Grok 4.7 fast X high. And when I run this, it's does a good job of like actually doing the thing and also their new fast mode, quote unquote, that they added for this model is ridiculously fast. It feels great to use. They've always leaned into that option of like being the fast model or whatever and this is no exception. It is really good
- 1:43
about like the modern frontier behaviors of handling lots of sub agents, running for long periods of time, checking its own work, doing lots of research. It's a good model that is capable of a lot of things. This is the type of model where if you're doing some very heavy pair programming-y type things where you're very in the loop, you're giving it very specific instructions on what to do
- 2:04
when, where, how, and why, it's going to excel at that. And it's also a very pleasant model to work with. The way it writes is quite nice. I like reading its outputs like this one, which is a little test prompt I did. The outputs you can see here are very clear and readable. It gives me the information I wanted, nothing more, nothing less. It's pretty good about asking decent questions, and
- 2:24
it's understanding of what you want is eh. Whenever there's some ambiguity in the prompt and it's not super clear on what exactly you're asking for, it'll come up with something to fill in those blanks. Sometimes it's really good, sometimes it's really bad, and well, I think that kind of leads me into just getting into the big issues with this model, which I want to start with the performance side of things. Because this
- 2:46
model is pitched as the frontier performance from SpaceX AI. This model is clearly positioned as the cheaper, faster, not quite as powerful, but still really useful model that would fit into a curve like this. Like this is the cursor bench where, you know, you have Fable 5.1 for the big task that's really powerful, really good at what it does, but it's more expensive. Then if you
- 3:07
need something cheaper, but still pretty capable, you kind of cross over into that Grok 4 7 line. And like this trend line right here of capability to price, Grok 4 7 clearly fits here on paper, and I think that makes sense for how it's positioned. Seems reasonable, all good. Problem is that's not reality at all. Because oh well, you'll notice is I unchecked a box on Opus 5.5, which came
- 3:29
out as of recording this yesterday. That completely demolishes everything on this chart. Now, this is also just a very, very good model. I probably will talk about this again in the future, but Opus 5.5 is excellent, and GPT-6 Soul, which came out today, is also excellent. And these are two models that fit into that same bucket on paper as Grok 4 7, and
- 3:50
Grok 4 7 does not hold a candle to either of them, unfortunately. I try to not focus too hard on stuff like, you know, one-shot design benches or 3D visualization benches or anything like that, because while they are good sniff tests of how capable a model is, but often they don't fully capture the reality of working with one of these models day-to-day to do more normal actual work, but I did have one example
- 4:11
here that was a side-by-side of both Grok 47 and Opus 5.5. It's a little thing, but what I did is on one of the sites I'm working on, I had this top bar, and this top bar, the like dark blue part would cut off right here, and I wanted it to slide back behind this sidebar, cuz I was experimenting with like the Apple sidebar thing. This was Opus's version, which looks really good. It nicely fades back there. This is
- 4:32
centered, it's all lined up. This looks good. I'm happy with this iteration of it. This was like three prompts to get it. This is the Grok 47 version, and this was after about five prompts bullying it repeatedly and cursor, like, "Please, just do this the way I want you to, and do it correctly." It looks awful. Like, all of this this top bar is just a disaster. I told it to add in some more padding so that this wouldn't
- 4:53
be up at the top of the screen, and then it moved this down more, so now this is awkwardly floating here. Like, this isn't good. I would not ship this. This feels awful. And again, as much as this is a very small, singular example, I think it does a very good job of showcasing what my general experience with this model has been. It's good. It can kind of get there, but it's not at
- 5:14
that same level as these other models. Before, that wouldn't be nearly as much of an issue, especially with the previous Grok models, because they were so much cheaper and had you could use so much more of them. Now, that's not the case. The the pricing here is unfortunately what I'm going to have to rant a lot about right after today's sponsor. Today's sponsor, Firecrawl, is one of the most essential pieces of my stack. It is the way my agents actually
- 5:36
search the web. They have a great search endpoint, scrape endpoint, and many more for letting your agents search the web and then get responses down in markdown that is easy for them to parse and work with. It's also open source, by the way. And they have what I would consider to be the gold standard for modern services of interacting with their API. They have a really good MCP server, a really good
- 5:56
set of skills, a great CLI, really good API, really good SDKs. All of it is right there. It's easy to interact with it from wherever you need to because often times I need maybe MCP for one of my normal like Codex agents, but then for a product I'm building, I need to stick web search into it, then I would use the SDK. Anyway you need to use it, you can use it. But I want to talk about one more thing, which is their new
- 6:17
developer index that is very impressive. Pretty much what they did here is they looked through the way people were using Firecrawl and realized that, you know, heavily agentic products that are doing a lot of dev work often end up searching the same things over and over again and generic web search isn't the best way to do that. So what they did is they made the developer index, which has over 70 million artifacts, which are readmes,
- 6:39
PRs, issues, OpenAI specs, skills, and much more for the tools we use every single day to improve the speed and quality of the search results your agents get while working in the background. I actually have a little example here that I had Astra find for me. A few days ago, it needed to do a web search to get a specific error on ESLint within my Z config. There was a
- 7:01
misconfiguration in one of my projects. And when they did the search for it on the normal search index, it ranked fourth. On the developer search index, it ranked second. So it's much quicker and easier to get the high signal thing that you actually want using the developer index. This is just one of many ways they are making search better. If you're using or building agents, you need to use these guys at
- 7:21
david7.link/firecrawl. If you go back to the post where Elon teased Grok-47 a while back, you'll notice that he said that Grok-47 would have even better token efficiency. And that is No. That that I'm I'm sorry, but that is just not true having used this thing. This is not an efficient model. It uses a lot of tokens, a lot more steps. Something like an Astra is
- 7:42
substantially more token efficient than it. This has always been OpenAI's edge. OpenAI has just been the kings of efficiency for really since the start of this year. They've been investing super heavily in that. It does seem like none of the other labs have quite caught up there. Even Anthropic with their new Opus 5.5 is not nearly as token efficient as something like Astra. Astra on some of these benchmarks is benching at the very top while doing like
- 8:04
probably a tenth of the tool calls as anything else. It's very impressive what they've done there. And this model on paper, you know, when they released it they were talking a ton about how hey, this is a new model. It is better than Grok-4.6 and it's the same price. So, originally Grok-4.6 was $2 in $6 out. Grok-4.7 is also $2 in $6 out. Great,
- 8:25
right? It's the same price, but pricing models is not the same as just the raw token price. If one model is 10x more expensive than another, but that one that's 10x more expensive does 1/10 of the tool calls and tokens as the other one, they're going to even out to be the same exact price. And oftentimes these more efficient models are going to be higher quality. So, on a true cost per
- 8:46
task metric, Grok-4.7 does not win out on that. But unfortunately, the price issues do go even deeper than this. When I was testing this out, I was looking through my usage limits and all this stuff, the pricing on this thing this Grok-4.7 made this table and chart and did this research. You look at this price table here where the advertised of two in six out, yep, that's what it is.
- 9:07
But you'll notice it says under 200k. So, under a 200k context window, you will get this price. But then, if you either turn on fast mode, which I showed at the start is really fast and great, or you go above that 200k token threshold, your price will double. So, you turn on either of those settings, your price doubles to four in 12 out. And then, if you stack both of those in,
- 9:28
you put on fast mode and over 200k, which is generally speaking what you want, you want the model to be fast and you want it to be working with longer context cuz longer context is quite useful to have. But now, once you're up at this 6 in 18 out, you are almost at the same price as something like Opus 5.5, which is 4 in 20 out. And then if you go to GPT-6 Soul, which is their new
- 9:50
like Soul model, which is very clearly in the same tier as something like an Opus or Grok 4.7, that is $2 in $10 out. So, on raw API price, these are going to be cheaper, and the OpenAI one is going to be much more efficient. So, this one will come out to be much cheaper than any of the other options. Opus will be similar in price, a little bit less efficient on the higher reasoning levels. But if you're running on like
- 10:11
high reasoning or whatever, it's really not that bad. And on CursorBench, their own benchmark, high reasoning, it completely demolishes Grok 4.7 in capability, and it is cheaper than high reasoning and X high reasoning, which are like the two reasoning levels that I would actually use for this thing. There are very few tasks where I would whip out like a Grok 4.7 low reasoning. Like that would just be, "Hey, can you read this file and tell me what's in it?"
- 10:31
This is not for real tasks in work these days. This is where I would want to be. I'd probably be high reasoning Opus 5.5. It ends up being cheaper for much higher quality outputs. And I think these pricing issues, the reason why I'm talking about this so much, is that I think that this hurts a Grok model far more than it hurts any other model from the other big two labs. Like Fable and
- 10:52
Astra are still very expensive and will burn your limits very quickly. They're not cheap or efficient models. Well, I Astra's pretty efficient, but it's not cheap. The point is, those are incredibly capable models that are a tier above everything else. They're genuinely novel in what they can do. Therefore, the premium on those tokens is worth it. It's worth dealing with the hassle of burning down your sub faster
- 11:14
or having to pay more on API. People are willing to do that to get that really powerful good token. Once you go down below that tier, and now you're into this sort of mid-pack that we've got right now, of all these models that are now competing on this line of like the useful day-to-day model that is not the smartest, most powerful, but also not dumb, well-priced, good speed, good efficiency, like the stuff that most
- 11:35
people will be using day-to-day for coding work or knowledge work or whatever. Once you're down into that category, Grok has historically had a big edge there, where Opus would burn limits much faster, Soul would burn limits much faster. Grok 4 6 and Grok 4 5, they felt borderline unlimited. Like in the Super Grok heavy sub, which is their $300 a month sub, I've been using
- 11:55
that, and previously the models would not burn my limits all that much at all. Now, just with Grok 4 7 having done some not an insane amount of work to be honest with you. Like I've done a lot with it, made a bunch of random stuff, and made some fixes to actual code bases, but nothing crazy. It's already burned 30% of my weekly limits after like a day of just pretty middling use.
- 12:18
If I was going off and doing long-running big coding agent sessions, like the kind of stuff I do with Fable or Astra or now Opus 5.5, it would incinerate it instantly. Like the limits here, it no longer feels unlimited, and that really hurts them. I think it hurts them more than anyone else, because once you're into that, you know, it's more of a constrained resource, I'm now looking
- 12:39
at this and I'm like, "Okay, I can probably get similar amounts of usage on Opus 5.5 as I can Grok 4 7." From what I've seen and what I've tested, Opus 5.5 is really easy on your limits, so you probably even get more of that. And Opus 5.5 is a much higher quality model than Grok 4 7, why would I pick this? There's no real compelling answers to that
- 13:00
outside of the biggest advantage that I think SpaceX AI has right now, which is just that their engineering is the best of the labs. Between Grokbot, they're like evolution of Hermes agents, it's really, really good. I love it, I use it all the time. Their Grok build CLI, which is, I would argue, one of the best CLIs that any of the labs have. Claude code's gotten really good, but Grok build is also excellent. I really enjoy
- 13:21
working with it. Their all the cursor stuff is really good right now. Their cloud agents are great. Their new project system is great. They have good surfaces to use Grok in. If you really want those services, Grok 4:7 is the way to use them. That's one angle and that's kind of fair. You can do that. But I think what's going to be interesting to see is how defensible that position is.
- 13:42
I made this very crude and kind of silly diagram to illustrate my thoughts on the three big US labs right now. Anthropic is the best at research of all the labs and they are not horrible at engineering, but they're not great at it. I mean, go look at the cloud desktop app. It's not in a phenomenal state. Open AI is kind of a hybrid where their research is pretty extraordinary in a lot of places and I would argue that in
- 14:03
a lot of ways Astra is still true frontier in a lot of capabilities, especially anything involving computer use. They are at the very top. They're at the true frontier of efficiency. They have really, really good research, but they also have much better engineering than Anthropic has. Like the computer use stuff, all their product services, the Codex desktop app has a lot of problems. It's not perfect by any
- 14:23
stretch of the imagination. Their sidebar makes me want to die. Some of the performance stuff isn't great. It's not all there, but they do have more of a balance than something like Anthropic does. Then on the other extreme, you end up with SpaceX AI, which is I would definitely the best engineering lab, but they're also the weakest engineering lab. Like Grok 4:7 would not come out of
- 14:43
these two labs right now. These two labs are far beyond this point. Who knows where things are going to end up going from here. Elon actually posted this in September, so way after the post I showed earlier and this was before the actual release of the model. It's a much more fair assessment of it where Grok 4:7 should roughly be on par with Opus 5, which I think is about right. Not 5.1, which ended up being 5.5 and yes,
- 15:05
Opus 5.5 is much better. Better in some ways? Yeah, it has definitely improved in some areas and I wouldn't go back to Grok 4:6. I still do prefer Grok 4:7. Worse in others? Already covered all those. We need to fix multimodal performance. I don't know if this is exactly the thing that I would zero in on as like the biggest thing that needs to be fixed. Uh computer uses awesome
- 15:25
and they I hope that they do fix that, but like this is not their biggest issue right now. According to him and yeah, I mean we saw what he said about Grok-4.7, take this with a grain of salt. But the idea here is that Grok-4.8 will be a noticeable improvement. Grok-4.9 is probably after Fable class. Grok-5 maybe better than anything, we shall see. And I think that's pretty correct, we shall see. I am very skeptical of SpaceX AI
- 15:47
fully out-researching OpenAI or Anthropic. I think the only place where they end up with a substantially better model than those two have is if they have a novel data source that the other two don't have that they can RL on that makes the model really good. If RL ends up just really mattering and being way more important than something like pre-training, cuz that's the other part that I didn't bring up earlier. Anthropic, they're unbelievably good at
- 16:09
pre-training. They have such good base models and they're able to make such strong foundations that they get that taste and discernment and whatever you want to call it that makes the Claude models feel so special. Their RL isn't always amazing. They do seem to be improving, especially with Opus-5.5, which seems to be smaller than Opus-5, but also way better than Opus-5. Probably a slight distillation on
- 16:30
whatever this rumored Fable-5.5 is. But really, the best path I could see SpaceX AI having on the model front would just be like constant iteration, really strong RL, like kind of what GLM is doing. Where they have this really impressive RL system that makes it very easy for them to rapidly iterate on new models and get them out fast with really good capability and squeeze those
- 16:50
capabilities into smaller, more efficient models. I think that is a big market share that is quite useful, but clearly they are targeting the big boy models. Time will tell if they end up actually getting there. I'm skeptical, but I'm hopeful. We we we will see. So really, what I think I'm trying to say is that Grok-4.7 is one step forward, two steps back. SpaceX AI has a long way to go and while they have gotten a good
- 17:11
model with good product services, they're not quite at the frontier yet. I hope they can fix these things and get to the point where they can truly compete with the other labs, but at least for right now, I don't see myself using Grok-1.7 as a heavy daily driver. It's really just going to live in the like it's a useful CLI to pull out every once in a while like the Grok build CLI for system management stuff. Yeah, I really like Grok bot, that'll still keep
- 17:32
getting used and Cursor is quite useful in a lot of cases, but for my main day-to-day work, Opus 5.5, Astra, these are still the kings. Like these are the top two models. I don't think I have anything outside of that. If you enjoyed the video, make sure you like and subscribe and I'll talk to you all soon.
Description
Grok 4.7 is a very rough release. It's a solid model, I enjoy using it, but the price is so so bad...
Thank you Firecrawl for sponsoring! Check them out at: https://davis7.link/firecrawl
elon’s original hype up - https://x.com/elonmusk/status/2082123...
more measured take on 4.7 from elon - https://x.com/elonmusk/status/2099458...
my pricing issues - https://x.com/davis7/status/210221370...
cursor bench - https://cursor.com/cursorbench
My X, Discord, and more: https://davis7.sh/
