I thought AI was gonna get cheaper?
TTTheo - t3.gg@t3dotgg · 572K subscribers
- Views
- 135K
- 134,605
- Outlier
- ×0.8
- Median 167K
- Length
- 18:18
- Published
- Oct 12
- Monday
- Comments
- 208
- 1.9K likes
Script
How it's written
- Words
- 3,700
- Words a minute
- 202
- Reading time
- 16 min
- Segments
- 13
The hook
0:00–0:25Intelligence too cheap to measure is AI's next frontier. According to Sam Altman back in July of 2025, I'm pretty sure that in all of 2025, my personal use of AI would probably amount to like $800 to $1,000. Versus this year, 2026, any given day, I can probably pull $1,000 in tokens. It seems like intelligence is getting way more expensive, especially now with models
Structure
Intelligence too cheap to measure is AI's next frontier. According to Sam Altman back in July of 2025, I'm pretty sure that in all of 2025, my personal use of AI would probably amount to like $800 to $1,000. Versus this year, 2026, any given day, I can probably pull $1,000 in tokens. It seems like intelligence is getting way more expensive, especially now with models like Fable and Astra that are literally more expensive per token and are doing more in it generating more tokens, massively increasing the cost that real-world businesses are seeing. It seems like everyone's AI bill is just going up and up and up with no sign of stopping. So, what happened here? Why aren't we achieving this new frontier as Sam Altman promised us? Why isn't intelligence getting cheaper? I feel like I was spending a few pennies per prompt last year, and now I'm spending tens, if not hundreds, of dollars. What went so wrong? Why are these companies charging so much more? I thought it was supposed to get cheaper. This video is inspired by Primagen, because he made a tweet referencing a real promise that Sam Altman had made, which is that he said AI would get 10 times cheaper year over year. And Prime said he's a bit confused because he thought that we would get this 10 times cheaper AI, but Astra's out and it's way more expensive. So, what went wrong? I got a lot of thoughts on this one.
Watch from 0:00The short version of what happened here is it did actually get cheaper. We're just not measuring it right. Long version, aka the video, will be the next thing after the sponsor break. Okay, look. I know this is an ad, but I just want to nerd out for a minute, because I've been building a lot more with Railway recently and I've had so much fun doing it. Let me get some bias out of the way first. I've loved Railway for years now. I was almost their fourth hire, but I instead chose to make my own startup and then my YouTube channel soon after. I know this team well. I love them and everything they're building is awesome. Railway had already built the easiest cloud for developers to build and ship real full-stack applications on. it. guys can everything from the database to the CDN and everything else in between. You could build your full app from scratch on Railway. And it was really nice when you needed persistent servers. Where things got genuinely magical is when they introduced their skills, CLIs, and MCPs to our agents. And now I can have agents just deploy and manage full stack apps without ever having to think about it. Even calling these apps almost feels misleading though, cuz one of those apps is a cloud I built on top of Railway. Yes, the cloud I've been working on for a while now, Lakebed, is entirely built on top of Railway and I'm so happy I did it this way. I have my full production environment as well as a staging environment that my agents provisioned for me, as well as an environment for one specific PR that again, my agent provisioned for me to test things out end to end. When I have any questions about how the infra is working or any ideas for how it could work better, I tell Claude or Codex about it. It connects to the MCP, it spins up a new environment, it figures it out, it verifies the changes, it ships to prod by merging on main. It will then check it on the environment through the MCP server and just know what's going on.
Watch from 1:22Ready for the spiciest part of this ad? I haven't looked at this dashboard in weeks cuz I don't have to, because my agents can manage all of it for me. Whether I'm building a one-off experiment to host my own analytics like I did with Slot Paws, or I'm trying to put together an HTML host like I did with Post Plan, or I'm building my own cloud with Lakebed, my default has been Railway for months now and I'm so thankful that changed. If you're worried about costs, don't be. I'm hosting a ton of stuff on Railway including our entire internal video pipeline where this ad's actually going to be uploaded, and my estimated costs for the month are under $50 for this insane amount of usage I'm getting. Ship on a platform that you and your agents will love at sodalite.link/railway. Sorry for the break, good to have you back. In order to explain why AI is actually getting cheaper, we're going to have to look at some numbers and talk more about how models work. The first thing we need to talk about is token cost. Flash bang warning for artificial analysis. When you look at model prices over time, the numbers are pretty scary. GPT-5 medium was $10 per million tokens out and a per million tokens in. Versus a model like GBT-6 Astra, which is $50 per mil out and 10 per mil in. That is a five to 10x increase on input and output tokens cost. That's crazy. That's terrifying. That's the exact opposite of what we were promised. We were promised 10x decreases in cost, but there's a 10x increase. Well, there are layers to this because token cost is one of the worst ways to measure cost. The three things we need to be considered of are first, yeah, sure, token cost. I'll talk a little bit about it. But the other two are where most of the important parts come from. The second one is how many
Watch from 2:52tokens it takes to complete a given piece of work. And the third is the types of work the models can complete. Specifically, how much intelligence per dollar, so to speak. So, we have token cost, tokens per dollar. We have tokens per task cost, how many tokens it take to solve a task. And then we have cost per level of intelligence. How much spend does it take? How much money do we have to spend to get a solution to a problem at a certain level of intelligence? So, yes, token costs did go up for the best of the best models, but let's be real here, there are also cheap options that are incredible. For example, a model like JLM 5.3 Flash, which I absolutely adore, it's one of my favorite model releases ever. This model is 15 cents per million input tokens and 50 cents per million out. It's also on sale in a lot of places for 50% off. So, it's actually 7 cents per mil in and 25 per mil out. That means that it's about a 30th the cost of GPT-5. For a 20th to a 30th the price per token, JLM 5.3 Flash will run circles around GPT-5 for a lot of real-world agentic stuff. So, again, if we look at the top-tier models, yes, cost per token's gone up. But if you look at other places, you'll see that we've also gotten really capable models that are way cheaper than before as well. So, yes, the top end has gotten crazier, but so has the bottom end. Both sides have seen meaningful improvements here. But as I mentioned before, cost per token is only a tiny piece of the formula here. I'll use a
Watch from 4:25really fun example. Let's look at Gemini 3 8 Flash. On this chart, it looks like a pretty good deal. It's $3.75 per million out. That makes it less than half the price of GPT-5 and less than a tenth the price of Astra and Fable. But then we hop up a little bit to cost per intelligence index task, and suddenly that difference gets a lot less different because 3 8 Flash uses so many tokens for the work that it's doing. So even though it is comically cheaper per token, the token waste that 3 8 Flash has results in this gap going from over 10x to under 3x. And 3.5 Flash was even worse here cuz it was so token hungry, getting to over half the cost of 5 6 Soul, even though 5 6 Soul is a way more expensive model. So once again, you need to factor all of these things in. Hilariously enough, Sonnet 5 was more expensive than Astra because Sonnet just burns tokens. So those are the first two dimensions to consider. Cost per token and the amount of tokens used to solve a given task. And to be clear, different tasks have different ways that they work here. For example, if the task is to ingest 800,000 tokens and categorize them in some way, the categorization and the output tokens aren't going to matter as much. The input tokens will, so you're always paying that cost basically no matter what. At that point, optimizing for the cheapest model for the input token read is probably the right call. If you're doing agentic dev work, things like caching, reliability, and token efficiency suddenly matter a hell of a lot more. We're also not looking at this
Watch from 5:56the right way. We're just looking at cost right here. What we want to see is cost versus intelligence. This chart is much more interesting, where you can see how expensive a run was and how well it performed in the benchmarks. They have this nice line now for the Pareto frontier, which is what is the best model at a given cost level. And you'll see GBD-6 Astra has a significant number of the spots here. And Fable ends up being way more expensive all the way over there, but you'll see at a given level of intelligence, cost can go down massively. Like if you're happy with a 45 to a 46 as the score here, your cost will be 82 cents per task. Where if you want to bump up into the 50s, suddenly it's $3.26. So for just a few more points, you're spending three times more money. I still don't think this is the best way to think about it though. I want to flip it. Instead of thinking about this as how much intelligence can you get per dollar, let's take a level of intelligence and see how much it costs over time. If we took at the capability of GPT-5 for example, how expensive would it be to get that same level of intelligence today with other cheaper options? I actually did a good bit of research here to find what models have similar scores to GPT-5. And it was harder than expected because we've had to overhaul almost all the benchmarks that we use over the last year because we saturated everything. I eventually stumbled on V4 Flash being roughly equivalent to 5 Codex High because the handful of benches they had overlapping were actually similar in score. Funny enough, the Deep Seek V4 Flash model is actually outperforming it in a bunch of
Watch from 7:28places like Humanities Last Exam, AI Amnitions, which is a benchmark measuring hallucination rates, but it got roughly the same overall intelligence index score. Deep Seek V4 Flash is 21 times cheaper than GPT-5 for input tokens and 55 times cheaper for output tokens. I couldn't find good numbers to measure the efficiency changes here, but even if you assumed that Deep Seek was 3x less efficient, the difference still nets out to around a 20x improvement. So if the goal was to get 10 times cheaper intelligence at a given level, we blew that out the park. We're at 20 times cheaper. Thankfully, this is actually being tracked by our friends at Artificial Analysis. They would pick a given level, which was intelligence less than 10 points before. That used to cost 60 cents per task, and that got all the way down to 2 cents per task over the course of about 2 years. From November of 2022 to July of 2024, we saw that cost go down massively. But then we start ranking things between 10 and 20 points. Initially, they were expensive all the way up to like $15 per task with O1 preview, cuz that model was super expensive. But you can see over time, it didn't take very long to drop. It was September of 2024 that it cost $15 for this level, and by February of 2025, just a few months later, Grok-3 mini was getting similar numbers at a way, way cheaper price. Pennies to what were dollars before. 16 cents to get a score
Watch from 9:00that was actually higher than O1 preview, cuz O1 preview only got an 11. And that was September of 2024, and by February of 2025, Grok-3 mini had four points on top of it at under a hundredth the price. Sorry, roughly a hundredth the price. Pretty crazy that the price has dropped by almost 99% in just 4 months. And Chad just made a good observation. Chad said, "The drops are faster each time." That is where things get really, really cool. The speed of this drop was a little crazy because the O1 preview and O1 models were more expensive than they should have been, but the drop happened nearly immediately after. And then we get to the 20s, where O3 was the first model to break 20, and from April to August that year, GPT-5 mini got a meaningfully higher score. Actually, no, it's like the same score. It's a 20 to 21. But instead of costing $1.55, it only cost 26 cents. Then DeepSeek 3.2 got the same score for 10 cents. And now there's models like Ling-3 flash that get the same score for 4.7 cents. But now models are even smarter. The ones in the 30s started at a dollar and 90 cents with 5.2. And not long after got to literally 3 cents. And now that we're in the 40s to the 50s, Opus 4.7 started us strong here at $3.85 per task, which also this is a fun trend.
Watch from 10:32Initially, the new model at the new level isn't that much more expensive. But then with 01, this new level cost us way more. Then the next new level cost us less more. Then the next one after was roughly the same. The next one after was slightly more expensive but barely. So like when a new level of intelligence happens, the cost isn't going up a whole bunch then even. And yes, all the axes are exponential here. The cost measure here is log, so the top is $16 and halfway through is a dollar. So remember that as we look. But now we get to where we are now, where you can see some of the most intense battlegrounds we've ever seen in LLM history, where when we got Opus 4.7, which was meaningful bump hitting a 41 on the intelligence index, we got 5.6 Tera not long after with a higher score at like almost half the price. Then GLM 5.3 getting an even higher score at under a third the price. And then GLM 5.3 Flash getting it nearly the same score at a hundredth the price. So while it might feel bad now because we just did another one of these bumps, at a given level of intelligence prices do drop massively, way more than 10x year over year. And when I said there was three things, I kind of lied cuz there's one last piece that's important to know. As intelligence has gotten more intelligent, the ways we use it have changed. We used to have the model auto complete a single function for us in our code base. Now we have it take a screenshot of a bug and get the change landed for us. The models can do so much
Watch from 11:50more. They're going end to end. We don't have to interfere as much because they stay on task better, too. So yes, our bills are getting higher. I'm not trying to gaslight you into thinking otherwise, but it's not because the token cost went up, it's because the usage went up. We are using these models so much more. They are running longer. They are doing a bigger tasks. They are completing work that we used to have to intervene with all the time. The fact that we are generating way more tokens per prompt is success. It's because the models got cheaper and better and more capable. This is kind of a Jevons paradox thing. When steam power got cheaper and coal got more accessible, we started spending much more money on all of those things because it suddenly was cheap enough and good enough to be usable in places it wasn't before. Our bills aren't going up because the models are more expensive, they're going up because we're using the expensive models more. It really is that simple. And this is where things get really fun. Like A4 in chat saying, "Imagine getting Fable 5.1 levels of intelligence for GLM 5 3 Flash pricing." It's going to happen. It's absolutely going to happen because I kind of danced around something earlier. I was comparing 5 3 Flash and 5 3 Flash pricing to GPT 5 pricing. And comparing the 25 to 50 cents per mil out on 5 3 Flash against the $10 per mil out on GPT 5 is unfair. It's unfair because it does not do enough credit to GLM 5 3 Flash. 5 3 Flash is a way, way better model than GPT 5. I would rather use GLM 5 3 Flash
Watch from 13:20for everything than be stuck using GPT 5 for anything because 5 3 Flash has a lot of the industry-wide advancements that have happened in the last year. The way it does vision is better. The way it handles tool calls is better. Most importantly, the way it manages long-running work in agentic workflows where the model goes and does things for hours instead of seconds is comically better. GLM 5 3 Flash is not just cheaper for a given level of capability than GPT 5, it's also just so much better in every realistic way. The only thing 53 flash doesn't have that GPT-5 does is a bit more knowledge that doesn't require searching. And chances are if you gave both a search tool and bench them on any knowledge anything, 53 flash will smoke GPT-5 like order of magnitude difference simply because it can do the searching well. It can use its tools well. It can get solutions better even if the model is dumber in terms of what we've stuffed into its brain. Small models with good tools massively outperform last year's smartest models even with tools because we've gotten so much better at that. And we're going to see more and more of these types of improvements. And this is coming from the guy who not that long ago said that we had peaked. Two years ago I did a video back when I was mostly blonde and mustached titled AI isn't going to keep improving because I thought we had kind of flatlined. And you can see here in the bench I had GPT-3.5 then GPT-4, 14, and 4.0 did not
Watch from 14:50meaningfully improve on the benchmarks that we were looking at. That's because we needed new benchmarks. I was so wrong. I've never in my life had a take that was that bad. And I I've learned my lesson. I was so wrong it's hilarious. You're going to make fun of me for being a guy who doesn't know what they're doing, make fun of that version of me not whatever you're mad about when I talk about compaction or something. I be people are mad at me for things that I'm right on and they are not mad at me for the things I was really really wrong on for some reason. But yeah, the point I'm trying to make is that we are still trying to find ways to measure these levels of intelligence. Things are so much better than I ever would have guessed they would have gotten way faster than I ever would have expected it to happen and way cheaper than I ever could have imagined it would be. The issue is that these things have gotten so cheap and so capable that we are just pissing money all over them. We are spending so much because they are so useful. And that doesn't mean that we failed. In fact, I would go as far as to say intelligence too cheap to measure, that isn't happening in the way that Sam may have implied here, but his previous promise of intelligence 10 times cheaper has been blown out of the water. We absolutely achieved that. The tasks that we used AI for last year are actually that much cheaper. The difference is we can do bigger and tougher things with it, which is, in my opinion, way more exciting. All that said, it's kind of crazy that you can rack up $1,500 of spend in a day using these models. I don't want to discount that with the things I'm saying.
Watch from 16:21All I'm saying is that you would never have been able to do that with GPT-5, not because it was so much cheaper, but because you just wouldn't be able to get that much work done. LLM spend is tied to the productivity we get from LLMs, and yes, we are getting more from that. Think I've said all I have to on this one. I hope you can better understand where our money's going now. And don't worry, things will continue to get cheaper, even if it doesn't look like it right now. Keep your head up and be excited for all the developments that'll continue to make models cheaper and more capable. I know I've been hyped about this, and I hope you are, too. Until next time, peace, nerds.
Watch from 17:47Full transcript
- 0:00
Intelligence too cheap to measure is AI's next frontier. According to Sam Altman back in July of 2025, I'm pretty sure that in all of 2025, my personal use of AI would probably amount to like $800 to $1,000. Versus this year, 2026, any given day, I can probably pull $1,000 in tokens. It seems like
- 0:22
intelligence is getting way more expensive, especially now with models like Fable and Astra that are literally more expensive per token and are doing more in it generating more tokens, massively increasing the cost that real-world businesses are seeing. It seems like everyone's AI bill is just going up and up and up with no sign of stopping. So, what happened here? Why aren't we achieving this new frontier as
- 0:44
Sam Altman promised us? Why isn't intelligence getting cheaper? I feel like I was spending a few pennies per prompt last year, and now I'm spending tens, if not hundreds, of dollars. What went so wrong? Why are these companies charging so much more? I thought it was supposed to get cheaper. This video is inspired by Primagen, because he made a tweet referencing a real promise that Sam Altman had made,
- 1:05
which is that he said AI would get 10 times cheaper year over year. And Prime said he's a bit confused because he thought that we would get this 10 times cheaper AI, but Astra's out and it's way more expensive. So, what went wrong? I got a lot of thoughts on this one. The short version of what happened here is it did actually get cheaper. We're
- 1:25
just not measuring it right. Long version, aka the video, will be the next thing after the sponsor break. Okay, look. I know this is an ad, but I just want to nerd out for a minute, because I've been building a lot more with Railway recently and I've had so much fun doing it. Let me get some bias out of the way first. I've loved Railway for years now. I was almost their fourth hire, but I instead chose to make my own startup and then my YouTube channel soon
- 1:46
after. I know this team well. I love them and everything they're building is awesome. Railway had already built the easiest cloud for developers to build and ship real full-stack applications on. it. guys can everything from the database to the CDN and everything else in between. You could build your full app from scratch on Railway. And it was really nice when you needed persistent servers. Where things got genuinely magical is when they introduced their
- 2:06
skills, CLIs, and MCPs to our agents. And now I can have agents just deploy and manage full stack apps without ever having to think about it. Even calling these apps almost feels misleading though, cuz one of those apps is a cloud I built on top of Railway. Yes, the cloud I've been working on for a while now, Lakebed, is entirely built on top of Railway and I'm so happy I did it this way. I have my full production
- 2:27
environment as well as a staging environment that my agents provisioned for me, as well as an environment for one specific PR that again, my agent provisioned for me to test things out end to end. When I have any questions about how the infra is working or any ideas for how it could work better, I tell Claude or Codex about it. It connects to the MCP, it spins up a new environment, it figures it out, it verifies the changes, it ships to prod
- 2:47
by merging on main. It will then check it on the environment through the MCP server and just know what's going on. Ready for the spiciest part of this ad? I haven't looked at this dashboard in weeks cuz I don't have to, because my agents can manage all of it for me. Whether I'm building a one-off experiment to host my own analytics like I did with Slot Paws, or I'm trying to put together an HTML host like I did with Post Plan, or I'm building my own cloud with Lakebed, my default has been
- 3:09
Railway for months now and I'm so thankful that changed. If you're worried about costs, don't be. I'm hosting a ton of stuff on Railway including our entire internal video pipeline where this ad's actually going to be uploaded, and my estimated costs for the month are under $50 for this insane amount of usage I'm getting. Ship on a platform that you and your agents will love at sodalite.link/railway. Sorry for the break, good to have you
- 3:30
back. In order to explain why AI is actually getting cheaper, we're going to have to look at some numbers and talk more about how models work. The first thing we need to talk about is token cost. Flash bang warning for artificial analysis. When you look at model prices over time, the numbers are pretty scary. GPT-5 medium was $10 per million tokens out
- 3:50
and a per million tokens in. Versus a model like GBT-6 Astra, which is $50 per mil out and 10 per mil in. That is a five to 10x increase on input and output tokens cost. That's crazy. That's terrifying. That's the exact opposite of what we were promised. We were promised 10x decreases in cost, but there's a 10x
- 4:10
increase. Well, there are layers to this because token cost is one of the worst ways to measure cost. The three things we need to be considered of are first, yeah, sure, token cost. I'll talk a little bit about it. But the other two are where most of the important parts come from. The second one is how many tokens it takes to complete a given piece of work. And the third is the types of work the models can complete.
- 4:30
Specifically, how much intelligence per dollar, so to speak. So, we have token cost, tokens per dollar. We have tokens per task cost, how many tokens it take to solve a task. And then we have cost per level of intelligence. How much spend does it take? How much money do we have to spend to get a solution to a problem at a certain level of intelligence? So, yes, token costs did
- 4:52
go up for the best of the best models, but let's be real here, there are also cheap options that are incredible. For example, a model like JLM 5.3 Flash, which I absolutely adore, it's one of my favorite model releases ever. This model is 15 cents per million input tokens and 50 cents per million out. It's also on sale in a lot of places for 50% off. So, it's actually 7 cents per mil in and 25
- 5:15
per mil out. That means that it's about a 30th the cost of GPT-5. For a 20th to a 30th the price per token, JLM 5.3 Flash will run circles around GPT-5 for a lot of real-world agentic stuff. So, again, if we look at the top-tier models, yes, cost per token's gone up.
- 5:36
But if you look at other places, you'll see that we've also gotten really capable models that are way cheaper than before as well. So, yes, the top end has gotten crazier, but so has the bottom end. Both sides have seen meaningful improvements here. But as I mentioned before, cost per token is only a tiny piece of the formula here. I'll use a
- 5:56
really fun example. Let's look at Gemini 3 8 Flash. On this chart, it looks like a pretty good deal. It's $3.75 per million out. That makes it less than half the price of GPT-5 and less than a tenth the price of Astra and Fable. But then we hop up a little bit to cost per intelligence index task, and suddenly that difference gets a lot less different because 3 8
- 6:18
Flash uses so many tokens for the work that it's doing. So even though it is comically cheaper per token, the token waste that 3 8 Flash has results in this gap going from over 10x to under 3x. And 3.5 Flash was even worse here cuz it was so token hungry, getting to over half the cost of 5 6 Soul, even though 5 6 Soul is a way more expensive model. So
- 6:40
once again, you need to factor all of these things in. Hilariously enough, Sonnet 5 was more expensive than Astra because Sonnet just burns tokens. So those are the first two dimensions to consider. Cost per token and the amount of tokens used to solve a given task. And to be clear, different tasks have different ways that they work here. For example,
- 7:01
if the task is to ingest 800,000 tokens and categorize them in some way, the categorization and the output tokens aren't going to matter as much. The input tokens will, so you're always paying that cost basically no matter what. At that point, optimizing for the cheapest model for the input token read is probably the right call. If you're doing agentic dev work, things like
- 7:21
caching, reliability, and token efficiency suddenly matter a hell of a lot more. We're also not looking at this the right way. We're just looking at cost right here. What we want to see is cost versus intelligence. This chart is much more interesting, where you can see how expensive a run was and how well it performed in the benchmarks. They have
- 7:42
this nice line now for the Pareto frontier, which is what is the best model at a given cost level. And you'll see GBD-6 Astra has a significant number of the spots here. And Fable ends up being way more expensive all the way over there, but you'll see at a given level of intelligence, cost can go down massively. Like if you're happy with a 45 to a 46 as the score here, your cost
- 8:05
will be 82 cents per task. Where if you want to bump up into the 50s, suddenly it's $3.26. So for just a few more points, you're spending three times more money. I still don't think this is the best way to think about it though. I want to flip it. Instead of thinking about this as how much intelligence can you get per dollar, let's take a level of intelligence and see how much it costs
- 8:25
over time. If we took at the capability of GPT-5 for example, how expensive would it be to get that same level of intelligence today with other cheaper options? I actually did a good bit of research here to find what models have similar scores to GPT-5. And it was harder than expected because we've had to overhaul almost all the benchmarks that we use over the last year because
- 8:46
we saturated everything. I eventually stumbled on V4 Flash being roughly equivalent to 5 Codex High because the handful of benches they had overlapping were actually similar in score. Funny enough, the Deep Seek V4 Flash model is actually outperforming it in a bunch of places like Humanities Last Exam, AI Amnitions, which is a benchmark measuring hallucination rates, but it got roughly the same overall
- 9:07
intelligence index score. Deep Seek V4 Flash is 21 times cheaper than GPT-5 for input tokens and 55 times cheaper for output tokens. I couldn't find good numbers to measure the efficiency changes here, but even if you assumed that Deep Seek was 3x less efficient, the difference still nets out to around a 20x improvement. So if the goal was to
- 9:30
get 10 times cheaper intelligence at a given level, we blew that out the park. We're at 20 times cheaper. Thankfully, this is actually being tracked by our friends at Artificial Analysis. They would pick a given level, which was intelligence less than 10 points before. That used to cost 60 cents per task, and that got all the way down to 2 cents per task over the course
- 9:53
of about 2 years. From November of 2022 to July of 2024, we saw that cost go down massively. But then we start ranking things between 10 and 20 points. Initially, they were expensive all the way up to like $15 per task with O1 preview, cuz that model was super expensive. But you can see over time, it didn't take very long to drop. It was
- 10:14
September of 2024 that it cost $15 for this level, and by February of 2025, just a few months later, Grok-3 mini was getting similar numbers at a way, way cheaper price. Pennies to what were dollars before. 16 cents to get a score that was actually higher than O1 preview, cuz O1 preview only got an 11.
- 10:35
And that was September of 2024, and by February of 2025, Grok-3 mini had four points on top of it at under a hundredth the price. Sorry, roughly a hundredth the price. Pretty crazy that the price has dropped by almost 99% in just 4 months. And Chad just made a good observation. Chad said, "The drops are faster each time." That is where things
- 10:56
get really, really cool. The speed of this drop was a little crazy because the O1 preview and O1 models were more expensive than they should have been, but the drop happened nearly immediately after. And then we get to the 20s, where O3 was the first model to break 20, and from April to August that year, GPT-5 mini got a meaningfully higher score.
- 11:17
Actually, no, it's like the same score. It's a 20 to 21. But instead of costing $1.55, it only cost 26 cents. Then DeepSeek 3.2 got the same score for 10 cents. And now there's models like Ling-3 flash that get the same score for 4.7 cents. But now models are even smarter. The ones in the 30s started at a dollar and 90 cents with 5.2.
- 11:38
And not long after got to literally 3 cents. And now that we're in the 40s to the 50s, Opus 4.7 started us strong here at $3.85 per task, which also this is a fun trend. Initially, the new model at the new level isn't that much more expensive. But then with 01, this new level cost us way more. Then the next new level cost
- 11:59
us less more. Then the next one after was roughly the same. The next one after was slightly more expensive but barely. So like when a new level of intelligence happens, the cost isn't going up a whole bunch then even. And yes, all the axes are exponential here. The cost measure here is log, so the top is $16 and halfway through is a dollar. So
- 12:20
remember that as we look. But now we get to where we are now, where you can see some of the most intense battlegrounds we've ever seen in LLM history, where when we got Opus 4.7, which was meaningful bump hitting a 41 on the intelligence index, we got 5.6 Tera not long after with a higher score at like almost half the price. Then GLM 5.3 getting an even higher score at
- 12:42
under a third the price. And then GLM 5.3 Flash getting it nearly the same score at a hundredth the price. So while it might feel bad now because we just did another one of these bumps, at a given level of intelligence prices do drop massively, way more than 10x year over year. And when I said there was three things, I kind of lied cuz there's one last piece that's important to know.
- 13:04
As intelligence has gotten more intelligent, the ways we use it have changed. We used to have the model auto complete a single function for us in our code base. Now we have it take a screenshot of a bug and get the change landed for us. The models can do so much more. They're going end to end. We don't have to interfere as much because they stay on task better, too. So yes, our
- 13:26
bills are getting higher. I'm not trying to gaslight you into thinking otherwise, but it's not because the token cost went up, it's because the usage went up. We are using these models so much more. They are running longer. They are doing a bigger tasks. They are completing work that we used to have to intervene with all the time. The fact that we are generating way more tokens per prompt is
- 13:46
success. It's because the models got cheaper and better and more capable. This is kind of a Jevons paradox thing. When steam power got cheaper and coal got more accessible, we started spending much more money on all of those things because it suddenly was cheap enough and good enough to be usable in places it wasn't before. Our bills aren't going up because the models are more expensive,
- 14:07
they're going up because we're using the expensive models more. It really is that simple. And this is where things get really fun. Like A4 in chat saying, "Imagine getting Fable 5.1 levels of intelligence for GLM 5 3 Flash pricing." It's going to happen. It's absolutely going to happen because I kind of danced around something earlier.
- 14:27
I was comparing 5 3 Flash and 5 3 Flash pricing to GPT 5 pricing. And comparing the 25 to 50 cents per mil out on 5 3 Flash against the $10 per mil out on GPT 5 is unfair. It's unfair because it does not do enough credit to GLM 5 3 Flash. 5 3 Flash is a way, way better model than GPT 5. I would rather use GLM 5 3 Flash
- 14:50
for everything than be stuck using GPT 5 for anything because 5 3 Flash has a lot of the industry-wide advancements that have happened in the last year. The way it does vision is better. The way it handles tool calls is better. Most importantly, the way it manages long-running work in agentic workflows where the model goes and does things for hours instead of seconds is comically
- 15:12
better. GLM 5 3 Flash is not just cheaper for a given level of capability than GPT 5, it's also just so much better in every realistic way. The only thing 53 flash doesn't have that GPT-5 does is a bit more knowledge that doesn't require searching. And chances are if you gave both a search tool and
- 15:32
bench them on any knowledge anything, 53 flash will smoke GPT-5 like order of magnitude difference simply because it can do the searching well. It can use its tools well. It can get solutions better even if the model is dumber in terms of what we've stuffed into its brain. Small models with good tools massively outperform last year's
- 15:53
smartest models even with tools because we've gotten so much better at that. And we're going to see more and more of these types of improvements. And this is coming from the guy who not that long ago said that we had peaked. Two years ago I did a video back when I was mostly blonde and mustached titled AI isn't going to keep improving because I thought we had kind of flatlined. And
- 16:15
you can see here in the bench I had GPT-3.5 then GPT-4, 14, and 4.0 did not meaningfully improve on the benchmarks that we were looking at. That's because we needed new benchmarks. I was so wrong. I've never in my life had a take that was that bad. And I I've learned my lesson. I was so wrong it's hilarious. You're going to make fun of me for being a guy who doesn't know
- 16:36
what they're doing, make fun of that version of me not whatever you're mad about when I talk about compaction or something. I be people are mad at me for things that I'm right on and they are not mad at me for the things I was really really wrong on for some reason. But yeah, the point I'm trying to make is that we are still trying to find ways to measure these levels of intelligence. Things are so much better than I ever would have guessed they would have gotten way faster than I ever would have
- 16:57
expected it to happen and way cheaper than I ever could have imagined it would be. The issue is that these things have gotten so cheap and so capable that we are just pissing money all over them. We are spending so much because they are so useful. And that doesn't mean that we failed. In fact, I would go as far as to say intelligence too cheap to measure, that isn't happening in the way that Sam
- 17:18
may have implied here, but his previous promise of intelligence 10 times cheaper has been blown out of the water. We absolutely achieved that. The tasks that we used AI for last year are actually that much cheaper. The difference is we can do bigger and tougher things with it, which is, in my opinion, way more exciting. All that said,
- 17:40
it's kind of crazy that you can rack up $1,500 of spend in a day using these models. I don't want to discount that with the things I'm saying. All I'm saying is that you would never have been able to do that with GPT-5, not because it was so much cheaper, but because you just wouldn't be able to get that much work done. LLM spend is tied to the productivity we get from LLMs, and yes, we are getting more from that.
- 18:00
Think I've said all I have to on this one. I hope you can better understand where our money's going now. And don't worry, things will continue to get cheaper, even if it doesn't look like it right now. Keep your head up and be excited for all the developments that'll continue to make models cheaper and more capable. I know I've been hyped about this, and I hope you are, too. Until next time, peace, nerds.
Description
Yes, AI has gotten cheaper and it's going to keep getting cheaper, just not in the way you see with your wallet at the moment, lets break it down.
SponsorThank you Railway for sponsoring!
Check them out at: https://soydev.link/railway
Sources:
https://x.com/ThePrimeagen/status/209...
Want to sponsor a video? Learn more here: https://soydev.link/sponsor-me
Check out my Twitch, Twitter, Discord more at https://t3.gg
S/O @Ph4seon3 for the awesome edit 🙏
#ai #programming #coding
