I love Ultrafast (it's unusable)
TTTheo - t3.gg@t3dotgg · 572K subscribers
- Views
- 145K
- 145,296
- Outlier
- ×0.9
- Median 167K
- Length
- 28:17
- Published
- Oct 6
- Tuesday
- Comments
- 236
- 2K likes
Script
How it's written
- Words
- 5,664
- Words a minute
- 200
- Reading time
- 24 min
- Segments
- 19
The hook
0:00–0:27Ultrafast may not have been ultra fast to get here, but now it is here. And not only can it build absurdly quickly with a model like Astra, it also can burn through your rate limits and bills even faster. It is still genuinely trippy when you're using it though. Like I have an app I just whipped together with Lakeb and I'll say make it light mode and funnier and we'll just watch as it live updates. This is all real time. So, I hope I don't stumble over any words
Structure
Ultrafast may not have been ultra fast to get here, but now it is here. And not only can it build absurdly quickly with a model like Astra, it also can burn through your rate limits and bills even faster. It is still genuinely trippy when you're using it though. Like I have an app I just whipped together with Lakeb and I'll say make it light mode and funnier and we'll just watch as it live updates. This is all real time. So, I hope I don't stumble over any words because I don't want to have to deal with that in post while still maintaining the real time. I was taking a snapshot of the page. Oh, it's already done. Yeah, again, fully real time. It's just flying through it. There's a ton of real use cases for this, like prototyping live, working in the loop, getting work done faster, creating experiences for users that feel way more immediate, computer use in a way that you can watch and feels even faster than a human using the computer, and more. But as powerful as it is to use a model that is this fast, the price is rough. Aster is already a little expensive at $50 per million output tokens. But on ultra fast, it goes up to $300 per million out. In, god forbid, this is a long context project cuz now you're at $450 per million tokens out. These bills are actually insane. When I was first testing out Ultraast, I used it for some basic day-to-day work. Like I had it review two small pull requests that I had open. Both were under a 100 lines of code, and they were quite expensive. I
Watch from 0:00want you to think about that. How much would you guess was the token spend to review two poll requests? And while you think about that, I hope you don't mind me making some of that money back with a quick break for today's sponsor. Going to ask you to think into your past. This might hurt, so I'm sorry in advance. I want you to think back to a particularly complex poll request that you had to review in the days before AI. You might have looked through every line of code and thought you saw everything that could possibly go wrong and left a bunch of feedback just to have it go out and realize things were broken. When that happened, you probably realized the best solution is to actually pull down the changes and test it yourself before giving the thumbs up. But that's slow and tedious and annoying. But if you don't do that, are you actually giving the right realistic feedback for that stuff? Because there are so many times where the code looks totally fine, but then when you go to run it, things work in unexpected ways. Today's sponsor is Graptile, and they asked themselves the same question because they were looking at what their agents were doing and realized that, yeah, they're pretty good at reading code and finding issues, but they do a hell of a lot better when they're given a real computer to run that code on. And that's why I'm so hyped on T-Rex. T-Rex is a new solution that Reptiles put together to allow your review agents to run on a real computer that can actually run and test the changes. So, it's no longer just looking at the code and giving it a thumbs up that it thinks it's fine. It's actually testing the code itself. We've been running T-Rex on a real code bases and we've seen the benefit immediately. For example, in one of Ben's projects for managing all of our YouTube stuff. This PR would have had a real regression in how we manage our YouTube off keys. But since Gretile ran it for real, it noticed the issue immediately and even provided evidence of these failures in
Watch from 1:30their artifacts. They even give you this nice little dashboard to see exactly what went wrong in these runs. Unsurprisingly, you get better feedback if you give your agents a better environment. Get better reviews and ship fewer bugs at soyv.link/grapile. Welcome back. Do you have a price in mind? I see people in chat guessing like $100 per PR. It was $600 of usage for both. It's insane. It's crazy how much money you can burn with this. Chat doesn't seem to believe me that it was 600 two zeros. Yeah. Yeah, it was. But if we go back here and look at the price and realize it went from $50 per mill out to 300 per mill out, you realize just how brutal this is. A 6x increase is rough, especially outside of the output tokens. The cash read cost going up to $6 per mill makes cash reads more expensive than they are to do normal reads on other models. And the cash right cost at 75 bucks is insane. It's like unbelievably high. I'm going to be so real with you guys. I just didn't think Ultra Fast was going to be for me at all. That would make any sense at all. And I still do largely believe that. I don't think almost anyone should be touching ultraast right now. It's just far too expensive, especially over the API rates. But even the amount you get in the new $500 a month ultra fast plan they offer is just it's not enough.
Watch from 3:00They are subsidizing it heavily. So in that ultra fast $500 plan, you get about $1,000 of usage per week, which takes a whopping 2.1 hours to burn when you're using ultraast. I refreshed my limits right before starting prep for this video. And I had 38% left in my Codeex account, the one that has the $500 plan. I sent two prompts. one to do this quick build. It took 38 seconds. And one to make it light mode and funnier, and it took 16 seconds. That knocked my usage down from 38% to 37. That's a 1% hit for under a minute of work. Let's ask it to do a bit more quick. How about turn this into a canban app with a live chat. Make the changes one at a time so I can see them appearing in the dev server as you go. I'm going to knock it down to medium as well. Deploy as you go to because LakeBed will auto update the live deploy when it changes as well, which I think is cool. But let's just go look at the local version for now. So again, all real time. We are not doing the usual cuts. I'm sorry Jeff, you're the one person who's going to have an easier time. And we see it as it updates while it is going.
Watch from 4:30It's full of useless subtext, which I'll ask it to fix, but it's going to keep making changes as it goes, and we'll see them come in live. And it's trippy. You can have a very different working relationship with a setup like this. I found that it like actually makes sense to split screen with your app and your model again because I'm not just asking it to go do the thing and then looking at it when it's done. So if you're struggling a lot with those flows where you ask it to do a thing, the thread disappears, then you come back to it later, this is very good for getting out of that, but it is still just far too expensive for it. I'm going to do I'm going to move this to the inapp browser. Oh, that's the deployed version. Yeah, to the inapp browser so we can see it as it goes a little more easily. And here's where we're going to start noticing the first thing that you'll want to think about when you use ultra fast mode. When your token generation took 10 minutes and your git commands or builds or whatever took under 30 seconds, that ratio meant that the token time was so much higher that you didn't care about optimizing those other
Watch from 5:49pieces. I almost feel like I want to use a different system prompt when I use ultra fast because I want to steer it away from things that aren't as important that take more time because when inference goes from 10 minutes to 30 seconds that 30 seconds of waiting on tool calls goes from a small percentage of your runtime to doubling your runtime. So, I just told it, for example, to not make changes to the preview browser, just change code when I ask because I don't want it to call tools that take seconds because I just want to see the updates while we go. So, let's do another here. There's a ton of useless space being taken up from the top. I want you to delete all unnecessary subtitles and information and really try to clear out the top nav so that way less vertical space is wasted. And now we get to watch it make these changes live. All gone. 11 seconds. It's very nice. And if this is the flow you want to be in the loop with the model, I think this makes a ton of sense. And where this gets really crazy is with models like Astra, they handle getting steered much better. So, I can do something like say make it dark mode by default again and say I don't want guests to be able to chat. It just made the dark mode change in 5 seconds. Allegedly working is a bit lame. Make the copy better. And I'm just sending these while it's doing other things, forcing it to steer, but it will do all of them at the same time. Fine.
Watch from 7:20Can I attach images? Make that work for attaching images in new tickets and chat. Signed in only. Also, new tasks should be signed in only. And you can just spam it with things you notice as it is working. And the result feels something that I've never like actually felt before coding where it's the flow state that you used to get when you were hopping between the code and the output and knowing what was affecting what and just flying between the two back in the day when we would actually write the code ourselves. But you can make way bigger changes this way too cuz you can build real things with this cuz it is the real Astra. It's actually running on Nvidia. I thought it was on Cerebras. I just recently learned it's not with this. It is magic. Combine it with voice and the experience feels fundamentally different. And I understand why all those OpenAI employees were so hyped about this. It's because of this unique feeling with the live back and forth. But it's also because they don't have to pay the bill because OpenAI employees get unlimited inference. That's of course they do. They're employees. But just from the handful of changes I made here, let's see the effect on my usage limits. I was at 37% before that like small handful of changes.
Watch from 8:51Now I'm down to 34 on the $500 plan. I disabled two features. I switched to dark mode and I'm now down 3% of my $500 plan. It's insane. I'm now going to ask 61 Soul on Medium to analyze that thread and see what the dollar value would have been. And while I do that, I will have a sip of a beer because this makes me very depressed. Apparently, OpenAI employees now have to get approval per task for ultra fast because even they are hitting compute restrictions with it because it's so expensive. That's a bit of a relief. No longer feels so much as inference for me and not for the as it did before. God, once you get used to ultra fast, everything else feels super slow. Yeah, if I was paying standard API prices for that Aster usage, it would have been $7.72, but instead it was $46.32. Yeah. Out of curiosity, I want to know what the price would have been using 6.1 soul instead. It would have been $110. So, if I use 61 soul at normal speed, it would have been $110. And it's absolutely capable of the same things here. But instead, I spent over 40x the same amount to do it with Astra ultraast. On one hand, this shows how insane of a value 61 soul is, but it also shows how absurd the cost is for ultraast. And it really is absurd. So, you're probably looking at these numbers and thinking that there's no world in which this makes sense. And I'm not going to say I disagree. I think it was insane of them to release this at this price. I don't want to show it. I don't want to push it. I don't think most people should use it. And it's not
Watch from 10:09surprising that someone like Primagen bought the $500 plan, saw it was the ultra fast plan, and decided to try it out, ended up burning over half his limit in an hour. They really should not have made this seem like a thing you just click. It's the same problem I have with the max reasoning level. Funny enough, when you give users a knob, they will turn it all the way. And that has resulted in a lot of bad decisions by both OpenAI for giving us the knob and the users for using the knob. And yeah, just it sucks. So, where is the value in this? I'm going to explain this in a strange way with a recently released app I built called Slopolytics. I don't know if you guys are like me in this way, but when I look at a site like this and I'm like, "Wow, this actually looks decent. The info is clearly communicated and it's easy to navigate and things work." That must have been made with an anthropic model, right? Like clearly this was done with Opus or Fable, right? Well, I actually built this entirely with Astra on Ultraast. Funny enough, I was working on it with Opus 55 and Ultrafast at the same time, and it ended up taking the Opus run 15 minutes or so before it had a first working draft. Meanwhile, it took Ultra Fast well under 2 minutes. It was close to about a minute and a half of ultraast work on Astra to get a first working draft. But it was Astra, so you can guess it was ugly. The first draft from Opus that
Watch from 11:39took almost 20 minutes looked meaningfully better than the first draft that took a minute and a half using Astro Ultraast. But by the time I saw the Opus version, the Astro one was better because it only took a minute and a half and I looked at it and I said, "Here are the issues I take with this version you just made." And then they updated live in my browser. I was like, "Okay, those things are better. These two things still suck. Can you try this?" And then it tried it. It's like, "Oh, I don't like that. Go back the other way. try this other thing instead. And I ended up sending more messages than I've sent in a thread in a long time. After a bit of finagling and learning some things I up in my existing local T3 code, I found the thread. And I think this will probably be the best way I can showcase why I enjoyed using ultra fast for this so much. I'd go as far as to say I might not have even finished slopytics if it wasn't for ultraast. It made it way more fun to build this way. So the first thing that happened is since I was in the thread, I noticed it trying to use my tsdev skill, which is a skill for using tail scale to host dev servers remotely, but I was doing this on my machine, so I didn't need that. And since I was actually paying attention, I corrected it. I'm on the machine. It can be local host. 30 seconds later, snapshot now has 688 variants. I noticed the load was bad though cuz it started making the page and it had spit out a URL at some point. So I went there and it didn't work. It ended up being roughly 2 minutes of work total before it had a local host version that had all of the data from artificial analysis and a decentish UI. But I noticed some of the bar charts were bad. They were all horizontal which didn't have the
Watch from 13:10comparison I wanted. So I said vertical bars make it all fit fine on a normal screen. I sent a screenshot 41 seconds later where it had a random error and this is what it looked like at the time. Better artificial analysis and it was it was sloppy. And I told it not acceptable. fixed some old CSS issue. Had it live. I said, "Use more logos in colors consistent with the labs. Use artificial analysis for inspiration." A minute later, it was updated. I I almost immediately respond. You'll notice the timestamps here. It responds at 1:14 a.m. I respond at 1:14 a.m. It responds at 1:13. I respond a minute later. Not even. I was in the loop for this. I had stopped paying attention to the Opus thread that was working on the same task. change my mind on sort. Averages are useless. Sort by the highest option. Get rid of average lines. This was all ideas that I had to make the sorting make more sense and to change how I was grouping the models. One of the things I think I did here that's the coolest, I don't exclusively show the max reasoning levels and then sort by tokens on max reasoning. And this is one of the problems I had with the artificial analysis dash. One of my final artificial analysis flashbang warnings ever because I will be showing slopalytics in the future. I just want to compare quickly. The numbers on the homepage here are for Opus 55 and Fable 5.1. These are all max reasoning, as are the costs. None of these numbers make sense. And it gets even worse if you go down to output tokens where you see Sonnet at 200,000 tokens, Opus at 120,000, and then Astra all the way here
Watch from 14:41at 27K. It makes even less sense when you turn on Opus 5.5 high, and it appeared somewhere. Can you find it? I happen to know where it went. It's here. Opus 55 is really far on the left. Even though 55 max is over there, it is impossible to see the relationships between models in a family in this view. So instead, I group the numbers by the model family. And then the order is based on which one has the highest number, which has some fun side effects. For example, when I put in sonnet, it's very much the worst model token efficiency wise. But if I turn off max, which is one of the things I have here is my little toggles between low, mid, high, and max. I can turn off max reasoning. And now sonnet has been moved a bit to the side because it is unreasonable on max effort, but it's actually quite reasonable on XI. And it's even more reasonable on high, dropping down to be lower than Opus for the amount of tokens it's using. Not only would I probably not have spent the time building this if I didn't have ultra fast, I don't think I would have come up with all of these UI patterns because I had to do back and forths where I asked it to do a thing, it didn't look right, and I would just ask it to do something else. And even though Astra is way, way, way, way worse at design than the anthropic models are, I made a better design because I was in the loop with it. And it's been a while since I was in the loop. I even made a change in T3 code where when threads are running they get auto hid because I don't care what the thread is doing until it's done unless of course the
Watch from 16:13model is working with me and we have this real quick back and forth. When I went back to this thread just now I noticed that I was on 61 soul without ultra fast cuz there is no ultra fast for 61 soul. I cannot wait till there is. But I was on 61 soul regular speeds and I made the switch not because I was so scared of wasting all my money, but because my food was in the oven and I wanted to be able to get up and do it and not feel like I was missing as much. So I ended up going and getting my food out the oven, coming back, and it was still working. But then I switched back to ultra fast and I was flying again. And what you'll see in this thread is a very different prompting style than I normally do. Picture with not acceptable. Use more logos and colors consistent with the labs. Change my mind on the sort. Averages are useless. Sort by highest. Get rid of all the useless titles and subtitles. It's so cringe. And like all these are being sent within like minutes of each other. 115, 116, 118, 114. That's like five prompts in 4 minutes. And all the replies are under a minute. Some of them are 20 seconds. The sort change 18 seconds. You don't have time to get distracted. Same with share button. It's just a URL. I was telling it to like remove things. Default view doesn't need to have a URL hash. Only do that when they make changes. Sort orders all wrong. Default should be highest last. Can I ever draw a line showing where it cuts off other models? Oh, and if I click it, it should persist. And hovering other models will show the gap in score and cost. This is another one of the crazy UX things I added that I would never have come up with if I didn't get to go back and forth this way. When I want to compare two models in this chart, let's say I want to compare Sonnet 5 on XI to other things like let's see how Fable 51. Now I have
Watch from 17:45this awesome comparison view that appears in the top left where it shows you the multiplier difference for the cost, the tokens, and the intelligence scores. I only came up with this because I was staring at it while it was being worked on and could have the back and forth with the model and try different things. I ended up doing a lot of back and forth here. I I forgot what the first version looked like. I screenshotted it because I hated it. Yet here it was like covering things up. I changed my mind three times as to where this card should go and just went back and forth a whole bunch until I was happy with where it ended up. I was so unhappy with how it was organizing the panel. So, I just told it I want these three columns and then it did it. UI feels a bit flickery when hovering around. Not sure the best solution. Maybe delay. It did it and I hated it. I changed my mind. This feels awful. It took 25 seconds to do it, 10 seconds to undo it. And I sent another follow-up. Panel should be top left, not right. Top right's a forbidden area. Need that for my face in my videos. Also, can we move the sidebar to the right in a way that doesn't suck? 32 seconds later, it was done. Hovering names in 2D chart should make the line more prominent and others more transparent. This is in these charts. So, I can hover five or saw it and it makes it clear which is which. That was a thing that I randomly came up with when I was scrolling around till I just threw it there. Then, it was fixed.
Watch from 19:16Good work. Sadly, it's still hideous and needs a bunch of cleanup. Get on that. spin up some separate sub aents to work on branding. And it did. It came up with a bunch of names. I hated all of them. I asked it to have other threads come up with things. It came up with a bunch of garbage. Turns out models are still bad at naming things even if they're fast. I came up with slopalytics. And here's where it became slopalytics. But like, do you see how many messages I did in this thread? There's like 50 plus. I never send that many messages in a thread. It felt entirely different to build this way. Also, I just ran real TPS numbers for my usage. I saw regular at 30 TPS, fast at 60ish, and ultra fast at over 300, between 320 and 340. Crazy. But I also found the cost for all of my threads working on this, that main thread was $36 and the follow-up where I did a bunch of additional work was $250. If I was not using ultra fast and instead I'd used a normal Astro, it would have went from $550 to 90. And if I had used Soul 61, which would have been just as good for this work, by the way, $12. So now we have to ask the question, is saving all that time worth $540 or so to you? The answer is probably no. And if you're comparing it that way, you should never ever ever use ultra fast. It's just straight up not worth it. It is far too expensive. It burns your limits hilariously quickly.
Watch from 20:32It made the $500 tier make no sense at all and is just it's strange. It's bad. It should not be included in the product without a bunch of warnings. So, that's why it's going to hurt me a lot to defend it because this is not a real comparison here. This is a comparison of how much Astro Ultra Fast costs compared to Soul 61. The difference here isn't I could have been more patient and saved $500 something dollars. The difference is that I just wouldn't have built it. If I had to wait between every step, I would have lost the context of what I'm thinking about. I would have lost track of the things I wanted to change and I have to rebuild that context every time a step finishes. So I found myself writing smaller requests and asking for smaller changes instead of doing what I do with Opus, which is look at the output, make a list of the 20 things I want changed, send it, and then expect it to come back in between 5 minutes and 2 hours. And this in the loop work is super useful for certain things like refining a UI. It's silly to put it this way, but I can make better UIs with Astra than I can with Fable and Opus. Not because Astra is better at design. It's objectively worse, but because design is all about how users interact with and perceive things. And if I could spend more time on that perception and I can spend more time in the loop here, I can fly. But this is where I have some hope and also where I think our friend Bot Cooper is catching on. Am I tweaking? Or if the multiplier stays 6x,
Watch from 21:58then 61 soul ultra fast would come in notably cheaper than 6 Astra and much cheaper than 6 Astra fast. Well, considering that 61 standard pricing for Soul is $12.3 and Astra's would have been 90. Yeah, in a hypothetical world where 61 soul comes out with an ultraast mode that happens to be the same price multiplier specifically, not only is it 10 times cheaper than Astra ultraast, it actually comes out around the price, if not cheaper than Astra Standard. Do you see where we're going here? The point of Astra Ultraast is not to be used. It is to be experimented with where lunatics like I can waste far too many tokens on it just to see what it's capable of and get a feel of these flows because we're hoping for a future where you can get these speeds at a price that doesn't make you feel sick. And if we do end up in that future where soul 61 is just as fast on ultra fast as Astra is at a tenth the price, I could actually see that becoming my default model. I already have 61 soul fast as my default for a ton of When I open a new project in fleet, which is my repo for managing things. Oh, it's currently pinned to Astra. I'm actually just going to fix that now because that is not what I want it to be on. I change it on other machines. This is now going to be 61 soul high fast because I think it's a really good balance of speed, capability, and performance in general when I'm trying to do tasks where I'm managing the machines in my network. not
Watch from 23:29for writing code I'm deploying but for working on my systems directly. I got one more thing for y'all. Tibo recently teased 6.1 coming soon. A lot of people assumed that was 61 Astra, but it was actually a reply that was cut off here about ultraast. This is a public confirmation that 6.1 soul ultraast is coming soon. I think that this might end up being cerebrus. Not sure just yet. Could be cerebras. It could just be the crazy Nvidia hacks they're doing right now. Either way, 61 soul ultraast is coming and I think it might be the differentiator that makes this $500 ultraast plan go from feeling overpriced for what you're getting to a genuinely incredible value. So now we are left with one important question. What should we use ultraast for? The only thing this is good for is super really high priority issues you're trying to resolve like an incident response for a high severity issue assuming you work at OpenAI because it is even for high severity incidents probably not worth the money unless the money isn't real. It's just so expensive it is basically impossible to justify. There is one other time you can justify it though and for me that time is right now. I have nine manual resets on my ultra fast account because I went to an event where they gave us six and then I have all the others that we've all been getting. One of those expires in an hour. That means I have one hour to burn 34% of this account's limit. That's a great time to use ultraast. And since our
Watch from 24:59friend Tibo made a pretty bold announcement today, over the next 28 days, each day we'll either ship one thing that is a clear improvement and relevant for most CEX and work users, or we'll ship a full reset. Let the improvements begin. This means there will be a lot of opportunities for me to burn my ultra fast because whenever a reset is promised, they're always late. Whenever they say like it'll be out at 10 a.m., it's out at noon at earliest. So, just start burning. Since I'm using Opus for my serious work anyways, my Codeex account being at zero is fine. It's not going to block me a whole lot. So, as soon as there is a whiff that a reset is coming, I burn that account to the ground. Sometimes the reset happens before I hit zero. Sometimes it happens an hour or two after I hit zero. Sometimes it happens the day after. I'm using my Ultraast when my account's about to reset anyways. So, that's when I use it. I don't know when you should use it. Maybe for incident response here and there. If you're an OpenAI employee, I'm sure you'll use it a lot for those types of things. Use it when you don't care to lose it. Don't upgrade to the $500 tier just for this yet. It's not worth it. With 61 soul ultra fast coming soon, it might suddenly become more worth it. And I'll be sure to keep you guys in the loop when I have more info on that. But for now, $500 tier, I don't think you should reach for it. What about the $200 tier, though? You don't get ultra fast, but you do get 61 Soul, you do get Astra, you get fast mode, you get a lot of other things, you get all these resets. Is the $200 plane still worth it? Well, for that I'm going to need another beer and probably another video that will be coming up very, very soon. So, make sure you're subscribed if you're not so I can cover all of these things and make sure you know what's going on in this crazy AI world. Now you know that Ultra Fast is worth avoiding. Hopefully soon you'll also know if that $200 plan is worth using or avoiding, too. Until next time, God. Peace nerds.
Watch from 26:29Full transcript
- 0:00
Ultrafast may not have been ultra fast to get here, but now it is here. And not only can it build absurdly quickly with a model like Astra, it also can burn through your rate limits and bills even faster. It is still genuinely trippy when you're using it though. Like I have an app I just whipped together with Lakeb and I'll say make it light mode and funnier and we'll just watch as it
- 0:22
live updates. This is all real time. So, I hope I don't stumble over any words because I don't want to have to deal with that in post while still maintaining the real time. I was taking a snapshot of the page. Oh, it's already done. Yeah, again, fully real time. It's just flying through it. There's a ton of real use cases for this, like prototyping live, working in
- 0:45
the loop, getting work done faster, creating experiences for users that feel way more immediate, computer use in a way that you can watch and feels even faster than a human using the computer, and more. But as powerful as it is to use a model that is this fast, the price is rough. Aster is already a little expensive at $50 per million output tokens. But on ultra fast, it goes up to
- 1:06
$300 per million out. In, god forbid, this is a long context project cuz now you're at $450 per million tokens out. These bills are actually insane. When I was first testing out Ultraast, I used it for some basic day-to-day work. Like I had it review two small pull requests that I had open. Both were under a 100 lines of
- 1:27
code, and they were quite expensive. I want you to think about that. How much would you guess was the token spend to review two poll requests? And while you think about that, I hope you don't mind me making some of that money back with a quick break for today's sponsor. Going to ask you to think into your past. This might hurt, so I'm sorry in advance. I want you to think back to a particularly complex poll request that you had to
- 1:47
review in the days before AI. You might have looked through every line of code and thought you saw everything that could possibly go wrong and left a bunch of feedback just to have it go out and realize things were broken. When that happened, you probably realized the best solution is to actually pull down the changes and test it yourself before giving the thumbs up. But that's slow and tedious and annoying. But if you don't do that, are you actually giving the right realistic feedback for that
- 2:09
stuff? Because there are so many times where the code looks totally fine, but then when you go to run it, things work in unexpected ways. Today's sponsor is Graptile, and they asked themselves the same question because they were looking at what their agents were doing and realized that, yeah, they're pretty good at reading code and finding issues, but they do a hell of a lot better when they're given a real computer to run that code on. And that's why I'm so hyped on T-Rex. T-Rex is a new solution
- 2:31
that Reptiles put together to allow your review agents to run on a real computer that can actually run and test the changes. So, it's no longer just looking at the code and giving it a thumbs up that it thinks it's fine. It's actually testing the code itself. We've been running T-Rex on a real code bases and we've seen the benefit immediately. For example, in one of Ben's projects for managing all of our YouTube stuff. This PR would have had a real regression in
- 2:52
how we manage our YouTube off keys. But since Gretile ran it for real, it noticed the issue immediately and even provided evidence of these failures in their artifacts. They even give you this nice little dashboard to see exactly what went wrong in these runs. Unsurprisingly, you get better feedback if you give your agents a better environment. Get better reviews and ship fewer bugs at soyv.link/grapile.
- 3:12
Welcome back. Do you have a price in mind? I see people in chat guessing like $100 per PR. It was $600 of usage for both. It's insane. It's crazy how much money you can burn with this. Chat doesn't seem to believe me that it was 600 two zeros. Yeah. Yeah, it was. But
- 3:32
if we go back here and look at the price and realize it went from $50 per mill out to 300 per mill out, you realize just how brutal this is. A 6x increase is rough, especially outside of the output tokens. The cash read cost going up to $6 per mill makes cash reads more
- 3:55
expensive than they are to do normal reads on other models. And the cash right cost at 75 bucks is insane. It's like unbelievably high. I'm going to be so real with you guys. I just didn't think Ultra Fast was going to be for me at all. That would make any sense at all. And I still do largely believe
- 4:16
that. I don't think almost anyone should be touching ultraast right now. It's just far too expensive, especially over the API rates. But even the amount you get in the new $500 a month ultra fast plan they offer is just it's not enough. They are subsidizing it heavily. So in that ultra fast $500 plan, you get about $1,000 of usage per week, which takes a
- 4:38
whopping 2.1 hours to burn when you're using ultraast. I refreshed my limits right before starting prep for this video. And I had 38% left in my Codeex account, the one that has the $500 plan. I sent two prompts. one to do this quick build. It took 38 seconds. And one to make it light mode and funnier, and it
- 4:58
took 16 seconds. That knocked my usage down from 38% to 37. That's a 1% hit for under a minute of work. Let's ask it to do a bit more quick. How about turn this into a canban app with a live chat. Make the changes one at a time so I can see them
- 5:19
appearing in the dev server as you go. I'm going to knock it down to medium as well. Deploy as you go to because LakeBed will auto update the live deploy when it changes as well, which I think is cool. But let's just go look at the local version for now. So again, all
- 5:39
real time. We are not doing the usual cuts. I'm sorry Jeff, you're the one person who's going to have an easier time. And we see it as it updates while it is going. It's full of useless subtext, which I'll ask it to fix, but it's going to keep making changes as it goes, and we'll see them come in live. And it's trippy. You
- 5:59
can have a very different working relationship with a setup like this. I found that it like actually makes sense to split screen with your app and your model again because I'm not just asking it to go do the thing and then looking at it when it's done. So if you're struggling a lot with those flows where you ask it to do a thing, the thread disappears, then you come back to it
- 6:19
later, this is very good for getting out of that, but it is still just far too expensive for it. I'm going to do I'm going to move this to the inapp browser. Oh, that's the deployed version. Yeah,
- 6:41
to the inapp browser so we can see it as it goes a little more easily. And here's where we're going to start noticing the first thing that you'll
- 7:03
want to think about when you use ultra fast mode. When your token generation took 10 minutes and your git commands or builds or whatever took under 30 seconds, that ratio meant that the token time was so much higher that you didn't care about optimizing those other pieces. I almost feel like I want to use a different system prompt when I use ultra fast because I want to steer it
- 7:25
away from things that aren't as important that take more time because when inference goes from 10 minutes to 30 seconds that 30 seconds of waiting on tool calls goes from a small percentage of your runtime to doubling your runtime. So, I just told it, for example, to not make changes to the preview browser, just change code when I ask because I don't want it to call
- 7:45
tools that take seconds because I just want to see the updates while we go. So, let's do another here. There's a ton of useless space being taken up from the top. I want you to delete all unnecessary subtitles and information and really try to clear out the top nav so that way less vertical space is
- 8:07
wasted. And now we get to watch it make these changes live. All gone. 11 seconds. It's very nice. And if this is the flow you want to be in the loop with the model, I think this makes a ton of sense. And where this gets really crazy is with models like Astra, they handle getting steered much
- 8:28
better. So, I can do something like say make it dark mode by default again and say I don't want guests to be able to chat. It just made the dark mode change in 5 seconds. Allegedly working is a bit lame. Make the copy better. And I'm just sending these while it's doing other things, forcing it to steer, but it will
- 8:49
do all of them at the same time. Fine. Can I attach images? Make that work for attaching images in new tickets and chat. Signed in only. Also, new tasks should be signed in only. And you can just spam it with things you notice as it is working. And the result feels
- 9:11
something that I've never like actually felt before coding where it's the flow state that you used to get when you were hopping between the code and the output and knowing what was affecting what and just flying between the two back in the day when we would actually write the code ourselves. But you can make way bigger changes this way too cuz you can build real things with this cuz it is
- 9:32
the real Astra. It's actually running on Nvidia. I thought it was on Cerebras. I just recently learned it's not with this. It is magic. Combine it with voice and the experience feels fundamentally different. And I understand why all those OpenAI employees were so hyped about this. It's because of this unique feeling with the live back and forth. But it's also because they don't have to
- 9:53
pay the bill because OpenAI employees get unlimited inference. That's of course they do. They're employees. But just from the handful of changes I made here, let's see the effect on my usage limits. I was at 37% before that like small handful of changes. Now I'm down to 34 on the $500 plan. I disabled two features. I switched to
- 10:15
dark mode and I'm now down 3% of my $500 plan. It's insane. I'm now going to ask 61 Soul on Medium to analyze that thread and see what the dollar value would have been. And while I do that, I will have a sip of a beer because this makes me very depressed. Apparently, OpenAI employees now have to get approval per task for ultra fast because even they are hitting
- 10:36
compute restrictions with it because it's so expensive. That's a bit of a relief. No longer feels so much as inference for me and not for the as it did before. God, once you get used to ultra fast, everything else feels super slow. Yeah, if I was paying standard API prices for that Aster usage, it would have been $7.72, but instead it was $46.32.
- 10:57
Yeah. Out of curiosity, I want to know what the price would have been using 6.1 soul instead. It would have been $110. So, if I use 61 soul at normal speed, it would have been $110. And it's absolutely capable of the same things here. But instead, I spent over 40x the same amount to do it with Astra ultraast. On one hand, this shows how insane of a value 61 soul is, but it
- 11:18
also shows how absurd the cost is for ultraast. And it really is absurd. So, you're probably looking at these numbers and thinking that there's no world in which this makes sense. And I'm not going to say I disagree. I think it was insane of them to release this at this price. I don't want to show it. I don't want to push it. I don't think most people should use it. And it's not
- 11:39
surprising that someone like Primagen bought the $500 plan, saw it was the ultra fast plan, and decided to try it out, ended up burning over half his limit in an hour. They really should not have made this seem like a thing you just click. It's the same problem I have with the max reasoning level. Funny enough, when you give users a knob, they will turn it all the way. And that has
- 12:01
resulted in a lot of bad decisions by both OpenAI for giving us the knob and the users for using the knob. And yeah, just it sucks. So, where is the value in this? I'm going to explain this in a strange way with a recently released app I built called Slopolytics. I don't know if you guys are like me in this way, but when I look at a site like this and I'm
- 12:22
like, "Wow, this actually looks decent. The info is clearly communicated and it's easy to navigate and things work." That must have been made with an anthropic model, right? Like clearly this was done with Opus or Fable, right? Well, I actually built this entirely with Astra on Ultraast. Funny enough, I was working on it with Opus 55 and
- 12:45
Ultrafast at the same time, and it ended up taking the Opus run 15 minutes or so before it had a first working draft. Meanwhile, it took Ultra Fast well under 2 minutes. It was close to about a minute and a half of ultraast work on Astra to get a first working draft. But it was Astra, so you can guess it was
- 13:07
ugly. The first draft from Opus that took almost 20 minutes looked meaningfully better than the first draft that took a minute and a half using Astro Ultraast. But by the time I saw the Opus version, the Astro one was better because it only took a minute and a half and I looked at it and I said, "Here are the issues I take with this version you just made." And then they
- 13:29
updated live in my browser. I was like, "Okay, those things are better. These two things still suck. Can you try this?" And then it tried it. It's like, "Oh, I don't like that. Go back the other way. try this other thing instead. And I ended up sending more messages than I've sent in a thread in a long time. After a bit of finagling and learning some things I up in my existing local T3 code, I found the thread. And I think this will probably
- 13:51
be the best way I can showcase why I enjoyed using ultra fast for this so much. I'd go as far as to say I might not have even finished slopytics if it wasn't for ultraast. It made it way more fun to build this way. So the first thing that happened is since I was in the thread, I noticed it trying to use my tsdev skill, which is a skill for using tail scale to host dev servers
- 14:11
remotely, but I was doing this on my machine, so I didn't need that. And since I was actually paying attention, I corrected it. I'm on the machine. It can be local host. 30 seconds later, snapshot now has 688 variants. I noticed the load was bad though cuz it started making the page and it had spit out a URL at some point. So I went there and it didn't work. It ended up being roughly 2 minutes of work total before
- 14:31
it had a local host version that had all of the data from artificial analysis and a decentish UI. But I noticed some of the bar charts were bad. They were all horizontal which didn't have the comparison I wanted. So I said vertical bars make it all fit fine on a normal screen. I sent a screenshot 41 seconds later where it had a random error and this is what it looked like at the time. Better artificial analysis and it was it
- 14:53
was sloppy. And I told it not acceptable. fixed some old CSS issue. Had it live. I said, "Use more logos in colors consistent with the labs. Use artificial analysis for inspiration." A minute later, it was updated. I I almost immediately respond. You'll notice the timestamps here. It responds at 1:14 a.m. I respond at 1:14 a.m. It responds
- 15:14
at 1:13. I respond a minute later. Not even. I was in the loop for this. I had stopped paying attention to the Opus thread that was working on the same task. change my mind on sort. Averages are useless. Sort by the highest option. Get rid of average lines. This was all ideas that I had to make the sorting make more sense and to change how I was
- 15:35
grouping the models. One of the things I think I did here that's the coolest, I don't exclusively show the max reasoning levels and then sort by tokens on max reasoning. And this is one of the problems I had with the artificial analysis dash. One of my final artificial analysis flashbang warnings ever because I will be showing slopalytics in the future. I just want to compare quickly. The numbers on the homepage here are for Opus 55 and Fable
- 15:57
5.1. These are all max reasoning, as are the costs. None of these numbers make sense. And it gets even worse if you go down to output tokens where you see Sonnet at 200,000 tokens, Opus at 120,000, and then Astra all the way here at 27K. It makes even less sense when you turn on Opus 5.5 high, and it
- 16:18
appeared somewhere. Can you find it? I happen to know where it went. It's here. Opus 55 is really far on the left. Even though 55 max is over there, it is impossible to see the relationships between models in a family in this view. So instead, I group the numbers by the model family. And then the order is based on which one has the highest
- 16:39
number, which has some fun side effects. For example, when I put in sonnet, it's very much the worst model token efficiency wise. But if I turn off max, which is one of the things I have here is my little toggles between low, mid, high, and max. I can turn off max reasoning. And now sonnet has been moved a bit to the side because it is unreasonable on max effort, but it's
- 17:00
actually quite reasonable on XI. And it's even more reasonable on high, dropping down to be lower than Opus for the amount of tokens it's using. Not only would I probably not have spent the time building this if I didn't have ultra fast, I don't think I would have come up with all of these UI patterns because I had to do back and forths
- 17:20
where I asked it to do a thing, it didn't look right, and I would just ask it to do something else. And even though Astra is way, way, way, way worse at design than the anthropic models are, I made a better design because I was in the loop with it. And it's been a while since I was in the loop. I even made a change in T3 code where when threads are running they get auto hid because I don't care what the thread is doing
- 17:42
until it's done unless of course the model is working with me and we have this real quick back and forth. When I went back to this thread just now I noticed that I was on 61 soul without ultra fast cuz there is no ultra fast for 61 soul. I cannot wait till there is. But I was on 61 soul regular speeds and I made the switch not because I was so scared of wasting all my money, but
- 18:02
because my food was in the oven and I wanted to be able to get up and do it and not feel like I was missing as much. So I ended up going and getting my food out the oven, coming back, and it was still working. But then I switched back to ultra fast and I was flying again. And what you'll see in this thread is a very different prompting style than I normally do. Picture with not acceptable. Use more logos and colors
- 18:22
consistent with the labs. Change my mind on the sort. Averages are useless. Sort by highest. Get rid of all the useless titles and subtitles. It's so cringe. And like all these are being sent within like minutes of each other. 115, 116, 118, 114. That's like five prompts in 4 minutes. And all the replies are under a minute. Some of them are 20 seconds. The sort change 18 seconds. You don't have
- 18:44
time to get distracted. Same with share button. It's just a URL. I was telling it to like remove things. Default view doesn't need to have a URL hash. Only do that when they make changes. Sort orders all wrong. Default should be highest last. Can I ever draw a line showing where it cuts off other models? Oh, and if I click it, it should persist. And hovering other models will show the gap in score and cost. This is another one of the crazy UX things I added that I
- 19:05
would never have come up with if I didn't get to go back and forth this way. When I want to compare two models in this chart, let's say I want to compare Sonnet 5 on XI to other things like let's see how Fable 51. Now I have this awesome comparison view that appears in the top left where it shows you the multiplier difference for the cost, the tokens, and the intelligence scores. I only came up with this because
- 19:27
I was staring at it while it was being worked on and could have the back and forth with the model and try different things. I ended up doing a lot of back and forth here. I I forgot what the first version looked like. I screenshotted it because I hated it. Yet here it was like covering things up. I changed my mind three times as to where this card should go and just went back and forth a whole bunch until I was
- 19:48
happy with where it ended up. I was so unhappy with how it was organizing the panel. So, I just told it I want these three columns and then it did it. UI feels a bit flickery when hovering around. Not sure the best solution. Maybe delay. It did it and I hated it. I changed my mind. This feels awful. It took 25 seconds to do it, 10 seconds to undo it. And I sent another follow-up. Panel should be top left, not right. Top
- 20:09
right's a forbidden area. Need that for my face in my videos. Also, can we move the sidebar to the right in a way that doesn't suck? 32 seconds later, it was done. Hovering names in 2D chart should make the line more prominent and others more transparent. This is in these charts. So, I can hover five or saw it and it makes it clear which is which. That was a thing that I randomly came up
- 20:29
with when I was scrolling around till I just threw it there. Then, it was fixed. Good work. Sadly, it's still hideous and needs a bunch of cleanup. Get on that. spin up some separate sub aents to work on branding. And it did. It came up with a bunch of names. I hated all of them. I asked it to have other threads come up with things. It came up with a bunch of garbage. Turns out models are still bad at naming things even if they're fast. I
- 20:51
came up with slopalytics. And here's where it became slopalytics. But like, do you see how many messages I did in this thread? There's like 50 plus. I never send that many messages in a thread. It felt entirely different to build this way. Also, I just ran real TPS numbers for my usage. I saw regular at 30 TPS, fast at 60ish, and ultra fast
- 21:14
at over 300, between 320 and 340. Crazy. But I also found the cost for all of my threads working on this, that main thread was $36 and the follow-up where I did a bunch of additional work was $250. If I was not using ultra fast and instead I'd used a normal Astro, it would have went from $550 to 90. And if
- 21:35
I had used Soul 61, which would have been just as good for this work, by the way, $12. So now we have to ask the question, is saving all that time worth $540 or so to you? The answer is probably no. And if you're comparing it that way, you should never ever ever use ultra fast. It's just straight up not worth it. It is far too expensive. It
- 21:56
burns your limits hilariously quickly. It made the $500 tier make no sense at all and is just it's strange. It's bad. It should not be included in the product without a bunch of warnings. So, that's why it's going to hurt me a lot to defend it because this is not a real comparison here. This is a comparison of how much Astro Ultra Fast costs compared to Soul 61. The difference here isn't I
- 22:18
could have been more patient and saved $500 something dollars. The difference is that I just wouldn't have built it. If I had to wait between every step, I would have lost the context of what I'm thinking about. I would have lost track of the things I wanted to change and I have to rebuild that context every time a step finishes. So I found myself writing smaller requests and asking for
- 22:38
smaller changes instead of doing what I do with Opus, which is look at the output, make a list of the 20 things I want changed, send it, and then expect it to come back in between 5 minutes and 2 hours. And this in the loop work is super useful for certain things like refining a UI. It's silly to put it this
- 22:59
way, but I can make better UIs with Astra than I can with Fable and Opus. Not because Astra is better at design. It's objectively worse, but because design is all about how users interact with and perceive things. And if I could spend more time on that perception and I can spend more time in the loop here, I can fly. But this is where I have some
- 23:20
hope and also where I think our friend Bot Cooper is catching on. Am I tweaking? Or if the multiplier stays 6x, then 61 soul ultra fast would come in notably cheaper than 6 Astra and much cheaper than 6 Astra fast. Well, considering that 61 standard pricing for Soul is $12.3
- 23:41
and Astra's would have been 90. Yeah, in a hypothetical world where 61 soul comes out with an ultraast mode that happens to be the same price multiplier specifically, not only is it 10 times cheaper than Astra ultraast, it actually comes out around the price, if not cheaper than Astra Standard. Do you see
- 24:01
where we're going here? The point of Astra Ultraast is not to be used. It is to be experimented with where lunatics like I can waste far too many tokens on it just to see what it's capable of and get a feel of these flows because we're hoping for a future where you can get these speeds at a price that doesn't make you feel sick. And if we do end up
- 24:23
in that future where soul 61 is just as fast on ultra fast as Astra is at a tenth the price, I could actually see that becoming my default model. I already have 61 soul fast as my default for a ton of When I open a new project in fleet, which is my repo for managing things. Oh, it's currently pinned to Astra. I'm actually just going to fix that now because that is not what
- 24:43
I want it to be on. I change it on other machines. This is now going to be 61 soul high fast because I think it's a really good balance of speed, capability, and performance in general when I'm trying to do tasks where I'm managing the machines in my network. not for writing code I'm deploying but for working on my systems directly. I got
- 25:04
one more thing for y'all. Tibo recently teased 6.1 coming soon. A lot of people assumed that was 61 Astra, but it was actually a reply that was cut off here about ultraast. This is a public confirmation that 6.1 soul ultraast is coming soon. I think that this might end up being cerebrus. Not sure just yet. Could be cerebras. It could just be the
- 25:25
crazy Nvidia hacks they're doing right now. Either way, 61 soul ultraast is coming and I think it might be the differentiator that makes this $500 ultraast plan go from feeling overpriced for what you're getting to a genuinely incredible value. So now we are left with one important question. What should we use ultraast for? The only thing this is good for is super really high
- 25:47
priority issues you're trying to resolve like an incident response for a high severity issue assuming you work at OpenAI because it is even for high severity incidents probably not worth the money unless the money isn't real. It's just so expensive it is basically impossible to justify. There is one other time you can justify it though and
- 26:09
for me that time is right now. I have nine manual resets on my ultra fast account because I went to an event where they gave us six and then I have all the others that we've all been getting. One of those expires in an hour. That means I have one hour to burn 34% of this account's limit. That's a great time to use ultraast. And since our
- 26:29
friend Tibo made a pretty bold announcement today, over the next 28 days, each day we'll either ship one thing that is a clear improvement and relevant for most CEX and work users, or we'll ship a full reset. Let the improvements begin. This means there will be a lot of opportunities for me to burn my ultra fast because whenever a reset is promised, they're always late.
- 26:51
Whenever they say like it'll be out at 10 a.m., it's out at noon at earliest. So, just start burning. Since I'm using Opus for my serious work anyways, my Codeex account being at zero is fine. It's not going to block me a whole lot. So, as soon as there is a whiff that a reset is coming, I burn that account to the ground. Sometimes the reset happens before I hit zero. Sometimes it happens
- 27:11
an hour or two after I hit zero. Sometimes it happens the day after. I'm using my Ultraast when my account's about to reset anyways. So, that's when I use it. I don't know when you should use it. Maybe for incident response here and there. If you're an OpenAI employee, I'm sure you'll use it a lot for those types of things. Use it when you don't care to lose it. Don't upgrade to the $500 tier just for this yet. It's not
- 27:31
worth it. With 61 soul ultra fast coming soon, it might suddenly become more worth it. And I'll be sure to keep you guys in the loop when I have more info on that. But for now, $500 tier, I don't think you should reach for it. What about the $200 tier, though? You don't get ultra fast, but you do get 61 Soul, you do get Astra, you get fast mode, you get a lot of other things, you get all these resets. Is the $200 plane still
- 27:53
worth it? Well, for that I'm going to need another beer and probably another video that will be coming up very, very soon. So, make sure you're subscribed if you're not so I can cover all of these things and make sure you know what's going on in this crazy AI world. Now you know that Ultra Fast is worth avoiding. Hopefully soon you'll also know if that $200 plan is worth using or avoiding, too. Until next time,
- 28:15
God. Peace nerds.
Description
I burned $600 reviewing two small pull requests with UltraFast, but building Slopalytics showed me why this speed changes how I work.
Thank you to Greptile for Sponsoring: https://soydev.link/greptile
Want to sponsor a video? Learn more here: https://soydev.link/sponsor-me
Check out my Twitch, Twitter, Discord more at https://t3.gg
S/O @Ph4seon3 for the awesome edit 🙏
#openai #programming #ultrafast
