So I've been testing Gemini 4...
TTTheo - t3.gg@t3dotgg · 572K subscribers
- Views
- 223K
- 223,347
- Outlier
- ×1.3
- Median 167K
- Length
- 38:30
- Published
- Oct 14
- Wednesday
- Comments
- 438
- 3.7K likes
Script
How it's written
- Words
- 7,921
- Words a minute
- 206
- Reading time
- 34 min
- Segments
- 27
The hook
0:00–0:26It finally happened. Gemini 4 Argon is here. Before we get started, I got a quick question. How long do you think it has been since we got a normal-sized large Gemini model release, like the Pro line? How many months do you think it has been? The gap between a model release like 5 6 Soul and 6 Astra was 1 to 2 months. The gap between 3 1 Pro and Gemini 4 Argon is 8 months. It's been since February. We have not gotten anything other than flash models from
Structure
It finally happened. Gemini 4 Argon is here. Before we get started, I got a quick question. How long do you think it has been since we got a normal-sized large Gemini model release, like the Pro line? How many months do you think it has been? The gap between a model release like 5 6 Soul and 6 Astra was 1 to 2 months. The gap between 3 1 Pro and Gemini 4 Argon is 8 months. It's been since February. We have not gotten anything other than flash models from Google. And now, after teasing Gemini 4 Argon over a week ago, they have finally put it out for all of us to use. With a drop like this, there's a lot we have to cover, from how it's priced to how good it is at front end, to all the 3D capabilities, the weird things that make Gemini models Gemini models, the unique capabilities in those models, and of course, the comparison with recent releases like Opus 5.5, GPT 6 1 Soul, and Muse Wonderful, which if you've seen my video on that, I am quite enjoying. A handful of my friends at Google actually stuck their necks out a bit to get me early access to this model. I'm very thankful for them, and I sincerely hope they don't get in trouble for what I'm going to say. There are things that are good about this model, and there are things that are less good about this model. Normally, this is where I'd make a joke about how I'm not a paid shill and the company isn't paying me, but in this particular case, I can't because Google does pay me quite a bit because I'm a YouTuber, as you might know, watching this video likely on YouTube.
Watch from 0:00All of that said, they are not paying me for my coverage for anti-gravity and Gemini stuff. In fact, my previous Gemini videos almost got me demonetized on YouTube, so this comes at a cost to me, if anything, which is why I hope you can forgive us for a quick break for today's sponsor. I got an interesting question for you. You want people like me to use your product because if you do, you probably shouldn't do a normal sign-in flow because I'm not going to be clicking those buttons myself nowadays. I do most of my work through my agents now, which means any platform that my agents choose needs to be usable by the agent. In particular, it needs to be able to be signed up for by your agents. Today's sponsor is WorkOS, and that might confuse you because they are a platform for user signing in, right? Well, they are. They are the best way to have humans and businesses signing into your app. So, if you're trying to get enterprise customers, then you should probably be using WorkOS. That's why everyone from OpenAI and Anthropic to Fowl, Cursor, Bolt, Vanta, Carta, and more are all using them. But, I want to talk about AuthMD because this is a super, super cool standard. The goal of AuthMD is to make it easier for agents to sign up for your apps. It's a new standard that's been designed by WorkOS in partnership with companies like Fire Crawl and Cloudflare. And the results speak for themselves. There's already a ton of companies adopting these standards like Neon, Monday, Parallel, and more. What makes it so cool is that agents can now register on behalf of a user. So, if you're trying to get business customers and you want their agents signing up, this is a great place to start. Get Auth Right for all your users at swyd.link/workos.
Watch from 1:16Now that that's out of the way, let's dive into the newest model from Google, Gemini 4 Argon. Flash banging warning for their website because it is not dark mode for some reason. Argon is now globally available today for developers and enterprises via the API and studio and Gemini platform, Google AI Ultra subscribers and anti-gravity, and Google AI Ultra and Pro subs in the Gemini app and AI mode in search. This is the God, I did not realize how bad Google's like products line had gotten until this. It's a mess. Model excels at deep work in complex real-world tasks across domains, including coding, finance, legal, cybersecurity, and science. Its depth and precision across multi-step tasks lead Argon to outperform other frontier models on benchmarks across domains, ranking number one on the VALS index, which measures model performance across critical economic sectors like finance, coding, legal, and tax work. And yes, indeed, Gemini 4 Argon does appear to be one of the best values here. Somehow they have a cost for wonderful, though, which is strange cuz I was never given token pricing. Okay, they're saying it's $4.00 in and $20.00 out. That's wrong. We don't have a cost for wonderful, and it won't be the same price as standard wonder, obviously. So, uh that makes me trust this a lot less. I'll be real. Regardless, I have my own benches. I've I've run a lot of benches on this model. I'll start with some benches that aren't mine. With Slot Politics, I already shared the numbers for Gemini here. You can see that its cost is not actually that great when you factor in how many tokens it uses on something like the Artificial Analysis
Watch from 2:35Intelligence Index, but its score is putting it above Astra and also above 61 Soul, but meaningfully lower than Opus 5.5 on high as well as obviously X high and max. And it is also more expensive than Opus 5.5 on high despite getting a lower score. Of course, Artificial Analysis Intelligence Index is not a perfect measure of model capabilities. It is not the best index for things like code, but it is a good general idea of where intelligence sets. I chose to show you these numbers first because I'm scared people's brains fall out whenever I show them the model price. Because if you see this $2 per mil in and $10 per mil out, your immediate assumption's going to be this model is cheap when you compare it to things like the 5.25 rates for Opus 5.5. This seems like a really good deal. It's not. It's a combination of reasons. The first one is that this model is it's token hungry. It's not particularly efficient, which is part of what puts it in the higher price range like we just saw on Artificial Analysis. The other really, really egregious problem is that it sucks at hitting cash. I have had the worst rate of cash reads from any model I've used recently with this release. I'm talking like 70 to 80% cash hit versus other models where I see 99% plus. It means that that 95% discount on cash reads is not going to happen as often and you're going to actually eat a lot of input cost on this model. This might be a bug in Anti-Gravity either in how it's actually sending things or how it's reporting data to me. It is slightly possible this is a bug in the MCP implementation that we use in T3 Code because you will never
Watch from 4:05catch me dead in their awful CLI, but I am not particularly confident in any of this and I am not interested in doing more work to figure it out because Google should be the ones doing this, It's me. They have a section about using it to solve complex engineering tasks saying that uh anti-gravity has helped teams increase agentic code submissions by more than 35% while reducing code rollbacks by nearly a third. Those are not great numbers considering that what they were using before was also not particularly good. The more interesting thing and I won't air out who this was, but I was talking to a Google employee in not one of the ones that Google knows I talked to, so don't go after the people who normally work with me. The way they described Argon to me wasn't it's a revolution or this model has changed how we build. The way it was described to me is I find myself reaching for Opus 5 less. Not 5.5, Opus 5. The one that I notoriously panned after realizing I was wrong in my first video. It is what it is. Let's see what this can do at least according to Google. Apparently they use it to untangle browser feature code in Chrome. Piping multiple clean code changes at a time, Argon helped the team finish an architectural refactor across hundreds of files completing months of work in weeks. This feels like it was written in 2024. It optimized a production Python code base in just a few hours. The Here they have some fun demos of the cool 3D stuff they've been doing with it.
Watch from 5:35God, why is it making sounds? That's horrible. This isn't a generated video. This is generated 3D models that they then went through with some type of like software rotoscope type thing to show the models in video form. Very cool. Animations are still a bit cringe. This YouTube player embed is broken and the overlay won't disappear. Classic Google-isms. I I've joked for a bit now, you'll know when Google has a good model when their software finally starts working. They got a quote from someone at Deloitte. Early impressions from Gemini 4 are very strong. For long-running tasks across these workflows, it completed successfully 100% of the time in our e-vals, maintaining context and running through multiple tools across impressive spans. Instead of the reasoning delivers significantly higher, clearer, yada yada, the text clips. Google being Google. I shouldn't be as annoyed as I am by this completed successfully 100% of the time in our evals thing because this depends entirely on what's being evaluated. For example, I have an eval that Gemini completed 100% of the time. It's Skatebench. I still actually can't believe this. I think it's cool as hell. Gemini for Argon got a perfect score on my private version of my benchmark for seeing how well models name skateboard tricks. For reference, Opus 5.5 on Max gets a 79%. This bench has proven to be weirdly difficult for most models to do.
Watch from 6:59In particular, some of the tricks that are like switch and fakie stance cuz skateboarding is a whole grammar around the way you're standing on the board. Google has always had a weird lead here with models like 3.8 flash getting a 96% but seeing the first perfect score was kind of trippy for me. Especially when I looked into it more and saw just how many errors it was hitting when it ran because the API was quite unstable during my early testing. I don't know if that's improved since but I was getting pretty intense error rates and I had to overhaul a bunch of my benchmark just to get around the fact that the endpoints error as often as they did. It was also relatively impressive in terms of its token utilization in the bench like it got answers fast and didn't overthink which is rare for a Gemini model and nice to see. So, yeah, if your task is somehow reminiscent to naming skateboard tricks, there's a decent chance you're going to have a good experience with Argon both in its like speed, token efficiency, but also its accuracy in doing the thing. But again, when you say a model has 100% success in our evals, the eval and its relation to the work you're doing matters and I don't think a lot of people are making a living naming skateboard tricks. I wish they were. I wish I could have that be my job but uh sadly it is not. They also say that the model has advanced reasoning and multimodal capabilities while maintaining strict grounding, yada yada yada. It's good at looking at documents, PDFs, videos, audio, things like that. I have historically seen a meaningful lead in Gemini models' ability to like do things with vision. Like giving it a picture of a motherboard and asking it to label the parts and draw that in like Python on top of it. The Google models have historically been very impressive in this space. I did not benchmark any
Watch from 8:23of that because I was just using it for code. But if you have complex vision tasks, it is definitely worth running them against Gemini 4 just to see if you have a real benefit. You might be surprised. Do note though, as we start to complain more about Googleisms, that the current price of $2 per mil in and $10 per mil out is an introductory price. This is Google's new favorite thing to do when they drop a model. They charge a reasonable price up until a certain date, and then they double the price after. So this model's actually going to be $4 per mil in and $20 per mil out starting December 1st. It's almost like they don't want you to start building heavily on top of the model. It's It's rough. They've been doing this for most of their model drops recently and it makes pricing weird. I I don't think it's as bad as what Meta did by not putting out a price at all for wonderful and only having it in the sub, but I still hate this. Just give us a price that you can actually serve it at. Lower the price in the future. Don't raise it. The only lab that regularly raises prices for models is our friends over at Google. And then they tell us to get started. So I guess we should. First, we'll start with one of the things Google models have historically been pretty strong at. The thing is of course front end. I have used Google models for front end stuff pretty often when they are the new thing. Specifically when 3 Pro and 3 1 Pro dropped, I found a lot of good uses. I built a lot of the UI and UX around Lawn, my alternative to Frame.io for video stuff. It's what we actually use for all of our video management for my company. Every one of my YouTube videos gets uploaded on this platform and managed by my team on Lawn, and a lot of the UI and UX here was built by Gemini 3
Watch from 9:531 Pro. I obviously did a lot more work on top of that using the Opus models at the time, but I found that the back and forth between those models allowed me to make actually unique-ish homepages at the time. So, let's see how that capability has carried through. I think this is quite good. I don't necessarily think a homepage should have this much text on it, but it is very nice-looking. I would not have guessed that this was AI-generated if I just stumbled on a page like this like browsing the web. It also made a nice little interactive section here where you can do things that is actually functional and not cringe. Like, this is decent. Believe me, I would talk a lot of crap if I could. This is fine. Hop into the other designs. That one had a bit of like font popping, that might just be the way that WhichAI is set up. By the way, WhichAI created by our friend Dara, really cool kid. Check it out whichai.dev. Very useful to see how the front-end capabilities vary across different models. All of the versions I'm showing now are using the official Anthropic design skill. So, if you aren't getting good designs out of the model, definitely try turning that on and see if you have better results. If I turn it off, you get a lot more cringe, a lot more of this like unnecessary text everywhere, a lot of fonts that are kind of gross, just not the best quality. Like, all of them have that like awful top nav, like every single one of them does. God, this retro one is so cringe.
Watch from 11:24And this is without any skills on. So, if you are using this for design, highly recommend Anthropic design skill. I don't recommend the taste skill though. That gets it back to where it was with the weird cringe top navs and as well as like the really consistent gross layout. The design skill seems to be a good balance that actually forces the model to make things that look decent and unique. So, for front-end, seems pretty good. Is it state-of-the-art? I still think Fable 5.1 is the best I've seen. Opus has its moments, but Fable overall makes the coolest, most unique front-ends and like homepages, marketing sites, that type of thing. I still don't think any model is good at just like doing things right the first try, and I prefer ones where I can and back and forth. For example, with Slapolitics, I used OpenAI models for this entire site. I did everything with Astra on ultrafast. Not because ultrafast is magically better at design, all OpenAI models kind of suck at design, but since I was using ultrafast, I could tell it what I wanted and ask for different versions and go back and forth until I got it just how I wanted without having to wait constantly throughout. So, I think that speed is really important when you're iterating on real UX, not like a marketing site, but like the buttons people press on the website and how they interface with it. Google models are fast, but that doesn't necessarily mean they get work done quickly. Anecdotally, having tested this a bunch myself, it's not a super fast model. It might generate tokens quickly, but it doesn't solve problems fast.
Watch from 12:45Next, we should talk a bit about the unique capabilities of Google models. This is strange because every lab kind of has something like this. For example, with OpenAI models, you can generate images and it will use their API to call the image gen tools and models that they have, where with Claude models, you can't do that. You have to call some other external service. Grok models have the ability to do cool things with the Twitter API. Like if I want to get good data out of Twitter, I can't do it through the API, it's way too expensive. So, I'll just tell Codex or Claude to call Grok, and by calling Grok, it gets access to the XAI APIs. A lot of different models have these types of capabilities. If they have access to unique data, you can get it through that model. So, why am I bringing this up? Gemini models have some unique capabilities, many of which are very useful to me. Why are they useful to me? I'm a YouTuber. That means I have to do a lot of stuff on YouTube, and Google gives very generous access to the YouTube API and capabilities to the Gemini models. If you want to get a transcript from a YouTube video, the Gemini API is one of the best ways to do it right now. And we've been using the Google models heavily for all of those types of things throughout my businesses. So, I will be using this model a lot. Not because I necessarily want to or I'm choosing to cuz it's so uniquely intelligent and such a good value. Just because Google has unique capabilities that they're providing their models through their APIs, through their data centers, through the things that they uniquely have access to. That said, I've actually found the Gemini models not that good at most Google services. I have a much better experience dealing with my email inbox through tools like Grok and Chat GPT
Watch from 14:07than I do with anything Gemini. In fact, every time the Gemini button appears in some app, it it scares me and is often not functional or just outright cringe. I think if you're a YouTuber or you work with YouTubers, Gemini will be a very useful thing to have in your pipeline. If you're doing a lot of unique vision work, historically the Gemini models have been good at that as well, so worth benchmarking. I don't think there is as much here as there used to be though, cuz a lot of the things that Google's models were uniquely good at have been maintained and achieved by others externally. There is some secret sauce in this model. There's something that makes it different. You can see it in SkateBench. It's the only model to get a perfect score and the only models to break the 90 range are Gemini and somehow Muse wonderful. Everything else is in the 80s, 70s, even 50s and 60s. So, there might be more secret hidden capabilities I'm not seeing in the model. I've heard people say that they're really good for internationalization, localization, multilingual stuff. I haven't seen too much of this unique capability in my own testing. I just wanted to call it out because it's not that common nowadays. Usually, since all the models are just throw all the data in pre-training and then RL it on code, we don't get as much unique capability in things anymore, but Google still has some and I thought it was worth noting. Now, we get into the meat, the things that y'all are most hyped about, things like the real day-to-day code work, the things I hate about Gemini itself and the Gemini ecosystem, Opus, Soul, and Muse and how they compare, and of course, the thing
Watch from 15:39everyone is hyped about, weirdly so, Fish Slap, my game that I make with every model to see how good they are at 3D. So, uh without further ado, I think it's time for fish slop. And no, the video is not encoding incorrectly. It did actually screw up the top of the water that badly. The UI is uh it's Gemini, to say the least. It did make the whole game loop where there's five tanks to get through, which is solid. Submarine models, not particularly great. It's not as bad as things like Muse and Grok, but it's not great. Considering how crazy all the 3D demos I saw other people posting were, I expected a lot more. I gave it the exact same prompt, exact same setup, exact same tools, and everything as the others had. It does animation decently well. Like, the way the fish moves is reasonable. The eyes are pointed in the right direction, which might not sound like much, but just a few months ago that was a rare skill. But, there are some uniquely Google-esque things here. Let me show you one of the funnier examples. So, games have to have physics and mechanics. My game has a mechanic where the fish eat the food. There's a problem. The food can fall to the bottom of the tank, and when it does, it doesn't expire.
Watch from 17:10The fish can go to the bottom of the tank to eat it, but they can't figure out how. So, the fish will come over cuz they're hungry. They'll try to eat the food that's on the floor, so they can't, and then they'll just sit there staring at the food trying to get to it until they die. It wouldn't be too bad if I could just like feed them more, but the game has a mechanic where you can only put so much food in the tank at a time, and it forgot to make these expire. So, it will actually just sit here forever. I also just realized the sound is not routing, so let me fix that quick. There we go. Sorry for the lack of sound for the first bit. As I said, you weren't missing out on too much. It's not particularly great-sounding game. But, this version was bad enough and broken enough that I wanted to give it one more shot. So, I did. I told about some of the things that were broken, like the shimmering on the ceiling of the tank, the food mechanic being broken where the food wouldn't expire and they couldn't get at the bottom, and that I was disappointed in the graphics and I wanted it to improve. So, it ran for a while. It came back already looking a hell of a lot better. Didn't touch the model for the sub, sadly, but it did fix a lot of the things involving like the feeding of the fish. And I was able to actually like play this version for a bit. But then you start to notice some of the the Gemini-isms. And the best I can put it is like it doesn't have big model smell. Big model smell is really hard to describe.
Watch from 18:26It's a a type of thoughtfulness and generalization that models do when they have like a better understanding of the world that comes from the model being bigger. I talked about this a bit in terms of 3D in my previous video about Muse, where I talked about how the model's 3D capabilities don't reflect how smart or capable the model is. 3D is a thing that they train in separately, like they add it on. This model clearly got more time with that 3D stuff, but it also has some stupid mistakes that I haven't seen other models do. Monsters spawn every minute. When you finish the wave, they drop a power up, and the power up lasts for 30 seconds. Why is there a power up that helps with my attack? I can't shoot anything after I get it, and they expire before I can use it. So, one of the power ups is legitimately useless because there will never be a monster in the tank when you have it on because it will disappear before the monsters come in. It's just like the type of silly thing that a human would not have done, that a smart model also would not have done, that a model that's goal is to just write code that seems good without thinking about it would absolutely do. Imagine that you were a part-time contributor to a game like this, and there was three power-ups made, you might throw out the idea of a power-up that does more damage and not even think about it and then add it. A smart model should have a good enough understanding of the system to know that that is stupid. Gemini doesn't. I think now is a better time than ever to start talking about the way this model behaves in real-world tasks. In particular,
Watch from 19:52hallucination. The thing I just showed you wasn't hallucination. It's not like the model imagined this thing that wasn't the case. It saw a power-up system, it came up with an idea for the power-up system, and it put it in the game. It might be a stupid power-up, doesn't make sense, but it didn't hallucinate anything. According to Artificial Analysis, the model actually doesn't hallucinate much. On their AI Omniscience benchmark, which is their attempt to measure hallucination, Gemini 4 Argon had a 15% hallucination rate, the lowest of any model scoring over a 45 on the intelligence index. And the S and D, you can see here in the official charts, Gemini 4 Argon has an impressively low hallucination rate. But the AI Omniscience bench is not just hallucinations. It's both incorrect answers as well as partial answers and refusals. So, if it gets the questions correctly, it gets points. If it gets them incorrectly, it loses points. If it says, "I can't answer this," it's neutral. When you look at the overall score, it's no longer first place, it's now in fourth because it gets few enough things right that its refusal to get things wrong does not really matter as much. It's just like, "Sorry, can't help." more often than others. Cool. Nice to see Google making their models hallucinate less. Still not a particularly good score. Sadly, from my experience, this benchmark is not that realistic because I have seen this model just be wrong a lot. Here is where we get into the real code work and where my actual breakdown of the model begins. I started by throwing it at TS Rust, a
Watch from 21:24project I built that I talked about in other videos where I rewrote the entire chess compiler in Rust using Opus 5.5. I had a bunch of different models audit the code base and describe it, things that could be improved, what is it worth knowing in this code base. I would take all the claims it made, validate them, and then put them through a panel of judges to rate how good or bad the model did. GPT-61 Soul came out on the top making 38 claims that were all true. It had zero wrong claims, and the claims were informative, accurate, and useful. So, all the judges gave it a pretty good score, averaged an 8.84. Opus 5.5 slightly lower, 8.78. It took way longer, 49 minutes versus 28 minutes, and it cost $24 instead of $4, and it got one of the claims wrong. Still a very good score, just made some mistakes and took too long and was too expensive. 6 Astra also had no wrong claims, was way faster at 20 minutes, still a bit too expensive. 61 Soul on X High, things improved. Also, funny, uh 6 Astra on Max scored worse than 6 Astra on X High. Once again, as I've been saying for a while, you should probably avoid the max reasoning levels. They do more damage than good. You might notice something's missing, though. Where is Gemini? Well, we hadn't scrolled down far enough. Yeah. Gemini for Argon, in my audit the code base bench, scored a 4.38, less than half of Opus's score and 61 Soul's score. It also took 43 minutes long, almost double the time 61 Soul took, and more than double it than it took on X High. On the bright side, it
Watch from 22:55would have only cost $3, but on the less bright side, of its 36 claims, 24 were false. The majority of the things it claimed about the code base were wrong. Do you understand how egregious that is? Duh. It means that when this model's doing real work in a real code base, it's as likely, if not more likely, to be wrong than right. And this has lined up with my experience using it. I I hate to be that guy, especially cuz I've likely burned some credibility with a lot of y'all with my Muse Wonder video. I don't care. Muse Wonder is a unique balance here where it is getting similar scores to Fable, way faster, and gets things wrong, but eight out of 33 is comparable to Fable 5.1's 11 of 33, and a whole different world than 24 of 36. I'm going to be so real, and we'll you'll see what I mean as we go deeper. I don't think anybody who finds this model on the same frontier level capability as things like Opus 5.5, Fable 5.1, 6 Astra, and 6.1 Soul, or even Muse Wonder. I don't think anyone who finds these models comparable is actually working on that real of stuff. Show me demos if you disagree. I would like to be wrong here. I want to have more faith in Google as a competitor, but the numbers aren't going to get better as we keep going. The next round was one where I audited a real bug in T3 code related to how the bulk remote update workflows were functioning. Opus 5.5 got the best score with an 822. It had five claims that were wrong of its 46. It was $26 to run, and took 20 minutes, but these are like good, useful numbers. 6.1 Soul got a lower score. It
Watch from 24:26missed some of the high severity things Opus 5.5 found, but it didn't say anything incorrect. It took 10 minutes, and it cost a dollar. Insane. This is why I like 6.1 Soul so much as a review model. Muse Wonder scored quite a bit lower. I think it missed a handful of claims, and it still had the same roughly error rate as Opus did, but it only took 6 minutes, and most importantly, in my opinion, it was super readable. So again, I like Muse Wonder cuz it's a different flavor, and finds useful stuff. Gemini for Argon is another flavor. It's not just a Claude distill like a lot of the Open Weight Labs are doing. So like, if you talk to something like GLM 5.3 Flash, Kimmy K3, or any of those other models, they all talk and act relatively similarly. All four of these models behave entirely differently. Gemini is too dumb to justify the flavor change. Muse Wonder Full is probably too expensive to justify it. We don't know cuz again, we don't have the price. But at least it's not lying 14 out of 40 times. Yeah. Muse has taken whatever I was hoping Gemini would do. Hopefully you're starting to see why I was nicer to Muse than people expected. Testing both the Muse model and the Gemini model at the same time really deeply reminded me of how much I don't like working with Gemini. Sadly, it gets worse. I didn't do a fancy visual for this one. This is an audit of my cloud product Lakebed. 61 Soul got a nine out of 10. Sonic 55 got an 8 and 1/2 out of 10, mostly cuz Sonic 5 on Mac's will just burn and burn and burn. So, it finds everything. Muse took a pretty big hit here going from the 825 from Opus 55 down to 5.75. It's not as big of
Watch from 25:58a hit as Gemini gets, 2.25. These are scores out of 10. Muse gave its findings a three. Opus, Fable, and Astra gave its findings a two. Of its 78 findings, 56 were wrong. That is a 70% failure rate. This model is more likely to gaslight your agents than it is to give useful feedback in reviews. If you do decide to use this model for real work for some reason, for the love of God, please put the things it does through Soul or Opus or some model that doesn't have this egregious rate of lying. I can't fathom how any bench measured this model is not hallucinating a lot when I don't I've never seen it not hallucinate in my day-to-day real-world code work. In my final run where I had it go through the T3 code architecture, it got a 3.95 out of 10. And you can also see again Muse wonderful being not like as good if not better than things like Fable and Astra, but neck and neck with Astra while also being much more pleasant to work with is worth noting. Again, to compare to Muse, it got 29 true claims, eight partially true, four false. Argon got 31 true, 14 partially true, 47 false. Gemini for Argon had more false findings than Muse wonderful had true findings. What the It's not just like kind of bad or a little more likely to do something wrong. No, it's like an order of magnitude gap here. It is lying 10 times
Watch from 27:30more than any of the other models I'm benching other than Muse wonder the non-full version. That one's just garbage. Throw it out. Don't touch it. But like the only bad models in this bench were Muse wonder standard and Gemini 4 Argon and they're both garbage. As you saw in my Muse video, I'm a little bit more forgiving if the model's pleasant to interact with. No points here either, Google. Walls upon walls of text full of things that don't really matter that much. Not very pleasant to read. But if the code works, that's fine then, right? Well, I had it do some work around the SQL light connection layer in T3 code in hopes of stabilizing it for heavy users on platforms like Windows where we're having some connectivity issues. I already knew a lot of the fixes. I was just curious if Gemini could find them. So, there's lots of users reporting issues with instability in connections. A lot appear to be SQL light related. What should we prioritize to fix this issue? I even gave it a hint with async SQL light. This is not the right solution. The right solution is pooling and having read pools at a single right. It did a hunt. It did a lot of the classic Gemini-isms of just reading every single file synchronously. It just kind of went on and on with that. It does correctly identify the split read pool and a single right connection piece.
Watch from 29:03I tell it to make a PR for priority one, which is the first section it had in here. Uh 30 minutes later, way longer than it should have taken, I've been told by some people that apparently Gemini is way slower in T3 code. You might think that's our fault. It literally cannot be. We're using their ACP bindings that they provide on their servers that we embed in our app. We actually did a bunch of weird to make it work at all because Google's a special snowflake. We'll talk more about the Google special snowflake Regardless, 30 minutes is an unreasonable amount of time to make this PR which when we open, we will see is closed. Because this change should not have been 1,200 lines of code. This change should have been 30. At this point, I was done. Especially when I read the PR which by the way, we have skills in the repo for how to format the PR that are really simple. It's supposed to open with a simple two to three sentence description of the exact problem the user had that they're trying to fix. We don't get that. We get a summary. We split the SQLite read accesses and write accesses in this particular file in the orchestration projection and query services that read snapshots and standalone select queries. Is this a run-on sentence? It is. It's unreadable slop that didn't follow any of the repo rules. Just garbage. So, I was already pretty ready to close it. Then I saw how big it was. Then I saw my favorite thing. CI failed. It just didn't write the code right. The check and the lint both failed and it never bothered to fix it or take care of any of the high or
Watch from 30:22higher level comments that it got from our review bots. So, I gave up. I opened up Opus 5.5. I pasted the thread ID in the PR and said, "This is opened by Gemini for Argon. I think the PR is garbage and doesn't really fix the problems that it barely stated. Go through the T3 code thread, figure out where it went wrong, give me your honest thoughts on the work that Gemini did. You agree it's slop, close the PR." I agree it's slop and I closed it with a comment that gives the reasons. Where it went wrong. Turn one, the diagnosis was mostly correct, but it had no evidence. Gemini found the real mechanism, the synchronous database sync runs on the main thread. Yada yada yada. Turn two, it fixed the wrong half of its own diagnosis. The reader pool still opens database sync on the main thread. A slow read on a reader connection blocks heartbeats the same way it does on the writer. The only effect is less waiting on this one semaphore call and that matters only when a fiber yields inside a transaction. What it's saying is it didn't work. It also called out that it didn't check prior art. There was already two PRs that fixed these problems. You might remember the previous video with Muse where I pointed out how much I liked that Muse said, "I'm not going to write code. Here's a PR that fixes it. You should merge this instead." This model doesn't do that. In fact, this model will hallucinate its own incorrect solution instead. The correct answer was in the PR tab on our GitHub and not only did it not find that, it invented its own wrong answer. It also solved it with a regex to route requests. I wish I was kidding. It's so bad. It was so garbage.
Watch from 31:52According to Opus, Gemini can read code and explain how it works. It does not test its own theory. It never asked whether a reader pool on the main thread stops the event loop from blocking. It answered your loose question with a confident roadmap and a percentage that it invented. When I told it to do priority one, it built that literally and at maximum size without checking whether priority one was even correct. So, I told it to go make the changes and it was 21 lines of code. They are now merged. They'll be in the next nightly and people have a better experience with SQLite. Real quick crash out about the instability of this model and the anti-gravity stuff. If I get my Anthropic account banned, that sucks, but I can open a new one with a new email address and I don't lose too much. If my OpenAI account gets banned, slightly more annoying, but again, I open a new one, I'm good. I might sign in to both of those with my Google account, but if I'm banned, it's just my sign-in to Anthropic or OpenAI that's banned. If my Gemini account gets banned, I might lose my YouTube channel cuz my Gemini account is my Google account. So, I could lose my email, I could lose my billing system for my credit cards, I could lose my saved passwords in Chrome, I could lose my access to my YouTube channel, I will lose my access to everything else I sign in to with Google. There's a real risk. They have promised that they will only ban you from Gemini and anti-gravity. They won't ban you from Google anymore. But, I've seen people getting their Google accounts banned still, so it's a risk.
Watch from 33:19As such, integrating their official ECP into T3 code felt risky. We chose to do it because I was told it should be fine, but I can also tell they're not putting a whole lot of effort into this, which you can see in places like here, where the model just failed outright to connect to the gateway in the middle of a run after reading endless files for an hour and 2 minutes. And then it just failed and refused to continue, and I had to restart this entire thread with none of its existing history. Horrible. And then there's a bunch of silly little Googleisms, like they don't report the model and then let you pick effort in the ECP. All of the model slugs are paired with the effort levels. So, you have to pick for Argon high, medium, or low as a model. Every other lab gets these things right. ECPs are pretty simple standard to implement and integrate. Google is Google, and they got all of this wrong. It is embarrassing how miserable it was to integrate anti-gravity and how high a percentage of the bugs in T3 code are just for anti-gravity. It has consistently been a thorn in my side since we first integrated it, and even having Gemini 4 did not make it better. It was a relief to have a reasonably usable UI for a Gemini model for the first time ever. I honestly thought it would make me like it more. But, now that I see it next to models that are competent, like Muse, Opus, even like Sonnet, honestly, it just hurts more. And now we get to
Watch from 34:37the end, comparing it to the other models. This feels unfair. Feels unfair to the other models. This is just garbage. Don't ask me about how good the usage is on the $200 plan or anything. I don't care. I'm not going to push the limits enough to know. It's probably practically unlimited because you're not going to do a whole lot with it cuz you're going to get frustrated and give up. It's just It's just bad. I'm going to put my money where my mouth is here and show you just how committed I am to this. I don't trust any Gemini models nor Muse Wonder. To be clear, Muse Wonder full is fine, but the standard Muse Wonder as well as Gemini 3, 4, etc. They do not contribute useful code the T3 code code base. I would like to find some way to auto close all pull requests that are filed using models like Gemini 4 Argon or the normal Muse Wonder when they are filed to T3 code. I am not joking. I will find a way to get this in later this week. I have wasted more time in tokens trying to vet code that was written by Gemini 4 than I would have spent writing the code with a smarter model in the first place. Gemini is not a useful model for anything in the code world other than maybe as like a first or second pass on a nice looking marketing page. As soon as I'm done with this video, I'm just not going to use Gemini for anything other than like fetching data for YouTube. This is one of the worst releases of the year. I don't care what the benchmarks say. I have been testing it constantly in real world work and I have not merged a thing
Watch from 36:08that this model has shipped. This feels like a throwback to 2024 with some of the hallucinations I see this model making. I was played up to this point because I'm thankful for the Google employees I've talked to. They have generally been one of the more receptive labs to my feedback and I tried to share this feedback with them early. I've also heard rumors that there is a better snapshot like carbon or something that's coming soon. I don't know how much better it's going to be, but it needs to be a lot because this model is garbage and I have no intention to ever use it again other than for fetching data on YouTube. Tell me what y'all think. Am I overreacting massively or is this model actually just garbage? They told me it was good at code and it's not good at code, so I'm I'm pretty done with it. Maybe you can find some cool uses. Let me know in the comments if you do. Until next time, I'm sticking with Muse. Okay, seriously, I'm probably just going to stick with Opus 5.5. You get the idea though. I just I'm out. Bye.
Watch from 37:39Full transcript
- 0:00
It finally happened. Gemini 4 Argon is here. Before we get started, I got a quick question. How long do you think it has been since we got a normal-sized large Gemini model release, like the Pro line? How many months do you think it has been? The gap between a model release like 5 6 Soul and 6 Astra was 1 to 2 months. The gap between 3 1 Pro and Gemini 4 Argon is 8 months. It's been
- 0:22
since February. We have not gotten anything other than flash models from Google. And now, after teasing Gemini 4 Argon over a week ago, they have finally put it out for all of us to use. With a drop like this, there's a lot we have to cover, from how it's priced to how good it is at front end, to all the 3D capabilities, the weird things that make Gemini models Gemini models, the unique capabilities in those models, and of
- 0:43
course, the comparison with recent releases like Opus 5.5, GPT 6 1 Soul, and Muse Wonderful, which if you've seen my video on that, I am quite enjoying. A handful of my friends at Google actually stuck their necks out a bit to get me early access to this model. I'm very thankful for them, and I sincerely hope they don't get in trouble for what I'm going to say. There are things that are good about this model, and there are things that are less good about this
- 1:05
model. Normally, this is where I'd make a joke about how I'm not a paid shill and the company isn't paying me, but in this particular case, I can't because Google does pay me quite a bit because I'm a YouTuber, as you might know, watching this video likely on YouTube. All of that said, they are not paying me for my coverage for anti-gravity and Gemini stuff. In fact, my previous Gemini videos almost got me demonetized on YouTube, so this comes at a cost to
- 1:25
me, if anything, which is why I hope you can forgive us for a quick break for today's sponsor. I got an interesting question for you. You want people like me to use your product because if you do, you probably shouldn't do a normal sign-in flow because I'm not going to be clicking those buttons myself nowadays. I do most of my work through my agents now, which means any platform that my agents choose needs to be usable by the agent. In particular, it needs to be
- 1:45
able to be signed up for by your agents. Today's sponsor is WorkOS, and that might confuse you because they are a platform for user signing in, right? Well, they are. They are the best way to have humans and businesses signing into your app. So, if you're trying to get enterprise customers, then you should probably be using WorkOS. That's why everyone from OpenAI and Anthropic to Fowl, Cursor, Bolt, Vanta, Carta, and more are all using them. But, I want to
- 2:06
talk about AuthMD because this is a super, super cool standard. The goal of AuthMD is to make it easier for agents to sign up for your apps. It's a new standard that's been designed by WorkOS in partnership with companies like Fire Crawl and Cloudflare. And the results speak for themselves. There's already a ton of companies adopting these standards like Neon, Monday, Parallel, and more. What makes it so cool is that agents can now register on behalf of a
- 2:26
user. So, if you're trying to get business customers and you want their agents signing up, this is a great place to start. Get Auth Right for all your users at swyd.link/workos. Now that that's out of the way, let's dive into the newest model from Google, Gemini 4 Argon. Flash banging warning for their website because it is not dark mode for some reason. Argon is now globally available today for developers and enterprises via the API and studio
- 2:48
and Gemini platform, Google AI Ultra subscribers and anti-gravity, and Google AI Ultra and Pro subs in the Gemini app and AI mode in search. This is the God, I did not realize how bad Google's like products line had gotten until this. It's a mess. Model excels at deep work in complex real-world tasks across domains, including coding, finance,
- 3:08
legal, cybersecurity, and science. Its depth and precision across multi-step tasks lead Argon to outperform other frontier models on benchmarks across domains, ranking number one on the VALS index, which measures model performance across critical economic sectors like finance, coding, legal, and tax work. And yes, indeed, Gemini 4 Argon does appear to be one of the best values
- 3:29
here. Somehow they have a cost for wonderful, though, which is strange cuz I was never given token pricing. Okay, they're saying it's $4.00 in and $20.00 out. That's wrong. We don't have a cost for wonderful, and it won't be the same price as standard wonder, obviously. So, uh that makes me trust this a lot less. I'll be real. Regardless, I have my own
- 3:49
benches. I've I've run a lot of benches on this model. I'll start with some benches that aren't mine. With Slot Politics, I already shared the numbers for Gemini here. You can see that its cost is not actually that great when you factor in how many tokens it uses on something like the Artificial Analysis Intelligence Index, but its score is putting it above Astra and also above 61
- 4:10
Soul, but meaningfully lower than Opus 5.5 on high as well as obviously X high and max. And it is also more expensive than Opus 5.5 on high despite getting a lower score. Of course, Artificial Analysis Intelligence Index is not a perfect measure of model capabilities. It is not the best index for things like code, but it is a good general idea of
- 4:30
where intelligence sets. I chose to show you these numbers first because I'm scared people's brains fall out whenever I show them the model price. Because if you see this $2 per mil in and $10 per mil out, your immediate assumption's going to be this model is cheap when you compare it to things like the 5.25 rates for Opus 5.5. This seems like a really good deal. It's not. It's a combination
- 4:51
of reasons. The first one is that this model is it's token hungry. It's not particularly efficient, which is part of what puts it in the higher price range like we just saw on Artificial Analysis. The other really, really egregious problem is that it sucks at hitting cash. I have had the worst rate of cash reads from any model I've used recently with this release. I'm talking like 70
- 5:13
to 80% cash hit versus other models where I see 99% plus. It means that that 95% discount on cash reads is not going to happen as often and you're going to actually eat a lot of input cost on this model. This might be a bug in Anti-Gravity either in how it's actually sending things or how it's reporting data to me. It is slightly possible this is a bug in the MCP implementation that
- 5:33
we use in T3 Code because you will never catch me dead in their awful CLI, but I am not particularly confident in any of this and I am not interested in doing more work to figure it out because Google should be the ones doing this, It's me. They have a section about using it to solve complex engineering tasks saying that uh anti-gravity has helped teams increase agentic code submissions by more than 35% while reducing code
- 5:56
rollbacks by nearly a third. Those are not great numbers considering that what they were using before was also not particularly good. The more interesting thing and I won't air out who this was, but I was talking to a Google employee in not one of the ones that Google knows I talked to, so don't go after the people who normally work with me. The way they described Argon to
- 6:16
me wasn't it's a revolution or this model has changed how we build. The way it was described to me is I find myself reaching for Opus 5 less. Not 5.5, Opus 5. The one that I notoriously panned after realizing I was wrong in my first video. It is what it is. Let's see what this can do at least according to Google. Apparently they use it to untangle browser feature code in Chrome.
- 6:38
Piping multiple clean code changes at a time, Argon helped the team finish an architectural refactor across hundreds of files completing months of work in weeks. This feels like it was written in 2024. It optimized a production Python code base in just a few hours. The Here they have some fun demos of the cool 3D stuff they've been doing with
- 6:58
it. God, why is it making sounds? That's horrible. This isn't a generated video. This is generated 3D models that they then went through with some type of like software rotoscope type thing to show the models in video form. Very cool. Animations are still a bit cringe. This YouTube player embed is broken and
- 7:19
the overlay won't disappear. Classic Google-isms. I I've joked for a bit now, you'll know when Google has a good model when their software finally starts working. They got a quote from someone at Deloitte. Early impressions from Gemini 4 are very strong. For long-running tasks across these workflows, it completed successfully 100% of the time in our e-vals, maintaining context and running through
- 7:40
multiple tools across impressive spans. Instead of the reasoning delivers significantly higher, clearer, yada yada, the text clips. Google being Google. I shouldn't be as annoyed as I am by this completed successfully 100% of the time in our evals thing because this depends entirely on what's being evaluated. For example, I have an eval that Gemini
- 8:02
completed 100% of the time. It's Skatebench. I still actually can't believe this. I think it's cool as hell. Gemini for Argon got a perfect score on my private version of my benchmark for seeing how well models name skateboard tricks. For reference, Opus 5.5 on Max gets a 79%. This bench has proven to be weirdly difficult for most models to do.
- 8:23
In particular, some of the tricks that are like switch and fakie stance cuz skateboarding is a whole grammar around the way you're standing on the board. Google has always had a weird lead here with models like 3.8 flash getting a 96% but seeing the first perfect score was kind of trippy for me. Especially when I looked into it more and saw just how many errors it was hitting when it ran because the API was quite unstable
- 8:44
during my early testing. I don't know if that's improved since but I was getting pretty intense error rates and I had to overhaul a bunch of my benchmark just to get around the fact that the endpoints error as often as they did. It was also relatively impressive in terms of its token utilization in the bench like it got answers fast and didn't overthink which is rare for a Gemini model and nice to see. So, yeah, if your task is
- 9:05
somehow reminiscent to naming skateboard tricks, there's a decent chance you're going to have a good experience with Argon both in its like speed, token efficiency, but also its accuracy in doing the thing. But again, when you say a model has 100% success in our evals, the eval and its relation to the work you're doing matters and I don't think a lot of people are making a living naming skateboard tricks. I wish they were. I
- 9:25
wish I could have that be my job but uh sadly it is not. They also say that the model has advanced reasoning and multimodal capabilities while maintaining strict grounding, yada yada yada. It's good at looking at documents, PDFs, videos, audio, things like that. I have historically seen a meaningful lead in Gemini models' ability to like do things with vision. Like giving it a picture of a motherboard and asking it
- 9:46
to label the parts and draw that in like Python on top of it. The Google models have historically been very impressive in this space. I did not benchmark any of that because I was just using it for code. But if you have complex vision tasks, it is definitely worth running them against Gemini 4 just to see if you have a real benefit. You might be surprised. Do note though, as we start to complain more about Googleisms, that
- 10:06
the current price of $2 per mil in and $10 per mil out is an introductory price. This is Google's new favorite thing to do when they drop a model. They charge a reasonable price up until a certain date, and then they double the price after. So this model's actually going to be $4 per mil in and $20 per mil out starting December 1st. It's almost like they don't want you to start
- 10:27
building heavily on top of the model. It's It's rough. They've been doing this for most of their model drops recently and it makes pricing weird. I I don't think it's as bad as what Meta did by not putting out a price at all for wonderful and only having it in the sub, but I still hate this. Just give us a price that you can actually serve it at. Lower the price in the future. Don't raise it. The only lab that regularly
- 10:47
raises prices for models is our friends over at Google. And then they tell us to get started. So I guess we should. First, we'll start with one of the things Google models have historically been pretty strong at. The thing is of course front end. I have used Google models for front end stuff pretty often when they are the new thing. Specifically when 3 Pro and 3 1 Pro dropped, I found a lot of good uses. I built a lot of the UI and UX around
- 11:09
Lawn, my alternative to Frame.io for video stuff. It's what we actually use for all of our video management for my company. Every one of my YouTube videos gets uploaded on this platform and managed by my team on Lawn, and a lot of the UI and UX here was built by Gemini 3 1 Pro. I obviously did a lot more work on top of that using the Opus models at the time, but I found that the back and
- 11:30
forth between those models allowed me to make actually unique-ish homepages at the time. So, let's see how that capability has carried through. I think this is quite good. I don't necessarily think a homepage should have this much text on it, but it is very nice-looking. I would not have guessed that this was AI-generated if I just stumbled on a page like this
- 11:51
like browsing the web. It also made a nice little interactive section here where you can do things that is actually functional and not cringe. Like, this is decent. Believe me, I would talk a lot of crap if I could. This is fine. Hop into the other designs. That one had a bit of like font popping, that might just be the way that WhichAI is set up. By the way, WhichAI created by our friend Dara, really cool kid. Check it
- 12:12
out whichai.dev. Very useful to see how the front-end capabilities vary across different models. All of the versions I'm showing now are using the official Anthropic design skill. So, if you aren't getting good designs out of the model, definitely try turning that on and see if you have better results. If I turn it off, you get a lot more cringe, a lot more of this like unnecessary text
- 12:33
everywhere, a lot of fonts that are kind of gross, just not the best quality. Like, all of them have that like awful top nav, like every single one of them does. God, this retro one is so cringe. And this is without any skills on. So, if you are using this for design, highly recommend Anthropic design skill. I don't recommend the taste skill though.
- 12:53
That gets it back to where it was with the weird cringe top navs and as well as like the really consistent gross layout. The design skill seems to be a good balance that actually forces the model to make things that look decent and unique. So, for front-end, seems pretty good. Is it state-of-the-art? I still think Fable 5.1 is the best I've seen. Opus has its moments, but Fable overall
- 13:13
makes the coolest, most unique front-ends and like homepages, marketing sites, that type of thing. I still don't think any model is good at just like doing things right the first try, and I prefer ones where I can and back and forth. For example, with Slapolitics, I used OpenAI models for this entire site. I did everything with Astra on ultrafast. Not because
- 13:34
ultrafast is magically better at design, all OpenAI models kind of suck at design, but since I was using ultrafast, I could tell it what I wanted and ask for different versions and go back and forth until I got it just how I wanted without having to wait constantly throughout. So, I think that speed is really important when you're iterating on real UX, not like a marketing site, but like the buttons people press on the website and how they interface with it.
- 13:55
Google models are fast, but that doesn't necessarily mean they get work done quickly. Anecdotally, having tested this a bunch myself, it's not a super fast model. It might generate tokens quickly, but it doesn't solve problems fast. Next, we should talk a bit about the unique capabilities of Google models. This is strange because every lab kind of has something like this. For example,
- 14:17
with OpenAI models, you can generate images and it will use their API to call the image gen tools and models that they have, where with Claude models, you can't do that. You have to call some other external service. Grok models have the ability to do cool things with the Twitter API. Like if I want to get good data out of Twitter, I can't do it through the API, it's way too expensive. So, I'll just tell Codex or Claude to
- 14:37
call Grok, and by calling Grok, it gets access to the XAI APIs. A lot of different models have these types of capabilities. If they have access to unique data, you can get it through that model. So, why am I bringing this up? Gemini models have some unique capabilities, many of which are very useful to me. Why are they useful to me? I'm a YouTuber. That means I have to do a lot of stuff on YouTube, and Google
- 14:57
gives very generous access to the YouTube API and capabilities to the Gemini models. If you want to get a transcript from a YouTube video, the Gemini API is one of the best ways to do it right now. And we've been using the Google models heavily for all of those types of things throughout my businesses. So, I will be using this model a lot. Not because I necessarily want to or I'm choosing to cuz it's so uniquely intelligent and such a good
- 15:18
value. Just because Google has unique capabilities that they're providing their models through their APIs, through their data centers, through the things that they uniquely have access to. That said, I've actually found the Gemini models not that good at most Google services. I have a much better experience dealing with my email inbox through tools like Grok and Chat GPT
- 15:39
than I do with anything Gemini. In fact, every time the Gemini button appears in some app, it it scares me and is often not functional or just outright cringe. I think if you're a YouTuber or you work with YouTubers, Gemini will be a very useful thing to have in your pipeline. If you're doing a lot of unique vision work, historically the Gemini models have been good at that as well, so worth
- 16:01
benchmarking. I don't think there is as much here as there used to be though, cuz a lot of the things that Google's models were uniquely good at have been maintained and achieved by others externally. There is some secret sauce in this model. There's something that makes it different. You can see it in SkateBench. It's the only model to get a perfect score and the only models to
- 16:21
break the 90 range are Gemini and somehow Muse wonderful. Everything else is in the 80s, 70s, even 50s and 60s. So, there might be more secret hidden capabilities I'm not seeing in the model. I've heard people say that they're really good for internationalization, localization, multilingual stuff. I haven't seen too much of this unique capability in my own testing. I just wanted to call it out
- 16:42
because it's not that common nowadays. Usually, since all the models are just throw all the data in pre-training and then RL it on code, we don't get as much unique capability in things anymore, but Google still has some and I thought it was worth noting. Now, we get into the meat, the things that y'all are most hyped about, things like the real day-to-day code work, the things I hate
- 17:04
about Gemini itself and the Gemini ecosystem, Opus, Soul, and Muse and how they compare, and of course, the thing everyone is hyped about, weirdly so, Fish Slap, my game that I make with every model to see how good they are at 3D. So, uh without further ado, I think it's time for fish slop. And no, the video is not encoding incorrectly. It
- 17:24
did actually screw up the top of the water that badly. The UI is uh it's Gemini, to say the least. It did make the whole game loop where there's five tanks to get through, which is solid. Submarine models, not particularly great. It's not as bad as things like Muse and Grok, but it's not great. Considering how crazy all the
- 17:45
3D demos I saw other people posting were, I expected a lot more. I gave it the exact same prompt, exact same setup, exact same tools, and everything as the others had. It does animation decently well. Like, the way the fish moves is reasonable. The eyes are pointed in the right direction, which might not sound like much, but just a few months ago that was a rare skill. But, there are some uniquely
- 18:07
Google-esque things here. Let me show you one of the funnier examples. So, games have to have physics and mechanics. My game has a mechanic where the fish eat the food. There's a problem. The food can fall to the bottom of the tank, and when it does, it doesn't expire. The fish can go to the bottom of the tank to eat it, but they can't figure
- 18:29
out how. So, the fish will come over cuz they're hungry. They'll try to eat the food that's on the floor, so they can't, and then they'll just sit there staring at the food trying to get to it until they die. It wouldn't be too bad if I could just like feed them more, but the game has a mechanic where you can only put so much food in the tank at a time, and it forgot to make these expire.
- 18:49
So, it will actually just sit here forever. I also just realized the sound is not routing, so let me fix that quick. There we go. Sorry for the lack of sound for the first bit. As I said, you weren't missing out on too much. It's not particularly great-sounding game. But, this version was bad enough and broken enough that I wanted to give it one more shot.
- 19:09
So, I did. I told about some of the things that were broken, like the shimmering on the ceiling of the tank, the food mechanic being broken where the food wouldn't expire and they couldn't get at the bottom, and that I was disappointed in the graphics and I wanted it to improve. So, it ran for a while. It came back already looking a hell of a lot better. Didn't touch the model for the sub,
- 19:31
sadly, but it did fix a lot of the things involving like the feeding of the fish. And I was able to actually like play this version for a bit. But then you start to notice some of the the Gemini-isms. And the best I can put it is like it doesn't have big model smell. Big model smell is really hard to describe.
- 19:52
It's a a type of thoughtfulness and generalization that models do when they have like a better understanding of the world that comes from the model being bigger. I talked about this a bit in terms of 3D in my previous video about Muse, where I talked about how the model's 3D capabilities don't reflect how smart or capable the model is. 3D is
- 20:12
a thing that they train in separately, like they add it on. This model clearly got more time with that 3D stuff, but it also has some stupid mistakes that I haven't seen other models do. Monsters spawn every minute. When you finish the wave, they drop a power up, and the power up lasts for 30 seconds. Why is there a power up
- 20:33
that helps with my attack? I can't shoot anything after I get it, and they expire before I can use it. So, one of the power ups is legitimately useless because there will never be a monster in the tank when you have it on because it will disappear before the monsters come in. It's just like the type of silly thing that a human would not have done, that a smart model also would not have
- 20:55
done, that a model that's goal is to just write code that seems good without thinking about it would absolutely do. Imagine that you were a part-time contributor to a game like this, and there was three power-ups made, you might throw out the idea of a power-up that does more damage and not even think about it and then add it. A smart model should have a good enough understanding of the system to know that that is stupid. Gemini doesn't. I think now is a
- 21:17
better time than ever to start talking about the way this model behaves in real-world tasks. In particular, hallucination. The thing I just showed you wasn't hallucination. It's not like the model imagined this thing that wasn't the case. It saw a power-up system, it came up with an idea for the power-up system, and it put it in the game. It might be a stupid power-up, doesn't make sense, but it didn't hallucinate anything. According to
- 21:39
Artificial Analysis, the model actually doesn't hallucinate much. On their AI Omniscience benchmark, which is their attempt to measure hallucination, Gemini 4 Argon had a 15% hallucination rate, the lowest of any model scoring over a 45 on the intelligence index. And the S and D, you can see here in the official charts, Gemini 4 Argon has an impressively low hallucination rate. But
- 22:01
the AI Omniscience bench is not just hallucinations. It's both incorrect answers as well as partial answers and refusals. So, if it gets the questions correctly, it gets points. If it gets them incorrectly, it loses points. If it says, "I can't answer this," it's neutral. When you look at the overall score, it's no longer first place, it's now in fourth because it gets few enough
- 22:22
things right that its refusal to get things wrong does not really matter as much. It's just like, "Sorry, can't help." more often than others. Cool. Nice to see Google making their models hallucinate less. Still not a particularly good score. Sadly, from my experience, this benchmark is not that realistic because I have seen this model
- 22:43
just be wrong a lot. Here is where we get into the real code work and where my actual breakdown of the model begins. I started by throwing it at TS Rust, a project I built that I talked about in other videos where I rewrote the entire chess compiler in Rust using Opus 5.5. I had a bunch of different models audit the code base and describe it, things
- 23:05
that could be improved, what is it worth knowing in this code base. I would take all the claims it made, validate them, and then put them through a panel of judges to rate how good or bad the model did. GPT-61 Soul came out on the top making 38 claims that were all true. It had zero wrong claims, and the claims were informative, accurate, and useful. So, all the judges gave it a pretty good score, averaged an 8.84. Opus 5.5
- 23:27
slightly lower, 8.78. It took way longer, 49 minutes versus 28 minutes, and it cost $24 instead of $4, and it got one of the claims wrong. Still a very good score, just made some mistakes and took too long and was too expensive. 6 Astra also had no wrong claims, was way faster at 20 minutes, still a bit too expensive. 61 Soul on X High, things
- 23:48
improved. Also, funny, uh 6 Astra on Max scored worse than 6 Astra on X High. Once again, as I've been saying for a while, you should probably avoid the max reasoning levels. They do more damage than good. You might notice something's missing, though. Where is Gemini? Well, we hadn't scrolled down far enough. Yeah. Gemini for Argon, in my audit the
- 24:10
code base bench, scored a 4.38, less than half of Opus's score and 61 Soul's score. It also took 43 minutes long, almost double the time 61 Soul took, and more than double it than it took on X High. On the bright side, it would have only cost $3, but on the less bright side, of its 36 claims, 24 were
- 24:31
false. The majority of the things it claimed about the code base were wrong. Do you understand how egregious that is? Duh. It means that when this model's doing real work in a real code base, it's as likely, if not more likely, to be wrong than right. And this has lined up with my experience using it. I I hate to be that guy, especially cuz I've
- 24:52
likely burned some credibility with a lot of y'all with my Muse Wonder video. I don't care. Muse Wonder is a unique balance here where it is getting similar scores to Fable, way faster, and gets things wrong, but eight out of 33 is comparable to Fable 5.1's 11 of 33, and a whole different world than 24 of 36. I'm going to be so real, and we'll
- 25:13
you'll see what I mean as we go deeper. I don't think anybody who finds this model on the same frontier level capability as things like Opus 5.5, Fable 5.1, 6 Astra, and 6.1 Soul, or even Muse Wonder. I don't think anyone who finds these models comparable is actually working on that real of stuff. Show me demos if you disagree. I would like to be wrong here. I want to have more faith in Google as a competitor,
- 25:35
but the numbers aren't going to get better as we keep going. The next round was one where I audited a real bug in T3 code related to how the bulk remote update workflows were functioning. Opus 5.5 got the best score with an 822. It had five claims that were wrong of its 46. It was $26 to run, and took 20 minutes, but these are like good, useful numbers. 6.1 Soul got a lower score. It
- 25:58
missed some of the high severity things Opus 5.5 found, but it didn't say anything incorrect. It took 10 minutes, and it cost a dollar. Insane. This is why I like 6.1 Soul so much as a review model. Muse Wonder scored quite a bit lower. I think it missed a handful of claims, and it still had the same roughly error rate as Opus did, but it only took 6 minutes, and most importantly, in my opinion, it was super readable. So again, I like Muse Wonder
- 26:20
cuz it's a different flavor, and finds useful stuff. Gemini for Argon is another flavor. It's not just a Claude distill like a lot of the Open Weight Labs are doing. So like, if you talk to something like GLM 5.3 Flash, Kimmy K3, or any of those other models, they all talk and act relatively similarly. All four of these models behave entirely differently. Gemini is too dumb to
- 26:40
justify the flavor change. Muse Wonder Full is probably too expensive to justify it. We don't know cuz again, we don't have the price. But at least it's not lying 14 out of 40 times. Yeah. Muse has taken whatever I was hoping Gemini would do. Hopefully you're starting to see why I was nicer to Muse than people expected. Testing both the Muse model and the
- 27:01
Gemini model at the same time really deeply reminded me of how much I don't like working with Gemini. Sadly, it gets worse. I didn't do a fancy visual for this one. This is an audit of my cloud product Lakebed. 61 Soul got a nine out of 10. Sonic 55 got an 8 and 1/2 out of 10, mostly cuz Sonic 5 on Mac's will just burn and burn and burn. So,
- 27:22
it finds everything. Muse took a pretty big hit here going from the 825 from Opus 55 down to 5.75. It's not as big of a hit as Gemini gets, 2.25. These are scores out of 10. Muse gave its findings a three. Opus, Fable, and Astra gave its findings a two. Of its 78 findings, 56 were wrong. That is a 70%
- 27:45
failure rate. This model is more likely to gaslight your agents than it is to give useful feedback in reviews. If you do decide to use this model for real work for some reason, for the love of God, please put the things it does through Soul or Opus or some model that doesn't have this egregious
- 28:06
rate of lying. I can't fathom how any bench measured this model is not hallucinating a lot when I don't I've never seen it not hallucinate in my day-to-day real-world code work. In my final run where I had it go through the T3 code architecture, it got a 3.95 out of 10. And you can also see again Muse wonderful being not like as good if not
- 28:27
better than things like Fable and Astra, but neck and neck with Astra while also being much more pleasant to work with is worth noting. Again, to compare to Muse, it got 29 true claims, eight partially true, four false. Argon got 31 true, 14 partially true, 47 false. Gemini for Argon had more false findings than Muse
- 28:50
wonderful had true findings. What the It's not just like kind of bad or a little more likely to do something wrong. No, it's like an order of magnitude gap here. It is lying 10 times more than any of the other models I'm benching other than Muse wonder the non-full version. That one's just garbage. Throw it out. Don't touch it. But like the only bad models in this
- 29:11
bench were Muse wonder standard and Gemini 4 Argon and they're both garbage. As you saw in my Muse video, I'm a little bit more forgiving if the model's pleasant to interact with. No points here either, Google. Walls upon walls of text full of things that don't really matter that much. Not very pleasant to read. But if the code
- 29:33
works, that's fine then, right? Well, I had it do some work around the SQL light connection layer in T3 code in hopes of stabilizing it for heavy users on platforms like Windows where we're having some connectivity issues. I already knew a lot of the fixes. I was just curious if Gemini could find them. So, there's lots of users reporting issues with instability in connections. A lot appear to be SQL light related.
- 29:53
What should we prioritize to fix this issue? I even gave it a hint with async SQL light. This is not the right solution. The right solution is pooling and having read pools at a single right. It did a hunt. It did a lot of the classic Gemini-isms of just reading every single file synchronously. It just kind of went on and on with that.
- 30:17
It does correctly identify the split read pool and a single right connection piece. I tell it to make a PR for priority one, which is the first section it had in here. Uh 30 minutes later, way longer than it should have taken, I've been told by some people that apparently Gemini is way slower in T3 code. You might think
- 30:37
that's our fault. It literally cannot be. We're using their ACP bindings that they provide on their servers that we embed in our app. We actually did a bunch of weird to make it work at all because Google's a special snowflake. We'll talk more about the Google special snowflake Regardless, 30 minutes is an unreasonable amount of time to make this PR which when we open, we will see
- 30:58
is closed. Because this change should not have been 1,200 lines of code. This change should have been 30. At this point, I was done. Especially when I read the PR which by the way, we have skills in the repo for how to format the PR that are really simple. It's supposed to open with a simple two to three sentence description of the exact problem the user had that they're trying to fix. We
- 31:19
don't get that. We get a summary. We split the SQLite read accesses and write accesses in this particular file in the orchestration projection and query services that read snapshots and standalone select queries. Is this a run-on sentence? It is. It's unreadable slop that didn't follow any of the repo rules. Just garbage. So, I was already pretty ready to close it. Then I saw how big it was. Then I saw my favorite
- 31:41
thing. CI failed. It just didn't write the code right. The check and the lint both failed and it never bothered to fix it or take care of any of the high or higher level comments that it got from our review bots. So, I gave up. I opened up Opus 5.5. I pasted the thread ID in
- 32:02
the PR and said, "This is opened by Gemini for Argon. I think the PR is garbage and doesn't really fix the problems that it barely stated. Go through the T3 code thread, figure out where it went wrong, give me your honest thoughts on the work that Gemini did. You agree it's slop, close the PR." I agree it's slop and I closed it with a comment that gives the reasons. Where it went wrong. Turn one, the diagnosis was mostly correct, but it had no evidence. Gemini found the real mechanism, the
- 32:23
synchronous database sync runs on the main thread. Yada yada yada. Turn two, it fixed the wrong half of its own diagnosis. The reader pool still opens database sync on the main thread. A slow read on a reader connection blocks heartbeats the same way it does on the writer. The only effect is less waiting on this one semaphore call and that matters only when a fiber yields inside a transaction.
- 32:44
What it's saying is it didn't work. It also called out that it didn't check prior art. There was already two PRs that fixed these problems. You might remember the previous video with Muse where I pointed out how much I liked that Muse said, "I'm not going to write code. Here's a PR that fixes it. You should merge this instead." This model doesn't do that. In fact, this model will hallucinate its own incorrect solution instead. The correct answer was
- 33:05
in the PR tab on our GitHub and not only did it not find that, it invented its own wrong answer. It also solved it with a regex to route requests. I wish I was kidding. It's so bad. It was so garbage. According to Opus, Gemini can read code and explain how it works. It does not test its own theory. It never asked whether a reader pool on the main thread
- 33:27
stops the event loop from blocking. It answered your loose question with a confident roadmap and a percentage that it invented. When I told it to do priority one, it built that literally and at maximum size without checking whether priority one was even correct. So, I told it to go make the changes and it was 21 lines of code. They are now merged. They'll be in the next nightly and people have a better
- 33:48
experience with SQLite. Real quick crash out about the instability of this model and the anti-gravity stuff. If I get my Anthropic account banned, that sucks, but I can open a new one with a new email address and I don't lose too much. If my OpenAI account gets banned, slightly more annoying, but again, I open a new one, I'm good. I might sign in to both of those with my Google account, but if I'm banned, it's just my
- 34:08
sign-in to Anthropic or OpenAI that's banned. If my Gemini account gets banned, I might lose my YouTube channel cuz my Gemini account is my Google account. So, I could lose my email, I could lose my billing system for my credit cards, I could lose my saved passwords in Chrome, I could lose my access to my YouTube channel, I will lose my access to everything else I sign in to with Google. There's a real risk. They have promised that they will only
- 34:29
ban you from Gemini and anti-gravity. They won't ban you from Google anymore. But, I've seen people getting their Google accounts banned still, so it's a risk. As such, integrating their official ECP into T3 code felt risky. We chose to do it because I was told it should be fine, but I can also tell they're not putting a whole lot of effort into this, which you can see in places like here, where
- 34:50
the model just failed outright to connect to the gateway in the middle of a run after reading endless files for an hour and 2 minutes. And then it just failed and refused to continue, and I had to restart this entire thread with none of its existing history. Horrible. And then there's a bunch of silly little Googleisms, like
- 35:11
they don't report the model and then let you pick effort in the ECP. All of the model slugs are paired with the effort levels. So, you have to pick for Argon high, medium, or low as a model. Every other lab gets these things right. ECPs are pretty simple standard to implement
- 35:32
and integrate. Google is Google, and they got all of this wrong. It is embarrassing how miserable it was to integrate anti-gravity and how high a percentage of the bugs in T3 code are just for anti-gravity. It has consistently been a thorn in my side since we first integrated it, and even having Gemini 4 did not make it better.
- 35:52
It was a relief to have a reasonably usable UI for a Gemini model for the first time ever. I honestly thought it would make me like it more. But, now that I see it next to models that are competent, like Muse, Opus, even like Sonnet, honestly, it just hurts more. And now we get to the end, comparing it to the other models. This feels unfair.
- 36:12
Feels unfair to the other models. This is just garbage. Don't ask me about how good the usage is on the $200 plan or anything. I don't care. I'm not going to push the limits enough to know. It's probably practically unlimited because you're not going to do a whole lot with it cuz you're going to get frustrated and give up. It's just It's just bad. I'm going to put my money where my mouth is here and show you just how committed I am to this. I don't trust any Gemini models nor Muse Wonder.
- 36:38
To be clear, Muse Wonder full is fine, but the standard Muse Wonder as well as Gemini 3, 4, etc. They do not contribute useful code the T3 code code base. I would like to find some way to auto close all pull requests that are filed using models like Gemini 4 Argon or the
- 37:00
normal Muse Wonder when they are filed to T3 code. I am not joking. I will find a way to get this in later this week. I have wasted more time in tokens trying to vet code that was written by Gemini 4 than I would have spent writing the code with a smarter model in the first place. Gemini is not a useful model for anything in the code
- 37:20
world other than maybe as like a first or second pass on a nice looking marketing page. As soon as I'm done with this video, I'm just not going to use Gemini for anything other than like fetching data for YouTube. This is one of the worst releases of the year. I don't care what the benchmarks say. I have been testing it constantly in real world work and I have not merged a thing that this model has shipped. This feels
- 37:41
like a throwback to 2024 with some of the hallucinations I see this model making. I was played up to this point because I'm thankful for the Google employees I've talked to. They have generally been one of the more receptive labs to my feedback and I tried to share this feedback with them early. I've also heard rumors that there is a better snapshot like carbon or something that's coming soon. I don't know how much better it's going to be, but it needs to
- 38:02
be a lot because this model is garbage and I have no intention to ever use it again other than for fetching data on YouTube. Tell me what y'all think. Am I overreacting massively or is this model actually just garbage? They told me it was good at code and it's not good at code, so I'm I'm pretty done with it. Maybe you can find some cool uses. Let me know in the comments if you do. Until next time, I'm sticking with Muse.
- 38:24
Okay, seriously, I'm probably just going to stick with Opus 5.5. You get the idea though. I just I'm out. Bye.
Description
Argon might be Google'ss Best Model Yet with Gemini 4, but is it 8 months to late?
SponsorThank you to WorkOS for Sponsoring! Check them out here:
Want to sponsor a video? Learn more here: https://soydev.link/sponsor-me
Check out my Twitch, Twitter, Discord more at https://t3.gg
S/O @Ph4seon3 for the awesome edit 🙏
#google #ai #programming
