Anthropic Rebuilt Git (yes really)
TTTheo - t3.gg@t3dotgg · 572K subscribers
- Views
- 60.1K
- 60,079
- Outlier
- ×0.4
- Median 167K
- Length
- 48:05
- Published
- Oct 16
- Friday
- Comments
- 156
- 1.1K likes
Script
How it's written
- Words
- 9,828
- Words a minute
- 204
- Reading time
- 42 min
- Segments
- 33
The hook
0:00–0:25For as long as I've been writing code, I've been using Git and GitHub. Funny enough, I actually started using Git before I wrote code cuz I used it to manage plugins for my Minecraft servers way way back when I was in like early high school. Git is core to the way that I build and work with software. But I have been starting to notice rough edges. A lot of the way that Git was built was around code being expensive and humans needing to review the code, approve the code, and control where this code goes because losing code or merging
Structure
For as long as I've been writing code, I've been using Git and GitHub. Funny enough, I actually started using Git before I wrote code cuz I used it to manage plugins for my Minecraft servers way way back when I was in like early high school. Git is core to the way that I build and work with software. But I have been starting to notice rough edges. A lot of the way that Git was built was around code being expensive and humans needing to review the code, approve the code, and control where this code goes because losing code or merging the wrong code was a very expensive and painful thing. The reality of how Git works has influenced the whole industry from how we work with our team to how products like GitHub are built to the way that we think about things that should be basic like secrets management. Shout out to my viral clip saying that we should be able to commit our environment variables. Seriously though, Git is not the right primitive for the way that things are moving right now. I'll be real with y'all. A lot of my problems with Git are coming from GitHub because it's just a slow and miserable platform. External estimates are measuring as much as 7% downtime across different things within GitHub. I can't tell you how many times we've had releases blocked because actions were failing or I haven't been able to merge an important change because the page wasn't loading. The fact that GitHub's uptime looks more like a Christmas tree than a green bar like it's supposed to should say everything you need to know.
Watch from 0:00But what if I told you GitHub wasn't the only problem here? that git itself is not the right solution for working heavily in parallel across a lot of people and even more agents. I've been thinking about this a lot because back in December I explored JJ and I loved it and ever since then I've been thinking about what would it look like to build something new for shipping really really fast with agents. It turns out I'm not the only one who's been thinking about this. Anthropic just put up an article about their new in-house source control system. Back in May, they were merging 2500 changes per day, but now in September, they're doing more than 30,000 changes merged a day. They are now bottlenecked by developer infrastructure that was built for humans and not for agents. And they've decided to build something of their own. It seems kind of crazy that agents are now smart enough to do things like this, but they really are. And I love this idea of challenging the fundamentals of how you're building to make the right solution for yourself. I have a feeling this one's going to be a really fun deep dive, especially now that I'm ripping things out like TypeScript for my own vibecoded alternatives. That said, I don't think everything should be ripped out for your own vibecoded alternatives. There's a lot of places where it makes sense to use a vendored product. Like for example, today's sponsor. It kind of feels like AI has solved most of modern webdev, but there are some things it hasn't solved. In particular, mobile. As good as it is at starting up an app and making small changes here and there, I find myself just fighting the model constantly when I'm trying to actually
Watch from 1:15get my apps ready to ship, fix real bugs, deal with Apple reviews, and all these other pieces. As I ran into more and more walls as we tried shipping the apps we were building, I hit up my friend Jamon at Infinite Red, and he has made life so much easier. If you're not familiar with these guys, they're industryleading experts in all things mobile, especially React Native and Expo, but also Swift and Cotlin as well. There's a reason that everyone from Domino's to Microsoft has called these guys in. They really know how to ship. I'm gonna be real with you guys. The role of a consultancy has fundamentally changed as a result of AI. You're not just hiring these guys to come in and do grunt code work anymore. You have agents for that. What you bring them in for is the expertise. Making sure you're not making simple mistakes. Getting your pipeline set up so you can actually build and ship the apps. Making it possible for your agents to actually verify their changes and know things are working. And just making sure everything start to finish is ready to go. In a world where we're all shipping faster, these guys can come in and fix anything from your team to your app. They can build a whole new app from scratch. They can take over development, or they can just give you the pipelines you need to do it yourselves. If you want to ship better apps sooner with less bugs, look no further than/infinite red. Man, I'm actually really excited to dive into this article. As soon as I saw it drop, I wanted to read it, but I decided to put it off for this filming. So, you're getting my organic take on this. If you're wondering why the site looks gross, it's because when I leave it on the default light mode that the blog is in, everyone yells at me. So, I'm using the dark reader extension to make it not just bright white. And this is the poop green brown color it chose.
Watch from 2:46Redesigning source control at Anthropic. As our merges per day grew more than 10fold, source control became our biggest development bottleneck. Here we share how we migrated to an in-house version control system in 5 months. I won't lie, the five months feels like one of those clawed timeline hallucinations when you ask it how long will this take and it's like oh two to three days and then it finishes the work in an hour. Regardless, this is impressive. Replacing Git at a company as big as Anthropic sounds insane. Last I checked, they had around 2,000 employees. So yeah, at Anthropic, coding is no longer the bottleneck to building systems or doing research. Claude is able to autonomously iterate on research loops, microoptimize complex programs and build a new services from the ground up at a speed and scale we've never seen before. The bottlenecks now all of the developer info that was built for human scale source control builds CI things like that. In May of 2026, Anthropic averaged around 2500 code changes merged a day. As of September, it's over 30,000 a day. We don't expect this rate of growth to slow anytime soon. The latent demand for merging code is practically infinite and every bottleneck we remove immediately leads to further increasing the amount of code that we produce. The way I've framed this in the past is derisking the merge button, making it so by the time a person's looking at merge, it's pretty likely the code is safe to merge. But also the other side where if you merge something you shouldn't have, the blast radius and the damage that it can do is reduced massively by systems to make it easy to test the code once it goes out with a limited set of users and
Watch from 4:10revert the things if they do actually end up breaking. And I've seen the difference when we introduce these types of things. We just ship faster. For example, I stayed up a bit too late last night because I was annoyed with some niche performance issues when you had a really big database in T3 code. So, I decided to fix as much of it as I possibly could in one sitting, as well as clearing out some old issues in PRs that were annoying me, as well as taking over the PRs that weren't quite implemented how I wanted. And between the hours of like midnight and I think around 3:00 or 4:00 a.m., I merged from here through to the top of this page to here, another full page of just Theo merges. You'll see t3.gg on the left for I think every single one of these. Some of them are multiple people, but they are me. We're still going all the way up. Okay, these were two I merged from others to here. That is three full pages of GitHub merge history that I did in 4 hours. I know it was four hours because I did it between nightly releases. We do every six hours because I wanted to have as much in as possible for the nightly. So before I went to bed, I could download it, test it, make sure all was good, fix anything that was broken, and revoke the build if it was bad, and then go to bed smiling, knowing that performance would be way better for everybody on the nightly when I woke up. The fact that I could confidently merge 50 plus PRs in a sitting, it just feels insane. Like
Watch from 5:40imagine going even a year back and telling yourself that you were merging 50 plus PRs a day. Hard to fathom. And if Anthropic was just saying these things and I hadn't experienced it myself, I'd be calling right now. But this is reality. If you're not moving this fast yet, you are either in like a place where you absolutely can't because people will die or you just don't realize how powerful these things are and you haven't taken the time to set up the tooling to make the stuff less risky. So let's dive in a bit here. This is interesting. They have blue and orange. Blue is pull requests. Orange is antique change requests. That must be their new thing. It lines up with the way they name Antique seems like their system for all of this. If you think this is anthropic being uniquely weird and silly here, there's a long history of companies reinventing source control for their specific needs. For example, Meta built a crazy system around Mercurial that I think is actually a hard fork of Mercurial. They built all of that to make their stacked diff system as strong as possible for when people were working on large arcs and approving pieces of it to merge it all in at once. Graphite is a whole company that was built on top of GitHub to bring the way meta worked with source control to people who were on GitHub because it was so far ahead of how we were building if we you were shipping fast. At my company, Ping Labs, we were heavy users of graphite. Now they're part of cursor and they're building an alternative to GitHub into cursor with origin. The point I'm trying to make here is that this isn't as unprecedented as it might seem. And many companies have found meaningfully better ways to work with and manage their source outside of Git. And Jold here, who used
Watch from 7:12to work at Meta and also works at Graphite and now cursor pointed out that a lot of the reason Meta built their systems and themselves instead of using off-the-shelf things is because their massive checkouts just didn't work well with Git. I vaguely remember the history here of Meta trying to convince the teams working on Git to make it work better for giant monor repos instead of doing separate repos for everything. The git maintainers didn't want to budge. So, they ended up going their own way. And that's fine. You shouldn't have to force the industry standard into your workflow if it doesn't fit. And it is nice seeing more companies challenge this cuz Git is not perfect and it certainly is not perfect for agents. So, let's learn more about what Anthropic has built. In order to support their insane growth, they built Scion, an in-house version control system, and they fully migrated their monor repo from Git in five months. Ready for a really silly fun fact? Both Anthropic and OpenAI are built in monor repos. So everything they're working on is different projects and folders inside of this one giant repo. If you've ever been curious why the sidebar in the chat GPT and Codex app is so bad, it's because of this. The vast majority of employees at OpenAI have one project, OpenAI, and they're still using github.com, by the way. Pretty much all of OpenAI's code lives in OpenAI/OPAI, which 404 is if you're not an employee. So for those saying I'm secretly employed, here's your proof. That makes a lot of the way they think and work and build strange to people who have more than one project.
Watch from 8:44So yeah, just a silly fun fact I wanted to include. Back to what Anthropic was saying here. Not only did they build Scion, which is their in-house version control system, they also built and shipped a code review surface called Antique alongside code search, a rich source control AM system, novel ways of managing and landing large amounts of reviewed code, and more, all in that same 5month period. In this post, they're going to focus on how they built Scion specifically. It makes for an interesting case study. Claude's ability to write code was both what put them in the situation in the first place and how they're able to build a highquality scalable system to deal with it. growing our monor repo. Before we explore what we built, we should acknowledge how unusual our workload is. Unsurprisingly, Anthropic uses Claude to write software far more than almost any other company. You might be surprised, guys. Okay, seriously though, they're uh they are definitely doing it more than we are, although we are doing quite a lot. But since they're using it so heavily, they're often the first to see where tools built for human scale start to strain. Source control is just one example of this. While we don't expect many organizations to have these problems in the very near term, we suspect it won't be too long before others who use AI code tools experience these same pressure points, which is why we want to share what we've built and what we learned. It's a decentralized system. Every clone is a complete self-contained repository that can compute diffs, read history, and blame, and check out any revision while fully offline. For the vast majority of repos, that is the right design, and it is a large part of why Git became the standard. But as their monor repo has grown, it increasingly doesn't fit that design space. The history of their main
Watch from 10:05branch alone is almost 100 gigs and a compressed working copy is over 25 gigs. They perform on the order of 25 million code checkouts per day for CI before counting developer environments, agent sandboxes, and research jobs. Fetched natively, that would be 600 pabytes of Git traffic a day. Git offers tools for working with a slice of a repository like shallow clones, sparse checkouts, and promiser remotes. Beyond these, there are standard workarounds for large monor repos like internal git mirrors and repository snapshots like seeds that are kept in blob storage. They've tried and used all of them at their peak. They operated several hundred internal git mirrors to deal with the load. This bought them time, but it became harder to operate as merge rates were increasing. Ces are falling further from the tip of Maine between refreshes and it took more and more care to keep operations that depend on history like diffs and blame working on these partial clones. Taking a step back, we noticed that like most companies, we were using Git as a centralized system with one authoritative copy, but nearly all of our clients were agents in jobs that are online by definition. This is a very interesting and worthwhile call out. You can't use claude if you're offline because claude is being served from a giant data center that has GPUs you don't. And unless you are plugged into those directly and somehow the internet for that data centers offline, you're online. Building everything around this idea that everyone needs everything synced just doesn't make that much sense. The properties that make Git excellent for distributed offline work
Watch from 11:36were actively undermining their ability to scale as the repo grew. So they asked themselves if they designed source control with the fact that agents are always online in mind from the start what would it look like how would it be different with agentic coding many common assumptions about how software is developed are evolving they embrace these changes when they built scion which allowed them to take a fundamentally different approach than git on several core principles source control for huge monor repos does have prior art like google's piper and meta's sapling I touching on meta stuff before so they were able to build scion with their hard one lessons in mind It is worth noting just how many people at Anthropic are ex Meta and ex Google and they were often working on these types of things at those companies like I know Boris was deep in developer tooling at Meta and now he is making cloud code. The first core change they made is again this idea that clients are always online since nearly all of the clients are agents in jobs. They designed Scion client and protocols around a server that's always reachable and they have clients fetch data when they need it. also clearly not using GitHub because even when you're online, you can't assume GitHub's online. The next point is this idea of client server asymmetry. In Git, every client is capable of being a full-fledged server. This is one of those things that makes Git Git. It is truly decentralized. Anything can be a Git server. But in Scion, the server answers questions about file history and blame, creates commits, performs merges and rebases, and so on. The client maintains a bounded local object cache.
Watch from 13:06It lays out the working copy files on disk. It monitors for file system changes and it gives users ergonomic ways to work with that server. It can compute very little on its own, but in Exchange, it requires way less data locally. Next, I talk about how the source control service can power many novel application types. With git, a CI job or tool that needs to ask a question about the repo, like who last changed the line or how two commits are related, it would have to maintain the whole local copy of that repo or hope that some vendor specific API is able to provide the answer. With Scion, all the source control queries that the client uses are also exposed via API. So in many cases, those jobs don't need the local repository at all. Many internal tools like their automated bisect and revert pipelines wouldn't be practical at the scale they're operating at without this cheap programmatic access to the source. They also call out that everything must be horizontally scalable. At their peak, they ran several hundred git mirrors which significantly strained capacity and no longer scaled once the repo outgrew the mirrors discs or memory limits. You know it's bad when your git mirrors are running out of disc space. And to be fair, discs did get a lot more expensive. I still can't believe that hard drives increased in price as much as they did. Thanks, Anthropic. In Scion, the repo's data is distributed and every component is independently scalable. There is no materialized copy of the repo anywhere, even on the server. There is no equivalent of pack files, just gets compressed bundles of objects, so there's no repo maintenance needed either. And then a very important piece, commits must be cheap, incremental, and lock free. Lock free is
Watch from 14:32particularly important. One person committing shouldn't block somebody else. Or more realistically, 500 agents committing shouldn't block another 500 agents. In Git, you commit locally and then you push in order to share your work. In Scion, every commit, rebase, or merge happens on the server. So, there's nothing to push. But each of these operations needs to be fast. The cost of creating a commit and updating its downstream data structures has to scale with the size of the change, not the overall size of the repo, or the server won't be able to keep up. The development life cycle in Scion discourages refs pointers to commits such as branches and tags except for the main ref or release branches. The reason for that is because global state that every server has to agree on is always going to cause scale issues. Globals are bad when you're operating this fast. Always. And then the most important piece, the working copy is optional and it is sparse by design. Because the repo is so large and they're checking out code tens of millions of times a day for CI, they want to avoid checkouts wherever they can. With Scion, a CI job that needs the diff between a commit and its merge base, for example, can ask the server just for that. The same goes for writes. Many write workloads can skip the checkout and function entirely over API. Scion's protocol is centered around sparse checkouts, which only contain the directories the client asked for. And there's also virtualized checkouts, which materialize files on demand. So when the job needs the file on the disk, the client only pays for what it's actually reading. I was playing with some of these ideas real late last year and early this year, trying to make it easier to manage state of like a codebase in a way where you don't have to have it all on your computer, but your agents can still access it. It's
Watch from 16:04dope to see them doing the real version of this cuz mine was a chaotic experiment. So, how does it actually work? Now that we've seen the principles, we can hopefully understand how it all comes together. These insights can apply to other source control systems or more broadly to other stages of the software development life cycle. Oh boy, we're now in the data structure section. This is where we're going to get into the fun details. The unit of work is a change set. I I see the meta inspiration here. And change sets are entries and write ahead logs. A change set stores most of what you'd expect from a commit in git. Where a git commit stores a pointer to the tree, a change set stores a list of the files that changed relative to its first parent. But where a git commit stores a pointer to the tree, a change set stores a list of the files that changed relative to the first parent along with content address pointers to the compressed file contents in blob storage. Change sets are created server side and are available everywhere instantly. Fascinating. I knew they were doing blob storage for a lot of this, but kind of crazy to see the like direct call out that the source code does not exist in the system and when you request the source it has to go fetch that from blob store and it's effectively just a database pointing at those blobs. Seems like all the metadata is in Spanner. Makes sense. Again, very Google inspired. Very interesting that they have separate author and committer fields. makes sense because the author is the person who could who triggered it and the committer is whatever agent did it or where it happened if you have like different systems that are doing these things. Very interesting. And other usual fields like author date, committer date, message, the parent, which we talked about before, that is the thing that this is in reference to. And then
Watch from 17:36the file changes which specifies the file that changed and then has pointers to the blobs in their blob storage. The structure lets them accept change sets with very high throughput because there's no pack files they have to deal with. No locking or crossover coordination and no refs to move around. What this is trying to say is you just have to append. You only are adding to the data set. You don't have to go through the whole history before appending it. You don't have to move pointers from here to there. You don't have to restructure the tree. You don't have to sync all of the data back and forth to make sure it's all in the right place. You just put up the change set and store the blobs in blob storage and you're done. Oh, is it not even storing the diffs for the file change? It's just storing the new state of the file. That would make sense. So, it's saying like these files changed. Here is where they point. But this thing in the storage is the full content of that file, not a diff. Because if it was a diff, recalculating would be hell. Change sets are recording only what changed. So on their own, they can't quickly answer questions like who edited a line or what commit affected a file. To answer those, Scion builds additional data structures from the change sets, including dedicated structures for file blame and path history, as well as skip lists for the commit graph. They call everything that isn't directly part of the change set derived data. And they compute it all through one generic mechanism. Every type of derived data is defined recursively. So its value for a change set is computed from its values for the change sets parents. The mechanism has three benefits. First, they can easily add new types of derived data when new
Watch from 19:06data structures would make some operations faster. Second, for each type, they can choose whether to compute it eagerly or defer the work, so they can avoid materializing data they don't need. And third, because the client never sees the internals of the structures, they can optimize them without changing the client. They've added new types of derived data many times as the system has grown, such as new indices for performing copy detection or path-based file search. makes a ton of sense to try and dumb this down. We're deep in the weeds here, which I I love. A big part of what makes Git so powerful is that you have all of the data when you clone the repo. So, if you want to find something like when this file changed, all of the things this person did, stuff like that, you can go through all the data yourself. The problem with this is if you want to add something new to Git or make some type of action faster, you're kind of screwed because you'd have to upend everybody's git. But if people don't have to clone the whole thing and they're just hitting an API for this, you can choose what you do and don't expose. And since the server here is full of cache layers, overprovisioned CPU, and so many other things, they look for the things clients want to request to get more information. They built some way to compute that that's efficient without having to change the data structure and now the clients can do the thing and the clients aren't going to cheat and do something weird which I've seen a lot. When you give Claude access to the internals of something, it will invent some crazy novel and also crazy slow and CPU inensive solutions to get what it's looking for. This hard API
Watch from 20:37boundary prevents that which is almost certainly a huge benefit. So how did they scale this? starts with a small fleet of Scion servers providing source control APIs for any operation a client needs like getting change sets, computing diffs, finding merge bases, and fetching blobs. Computing diffs is a fun thing to realize here because again it's storing the file content at the end of a change. So when you are asking what changed between these two like change sets, you don't just look at the diff because you don't have a diff. The diff is computed when you say here are two change sets. change between them and it does that by grabbing the blobs for all the files that are affected and seeing how they differ between those states. All source control data is content addressed. So a piece of data never changes once it's stored. This is also important. The ability to override history in git isn't too bad because whenever you force push, you're changing all the commit hashes. But if you had the ability to edit the contents of a given hash, that would fundamentally break git at its core. Immutability is always more scalable than mutable stuff pretty much 100% of the time. So building the whole system around these change sets being fixed and permanent and all of the flexible stuff being calculations done on top. This is touching my functional programming heart deep at its core. And since they have these immutable content addressed change sets, all servers can cach them without checking with the others. As soon as it's sent up to one server, it knows that change set's unique. it can just blindly trust that. As a result of this,
Watch from 22:07the servers are horizontally scalable and they require very little coordination. They're essentially large caches that can also compute derived data on the fly and write the results back to Spanner. Scion's many consumers no longer need to maintain local repos. Fetching and maintaining the repo used to be a significant maintenance burden for the CI jobs and tools that needed to ask source control questions, especially around commit, ancestry, and blame. Early in the migration, before they even had their own client, repo format and protocols, Scion was only used to answer these types of metadata queries. Fetching file contents and materializing working copies was far more data inensive operations. So they needed to design something of their own there. Scion's SCN client deviates from the Gits fetch protocols in several important ways and as a result it lends itself to highly scalable architectures that require significantly less CPU than git servers do. Science negotiation doesn't tell the server what the client already has. Instead, it operates exclusively on trees and blobs that the client wants. Starting from a root tree, the client checks its local store for missing trees and requests them recursively. Then it repeats the process for the blobs. So, it starts by finding the change sets that it needs to do the thing it wants to do. And once it has those, it goes and collects all of the blobs to have the files it needs. And then it can actually start working. Every request is for a specific object that never changes. So, any cache can answer it. And the architecture closely resembles traditional tiered caches. The process is also efficient and resumable because there's no single multi- gigabyte pack file response. Sparse and even virtualized checkouts don't need
Watch from 23:38special support because the client simply doesn't ask for trees it doesn't want. This paragraph is using a bunch of terms that they did not define and they are using in unique ways. So I will do my best to translate from claudeisms to English here. The clients don't connect directly to the Scion servers because those are managing all of the change sets. They're talking to a Damon over a socket and that Damon stores objects for the whole node. So, it's effectively keeping a cache of like what was the most recent version of a given file in a given file path. This allows it to respond to requests for files without having to go down the whole tree to compute the current state of things. I really hate how they use node here. Each of these dammons that does this caching gets periodically reset to be on the latest snapshot so that it isn't like recalculating all the things that have changed in the last 3 days. They update it constantly. And let's be real with the way people are spinning up these agents, spinning up CI, all of these things. They're probably starting that work from latest. So it makes a lot of sense to get the cache up to date that way and then just resolve things that have changed since the cache server stream objects directly from memory instead of like storing it in the hard drive which makes it way faster to get that data when you're on a client on the other side. And it stores this in the same format that it would have been on disk so that it doesn't have to change the shape or format of the data when it comes out of memory and gets onto a drive after going over the network. When caches are missing objects, they ask the authoritative scion servers, which is the layer we were talking about before,
Watch from 25:09which will read from Spanner or cloud storage as needed in order to get the updated data and the updated files in that cache layer. The Scion Damon, Scion cache, and Scion server all respond to identical RPCs for this protocol. So, the client doesn't need to know the cache topology. Any tier could theoretically talk to any other, but they strictly enforce the tiering to isolate failure domains. They want to avoid a situation where a failure in a regional scion cache overwhelms the upstream servers and brings down the entire system. So again they have that cluster of scion servers that are the like proper management of the real world state of things the authoritative layer that is the layer that stores in spanner for the change sets and the cloud storage for the actual contents of the files. There is then a cache layer that is what gets hit often for your actual work that is regional. So you don't have to like fetch across the world to hit this server. This is what will hit the scion server to get a reasonable state in the cache. Developer machines directly hit the scion cache when you're like pulling down the code to do work. And if the scan cache is missing things that you need, it will go pull it from the server. But if you're working on CI, even this isn't the right abstraction. And that's where this Damon comes in that we were just talking about that gets periodic snapshots so that it's ready to spin up CI jobs as quickly as possible. Each tier of this system maintains a cache hit rate of over 99% and it doesn't require longived state.
Watch from 26:40So it's easily scaled up if needed. Their infrastructure footprint is 200x smaller than their previous git mirroring infra in terms of CPU and memory requirements. That's insane. again like and this is like the crazy idea that has been like hurting me a bit. The realization that like all software was built to be somewhat generalizable. Now we can build software for our specific needs. And like it's a silly thing to pivot to, but hear me out. One of my weird YOLO projects right now is called Lust Lust Lang to be specific. on trying to take a subset of Rust that gets you most of the benefits of Rust, but make it so that it could be compiled and built a hundred times faster because the build times for TS Rust, which is my rewrite of the Tyer compiler in Rust, are too slow. This is an insane project that I didn't think would make any progress, but it's already able to compile over 40% of the files from TS Rust, and I am expecting by the end of the week it might actually work. Would this work for most Rust projects? no. Absolutely not. That is not what this is for. This is a bespoke experiment to see if I take a given codebase that I'm not happy with compile times on. Can I invent a new language to make it compile faster? It's like insane that this is even a thing you can think, much less do. But I'm using a model that's a lot dumber than what they have access to. I'm using the distilled version of two model generations away from what they have internally. They're rocking Fable Ultra
Watch from 28:02Max 7 internally and I'm here with Opus 55 more than happy to do these types of things. The fact that we can build this way now is just crazy. And while this system might not be 200 times smaller and faster for some cases, it is for theirs. And I would expect more and more companies that have these types of big painful scaling things to go reinvent the painful pieces that are slowing them down. And uh fun fact, SJ, the borrow checker is not why build times suck in Rust. It's a bunch of weird LLVM Anyways, CI and research jobs are able to check out code two orders of magnitude faster, seconds rather than minutes, which frees up a sizable amount of capacity across the fleet. CI alone had a 25% CPU capacity when neither clients nor caches need to perform repo maintenance, which simplifies the life cycle management of our jobs. Out of curiosity, I'm trying to get the hard numbers for how much time T3 code builds and CI have spent dealing with GitHub and Git like clones. It's going to be a number that hurts me a lot. We'll come back and check on that in a bit, but I have a feeling I'm going to feel a lot of pain as soon as I see that. So, what did they learn building Scion? They started building it around the same time as Mythos Preview and Claude Tag becoming available internally. Both of these releases simultaneously amplified the need for Scion and meaningfully accelerated how quickly they'd be able to build it. Here's a particularly striking example. Earlier in the year, they had largely handwritten the first version of the Git to Scion ingestion pipeline as a prototype. They thought it was good until they got their hands on
Watch from 29:33early internal snapshots of Mythos preview. They asked it to rewrite the whole thing and with no further instructions, it produced a version that consumed 20x less memory and was four times faster while preserving all of the implicit invariations around ingestion, ordering, and data structures. This is the history of changes to Scion and Antique. I will say worth noting here, they were bragging about 30k changes a day, but they've only made 12,250 changes to this architecture total. So their daily velocity is more than double the amount of like requests, commits, change sets, whatever you want to call them, that have landed for the whole of their source control. Kind of crazy. But they learned building and operating Scion offers some interesting lessons for what building agent scale software will look like. Rapid development with small teams is going to become more common. Can confirm. A team of seven people built both Scion and Antique, their code review tool, which they could have easily written a whole separate blog post about considering how awful code review is. I feel it. I'm thankful that we have so many wonderful sponsors working on that now. They also follow with not to mention half a dozen related systems that they fully migrated all of in 5 months. They did this all while the volume of code merged per day was increasing 10x. In the past, developing and deploying source control systems like this would have taken years of iterative migrations with many more engineers. We think the sort of fast, nimble migration will become more common in the future. Yeah, it just will.
Watch from 31:04Switching cost has gone down more than almost anything. Historically, people adopt a technology, a solution, a vendor, and then they kind of are stuck there because switching off is more expensive than whatever cost they're incurring staying. Now, it's easier to write your own alternative and move to it. It's insane. I personally threw together a solution to clone and mirror and backfill all of our analytics data that I have on one particular analytics provider so that I have it directly on my own managed click house on top of R2. It ended up being more than 10x cheaper and way easier for my agents to interface with. And I did this part-time with a handful of threads on the side while I was working on my real day-to-day work. And these types of things are miserable to do without AI. Like this sucked long ago and now it's relatively trivial to set this stuff up yourself. Like truly crazy. As Chad is pointing out, I literally wrote my own TypeScript compiler and it's now running thousands of times a day in T3 Code. It's over 40,000 weekly downloads. And I threw this together kind of as like a joke to see if it was possible. Truly unbelievable. Oh, I got my numbers. Of all of my CI, almost a thousand hours of the CI on T3 code was spent waiting for Git checkouts. That's 42 days of waiting for your git pull. And that's just CI.
Watch from 32:33That's not counting people doing it locally. Yeah. The estimates just for T3 code time spent waiting for git pull is between $350 and $600 just for the renting of a CPU pulling down the changes. Insane. And that's for a small project by a startup with fiveish employees depending on how you cut. Imagine how much crazier that could get at this level of scale. I do like this section. This is a thing I've never heard from Anthropic. Engineers are no less valuable, but their work is changing. Thank you, Anthropic, for not saying engineers don't matter anymore. Nice change. All the engineers who worked on the system would now rate Claude as better at writing code than them. But the engineers still supplied the invaluable understanding of the highlevel interystem components in the design level trade-offs, making informed choices around the problems they wanted to solve and the scale they knew they needed to achieve. Very often they found that the role of engineers was to tell Claude no. Agents will sometimes violate core system constraints to satisfy a pattern that should never have been allowed in the first place. Okay, maybe they are using the same Claude that we are. This was particularly common in their migrations. For example, there were multiple systems that tried to brute force looking up when a change merged domain by linearly reading the commit log instead of asking the code review tool. Anthropic also makes extensive use of adversarial review. And while this is incredibly effective in many ways, can also be myopic. So skilled engineers need to steer Claude away from overengineering parts of the system in response to hypothetical concerns. Yep, this is even worse when
Watch from 33:50you have feedback from other model families because they're worse at convincing each other. Making sure that your AI review systems don't accidentally cause huge overengineering and scope creep is a genuine real challenge. For example, one such review was overly defensive about trusting other internal services resulted in Claude adding excessive validation and complexity. Our team members still review every change request for the system, but they now examine the intent of the change as opposed to reading every line of code. Oh, how the world has changed. We were able to substantially improve optimization and quality by virtue of Claude's ability to write immense amounts of code much faster than they could themselves. Scion is among the most well- tested and well-defended code bases they've ever worked on. Even on their tight timelines, they were able to do hermetic simulations of every non-trivial infrastructure and source control interaction that they've ever encountered, as well as fuzzing and sophisticated rate limiting. It still blows me away that I can be like, "Hey, Claude, build a fuzzer for this system. Find bugs." and it just does. Surreal. Claude also found significant optimizations. For example, algorithmic optimizations to ancestry queries that missed one of their indexes, which would have taken an expert days or weeks of analyzing and tuning and then did it all again the next day. This one's particularly funny cuz I just filmed my video about Gemini 4 argan. Gemini 4 was hallucinating columns in my SQLite tables. After reading the file that is my whole schema, it will then fire a query for columns that don't exist.
Watch from 35:20Meanwhile, Claude is doing algorithmic optimizations to get around missing indexes. Anybody who says Gemini is in a good state right now is not trustable as an engineer. It's pretty rough. Yeah, Gemini's imagination is incredible. It hallucinates so well. It's truly a creative model. You can see here the difference in everyday source control commands when they run their SCN system instead of git. Pulls are 25x faster. Checkouts are 14x faster. Rebase is 5x faster. Status is 207 times faster. Commits are only 30% faster. Disappointing. But diffs are six times faster. New work trees. Oh god, this is one of the biggest issues. And like we have to deal this in T3 code. Making work trees when you have to clone everything is really expensive. 84x faster is a huge win there. Pull latest is 20 times faster. Commits are 24 times faster. And status for agent environments is 20x faster. Huge. Oh, this first section was developer environments versus agent environments. That makes sense. Yeah. All of these wins are massive. Alongside all the performance wins, they also laid the groundwork for new forms of monor repo security. Managing safety, security, and compliance in massive monor repos is as important as managing the size and scale. They built Scion with these in mind from the start. It supports rich access controls on repositories as well as the ability to apply permissions to individual directories in order to protect the most sensitive code without losing the benefits of the monor repo.
Watch from 36:52Look who was right about committing your environment variables, The theo wins never end. can't wait for another animate profile picture on Twitter to say I'm wrong because they don't have a job. Furthermore, the code review tools that they built can express complex semantic constraints on approvals which allows them to differentiate simple things like comment only changes from sensitive logical changes so that human reviewers can focus on the most consequential parts. You you don't need to do this. You don't have to do this programmatically. You don't have to build code to change the categorization of the change sets. I don't know if you guys have heard about this. There's this thing called claude. You can give it code and ask it a question and you get an answer. This actually might be the thing I'm most bearish on in this whole thing. If they are compute constrained enough that they're not running claude to summarize and filter their diffs and they feel like they need to make programmatic solutions for this instead, maybe they are out of comput. Maybe we are screwed. With all of this, they claim that a new level of operational excellence is possible. And you know what? To their credit, it is. Their uptime is significantly better than it was in the past. It was bad bad before. I can't remember the last time cloud was down. The website goes down decently often, but like cloud code and the cloud APIs, those are pretty reliable lately. Funny enough, I think I've had more problems with OpenAI and codeex outages than I have with cloud recently, which is unbelievable considering how big the gap was before.
Watch from 38:20So yeah, if they're saying that this is helping make operational excellence more possible, I'll believe them because they are they are fixing their downtime problems. Their team uses Claude Tag, which if you don't know, that's their Slackbot that they're all obsessed with. They use it heavily and it helps them monitor each component of the system. For example, they have Slack channels where they tasked Claude with being obsessively focused on the health of single components. They call these super fans. Also, they refer to them as claudes plural. That hurts me a bit. Guys, why why do you have to do this anthropic? I I was defending you and it felt good. Now you're doing cringe like this. We call these Claude super fans. I'm like imagining the person saying this to me and somehow their face is more punchable than mine. They have a Scion Damon superfan, a Scion cash superfan, an antique merge Q superfan, and so on. They have access to the telemetry and logs, and they act like experienced engineers who have nothing better to do than refresh charts and look for anything strange. Are you talking That's my job. They do call out they have traditional alerts as well, but the super fans detect subtle and novel issues that make those alerts seem one-dimensional by comparison. That does actually seem kind of cool. Another thing earlier I said where they were reducing compute with the programmatic thing. This is the opposite. This is them burning compute for fun. And here's what's probably the most important lesson in all of this. We're completely rethinking our priors. Engineers, for good reasons, get pretty fixed in their way of doing things because in order to be a worldclass engineer, you need to have a system in your head on how to do
Watch from 39:46the thing. And generally speaking, the hot, hyped up, cool, new thing is not the best long-term solution. So, a lot of the best engineers in the world have gotten used to this and are inherently skeptical by default. They trust what they know and what they know works and they're hesitant to try new things because they've been burned so many times. You got to get over that. Things are changing too fast. You have to flip the way you think about this almost like upside down where you have this knowledge of what does and doesn't work which leads you to picking certain tools. Now you need to do the opposite. You need to know what does and doesn't work and where it breaks and where it functions and use that to reinvent the things you've trusted your whole life to be perfect for your needs. I have been writing Typescript for almost a decade at this point and I rewrote TypeScript itself for fun. Let's see how anthropic explained this phenomena. This project prompted them to take a very hard look at other workflows that were designed for human scale. The proper merge Q for example just doesn't make sense at agent scale. What does CI even mean when hundreds or even thousands of changes can merge while tests are running? We haven't been able to effectively replicate the industry standard of a global repo lock on the merge queue. So instead, they've invested in postmerge in pre-eploy automated testing. Look at that. Theo was right again. Sorry to freak out about these things again, but you guys have no idea how long I've been harassed for thinking it's more important to be able to undo a bad change than to add thousands of layers in front of the changes to make it
Watch from 41:16slightly harder to make a bad change. So much of engineering has been focused on increasing friction more and more and more to lower the likelihood something breaks. I've been focused on make the friction as low as possible. Both to get code out, but more importantly to fix something when it breaks. Because no matter how strict your processes are, you will ship something broken. I don't care who the you are. If you actually think you will never ship broken code or that you have built your system in a way where it will never ever have bugs, I hope that your mom's basement doesn't smell too bad because in the real world things break and when things break you need to fix them and focusing on ways to fix things and reduce this damage that broken things can do is incredibly important. They do call out that by working this way sometimes concurrent changes conflict but Claude can typically fix these conflicts faster than humans can even identify them. I've been thinking a lot about this. The idea I've had is what if the merge Q wasn't PRs that need to be merged one after another? What if the merge Q was a super general here are things that we're happy to have go in. Hey Claude, land all of this. And it would try to combine like once something has the the merge thumbs up, Claude tries to squash as many of those chunks into individual things as possible like in groups and then pull them into main as effectively as it can dealing with the conflicts as it goes because Claude is now better at dealing with merge conflicts than I am. And now we got an interesting end here contributing to the wider ecosystem. We see Scion as just one part of the answer to the scaling
Watch from 42:46concerns that increasingly capable AI agents have raised where sharing these details is an example of how teams can tackle similar challenges. Though they don't expect their solution or their considerations to suit every case. They specify this is in no means a suitable replacement for Git and general use. The concepts I've outlined here really only make sense for truly enormous monor repos with either tens of thousands of engineers or swarms of agents prolifically writing code. Plus, Anthropic's unique in that their ability to spend tokens for internal use far exceeds what any other business would realistically want to spend. Never thought I'd see the day. To be fair, as I've talked about many times, tokens are getting cheaper, way cheaper. A given level of intelligence is achievable at lower and lower prices constantly. The fable rewrite of bun into Rust was like 300k plus of Fable tokens. I was able to do a bolder rewrite which was Typescript in Rust myself in around 20 to $25,000 of Opus 55 tokens. The speed that these prices are going down is insane. So, it is absolutely possible by this time next year, you'd be able to write your own similar thing to Scion, your own alternative to Git for your company for just a few thousand in tokens. They also call out a thing that is important to understand. They are doing this both to address their own software development needs, but also to test Claude's capabilities to see what it can do. Practically speaking, it's difficult to offer Scion in a form that would be broadly useful to other organizations.
Watch from 44:18It diverges significantly from git, meaning companies would need to migrate every piece of the development life cycle and it requires its own code review, code search, and CI. Their internal version of these components are highly bespoke and they would be difficult for other organizations to adopt and they worry that attempting to build a public version that tries to please everybody wouldn't actually be very useful. Okay, I know what this paragraph is here for. There was a conversation internally about whether or not they could sell what they built. The people who built it said, "No, you. We're not going to do that. Here's all the reasons why it doesn't make sense." Enterprise sales finally probably caved. It's like, "Okay, whatever. We don't have to sell this." And then they wanted to talk about it. And comms is like, "Wait, wait, wait. People are going to ask for this." So, they had to take all the arguments they had internally and do this so that I can justify now as I read this to you why they are not open sourcing this for others to use. Okay, more justification. Let's see this part. However, they anticipate that this class of problems will become more commonplace over the next 6 to 12 months and it would be an inefficient use of engineering resources and tokens every company using Git to face similar challenges alone. Given that, we think the most impactful thing we can do is share our thinking so that other devs can apply some of the most successful parts of Scion to Git without departing from the overall model. A Git server doesn't have to work the same way as a Git client does. Novel server architectures exist that provide significant scaling improvements. Git platforms can provide comprehensive serverside RPCs like Scion, perhaps even in a standardized way across forges that the Git clients can learn to talk to.
Watch from 45:38Negotiation protocol doesn't have to look exactly like it does today either in the various sparse and shallow features in Git can be made to compose well. Lastly, they'd like to acknowledge that the Git community has spent nearly two decades scaling the platform for millions of devs. Their contribution is far narrower. They're sharing their early views into how agentic software development interacts with these systems. They can partner with GitHub, GitLab, and the open source community to support the changing needs of devs. We're all in it together as the software development life cycle intersects with the AI exponential. It's crazy how something like this would have felt comical a year ago, and now it's almost like, yeah, of course, makes a lot of sense. I'm going to go fight the urge to make my own git clone. It's going to be hard, but I'll do my best. I hope you enjoyed this deep dive. I know I did. I love these types of nerdy rants and this was a very fun one for me getting to go deep on how source control works and how to make it scale the way that agents need. You still enjoy these deep dives or do you just want to see me vibe code more? Let me know in the comments. Until next time, he sns.
Watch from 47:05Full transcript
- 0:00
For as long as I've been writing code, I've been using Git and GitHub. Funny enough, I actually started using Git before I wrote code cuz I used it to manage plugins for my Minecraft servers way way back when I was in like early high school. Git is core to the way that I build and work with software. But I have been starting to notice rough edges. A lot of the way that Git was built was around code being expensive
- 0:20
and humans needing to review the code, approve the code, and control where this code goes because losing code or merging the wrong code was a very expensive and painful thing. The reality of how Git works has influenced the whole industry from how we work with our team to how products like GitHub are built to the way that we think about things that should be basic like secrets management. Shout out to my viral clip saying that
- 0:40
we should be able to commit our environment variables. Seriously though, Git is not the right primitive for the way that things are moving right now. I'll be real with y'all. A lot of my problems with Git are coming from GitHub because it's just a slow and miserable platform. External estimates are measuring as much as 7% downtime across different things within GitHub. I can't
- 1:00
tell you how many times we've had releases blocked because actions were failing or I haven't been able to merge an important change because the page wasn't loading. The fact that GitHub's uptime looks more like a Christmas tree than a green bar like it's supposed to should say everything you need to know. But what if I told you GitHub wasn't the only problem here? that git itself is not the right solution for working
- 1:23
heavily in parallel across a lot of people and even more agents. I've been thinking about this a lot because back in December I explored JJ and I loved it and ever since then I've been thinking about what would it look like to build something new for shipping really really fast with agents. It turns out I'm not the only one who's been thinking about
- 1:43
this. Anthropic just put up an article about their new in-house source control system. Back in May, they were merging 2500 changes per day, but now in September, they're doing more than 30,000 changes merged a day. They are now bottlenecked by developer infrastructure that was built for humans and not for agents. And they've decided
- 2:03
to build something of their own. It seems kind of crazy that agents are now smart enough to do things like this, but they really are. And I love this idea of challenging the fundamentals of how you're building to make the right solution for yourself. I have a feeling this one's going to be a really fun deep dive, especially now that I'm ripping things out like TypeScript for my own vibecoded alternatives. That said, I
- 2:25
don't think everything should be ripped out for your own vibecoded alternatives. There's a lot of places where it makes sense to use a vendored product. Like for example, today's sponsor. It kind of feels like AI has solved most of modern webdev, but there are some things it hasn't solved. In particular, mobile. As good as it is at starting up an app and making small changes here and there, I find myself just fighting the model constantly when I'm trying to actually
- 2:46
get my apps ready to ship, fix real bugs, deal with Apple reviews, and all these other pieces. As I ran into more and more walls as we tried shipping the apps we were building, I hit up my friend Jamon at Infinite Red, and he has made life so much easier. If you're not familiar with these guys, they're industryleading experts in all things mobile, especially React Native and Expo, but also Swift and Cotlin as well. There's a reason that everyone from Domino's to Microsoft has called these
- 3:07
guys in. They really know how to ship. I'm gonna be real with you guys. The role of a consultancy has fundamentally changed as a result of AI. You're not just hiring these guys to come in and do grunt code work anymore. You have agents for that. What you bring them in for is the expertise. Making sure you're not making simple mistakes. Getting your pipeline set up so you can actually build and ship the apps. Making it possible for your agents to actually
- 3:28
verify their changes and know things are working. And just making sure everything start to finish is ready to go. In a world where we're all shipping faster, these guys can come in and fix anything from your team to your app. They can build a whole new app from scratch. They can take over development, or they can just give you the pipelines you need to do it yourselves. If you want to ship better apps sooner with less bugs, look no further than/infinite
- 3:48
red. Man, I'm actually really excited to dive into this article. As soon as I saw it drop, I wanted to read it, but I decided to put it off for this filming. So, you're getting my organic take on this. If you're wondering why the site looks gross, it's because when I leave it on the default light mode that the blog is in, everyone yells at me. So, I'm using the dark reader extension to make it not just bright white. And this is the poop green brown color it chose.
- 4:10
Redesigning source control at Anthropic. As our merges per day grew more than 10fold, source control became our biggest development bottleneck. Here we share how we migrated to an in-house version control system in 5 months. I won't lie, the five months feels like one of those clawed timeline hallucinations when you ask it how long will this take and it's like oh two to three days and then it finishes the work
- 4:31
in an hour. Regardless, this is impressive. Replacing Git at a company as big as Anthropic sounds insane. Last I checked, they had around 2,000 employees. So yeah, at Anthropic, coding is no longer the bottleneck to building systems or doing research. Claude is able to autonomously iterate on research loops, microoptimize complex programs and build a new services from the ground
- 4:53
up at a speed and scale we've never seen before. The bottlenecks now all of the developer info that was built for human scale source control builds CI things like that. In May of 2026, Anthropic averaged around 2500 code changes merged a day. As of September, it's over 30,000 a day. We don't expect this rate of growth to slow anytime soon. The latent
- 5:13
demand for merging code is practically infinite and every bottleneck we remove immediately leads to further increasing the amount of code that we produce. The way I've framed this in the past is derisking the merge button, making it so by the time a person's looking at merge, it's pretty likely the code is safe to merge. But also the other side where if you merge something you shouldn't have, the blast radius and the damage that it
- 5:34
can do is reduced massively by systems to make it easy to test the code once it goes out with a limited set of users and revert the things if they do actually end up breaking. And I've seen the difference when we introduce these types of things. We just ship faster. For example, I stayed up a bit too late last night because I was annoyed with some niche performance issues when you had a really big database in T3 code. So, I
- 5:55
decided to fix as much of it as I possibly could in one sitting, as well as clearing out some old issues in PRs that were annoying me, as well as taking over the PRs that weren't quite implemented how I wanted. And between the hours of like midnight and I think around 3:00 or 4:00 a.m., I merged from here through to the top of this page
- 6:19
to here, another full page of just Theo merges. You'll see t3.gg on the left for I think every single one of these. Some of them are multiple people, but they are me. We're still going all the way up. Okay, these were two I merged from others to here.
- 6:41
That is three full pages of GitHub merge history that I did in 4 hours. I know it was four hours because I did it between nightly releases. We do every six hours because I wanted to have as much in as possible for the nightly. So before I went to bed, I could download it, test it, make sure all was good, fix anything that was broken, and revoke the build if it was bad, and then go to bed smiling,
- 7:02
knowing that performance would be way better for everybody on the nightly when I woke up. The fact that I could confidently merge 50 plus PRs in a sitting, it just feels insane. Like imagine going even a year back and telling yourself that you were merging 50 plus PRs a day. Hard to fathom. And if Anthropic was just saying these things and I hadn't experienced it myself, I'd be calling right now. But this is reality. If you're not
- 7:24
moving this fast yet, you are either in like a place where you absolutely can't because people will die or you just don't realize how powerful these things are and you haven't taken the time to set up the tooling to make the stuff less risky. So let's dive in a bit here. This is interesting. They have blue and orange. Blue is pull requests. Orange is antique change requests. That must be
- 7:45
their new thing. It lines up with the way they name Antique seems like their system for all of this. If you think this is anthropic being uniquely weird and silly here, there's a long history of companies reinventing source control for their specific needs. For example, Meta built a crazy system around Mercurial that I think is actually a hard fork of Mercurial. They
- 8:06
built all of that to make their stacked diff system as strong as possible for when people were working on large arcs and approving pieces of it to merge it all in at once. Graphite is a whole company that was built on top of GitHub to bring the way meta worked with source control to people who were on GitHub because it was so far ahead of how we were building if we you were shipping fast. At my company, Ping Labs, we were
- 8:28
heavy users of graphite. Now they're part of cursor and they're building an alternative to GitHub into cursor with origin. The point I'm trying to make here is that this isn't as unprecedented as it might seem. And many companies have found meaningfully better ways to work with and manage their source outside of Git. And Jold here, who used to work at Meta and also works at Graphite and now cursor pointed out that a lot of the reason Meta built their
- 8:50
systems and themselves instead of using off-the-shelf things is because their massive checkouts just didn't work well with Git. I vaguely remember the history here of Meta trying to convince the teams working on Git to make it work better for giant monor repos instead of doing separate repos for everything. The git maintainers didn't want to budge. So, they ended up going their own way. And that's fine. You shouldn't have to
- 9:11
force the industry standard into your workflow if it doesn't fit. And it is nice seeing more companies challenge this cuz Git is not perfect and it certainly is not perfect for agents. So, let's learn more about what Anthropic has built. In order to support their insane growth, they built Scion, an in-house version control system, and they fully migrated their monor repo from Git in five months. Ready for a really silly fun fact? Both Anthropic
- 9:33
and OpenAI are built in monor repos. So everything they're working on is different projects and folders inside of this one giant repo. If you've ever been curious why the sidebar in the chat GPT and Codex app is so bad, it's because of this. The vast majority of employees at OpenAI have one project, OpenAI, and they're still using github.com, by the way. Pretty much all of OpenAI's code
- 9:53
lives in OpenAI/OPAI, which 404 is if you're not an employee. So for those saying I'm secretly employed, here's your proof. That makes a lot of the way they think and work and build strange to people who have more than one project. So yeah, just a silly fun fact I wanted to include. Back to what Anthropic was saying here. Not only did they build Scion, which is their in-house version control system, they also built and
- 10:14
shipped a code review surface called Antique alongside code search, a rich source control AM system, novel ways of managing and landing large amounts of reviewed code, and more, all in that same 5month period. In this post, they're going to focus on how they built Scion specifically. It makes for an interesting case study. Claude's ability to write code was both what put them in the situation in the first place and how
- 10:36
they're able to build a highquality scalable system to deal with it. growing our monor repo. Before we explore what we built, we should acknowledge how unusual our workload is. Unsurprisingly, Anthropic uses Claude to write software far more than almost any other company. You might be surprised, guys. Okay, seriously though, they're uh they are definitely doing it more than we are,
- 10:56
although we are doing quite a lot. But since they're using it so heavily, they're often the first to see where tools built for human scale start to strain. Source control is just one example of this. While we don't expect many organizations to have these problems in the very near term, we suspect it won't be too long before others who use AI code tools experience these same pressure points, which is why we want to share what we've built and
- 11:16
what we learned. It's a decentralized system. Every clone is a complete self-contained repository that can compute diffs, read history, and blame, and check out any revision while fully offline. For the vast majority of repos, that is the right design, and it is a large part of why Git became the standard. But as their monor repo has grown, it increasingly doesn't fit that design space. The history of their main branch alone is almost 100 gigs and a
- 11:38
compressed working copy is over 25 gigs. They perform on the order of 25 million code checkouts per day for CI before counting developer environments, agent sandboxes, and research jobs. Fetched natively, that would be 600 pabytes of Git traffic a day. Git offers tools for working with a slice of a repository
- 11:58
like shallow clones, sparse checkouts, and promiser remotes. Beyond these, there are standard workarounds for large monor repos like internal git mirrors and repository snapshots like seeds that are kept in blob storage. They've tried and used all of them at their peak. They operated several hundred internal git mirrors to deal with the load. This bought them time, but it became harder to operate as merge rates were
- 12:18
increasing. Ces are falling further from the tip of Maine between refreshes and it took more and more care to keep operations that depend on history like diffs and blame working on these partial clones. Taking a step back, we noticed that like most companies, we were using Git as a centralized system with one authoritative copy, but nearly all of our clients were agents in jobs that are
- 12:39
online by definition. This is a very interesting and worthwhile call out. You can't use claude if you're offline because claude is being served from a giant data center that has GPUs you don't. And unless you are plugged into those directly and somehow the internet for that data centers offline, you're online. Building everything around this idea that everyone needs everything
- 13:00
synced just doesn't make that much sense. The properties that make Git excellent for distributed offline work were actively undermining their ability to scale as the repo grew. So they asked themselves if they designed source control with the fact that agents are always online in mind from the start what would it look like how would it be different with agentic coding many common assumptions about how software is
- 13:21
developed are evolving they embrace these changes when they built scion which allowed them to take a fundamentally different approach than git on several core principles source control for huge monor repos does have prior art like google's piper and meta's sapling I touching on meta stuff before so they were able to build scion with their hard one lessons in mind It is worth noting just how many people at
- 13:42
Anthropic are ex Meta and ex Google and they were often working on these types of things at those companies like I know Boris was deep in developer tooling at Meta and now he is making cloud code. The first core change they made is again this idea that clients are always online since nearly all of the clients are agents in jobs. They designed Scion client and protocols around a server
- 14:02
that's always reachable and they have clients fetch data when they need it. also clearly not using GitHub because even when you're online, you can't assume GitHub's online. The next point is this idea of client server asymmetry. In Git, every client is capable of being a full-fledged server. This is one of those things that makes Git Git. It is truly decentralized. Anything can be a Git server. But in Scion, the server
- 14:24
answers questions about file history and blame, creates commits, performs merges and rebases, and so on. The client maintains a bounded local object cache. It lays out the working copy files on disk. It monitors for file system changes and it gives users ergonomic ways to work with that server. It can compute very little on its own, but in Exchange, it requires way less data
- 14:45
locally. Next, I talk about how the source control service can power many novel application types. With git, a CI job or tool that needs to ask a question about the repo, like who last changed the line or how two commits are related, it would have to maintain the whole local copy of that repo or hope that some vendor specific API is able to provide the answer. With Scion, all the
- 15:05
source control queries that the client uses are also exposed via API. So in many cases, those jobs don't need the local repository at all. Many internal tools like their automated bisect and revert pipelines wouldn't be practical at the scale they're operating at without this cheap programmatic access to the source. They also call out that everything must be horizontally scalable. At their peak, they ran
- 15:26
several hundred git mirrors which significantly strained capacity and no longer scaled once the repo outgrew the mirrors discs or memory limits. You know it's bad when your git mirrors are running out of disc space. And to be fair, discs did get a lot more expensive. I still can't believe that hard drives increased in price as much as they did. Thanks, Anthropic. In Scion, the repo's data is distributed
- 15:46
and every component is independently scalable. There is no materialized copy of the repo anywhere, even on the server. There is no equivalent of pack files, just gets compressed bundles of objects, so there's no repo maintenance needed either. And then a very important piece, commits must be cheap, incremental, and lock free. Lock free is particularly important. One person committing shouldn't block somebody
- 16:07
else. Or more realistically, 500 agents committing shouldn't block another 500 agents. In Git, you commit locally and then you push in order to share your work. In Scion, every commit, rebase, or merge happens on the server. So, there's nothing to push. But each of these operations needs to be fast. The cost of creating a commit and updating its downstream data structures has to scale with the size of the change, not the
- 16:28
overall size of the repo, or the server won't be able to keep up. The development life cycle in Scion discourages refs pointers to commits such as branches and tags except for the main ref or release branches. The reason for that is because global state that every server has to agree on is always going to cause scale issues. Globals are bad when you're operating this fast. Always. And then the most important piece, the working copy is optional and
- 16:50
it is sparse by design. Because the repo is so large and they're checking out code tens of millions of times a day for CI, they want to avoid checkouts wherever they can. With Scion, a CI job that needs the diff between a commit and its merge base, for example, can ask the server just for that. The same goes for writes. Many write workloads can skip the checkout and function entirely over
- 17:11
API. Scion's protocol is centered around sparse checkouts, which only contain the directories the client asked for. And there's also virtualized checkouts, which materialize files on demand. So when the job needs the file on the disk, the client only pays for what it's actually reading. I was playing with some of these ideas real late last year and early this year, trying to make it easier to manage state of like a
- 17:31
codebase in a way where you don't have to have it all on your computer, but your agents can still access it. It's dope to see them doing the real version of this cuz mine was a chaotic experiment. So, how does it actually work? Now that we've seen the principles, we can hopefully understand how it all comes together. These insights can apply to other source control systems or more broadly to other stages of the software development life cycle. Oh boy, we're now in the data
- 17:53
structure section. This is where we're going to get into the fun details. The unit of work is a change set. I I see the meta inspiration here. And change sets are entries and write ahead logs. A change set stores most of what you'd expect from a commit in git. Where a git commit stores a pointer to the tree, a change set stores a list of the files that changed relative to its first parent. But where a git commit stores a pointer to the tree, a change set stores
- 18:15
a list of the files that changed relative to the first parent along with content address pointers to the compressed file contents in blob storage. Change sets are created server side and are available everywhere instantly. Fascinating. I knew they were doing blob storage for a lot of this, but kind of crazy to see the like direct call out that the source code does not exist in the system and when you request
- 18:36
the source it has to go fetch that from blob store and it's effectively just a database pointing at those blobs. Seems like all the metadata is in Spanner. Makes sense. Again, very Google inspired. Very interesting that they have separate author and committer fields. makes sense because the author is the person who could who triggered it and the committer is whatever agent did it or where it happened if you have like
- 18:56
different systems that are doing these things. Very interesting. And other usual fields like author date, committer date, message, the parent, which we talked about before, that is the thing that this is in reference to. And then the file changes which specifies the file that changed and then has pointers to the blobs in their blob storage. The structure lets them accept change sets
- 19:16
with very high throughput because there's no pack files they have to deal with. No locking or crossover coordination and no refs to move around. What this is trying to say is you just have to append. You only are adding to the data set. You don't have to go through the whole history before appending it. You don't have to move pointers from here to there. You don't have to restructure the tree. You don't have to sync all of the data back and
- 19:37
forth to make sure it's all in the right place. You just put up the change set and store the blobs in blob storage and you're done. Oh, is it not even storing the diffs for the file change? It's just storing the new state of the file. That would make sense. So, it's saying like these files changed. Here is where they point. But this thing in the storage is the full content of that file, not a
- 19:58
diff. Because if it was a diff, recalculating would be hell. Change sets are recording only what changed. So on their own, they can't quickly answer questions like who edited a line or what commit affected a file. To answer those, Scion builds additional data structures from the change sets, including dedicated structures for file blame and path history, as well as skip lists for the commit graph. They call everything
- 20:19
that isn't directly part of the change set derived data. And they compute it all through one generic mechanism. Every type of derived data is defined recursively. So its value for a change set is computed from its values for the change sets parents. The mechanism has three benefits. First, they can easily add new types of derived data when new data structures would make some operations faster. Second, for each
- 20:40
type, they can choose whether to compute it eagerly or defer the work, so they can avoid materializing data they don't need. And third, because the client never sees the internals of the structures, they can optimize them without changing the client. They've added new types of derived data many times as the system has grown, such as new indices for performing copy detection or path-based file search.
- 21:00
makes a ton of sense to try and dumb this down. We're deep in the weeds here, which I I love. A big part of what makes Git so powerful is that you have all of the data when you clone the repo. So, if you want to find something like when this file changed, all of the things this person did, stuff like that, you can go through all the data yourself. The problem with this is if you want to
- 21:22
add something new to Git or make some type of action faster, you're kind of screwed because you'd have to upend everybody's git. But if people don't have to clone the whole thing and they're just hitting an API for this, you can choose what you do and don't expose. And since the server here is full of cache layers, overprovisioned CPU, and so many other things, they look
- 21:43
for the things clients want to request to get more information. They built some way to compute that that's efficient without having to change the data structure and now the clients can do the thing and the clients aren't going to cheat and do something weird which I've seen a lot. When you give Claude access to the internals of something, it will invent some crazy novel and also crazy slow and CPU inensive solutions to get
- 22:05
what it's looking for. This hard API boundary prevents that which is almost certainly a huge benefit. So how did they scale this? starts with a small fleet of Scion servers providing source control APIs for any operation a client needs like getting change sets, computing diffs, finding merge bases, and fetching blobs. Computing diffs is a fun thing to realize here because again
- 22:25
it's storing the file content at the end of a change. So when you are asking what changed between these two like change sets, you don't just look at the diff because you don't have a diff. The diff is computed when you say here are two change sets. change between them and it does that by grabbing the blobs for all the files that are affected and seeing
- 22:47
how they differ between those states. All source control data is content addressed. So a piece of data never changes once it's stored. This is also important. The ability to override history in git isn't too bad because whenever you force push, you're changing all the commit hashes. But if you had the ability to edit the contents of a given hash, that would fundamentally break git at its core. Immutability is
- 23:09
always more scalable than mutable stuff pretty much 100% of the time. So building the whole system around these change sets being fixed and permanent and all of the flexible stuff being calculations done on top. This is touching my functional programming heart deep at its core. And since they have these immutable content addressed change sets, all servers can cach them without
- 23:30
checking with the others. As soon as it's sent up to one server, it knows that change set's unique. it can just blindly trust that. As a result of this, the servers are horizontally scalable and they require very little coordination. They're essentially large caches that can also compute derived data on the fly and write the results back to Spanner. Scion's many consumers no longer need to maintain local repos.
- 23:52
Fetching and maintaining the repo used to be a significant maintenance burden for the CI jobs and tools that needed to ask source control questions, especially around commit, ancestry, and blame. Early in the migration, before they even had their own client, repo format and protocols, Scion was only used to answer these types of metadata queries. Fetching file contents and materializing working copies was far more data inensive operations. So they needed to
- 24:13
design something of their own there. Scion's SCN client deviates from the Gits fetch protocols in several important ways and as a result it lends itself to highly scalable architectures that require significantly less CPU than git servers do. Science negotiation doesn't tell the server what the client already has. Instead, it operates exclusively on trees and blobs that the
- 24:33
client wants. Starting from a root tree, the client checks its local store for missing trees and requests them recursively. Then it repeats the process for the blobs. So, it starts by finding the change sets that it needs to do the thing it wants to do. And once it has those, it goes and collects all of the blobs to have the files it needs. And then it can actually start working.
- 24:53
Every request is for a specific object that never changes. So, any cache can answer it. And the architecture closely resembles traditional tiered caches. The process is also efficient and resumable because there's no single multi- gigabyte pack file response. Sparse and even virtualized checkouts don't need special support because the client simply doesn't ask for trees it doesn't want. This paragraph is using a bunch of
- 25:15
terms that they did not define and they are using in unique ways. So I will do my best to translate from claudeisms to English here. The clients don't connect directly to the Scion servers because those are managing all of the change sets. They're talking to a Damon over a socket and that Damon stores objects for the whole node. So, it's effectively
- 25:35
keeping a cache of like what was the most recent version of a given file in a given file path. This allows it to respond to requests for files without having to go down the whole tree to compute the current state of things. I really hate how they use node here. Each of these dammons that does this caching gets periodically reset to be on the latest snapshot so that it isn't like
- 25:58
recalculating all the things that have changed in the last 3 days. They update it constantly. And let's be real with the way people are spinning up these agents, spinning up CI, all of these things. They're probably starting that work from latest. So it makes a lot of sense to get the cache up to date that way and then just resolve things that have changed since the cache server stream objects directly from memory
- 26:18
instead of like storing it in the hard drive which makes it way faster to get that data when you're on a client on the other side. And it stores this in the same format that it would have been on disk so that it doesn't have to change the shape or format of the data when it comes out of memory and gets onto a drive after going over the network. When caches are missing objects, they ask the authoritative scion servers, which is
- 26:38
the layer we were talking about before, which will read from Spanner or cloud storage as needed in order to get the updated data and the updated files in that cache layer. The Scion Damon, Scion cache, and Scion server all respond to identical RPCs for this protocol. So, the client doesn't need to know the cache topology. Any tier could theoretically talk to any other, but they strictly enforce the tiering to
- 26:59
isolate failure domains. They want to avoid a situation where a failure in a regional scion cache overwhelms the upstream servers and brings down the entire system. So again they have that cluster of scion servers that are the like proper management of the real world state of things the authoritative layer that is the layer that stores in spanner for the change sets and the cloud
- 27:20
storage for the actual contents of the files. There is then a cache layer that is what gets hit often for your actual work that is regional. So you don't have to like fetch across the world to hit this server. This is what will hit the scion server to get a reasonable state in the cache. Developer machines directly hit the scion cache when you're
- 27:40
like pulling down the code to do work. And if the scan cache is missing things that you need, it will go pull it from the server. But if you're working on CI, even this isn't the right abstraction. And that's where this Damon comes in that we were just talking about that gets periodic snapshots so that it's ready to spin up CI jobs as quickly as possible. Each tier of this system maintains a cache hit rate of over 99%
- 28:00
and it doesn't require longived state. So it's easily scaled up if needed. Their infrastructure footprint is 200x smaller than their previous git mirroring infra in terms of CPU and memory requirements. That's insane. again like and this is like the crazy idea that has been like hurting me a bit. The realization that like all
- 28:21
software was built to be somewhat generalizable. Now we can build software for our specific needs. And like it's a silly thing to pivot to, but hear me out. One of my weird YOLO projects right now is called Lust Lust Lang to be specific. on trying to take a subset of Rust that gets you most of the benefits of Rust, but make it so that it could be
- 28:43
compiled and built a hundred times faster because the build times for TS Rust, which is my rewrite of the Tyer compiler in Rust, are too slow. This is an insane project that I didn't think would make any progress, but it's already able to compile over 40% of the files from TS Rust, and I am expecting by the end of the week it might actually work. Would this work for most Rust
- 29:04
projects? no. Absolutely not. That is not what this is for. This is a bespoke experiment to see if I take a given codebase that I'm not happy with compile times on. Can I invent a new language to make it compile faster? It's like insane that this is even a thing you can think, much less do. But I'm using a model that's a lot dumber than
- 29:25
what they have access to. I'm using the distilled version of two model generations away from what they have internally. They're rocking Fable Ultra Max 7 internally and I'm here with Opus 55 more than happy to do these types of things. The fact that we can build this way now is just crazy. And while this system might not be 200 times smaller and faster for some cases, it is for
- 29:47
theirs. And I would expect more and more companies that have these types of big painful scaling things to go reinvent the painful pieces that are slowing them down. And uh fun fact, SJ, the borrow checker is not why build times suck in Rust. It's a bunch of weird LLVM Anyways, CI and research jobs are able to check out code two orders of
- 30:08
magnitude faster, seconds rather than minutes, which frees up a sizable amount of capacity across the fleet. CI alone had a 25% CPU capacity when neither clients nor caches need to perform repo maintenance, which simplifies the life cycle management of our jobs. Out of curiosity, I'm trying to get the hard numbers for how much time T3 code builds
- 30:29
and CI have spent dealing with GitHub and Git like clones. It's going to be a number that hurts me a lot. We'll come back and check on that in a bit, but I have a feeling I'm going to feel a lot of pain as soon as I see that. So, what did they learn building Scion? They started building it around the same time as Mythos Preview and Claude Tag becoming available internally. Both of
- 30:49
these releases simultaneously amplified the need for Scion and meaningfully accelerated how quickly they'd be able to build it. Here's a particularly striking example. Earlier in the year, they had largely handwritten the first version of the Git to Scion ingestion pipeline as a prototype. They thought it was good until they got their hands on early internal snapshots of Mythos preview. They asked it to rewrite the whole thing and with no further instructions, it produced a version that
- 31:10
consumed 20x less memory and was four times faster while preserving all of the implicit invariations around ingestion, ordering, and data structures. This is the history of changes to Scion and Antique. I will say worth noting here, they were bragging about 30k changes a day, but they've only made 12,250 changes to this architecture total. So
- 31:32
their daily velocity is more than double the amount of like requests, commits, change sets, whatever you want to call them, that have landed for the whole of their source control. Kind of crazy. But they learned building and operating Scion offers some interesting lessons for what building agent scale software will look like. Rapid development with
- 31:53
small teams is going to become more common. Can confirm. A team of seven people built both Scion and Antique, their code review tool, which they could have easily written a whole separate blog post about considering how awful code review is. I feel it. I'm thankful that we have so many wonderful sponsors working on that now. They also follow with not to mention half a dozen related systems that they fully migrated all of
- 32:15
in 5 months. They did this all while the volume of code merged per day was increasing 10x. In the past, developing and deploying source control systems like this would have taken years of iterative migrations with many more engineers. We think the sort of fast, nimble migration will become more common in the future. Yeah, it just will. Switching cost has gone down more than
- 32:35
almost anything. Historically, people adopt a technology, a solution, a vendor, and then they kind of are stuck there because switching off is more expensive than whatever cost they're incurring staying. Now, it's easier to write your own alternative and move to it. It's insane. I personally threw together a solution to clone and mirror and backfill all of our analytics data that
- 32:56
I have on one particular analytics provider so that I have it directly on my own managed click house on top of R2. It ended up being more than 10x cheaper and way easier for my agents to interface with. And I did this part-time with a handful of threads on the side while I was working on my real day-to-day work. And these types of things are miserable to do without AI. Like this sucked long ago and now it's
- 33:18
relatively trivial to set this stuff up yourself. Like truly crazy. As Chad is pointing out, I literally wrote my own TypeScript compiler and it's now running thousands of times a day in T3 Code. It's over 40,000 weekly downloads. And I threw this together kind of as like a joke to see if it was possible. Truly unbelievable. Oh, I got my numbers. Of all of my CI, almost a thousand hours of
- 33:40
the CI on T3 code was spent waiting for Git checkouts. That's 42 days of waiting for your git pull. And that's just CI. That's not counting people doing it locally. Yeah. The estimates just for T3 code time spent waiting for git pull is between $350 and $600 just for the renting of a CPU pulling down the
- 34:03
changes. Insane. And that's for a small project by a startup with fiveish employees depending on how you cut. Imagine how much crazier that could get at this level of scale. I do like this section. This is a thing I've never heard from Anthropic. Engineers are no less valuable, but their work is changing. Thank you, Anthropic, for not saying engineers don't matter anymore.
- 34:23
Nice change. All the engineers who worked on the system would now rate Claude as better at writing code than them. But the engineers still supplied the invaluable understanding of the highlevel interystem components in the design level trade-offs, making informed choices around the problems they wanted to solve and the scale they knew they needed to achieve. Very often they found that the role of engineers was to tell
- 34:43
Claude no. Agents will sometimes violate core system constraints to satisfy a pattern that should never have been allowed in the first place. Okay, maybe they are using the same Claude that we are. This was particularly common in their migrations. For example, there were multiple systems that tried to brute force looking up when a change merged domain by linearly reading the commit log instead of asking the code review tool. Anthropic also makes
- 35:05
extensive use of adversarial review. And while this is incredibly effective in many ways, can also be myopic. So skilled engineers need to steer Claude away from overengineering parts of the system in response to hypothetical concerns. Yep, this is even worse when you have feedback from other model families because they're worse at convincing each other. Making sure that your AI review systems don't
- 35:26
accidentally cause huge overengineering and scope creep is a genuine real challenge. For example, one such review was overly defensive about trusting other internal services resulted in Claude adding excessive validation and complexity. Our team members still review every change request for the system, but they now examine the intent of the change as opposed to reading every line of code. Oh, how the world
- 35:48
has changed. We were able to substantially improve optimization and quality by virtue of Claude's ability to write immense amounts of code much faster than they could themselves. Scion is among the most well- tested and well-defended code bases they've ever worked on. Even on their tight timelines, they were able to do hermetic simulations of every non-trivial infrastructure and source control interaction that they've ever
- 36:08
encountered, as well as fuzzing and sophisticated rate limiting. It still blows me away that I can be like, "Hey, Claude, build a fuzzer for this system. Find bugs." and it just does. Surreal. Claude also found significant optimizations. For example, algorithmic optimizations to ancestry queries that missed one of their indexes, which would have taken an expert days or weeks of
- 36:29
analyzing and tuning and then did it all again the next day. This one's particularly funny cuz I just filmed my video about Gemini 4 argan. Gemini 4 was hallucinating columns in my SQLite tables. After reading the file that is my whole schema, it will then fire a query for columns that don't exist.
- 36:52
Meanwhile, Claude is doing algorithmic optimizations to get around missing indexes. Anybody who says Gemini is in a good state right now is not trustable as an engineer. It's pretty rough. Yeah, Gemini's imagination is incredible. It hallucinates so well. It's truly a creative model. You can see
- 37:12
here the difference in everyday source control commands when they run their SCN system instead of git. Pulls are 25x faster. Checkouts are 14x faster. Rebase is 5x faster. Status is 207 times faster. Commits are only 30% faster. Disappointing. But diffs are six times faster. New work trees. Oh god, this is
- 37:32
one of the biggest issues. And like we have to deal this in T3 code. Making work trees when you have to clone everything is really expensive. 84x faster is a huge win there. Pull latest is 20 times faster. Commits are 24 times faster. And status for agent environments is 20x faster. Huge. Oh, this first section was developer environments versus agent environments.
- 37:53
That makes sense. Yeah. All of these wins are massive. Alongside all the performance wins, they also laid the groundwork for new forms of monor repo security. Managing safety, security, and compliance in massive monor repos is as important as managing the size and scale. They built Scion with these in mind from the start. It supports rich access controls on repositories as well as the ability to apply permissions to
- 38:14
individual directories in order to protect the most sensitive code without losing the benefits of the monor repo. Look who was right about committing your environment variables, The theo wins never end. can't wait for another animate profile picture on Twitter to say I'm wrong because they don't have a job. Furthermore, the code review tools that they built can express complex semantic constraints on
- 38:36
approvals which allows them to differentiate simple things like comment only changes from sensitive logical changes so that human reviewers can focus on the most consequential parts. You you don't need to do this. You don't have to do this programmatically. You don't have to build code to change the categorization of the change sets. I don't know if you guys have heard about this. There's this
- 38:56
thing called claude. You can give it code and ask it a question and you get an answer. This actually might be the thing I'm most bearish on in this whole thing. If they are compute constrained enough that they're not running claude to summarize and filter their diffs and they feel like they need to make programmatic solutions for this instead, maybe they are out of comput. Maybe we are screwed. With all of this, they
- 39:17
claim that a new level of operational excellence is possible. And you know what? To their credit, it is. Their uptime is significantly better than it was in the past. It was bad bad before. I can't remember the last time cloud was down. The website goes down decently often, but like cloud code and the cloud APIs, those are pretty
- 39:37
reliable lately. Funny enough, I think I've had more problems with OpenAI and codeex outages than I have with cloud recently, which is unbelievable considering how big the gap was before. So yeah, if they're saying that this is helping make operational excellence more possible, I'll believe them because they are they are fixing their downtime problems. Their team uses Claude Tag, which if you don't know, that's their Slackbot that they're all obsessed with.
- 39:58
They use it heavily and it helps them monitor each component of the system. For example, they have Slack channels where they tasked Claude with being obsessively focused on the health of single components. They call these super fans. Also, they refer to them as claudes plural. That hurts me a bit. Guys, why why do you have to do this anthropic? I I was defending you and it felt good. Now you're doing cringe like this. We call these Claude super
- 40:19
fans. I'm like imagining the person saying this to me and somehow their face is more punchable than mine. They have a Scion Damon superfan, a Scion cash superfan, an antique merge Q superfan, and so on. They have access to the telemetry and logs, and they act like experienced engineers who have nothing better to do than refresh charts and
- 40:40
look for anything strange. Are you talking That's my job. They do call out they have traditional alerts as well, but the super fans detect subtle and novel issues that make those alerts seem one-dimensional by comparison. That does actually seem kind of cool. Another thing earlier I said where they were reducing compute with the programmatic thing. This is the opposite. This is them burning compute for fun. And here's what's probably the most important lesson in all of this. We're completely
- 41:02
rethinking our priors. Engineers, for good reasons, get pretty fixed in their way of doing things because in order to be a worldclass engineer, you need to have a system in your head on how to do the thing. And generally speaking, the hot, hyped up, cool, new thing is not the best long-term solution. So, a lot
- 41:24
of the best engineers in the world have gotten used to this and are inherently skeptical by default. They trust what they know and what they know works and they're hesitant to try new things because they've been burned so many times. You got to get over that. Things are changing too fast. You have to flip the way you think about this almost like upside down where you have this knowledge of what does and doesn't work
- 41:45
which leads you to picking certain tools. Now you need to do the opposite. You need to know what does and doesn't work and where it breaks and where it functions and use that to reinvent the things you've trusted your whole life to be perfect for your needs. I have been writing Typescript for almost a decade at this point and I rewrote TypeScript itself for fun. Let's see how anthropic explained this phenomena. This project
- 42:07
prompted them to take a very hard look at other workflows that were designed for human scale. The proper merge Q for example just doesn't make sense at agent scale. What does CI even mean when hundreds or even thousands of changes can merge while tests are running? We haven't been able to effectively replicate the industry standard of a global repo lock on the merge queue. So instead, they've invested in postmerge in pre-eploy automated testing. Look at
- 42:28
that. Theo was right again. Sorry to freak out about these things again, but you guys have no idea how long I've been harassed for thinking it's more important to be able to undo a bad change than to add thousands of layers in front of the changes to make it slightly harder to make a bad change. So
- 42:48
much of engineering has been focused on increasing friction more and more and more to lower the likelihood something breaks. I've been focused on make the friction as low as possible. Both to get code out, but more importantly to fix something when it breaks. Because no matter how strict your processes are, you will ship something broken. I don't care who the you are. If you
- 43:09
actually think you will never ship broken code or that you have built your system in a way where it will never ever have bugs, I hope that your mom's basement doesn't smell too bad because in the real world things break and when things break you need to fix them and focusing on ways to fix things and reduce this damage that broken things can do is incredibly important. They do call out that by working this way
- 43:30
sometimes concurrent changes conflict but Claude can typically fix these conflicts faster than humans can even identify them. I've been thinking a lot about this. The idea I've had is what if the merge Q wasn't PRs that need to be merged one after another? What if the merge Q was a super general here are things that we're happy to have go in. Hey Claude, land all of this. And it
- 43:52
would try to combine like once something has the the merge thumbs up, Claude tries to squash as many of those chunks into individual things as possible like in groups and then pull them into main as effectively as it can dealing with the conflicts as it goes because Claude is now better at dealing with merge conflicts than I am. And now we got an
- 44:12
interesting end here contributing to the wider ecosystem. We see Scion as just one part of the answer to the scaling concerns that increasingly capable AI agents have raised where sharing these details is an example of how teams can tackle similar challenges. Though they don't expect their solution or their considerations to suit every case. They specify this is in no means a suitable replacement for Git and general use. The
- 44:32
concepts I've outlined here really only make sense for truly enormous monor repos with either tens of thousands of engineers or swarms of agents prolifically writing code. Plus, Anthropic's unique in that their ability to spend tokens for internal use far exceeds what any other business would realistically want to spend. Never thought I'd see the day. To be fair, as I've talked about many times, tokens are
- 44:54
getting cheaper, way cheaper. A given level of intelligence is achievable at lower and lower prices constantly. The fable rewrite of bun into Rust was like 300k plus of Fable tokens. I was able to do a bolder rewrite which was Typescript in Rust myself in around 20 to $25,000 of Opus 55 tokens. The speed that these
- 45:15
prices are going down is insane. So, it is absolutely possible by this time next year, you'd be able to write your own similar thing to Scion, your own alternative to Git for your company for just a few thousand in tokens. They also call out a thing that is important to understand. They are doing this both to address their own software development needs, but also to test Claude's capabilities to see what it can do. Practically speaking, it's difficult to offer Scion in a form that would be
- 45:36
broadly useful to other organizations. It diverges significantly from git, meaning companies would need to migrate every piece of the development life cycle and it requires its own code review, code search, and CI. Their internal version of these components are highly bespoke and they would be difficult for other organizations to adopt and they worry that attempting to build a public version that tries to please everybody wouldn't actually be very useful. Okay, I know what this
- 45:58
paragraph is here for. There was a conversation internally about whether or not they could sell what they built. The people who built it said, "No, you. We're not going to do that. Here's all the reasons why it doesn't make sense." Enterprise sales finally probably caved. It's like, "Okay, whatever. We don't have to sell this." And then they wanted to talk about it. And comms is like, "Wait, wait, wait. People are going to ask for this." So, they had to take all
- 46:18
the arguments they had internally and do this so that I can justify now as I read this to you why they are not open sourcing this for others to use. Okay, more justification. Let's see this part. However, they anticipate that this class of problems will become more commonplace over the next 6 to 12 months and it would be an inefficient use of engineering resources and tokens every company using Git to face similar
- 46:38
challenges alone. Given that, we think the most impactful thing we can do is share our thinking so that other devs can apply some of the most successful parts of Scion to Git without departing from the overall model. A Git server doesn't have to work the same way as a Git client does. Novel server architectures exist that provide significant scaling improvements. Git platforms can provide comprehensive
- 46:58
serverside RPCs like Scion, perhaps even in a standardized way across forges that the Git clients can learn to talk to. Negotiation protocol doesn't have to look exactly like it does today either in the various sparse and shallow features in Git can be made to compose well. Lastly, they'd like to acknowledge that the Git community has spent nearly two decades scaling the platform for millions of devs. Their contribution is
- 47:20
far narrower. They're sharing their early views into how agentic software development interacts with these systems. They can partner with GitHub, GitLab, and the open source community to support the changing needs of devs. We're all in it together as the software development life cycle intersects with the AI exponential. It's crazy how something like this would have felt comical a year ago, and now it's almost
- 47:40
like, yeah, of course, makes a lot of sense. I'm going to go fight the urge to make my own git clone. It's going to be hard, but I'll do my best. I hope you enjoyed this deep dive. I know I did. I love these types of nerdy rants and this was a very fun one for me getting to go deep on how source control works and how to make it scale the way that agents need. You still enjoy these deep dives or do you just want to see me vibe code
- 48:01
more? Let me know in the comments. Until next time, he sns.
Description
In just 5 months, Anthropic went from 2,500 merges per day to over 30,000 this September and they wrote a blog explaining just how that was possible with their new system, Scion
SponsorThank you Infinite Red for sponsoring! Check them out here:
Sources:
https://www.anthropic.com/engineering...
Want to sponsor a video? Learn more here: https://soydev.link/sponsor-me
Check out my Twitch, Twitter, Discord more at https://t3.gg
S/O @Ph4seon3 for the awesome edit 🙏
#ai #claude #openai
