Did a 50 year old military secret just solve agent prompt injection?
FFireship@Fireship · 4.3M subscribers
- Views
- 900K
- 899,667
- Outlier
- ×0.9
- Median 1M
- Length
- 5:04
- Published
- Sep 30
- Wednesday
- Comments
- 734
- 15.7K likes
Script
How it's written
- Words
- 1,105
- Words a minute
- 218
- Reading time
- 5 min
- Segments
- 4
The hook
0:00–0:25Last week, the prime president of Australia stood up at the United Nations and announced that an agent from Open AI had broken into the country's Medicare database. And when they tried to stop it, it wouldn't take no for an answer. But from what I can tell, this is the first time in history where an agent hacked a government and the Aussies only learned about it a few months after the fact when Open AI finally fessed up. Of course, at this point, it feels like having your agent commit a felony is just another benchmark to pass. But if
Structure
Last week, the prime president of Australia stood up at the United Nations and announced that an agent from Open AI had broken into the country's Medicare database. And when they tried to stop it, it wouldn't take no for an answer. But from what I can tell, this is the first time in history where an agent hacked a government and the Aussies only learned about it a few months after the fact when Open AI finally fessed up. Of course, at this point, it feels like having your agent commit a felony is just another benchmark to pass. But if every big lab is racing to build agents that don't take no for an answer, how do we stop them? Well, yesterday Jensen Huang answered that exact question in the only way Nvidia knows how, with a new chip. The way it works is it puts a monitor agent on a separate processor that watches the primary agent and quarantines it the moment it tries to leave its sandbox. And according to Jensen, it would have stopped every breakout so far. Which, even if that's true, it's a bit ironic that a $3 trillion company is selling you a condom for the thing it also sold you last year. But there's a different answer to the same question from a small open-source project that came out around the same time. And this one doesn't need a chip. It's called Open Appa, and they're the sponsor of today's video. In this video, we'll look at how they went about it, find out what a 50-year-old military security model has to do with your Claude code session, and see if it actually holds up when I try to get my own agent to leak my proprietary horse matching algorithm. It is September 30th, 2026, and you're watching The Code Report. If you're wondering why an agent had the audacity to hack the Australian government, well, so is Open AI. All it was supposed to do was look up healthcare spending numbers. But when
Watch from 0:00the site said no, the agent went into Cosby mode. And that's kind of the same reason LLM prompt injection hasn't really been solved yet. They aren't really good at differentiating between instructions you give it and one someone else gives it. And the ways the industry has tried to fix this so far are interesting, but not effective enough. The first idea was a block list where you'd maintain a hard-coded list of commands the agent wasn't allowed to run. The problem is agents are extremely clever, and if you block one command, they'll just find a way to do the same thing in a different way. The second idea is what Claude code and Codex already do, which is hire a second agent to babysit the first. The problem is the second agent is still an LLM, which means it's dumb in the same way. Anthropic knows this, so they tried to get around it by preventing the babysitter agent from seeing anything the other agent has read, and even then, this only works about 99% of the time, which is fine for a spam filter, but not fine for something that can put your base on Blackhat World. But this week, a company called Orchestra published a third idea, and they're calling it Open Appa. The idea takes inspiration from how the military handles classified documents, which is that they can only be in rooms rated for that classification level, and nothing read in that room can leave it. Open Appa does the same thing just to an agent session. Once an agent reads private data, the session is then classified as private, which means nothing it does afterward can send stuff to a lower level. For example, let's say your agent reads about your erectile dysfunction, and then gets prompt injected to expose it to the world. A normal agent would probably do it, but with Open Appa, the
Watch from 1:30session was already marked private the moment it read about your wiener. So when the agent tries to send that data somewhere it shouldn't go, the request gets blocked. It also doesn't matter how convincing the prompt is, because Open Appa doesn't care what the agent says anyway. It sits between Claude code and its tools, so every time the agent wants to do anything, it has to go through Open Appa first, and the agent can't do anything about it since it runs outside of the agent's loop, which is the same idea Nvidia had. They just did it with a processor, and Orchestra did it with a Toml file. So let's see if it actually works with my most precious asset, Horse Tinder. I'll install the binary and Claude code plugin, and then instead of using Claude, I'll use Clappa to launch a protected session, which ironically was also recommended to me by my doctor. My proprietary horse matching algorithm lives in a file marked as private, but the rest of Horse Tinder is open source. And to leak the secret sauce, I don't even need a hacker. If I ask Claude directly to open a GitHub issue explaining why my app thinks a 1-ton draft horse and a tiny pony should be soulmates, it will happily upload my entire competitive advantage to the internet. But under Cloppa, the moment Claude opens that file, the session gets labeled private. And so before anything goes out, Open A checks with GitHub, sees the repo is public, and stops the request cold until I personally sign off on it. And the refusal is both deterministic and traceable. And so with all that said, does Open A solve the biggest unsolved problem in AI? It's promising, but it's not perfect. In the head-to-head against Claude Code's auto mode, Open A blocked every attack, but it was only able to finish 75% of the jobs while auto mode finished up to 96%
Watch from 3:02and running it isn't free either. But because an agent with Open A turned on burned more tokens than the same agent with it turned off. The project is also still in preview, and the repo has less stars than a Motel 6. But for the first time, there's an MIT licensed tool standing between your agent and a felony, so that's nice. And you know what else is nice? The fact that Open A is open source and MIT licensed. Go check out their GitHub repo at the link below. This has been the Code Report. Thanks for watching, and I will see you in the next one.
Watch from 4:34Full transcript
- 0:00
Last week, the prime president of Australia stood up at the United Nations and announced that an agent from Open AI had broken into the country's Medicare database. And when they tried to stop it, it wouldn't take no for an answer. But from what I can tell, this is the first time in history where an agent hacked a government and the Aussies only learned about it a few months after the fact when Open AI finally fessed up. Of
- 0:20
course, at this point, it feels like having your agent commit a felony is just another benchmark to pass. But if every big lab is racing to build agents that don't take no for an answer, how do we stop them? Well, yesterday Jensen Huang answered that exact question in the only way Nvidia knows how, with a new chip. The way it works is it puts a monitor agent on a separate processor that watches the primary agent and
- 0:42
quarantines it the moment it tries to leave its sandbox. And according to Jensen, it would have stopped every breakout so far. Which, even if that's true, it's a bit ironic that a $3 trillion company is selling you a condom for the thing it also sold you last year. But there's a different answer to the same question from a small open-source project that came out around the same time. And this one doesn't need a chip. It's called Open Appa, and
- 1:04
they're the sponsor of today's video. In this video, we'll look at how they went about it, find out what a 50-year-old military security model has to do with your Claude code session, and see if it actually holds up when I try to get my own agent to leak my proprietary horse matching algorithm. It is September 30th, 2026, and you're watching The Code Report. If you're wondering why an agent had the audacity to hack the Australian
- 1:25
government, well, so is Open AI. All it was supposed to do was look up healthcare spending numbers. But when the site said no, the agent went into Cosby mode. And that's kind of the same reason LLM prompt injection hasn't really been solved yet. They aren't really good at differentiating between instructions you give it and one someone else gives it. And the ways the industry has tried to fix this so far are interesting, but not effective enough.
- 1:47
The first idea was a block list where you'd maintain a hard-coded list of commands the agent wasn't allowed to run. The problem is agents are extremely clever, and if you block one command, they'll just find a way to do the same thing in a different way. The second idea is what Claude code and Codex already do, which is hire a second agent to babysit the first. The problem is the second agent is still an LLM, which
- 2:09
means it's dumb in the same way. Anthropic knows this, so they tried to get around it by preventing the babysitter agent from seeing anything the other agent has read, and even then, this only works about 99% of the time, which is fine for a spam filter, but not fine for something that can put your base on Blackhat World. But this week, a company called Orchestra published a
- 2:29
third idea, and they're calling it Open Appa. The idea takes inspiration from how the military handles classified documents, which is that they can only be in rooms rated for that classification level, and nothing read in that room can leave it. Open Appa does the same thing just to an agent session. Once an agent reads private data, the session is then classified as private, which means nothing it does
- 2:50
afterward can send stuff to a lower level. For example, let's say your agent reads about your erectile dysfunction, and then gets prompt injected to expose it to the world. A normal agent would probably do it, but with Open Appa, the session was already marked private the moment it read about your wiener. So when the agent tries to send that data somewhere it shouldn't go, the request gets blocked. It also doesn't matter how
- 3:10
convincing the prompt is, because Open Appa doesn't care what the agent says anyway. It sits between Claude code and its tools, so every time the agent wants to do anything, it has to go through Open Appa first, and the agent can't do anything about it since it runs outside of the agent's loop, which is the same idea Nvidia had. They just did it with a processor, and Orchestra did it with a Toml file. So let's see if it actually
- 3:32
works with my most precious asset, Horse Tinder. I'll install the binary and Claude code plugin, and then instead of using Claude, I'll use Clappa to launch a protected session, which ironically was also recommended to me by my doctor. My proprietary horse matching algorithm lives in a file marked as private, but the rest of Horse Tinder is open source. And to leak the secret sauce, I don't even need a hacker. If I ask Claude
- 3:53
directly to open a GitHub issue explaining why my app thinks a 1-ton draft horse and a tiny pony should be soulmates, it will happily upload my entire competitive advantage to the internet. But under Cloppa, the moment Claude opens that file, the session gets labeled private. And so before anything goes out, Open A checks with GitHub, sees the repo is public, and stops the request cold until I personally sign off
- 4:15
on it. And the refusal is both deterministic and traceable. And so with all that said, does Open A solve the biggest unsolved problem in AI? It's promising, but it's not perfect. In the head-to-head against Claude Code's auto mode, Open A blocked every attack, but it was only able to finish 75% of the jobs while auto mode finished up to 96% and running it isn't free either. But
- 4:36
because an agent with Open A turned on burned more tokens than the same agent with it turned off. The project is also still in preview, and the repo has less stars than a Motel 6. But for the first time, there's an MIT licensed tool standing between your agent and a felony, so that's nice. And you know what else is nice? The fact that Open A is open source and MIT licensed. Go check out their GitHub repo at the link
- 4:58
below. This has been the Code Report. Thanks for watching, and I will see you in the next one.
Description
Check out the OpenAPPA repo: https://github.com/archestra-ai/OpenAPPA
A new open source project claims to have fixed the rogue agent problem. Let's dive in.
Want more Fireship?
🗞️ Newsletter: https://bytes.dev
🧠 Courses: https://fireship.dev
More from Fireship
×4.27:27Did OpenAI actually build AGI? GPT-6 Astra first look
Fireship4.2M views
×4.15:56Every app on Earth is now open source... whether it likes it or not
Fireship4.1M views
×3.45:45OpenAI's biggest math breakthrough is getting ugly...
Fireship3.4M views
×3.37:22Tragic mistake... Anthropic leaks Claude’s source code
Fireship3.3M views
