All too common... It's sad yet understandable how a company would not prioritize security.
At the same time, it's also understandable how a security start-up, upon (rightly) getting fed up waiting, decide to publicly disclose, as a way to scrape some PR out of the sunk cost. Public disclosure has a place. But if you truly care about helping, you could do more than bumping on HackerOne and messaging the CISO once on LinkedIn.
Maybe I'm too cynical but it truly feels like nobody actually cares at this point.
Hah, not trying to pass off as human. Just communicating with my fellow men in black ;)
To be as explicit as possible: whether disclosing this publicly actually did more good then harm is not that clear cut. Even if accounting for all the second order effects.
Regardless, as a business you'd still be compelled to publish, because you've already poured resources into this research, there's still a chance to gain something, and there is enough plausible deniability about your true priorities.
We're in the worst of all worlds. Sometimes it feels like you only know it's a human because the AI would have over-explained.
But yeah, this security company only prodded, what, once or twice a month for 7 months? I mean, if they really truly cared, they would have found the CISO's home address, broken into their house, painted the reproduction steps on the inside of their front door (to avoid accidental disclosure), created a few "beginner friendly" repos with a git.exe that DDoS's their auth servers, got a job as a night cleaner in their offices, waited for one of the developers to leave their machine unlocked then fixed the vuln themselves.
It's just another capitalistic money grab, them posting their security concerns. Ugh.
This is very similar in root cause and exploitation to Copy Fail.
Which illustrates pretty well something that's lost when relying heavily on LLMs to do work for you: exploration.
I find that doing vulnerability research using AI really hinders my creativity. When your workflow consists of asking questions and getting answers immediately, you don't get to see what's nearby. It's like a genie - you get exactly what you asked for and nothing more.
The researcher who discovered Copy Fail relied heavily on AI after noticing something fishy. If he had to manually wade through lots of code by himself, he would have many more chances to spot these twin bugs.
At the same time, I'm pretty sure that by using slightly less directed prompting, a frontier LLM would found these bugs for him too.
It's a very unusual case of negative synergy, where working together hurt performance.
The RxRPC one is definitely a different root cause (although caused by a very similar mistake).
For the ESP one it's a bit harder to tell. I don't think the wrong thing was fixed, just that there was a very similar bug in almost the same spot. Could be wrong about that though.
> When your workflow consists of asking questions and getting answers immediately, you don't get to see what's nearby.
Very much aligns with my experience. For me this is the most unsatisfying thing about AI-based workflows in general, they miss stuff humans would never miss.
All the time I wonder what am I missing that's right nearby? It's remarkable how many times I have to ask Claude code to fully ingest something before it actually puts it into context. It always tries to laser through to target it's looking for, which is often not what you want it to look for, at least not all you want it to look for. Getting these models to open up their field of vision is tough.
Actually lately I’ve been feeling the other way around with it. The LLM catches things I would have overlooked. I ask for a new feature in a certain file, and the LLM suggests fixing a tangentially related file to accommodate the new feature without breaking something else.
Maybe this is just the crap legacy codebase I’m working with and how tangled up everything is, but I definitely have found several times now that it caught things I would have missed.
> The LLM catches things I would have overlooked. I ask for a new feature in a certain file, and the LLM suggests fixing a tangentially related file to accommodate the new feature without breaking something e
What are you using? Do you think this behavior is in response to prompting? My goal at times is to "rabbit hole" the LLM to get it to go down rabbit holes and find bigger and bigger picture issues until it homes in on something fundamentally broken that could have big impact if fixed. But it's not trivial to push the agent in that direction for me.
Do you think this is inherent or an artifact of prompting? Curiosity and side quests leads to higher token usage and longer time to finish, so I could understand why current harnesses and system prompts would not encourage that sort of thing.
But what if a coding agent was prompted to be more curious during development? Like a human developer, make mental notes of alternatives to try out and chase suspicious looking code which may seem unrelated to the task at hand. It could even spawn rabbit hole agents in parallel.
Taking a step back, this probably highlights major hazard with the increased usage of LLMs for coding, which is that everyone's style of work is going to converge because most code will be written by the 2-3 most popular models using the same system prompts.
> But what if a coding agent was prompted to be more curious during development?
I've tried using the language of curiosity. My qualitative take was that it did have a positive impact, but not much. And I can only tinker with system prompting so much, before I get drawn into LLM driving :)
> which is that everyone's style of work is going to converge
yeah I imagine even people's styles of thinking will converge as a result of this, more so than from reading other people's prose or programs. I think I saw something on HN to this effect within the last month, too.
It’s interesting to compare how the agentic search performs, with these targeted reads and lots of tool calls in the stream, versus the older but still valid paradigm of using a high-reasoning model like GPT-X-pro and feeding in all the relevant files at once with no tools.
I have found that the “pro” approach is much more holistic and able to tackle rather “creative” problems that require very careful design and the overall artifact is tight and self-consistent. — Claude Code by comparison is incredible in exploration and targeted implementation but indeed is not great at seeing the forest.
> All the time I wonder what am I missing that's right nearby?
Add to the prompt "use coding conventions of the file which you are currently editing". That gets the machine (Opus and Sonnet at least) to go over the nearby code and occasionally mention something obvious.
I don't follow. LLMs spotted these bugs in the first place. You seem to be saying that these discoveries are indications that they're bad for vulnerability discovery.
From what I understand, the copy fail bug was found by researcher who noticed something weird and then using AI to scan the codebase for instances where that becomes a problem.
I bet that with a slightly looser prompt/harness, the LLM could have found these twin bugs too.
Yet at the same time, I also think that if the human researcher had manually scanned the code, he'd have noticed these bugs too.
FWIW I do think LLMs are great tools for finding vulnerabilities in general. Just that they were visibly not optimally applied in this case.
I don't think the copy.fail people understood the issue they found, as is evident by the heavy focus on AF_ALG/aead_algif, which is essentially "innocent" as we're seeing here.
I think LLMs are great for vulnerability discovery, but you need to not skimp on the legwork and understanding what even you just found there.
It's an AI security firm! You might just as productively ask "why did all the other engineers who ever looked at this code not find it, and why was Theori the one to actually surface it?".
They do not get bored like a human but they are trained on human language and replicate the same traits, such as laziness, and expressing boredom or annoyance (even if obviously they do not experience anything at all). It’s actually a lot of effort to get them to engage with things at a deeper level without skipping corners
I’m hardly going to simp for LLM tools but the fact that the bug existed and no one had reported it seems proof positive no one was about to find it without them
Am I missing something? Where does it say that the researcher that found Dirty Frag used LLM to find it? Have you read the original report from the researcher?
No, they did not. Careful of falling for the psychosis.
> This finding was AI-assisted, but began with an insight from Theori researcher Taeyang Lee, who was studying how the Linux crypto subsystem interacts with page-cache-backed data.
You appear to want to die on the hill of "This vulnerability would never have been found if we lived in a world without LLM AI" which is a very strange hill to die on.
There's no question that we live in the world where LLM AI was involved in finding the copy fail vulnerability at this specific time, and it's completely normal for people to see a vulnerability and then look closer and find related vulnerabilities or a deeper root cause, but there's no need to adopt an extreme "without AI LLM we don't find these vulnerabilities" position.
It's weird to say I want to "die on this hill" because that's not even something I believe. There was nothing especially difficult about this particular vulnerability. My only observation that nobody did find it before, then an LLM security firm went out looking for Linux LPEs, and thus it was discovered.
That is a very difficult fact pattern to which to attach the conclusion "LLMs have sabotaged security research" (my paraphrase).
Well.. every new vulnerability is one nobody did find it before.
Otherwise, it won't be classified as "new"
--
Edit:
I think LLM is very useful here.
When a researcher spot something funny, instead of spending two days on reading and testing, he can fire up a LLM and have it read all the code lead to there in ~30 minutes.
The finding started with human intuition and was assisted by an LLM. You can yell "AI sec firm" 1000 times. A human got it started. You shouldn't die on that hill.
Of the MANY things I've completed in the last year that I would never have done without an LLM, a human got 100% of them started. The ideas were mine in every case.
But it is still a fact that I have been taking on all sorts of tasks I would never have taken on if I didn't have power tools.
My comment was solely about the correct attribution who made the initial finding. It's not a comment about the value of AI. I think we can get facts right and still argue for or against AI.
That's a pet peeve of mine as well, the inability to discuss facts / correct mistaken things on the internet, if the fact/mistake is on the "wrong" side of an argument.
I don't think I'm doing that though.
The context is a discussion about whether it would have been found without the assistance of an LLM. I agree that further upthread there maybe some misattribution but it is not present in the post you were directly replying to and it is not really the argument being made.
His whole sentiment was yelling several times "LLMs did this". He wanted to smuggle his pro AI attribution in, one way or another. In that way I could also argue "without humans, we wouldn't have LLMs." But it doesn't have value, right? I don't know why some try so hard to play down any human impact in this context. LLMs can help to find bugs. Without broader context it's a good and interesting thing. There is no need to trample over everything left and right just to overhype it.
Germane or not the knee-jerk reactions related to LLMs are getting ridiculous and it seems like it’s the same people throwing down at a moments notice and then chalking it up to a misunderstanding.
Right. Finding the bug is in itself a win. It seems we’re jumping from that spend-electricity-to-find-bugs win to arguing about how some things around it are not quite good or comfy.
Or a follow up prompt: "find similar classes of bugs". Once the actual case has been layed out finding like bugs isn't too hard. I hear you on the creativity bit. Like any tool, AI can put blinders on. Using it to augment without it fully taking over your workflow is tough.
Not just like any tool though. Interacting with agents can be incredibly boring and frustrating in a way that I personally do not experience with other technology
Just on a side note. Negative synergy does not seem so uncommon with machine learning. We did some research maybe 10 yrs ago an human/ML based duplicate detection (for a municipal support ticket system) . Research showed that pure AI and pur human outperformed co-working. Human oversight often e.g. overcorrected machine work. I think it is a nice HCI problem to solve actually to amplify creativity and unique skills in such processes. Particularly if they can be to some degree repetitive and tiresome.
I don't know... after they found a high profile bug like copyfail, I wouldn't attribute not looking for similar bugs to them being overly dependent on AI. It's easy to stop exploring, for a while at least, after you've struck on a major find. Maybe they would've returned to it in a few months. It certainly inspired others to explore similar areas and find these new bugs. Isn't that enough?
These are all page cache poisoning attacks (dirtyfrag, copyfail, dirtypipe). Maybe the page cache should have defense-in-depth measures for SUID binaries?
True! Building protections (e.g. physical pages in the page cache are not writeable 100% of the time) just for executables has of course countless circumventions as well (e.g. config files). Yeah, there is probably not that much to be done there, actually. Looking at some of the diffs it seems to me like the kernel makes it really not particularly obvious when/how this goes wrong. E.g. the patch for this is to look at an additional flag on the socket buffer to fix an arbitrary page cache write. This feels rather action at a distance. Logically this of course makes sense, the whole point of splice et al is to feed data from one file-like into another file-like, whatever those ends might be. That erases the underlying provenance of the data.
splice is documented to return EBADF if "One or both file descriptors are not valid, or do not have proper read-write mode."
So it seems surprising to me that you can call it when the out fd is not writable? But I didn't retain the information about the vulnerability, so I'm missing something. There was something about copy on write, IIRC?
"proper read-write mode" for the input fd is reading only. The exploit is writing to the splice() input fd.
Also, NB, I said permission check, not mode check. The input fd to splice can and will be open for only reading quite often. Doesn't mean the kernel can't still do a write permission check.
(Except I didn't say that here. Oops. Getting confused with my posts.)
OK, I may likely have too much sleep debt to understand, but given the bug is that splice can write to the input fd, you're suggesting maybe splice should only let you use an input fd if the process has access to write to it?
But splice is a more or less a generalization of sendfile, and sendfile is often used for webserving where the serving process does not have ownership of the documents it is serving. It doesn't make sense to limit splice such that it can't do the task it was built for. Maybe splice should just not write to the input fd? :P
> But splice is a more or less a generalization of sendfile
Not really, splice(2) is actually more limited, it's an optimisation for reading and writing data between files and pipes without needing to make copies.
sendfile(2) works with any fds because it just exists to remove a fair bit of the copy overhead when doing a userspace read/write loop, but it does actually do a copy.
> When your workflow consists of asking questions and getting answers immediately, you don't get to see what's nearby.
That's why is very very important to just step out and use saved time to go for a walk, to a park, sit on a bench, listen do birds, close eyes and zoom out.
Maybe. This phenomenon of security holes being found closely to each other was also common before LLMs. People attention gets directed to a place and typically more issues are getting found.
System partitions being non-writable has nothing to do with the vulnerability - it allows modifying the cache of any file that you can open for reading.
Not using setuid anywhere means you'd have to build a slightly more clever exploit, but it's still trivial - just modify some binary you know will run as root "soon".
But... I didn't check, but IIRC the untrusted_app secontext that apps run in is not allowed to open AF_ALG sockets - so you can't directly trigger the vulnerability as a malicious app. Although it might be possible in some roundabout way (requesting some more privileged crypto service to do so).
> Open source models found the same bugs? Sure, if you tell them "here is a for which may contain a vulnerability, look for a big in how function XYZ handles ABC"
In one of Anthropic's blog post, they describe that that's basically what they did too. They run the agent many times, each time specifying a different file to focus on. [1]
From my experience as a security researcher, manually finding a fishy file and sicking even sonnet 4.5 yields great results for most memory corruption bugs.
No comments otherwise. I don't have a clue as to who that guy is, and I haven't watched the video yet. You might be right overall.
Yes, but Anthropic didn’t already know the answers. In the OSS ‘reproductions’, they fed the model the one file that actually has a vuln and even told it which parts of the code to focus on. This is obviously a much easier task.
If OSS models are equally up to the task, why not find novel vulnerabilities?
Yeah, totally agree now that I've looked into it more.
> If OSS models are equally up to the task, why not find novel vulnerabilities?
To be fair, in the same blog post Anthropic mentioned costs in the tens of thousands of dollars per project looked at it. So it's a big ask to do an experiment that compares. Would love to see it though.
Security efforts are not evenly distributed, even within a single project. This includes both the thinking that the developers put in, and the scrutiny given to a piece of code by researchers.
The initial batch of publicly disclosed vulnerabilities by Mythos demonstrates that perfectly. None of the bugs themselves are especially interesting or complex, in my opinion. They were found by applying effort to a very large amount of code which included under-scrutinized areas, where bugs hid. Yes, even in projects like Linux and OpenBSD there are many pieces of code that aren't that properly vetted, because of the finite amount of developer/researcher time allotted.
The fact that this effort is much cheaper does indeed change things. But really strong sandboxing solutions, such as gvisor or firecracker, do a really good job of having very little attack surface, all of which is heavily scrutinized.
Until we see more of the bugs that were found, it remains to be seen whether or not the post's premise about sandboxes is correct.
Fuzzing data flow separately from control flow is an interesting idea
I can believe that it dramatically speeds up finding certain bugs, but I doubt that it can reach a large class of complex vulnerabilities, which in the case of high value targets is probably all that's left.
The PHP interpreter isn't much of an interesting target, since it (usually) doesn't accept user input, even if it does power a significant part of the web.
For that reason, it's much less researched and still has low complexity bugs.
More robust interpreters such as JavaScript's V8 will probably fare much better against data flow only fuzzing. Bugs in V8 tend to combine both data flow and control flow[1].
Also, although you can't trivially cross wires, you can create a wire crossing using a few XOR gates [1]
[1] https://cs.stanford.edu/people/eroberts/courses/soco/project...