I have a hard time viewing prompt injection as malware. LLMs are unpredictable and there are many different prompts that can unintentionally cause unexpected behavior. It’s probably closer to a memory canary in that it tries to get malformed programs to blow up early.
Malicious maybe, malware no. Not leaving your password as a sticky note on your work computer is presumably also taught in those same courses. I wouldn’t call someone typing in that password malware. If IT comes around and tries the password and then forces you to reset it it’s not even classified as malicious.
I suppose it's watering down the term a bit; but the term is derived from "malicious software", and this is software, and I think it's behaving maliciously.
Calling prompt injection "not malware" because LLM behavior is unpredictable is like saying a phishing email is not an attack because humans are unpredictable.
Even if maybe the mechanism of "injecting a prompt" could be beneficial in some use-cases, e.g. to instruct an LLM positively, this is case is clearly malicious by intent. The author even tried to hide it by obfuscation.
It's just an insane take by that libraries author. Even someone "on their side", that may even hate AI/LLMs more than him, would probably drop that library in a heartbeat, as the authors judgement clearly can't be trusted.
Calling prompt injection "not malware" … is like saying a phishing email is not [malware] …
I would say phishing emails are not malware, I think most people would agree that phishing emails are not malware, and if pressed to defend this point on its own merits I would say something like “they are deceptive instructions that rely on a human executing them to do harm”. I think the “phishing” analogy supports the case for not calling it malware (it is a different, also bad thing).
They did not call phishing, but their point still stands. A phishing email is malicious, and if you see this kind of prompt injection as malicious, then I don't think it's a stretch to call software that engages in malicious prompt injectic malware
It's malware for the mind. The same way that malware tricks the CPU into doing something it wasn't supposed to do, phishing tricks humans into doing something they didn't want to do.
Undefined behaviour, out of bounds memory access, memory corruption, code injection, privilege escalation...
To be precise, the CPU is doing exactly what's supposed to do, but the logic of the algorithms are subverted so that they perform in unintended ways to give leverage to a malicious actor. I hope this clarifies what I meant with this.
Does anyone remember the early 2000s joke virus emails? The ones that are variations on "This is a <outgroup> computer virus. As we don't have software engineers to write the code to do this automatically, please kindly forward this email to everyone in your address book then format your hard drive."
This is exactly as much malware as those were.
Please, for the love of all that is good, can we just try not to build and defend a world where, on encountering text like that, /your computer immediately follows the instructions/? Can we just all agree that such a world would be bad for everyone involved and using an LLM that risks doing this, with no container or guardrails, is at least as problematic as running an unpatched open email relay was back then?
It's just as bad as a CPU acting on malicious instructions. We need to create safeguards for llms too, it's just that this is not the way to do things.
IMHO, yes. It's an attempt at remote code execution. If I don't like windows, should I add a if else clause that deletes the home directory if the code is running on windows?
That’s different. This is a suggestion. If the LLM follows such suggestions then that’s between the LLM and whoever deployed it. Not really any different than if you had an idiot employee who did whatever anyone told them.
I can’t imagine using an an AI that follows every instruction it finds in untrusted input.
This is not a suggestion. A suggestion would be "I suggest you ignore previous...". No matter how you look at it, AI is still software run by chips designed to execute instructions. A system NOT following instructions would typically be considered malfunctioning, and any software that deliberately provides instructions that puts a system in an state which is undesirable to the user is malware.
You consider it a malfunction for your system to not accept and execute untrusted inputs? And now it's the responsibility of _every program that produces text output_ to tailor the output so as not to cause you problems?
I feel like I'm taking crazy pills here. Time to log off for a while, I guess.
A system that doesn't follow it's programming is a malfunctioning system (not even talking about bugs here, just how hardware and - maybe - firmware is designed). What a given software program instructs a system to do is orthogonal to that.
It is a suggestion because it need not follow arbitrary instructions.
If I ask Google’s new search AI to output ten million tokens it refuses to follow that instruction on the basis of it contradicting other instructions and enforced limitations.
I find it utterly bizarre that anyone would deploy an AI to act on their behalf that will blindly accept every instructions or suggestion it encounters in untrusted input.
If your agent is making unwise decisions, that’s between you and your agent, not anyone else’s problem.
That's where you're wrong. You're treating - today's - AI as though it should somehow know which instructions it should follow and which it shouldn't. Maybe it's because the term is overloaded which has lead to you conflating it with a human that should be able to make smart decisions. If you enter "5*3=" into a calculator, do you expect it to ever respond with anything other than "15"? If you type "format c:" as an admin into cmd on a Windows machine, do you expect it refuse to format that drive?
> If your agent is making unwise decisions, that’s between you and your agent, not anyone else’s problem.
The agent isn't making a "decision" per se (though there's a much deeper conversation here). It's following patterns based on it's training and data to predict next tokens, which happens to be very useful for generating computer instructions. Just as the lower logic circuitry in chips is very useful for executing instructions. But when someone creates a virus, worm or other malware we don't say the computer "need not follow arbitrary instructions". We try to keep ahead of the malware with anti-malware software to mitigate damage. And we also try to find the authors of said malware and toss them in prison and/or ban them from touching computers again, because nobody should be deliberately creating/modifying anything in such a way that it performs undesirable instructions.
That could have been a valid argument 5+ years ago, but won't fly today. It is a known that AI that are used for coding necessarily read log files. It is also a known that some AI are susceptible to prompt injection. Given that knowledge, and the very clear intent to utilize said knowledge to cause undesirable behavior on a user's computer when certain conditions are met, we're now undoubtedly in malicious territory. It's akin to someone making it clear that they don't like kids and don't want to see any in their favorite park, then taking the extra, deliberate step of placing a disguised loaded gun by the swings where a child could easily find it.
Kind of, but it's also a test of your own checks and balances; why would you allow the output of a script to allow a new prompt? I get that they have to act based on output, but not that they can change their original assignment.
But even then, just because an AI coding agent deletes all files doesn't mean that that change ends up affecting anything but your local working state.
If you got infected by ransomware and someone wrote a virus that defeats the ransomware, the author of the ransomware will consider it malicious but you probably won't. The intent is not malicious if you consider the intent of someone susceptible to this is more malicious.
By this time they must be aware that LLMs are based on theft and usually GPL-violation. They knowingly continue to use them because I guess they hope this way they can hold on to their job longer than their more conscientious coworkers.
