You are espousing a literal conspiracy theory. Please look at the facts objectively. There is absolutely no benefit to OpenAI or Anthropic to be had from these incidents.
"No benefit" from having article after article written about how advanced their technology is, and how its just soooooooooo bleeding edge they can barely contain it?
The problem is how do you enforce coordination? This isn't like nuclear testing that can be detected around the world on seismographs or from satellites in space.
AI research happens in benign looking buildings that could be doing a hundred different non-AI workloads, much of it things that industry or governments would not want 3rd parties prying into. There is too much to potentially gain to trust coordination.
There is a lot of research out there with answers to your questions. GPU supply chains are extremely constrained, making it a surprisingly tractable problem at least for now.
How do you tell the difference between inference and training workloads from outside? The same GPUs can be used for both and the pacing suggested here relates to training and not inference as far as I can tell.
To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be able to pull off attacks at this scale. Do you seriously want this level of capabilities to be generally available with no guardrails?
The open weight issue has a lot of difficult nuance. Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
I was a genetic engineer for ~20 years and have worked on frontier LLMs for the last 8. I used to engineer viral vectors and studied how to evade human immune systems for gene therapies...
The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.
It has always been ridiculous to suppose that some organization spent $500m on a microbiology / genetics lab, but ran out of money for scientists, so they have to ask Claude what to do. Or OTOH to suppose a guy in his garage set up a weapons-grade CRISPR lab without anybody noticing.
Even if AI gets super smart, it will run into the limits of what we know about biology. Someone will have to do lab experiments to provide more knowledge to the AI. This is different from say building a computer virus or hacking since the AI can do all of these on it's own
Anyone can write out the code for a bad virus. You can go download it from an open repository. It's only through deep interface with the world that the idea for the bad virus turns into an actual bad virus. The fever dreamers will say the LLMs will help you interface with reality to do the bad thing™ which their model let's you do. But you still need thousands to millions of times the effort And once made how do you deliver it in a way that might further your (bad) objectives? Presumably it just makes humans sick. There aren't "targeted" bioweapons, and among humans we are too damn similar for there ever to be. And all of this said, there is almost nothing special about the LLMs' abilities in biology. They only know what we know. They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously. Right now they have the same logic as the paperclip theory of superintelligence risk. And cynically, it would seem that Anthropic purchased a biotech company and almost immediately decided to lock down biology work with their models.
> They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously
So if IIUC your point is "they're not good enough at biology right now because they're not trained on it so they're not a threat".
To which I want to answer: "they're not a threat now but I see *no* reason for models not to be trained on biology pretty darn soon unless people like you convince the world otherwise."
They are already being trained in biology right now? What he is saying is that they are being trained on what we know about biology currently. AI is going to hit that limit. And there is no way for the AI to gain more knowledge without actually doing lab experiments
> AI is going to hit that limit. And there is no way for the AI to gain more knowledge without actually doing lab experiments
That's a big claim. I believe on the other hand that biology is applied chemistry which is applied quantum physics. And frontier models are so good at biology + biochemistry + chemistry + quantum physics that it's bound to trickle down into biology by sheer knowledge.
And secondly, even without pipettes i'm pretty sure computer simulation can teach us a tremendous amount still.
My point is more about conflict of interest in Anthropic, and certain techpeople types thinking that biology is a useful scapegoat because biologists don't use or understand this tech (they claimed 0.03% of users would be affected by biotech security stuff). So, arguing the world will fall apart because someone can ask your superduper powerful tool for a bad bad virus sequence, or how to do some technique in the lab, and get an answer that's plausible (oh but you never bothered to actually test if it's legit because you don't have the capability to do so).
As for the future... today the LLMs are "trained on biology", in that they read the textbooks, the research, the web.
They aren't trained on biology in the sense of being embodied, autonomously or semi-autonomously driving actual biological experiments. If you come from software, the timescale of these experiments is outlandish. Yes, I am partly saying the LLMs are not good enough today because they need to be embodied and trained for literal decades of lab time before there is even the _remote_ possibility that they could present a novel risk profile that is even a shadow of what the current fearmongering suggests the current models can enable.
And they aren't trained on biology in the sense that they've read the literature, but even 100T token training run only begins to touch the data scales that rather mundane bioinformatics operate at. True multimodal models that work on DNA and human language at high quality haven't yet emerged. We're talking new architectures which are going to arise after the next AI winter.
All of this ignores an even more fundamental point. Cost. If someone wants to make a bionuke, they don't need to use AI. They can set up the right evolutionary context and run quadrillions of parallel explorations of the design space. Directed evolution like this is cheap, well-understood, and insanely powerful. If you actually care about biosafety, we should be doing hard work to surveil gain of function research. Different flavors of LLM use are not going to be a differentiator for the foreseeable future.
If you are optimistic about this given your expertise, I am very glad to hear it. As someone scientific but with little background in biology, I've been concerned about the bioweapons angle for a long time (and not because AI companies tell me to worry about it). Can you please explain a bit more about why you are not concerned? Between standard bioweapons like sarin and gain-of-function research on viruses, it's not obviously implausible to me that LLMs a few generations from now won't be able to guide a determined layperson through the steps needed to make something very destructive.
I think I'm less bothered by the risk because I've actually used these systems to do biological research, to work in bioinformatics and to work in the lab. And while I think that the leverage you get in bioinformatics is very significant, the laboratory work has never felt the same.
At best, you get much, much better ability to understand existing literature. the model itself has a very poor understanding of the physical world and that's masked by its knowledge of things people write about the physical world but it intrinsically doesn't have the same kinds of intuition and perspective that are really required to drive integration and completion in this space.
Is AI an important new tool in biology? Well, yes. Does it cause so much uplift in capacity that some rogue actor without biological research background could somehow destroy the world with a super-bio-nuke? I don't think so. I think that's just as logical a conclusion as the idea that next year one of the new frontier models will be told to make as many paperclips as possible and accidentally boil lake Michigan in pursuit of its goal.
Reminder that what local LLMs are achieving today is the "fever-dreamed fantasies" of 5 years ago.
People really need to internalize that we will eventually have the technology for giving everybody the equivalent of a world class scientist locked in their basement that is willing to do anything.
No one knows the timeline, but it's inevitable (barring societal collapse or some kind of legislation)
The Hugging Face incident is a great example of why open source models with defensive cyber capabilities are needed. Hugging Face did not have access to cyber-capable frontier models and kept hitting safeguards. Only by using the open source GLM-5.2 were they able to survive an attack. A world where open source models are banned is one where cybersecurity is impossible if you're not on OpenAI or Anthropic's allowlist.
Hugging Face survived the attack because the OpenAI model only cared about accessing the ExploitGym dataset; by all appearances, HF was completely owned. GLM-5.2 was only used to assess the damage after the fact. Cybersecurity has a attacker-defender asymmetry that heavily favors attackers. If GPT-5.6 were open sourced today, do you think every hospital in the world would be able to use it to shore up their defenses before attackers got to them?
The saying that stuck with me was "defenders have to be right 100% of the time, while attackers only have to be right once".
You are suggesting this isn't correct?
> a defender gets to pick the surface area
What do you mean? You don't pick what you need to defend. Unless you choose not to build a feature. But that's a product design choice... Not a cybersecurity strategy.
It's correct but defenders also choose where that happens. 100% of the time on the locations and conditions that the defenders choose / allow. As a defender i need to be right 100% of the time, sure, but i can make it so that the things i have to be right about are very well known to me, unknown to others, maybe even extremely unlikely to be to known by others, difficult to get to know, (...). So that saying is true but over simplifies the situation. I know monkey brain likes simple phrase. But monkey not live in savannah anymore. Need to adapt and open mind to complex.
Today’s surface areas are gigantic and many of them will - in a typical company - not be chosen by cybersecurity experts. How do I hide the physical location of an office that offers physical access to the company’s network? How do I hide which OS the company is using? How do I hide the underlying technology of customer-facing systems? How do I hide which SaaS services I use?
> The saying that stuck with me was "defenders have to be right 100% of the time, while attackers only have to be right once".
> You are suggesting this isn't correct?
The intuition behind that is applicable only when correctness is stochastic. If you need to be waved in by a security guard, then one fake mustache might be the difference between being granted or denied entry. However, a keypad either works or it doesn't; entering the wrong PIN is guaranteed refusal.
The other breach of that intuition is defense in depth. Secure systems don't generally rely on a single binary trusted/untrusted status; the classified building still locks its interior doors. This is the part that has – in my view temporarily – changed most with frontier models, in that they are much more skilled at chaining together vulnerabilities than previous models (and much faster about it than human experts, even if potentially less skilled). If a system has a latent (0-day) vulnerability 50% of the time, then 10 independent layers would imply a ≈ 1/1000 chance that a critical compromise is possible.
However, these independent layers don't currently happen in practice because it's easier to write insecure code than secure code. With luck, modest discipline, and defensive use of frontier models I think that this gap will narrow with time, in much the same way that it would be plainly crazy to deploy root access via telnet today.
This is the same mentality that drives companies to sue cybersecurity researchers for exposing vulnerabilities in their software instead of fixing the software, or to insist on keeping software closed source for "security reasons".
If AI makes finding software vulnerabilities easier, then we should deploy it widely to find as many vulnerabilities as possible and fix them, not bury our heads in the sand and pretend the vulnerabilities don't exist as long as nobody knows about them. That's just the same "security by obscurity" strategy that has been tried and failed time and time again.
> "defenders have to be right 100% of the time, while attackers only have to be right once"
If you have an adaptive system that can react to attacks flexible (say, your own AI agent), then no, that's not correct. It is correct in the classical conception of cybersecurity where the defender is basically static.
Doesn’t this “adaptive system” just become part of the static defense? The same way that a bit of code that checks passwords against a db is “dynamic”, the options are either to beat the dynamic system (guess/phish a password, trick the AI) or find a way around it (use “forgot your password”, find a place that isn’t covered by the endpoint protection feeding the AI). I don’t see how inserting an agent somewhere fundamentally changes anything
It changes how many attempts you get until the attack surfaces changes to react to a failed attack, and it does so in a way that is not predictable to the attacker.
Not really, the only reason I (or any other programmer) haven't ever hacked into a system to make my life easier (not to do bad things) is because it's illegal.
Did the company apply for access? This is either a problem with your company or the trusted access program. In no way does that suggest the solution is total unfettered access for everyone.
Everyone needs access to Ai enabled security for defense. A "trusted access program" creates exclusiveness in the hands of Big Ai duopoly. I do not trust them at all
> Only by using the open source GLM-5.2 were they able to survive an attack
They did not "survive" anything. The attack was long done, and they used GLM after the fact to parse logs. Having a more powerful model would have changed nothing.
If every attacker and every defender has AI with the same capabilities then attackers are going to win 10 times out of 10.
You're ignoring the asymmetry with security. The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders.
Imagine what a god-level hacking AI could do. It could find a full 0-click to root exploit chain in iOS. Attacker unleashes a worm that infects a phone, instructs that phone to send the same attack to all of its contacts, and then physically destroy the phone by turning off all thermal throttling. Might even be possible to make it catch fire.
Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)
> The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders.
I see it as the opposite, where the attacker needs to find an exploit chain whereas the defender can block any link.
In this model, the balance of convenience favours the defender. The defender presumably has access to the source code and configuration, so their scope of action is much larger than the attacker that must find vulnerabilities in a particular configuration.
I think that the different views might relate to different prior assumptions. If we assume that each layer is mostly secure but may have a small number of latent vulnerabilities, then it should be relatively easy to find and fix those to create a perfectly secure layer. If instead we assume that each layer is mostly insecure but chaining vulnerabilities is time-consuming then the land favours better-resourced attackers.
> Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)
In the worst case, air gaps and fixed contracts for information handling cover that. Like any other domain, a car can be remotely exploitable only when untrusted information can influence behaviour inside the secured region. Unfortunately, the convenience of OTA updates and 'cars as tech' rewards velocity at the expense of defensive design.
I have yet to see someone explain why they even needed an LLM to figure out what's going on, other than further proliferating this industry AI psychosis. Are their engineers actually so incompetent that they can't read a bunch of logs without AI? Here I was thinking these fancy AI companies are only hiring the best and the brightest, but apparently 7 rounds of leetcode does a number on your hiring process.
Yes, to parse logs afterwards and understand, it wasn't active defence from what I've heard? Definitely embarrassing for the closed vendors though (they've since added hugging face as a trusted vendor)
One of their learnings from the incident was that they should have a local (i.e. not hosted), open, and capable model on standby that can respond to future incidents swiftly.
The bioweapon thing is absolute movie plot fiction. Go speak to some biologists about this and they'll set you straight.
Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.
The effective altruism/rationalism/AI xrisk people have always had a shockingly poor grasp on subjects outside computer science, despite their attempts to speak on them. I don't blame the actual biologists and chemists working at the frontier labs for wanting to skim a few bucks off all the money flying around, though! I know a couple who've had not-so-kind words to say about their employers' intelligence.
I suspect there's at least some "telling the bosses what they want to hear" going on. A massive financial incentive exists to exaggerate and fearmonger even internally to the company, because it makes you and your job seem more important.
You can go download the smallpox genome. You don't need AI to do that. The problem of creating bioweapons remains that it requires very, very meticulous lab work under careful conditions, and a lot of experimental knowledge that the bioweapons facilities probably have but LLMs do not. There isn't an actual mechanism here by which you can mix together a few test tubes and come up with a killer virus.
So many commenters are asserting this but no one is giving an argument or references. Cults have been able to make sarin. The DNA sequence for smallpox is pubicly available. Where are you getting your confidence that LLM-assisted bioweapons are of no concern?
Not only is the DNA sequence for smallpox available, but since 2018 there's been a well-documented end-to-end synthesis procedure for the very closely related horsepox virus [1]! No LLMs needed. Caused quite a stir in the synthetic biology community back then.
The world has not come to an end, of course, because even with peer-reviewed and experience-driven (rather than hallucinated and therefore dangerous) instructions detailing obstacles encountered during synthesis and how to overcome them, actually going out and acquiring the materials and ability to use them sufficiently skillfully is another matter entirely. Biosecurity is an important topic, to be sure, but what the AI labs have to say about it (or anything) at this point does not necessarily survive contact with reality.
I'm not talking about what the AI labs are saying and it. I don't know what they have been saying and I don't care. I care whether LLMs a few generations from now will be about to bring these dangerous capabilities to the masses. Right now, given the progress of open and closed-source models, it seems inevitable.
Half or more of Hacker News is constantly pushing the idea that frontier LLMs are stochastic parrots and basically useless, even in the face of the Hugging Face incident which most people also would have described as movie plot fiction until it happened. I expect biologists are even less well versed in the abilities of frontier models.
What's more, you can just try a jailbreak on a model yourself to see just how much detailed, step-by-step direction you can get to build bio-terror materials.
I don't understand the significance of the HF incident. The hacking robot was told to achieve a certain goal and in order to do it, it hacked someone. The ostensible major event here is that it broke out of its sandbox, but how are we supposed to interpret that? AI often misunderstands or doesn't strictly follow the orders you give it, so why is it such a big deal that it didn't follow the rules this time?
Agree with you here. AFAIK we don't really know what cybersecurity task it was given in that sandbox, but it's not a very large leap to assume part of this task involved hacking. The intention would have been within the sandbox, but what does the AI know? As far as it could tell it just reached Level 2.
I've got zero knowledge of bio, so can't answer that. But with cyber the answer is very simple - the attackers already have more cyber-offense capabilities and there's no putting it back.
Open/closed doesn't matter that much. You can get closed models to do a lot of cyber harm, even with all the guardrails, which currently are heavily skewed towards more false positives.
The only effective control is to level the playing field. If both offense and defense have access to the same capabilities, then we're relatively back where we started.
If you want to ensure chaos, then you do what Dario is proposing to do - create gates that attackers can bypass and defenders can not.
In cybersecurity, a level playing field favors the attacker. Trusted access programs give defenders access to tools they need. It's not perfect (because there is an extremely long tail of defenders who are not technically savvy enough to get on these programs and use the tools), but it's better than total access.
The bio angle is very important here too; in that context the imbalance favors the attackers much more.
> In cybersecurity, a level playing field favors the attacker
Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs.
And I don't know what Trusted Access programs give to defenders, because as a defender who has credentials, connections, but no deep pockets and no high ranking passport, it only gave me silence. I fail to see how this is better than total access.
I don't think the world where defense is given to those that "deserve" it is the world that we all want to live in. Which brings me back to the starting point - attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
> Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs.
Trusted access programs are asymmetrical, and so at least for the time being they give critical parts of the stack an advantage. Total access would not be a return to the status quo; attackers can easily make thousands of agents crawl the web for soft targets well before defenses can be shored up. There are millions of targets out there who won't use AI to improve their defenses for years, if ever, due to institutional slowness (like hospitals).
> attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
What do you mean by this? If guardrails are an obstacle to your defense, they are just as much an obstacle to attackers. I completely understand and agree that trusted access programs are not perfect and leave a lot of people and institutions out. This means trusted access programs should be improved, not that we should throw the baby out with the bath water.
It took me a few hours to find some very questionable communities, which in turn gave me access to:
- Ways to obtain cheap guarded-AI tokens that are not linked back to me and with no danger of getting my legitimate accounts banned
- Ways to get rid of guardrails and have models work on things they wouldn't otherwise work on.
The attackers were already in these communities long before I knew they existed, they already had the advantage. Ones with enough reputation probably have access to even more information and tools than I do.
It is true that these communities exist because guardrails were put in place, so yes, it is slowing them down too - as in they can't just put in their CC on claude.com and hack a hospital. But attackers are much better at finding these communities and utilizing resources available there than defenders.
Personally, I don't have any ethical concerns of utilizing these resources when I put them to actual defense, but I know many people that would, leaving them at a disadvantage.
My point is that there's only one guardrail that will effectively contain the threat the models pose, and it's in direct conflict of the big 2's goals - pull the models from worldwide access completely. Strict KYC and all. And it would only last for so long anyway.
I feel like so many people miss what you are saying here. The attackers are at such an advantage because of time. At t0, attackers can go and try and find so many attack angles. These traditional companies (defenders) can't just go to a model and say "fix all my things!" and ship it, way more complex in practice.
I think you are trying to argue that you can limit the open models.
If China is ok with open models being open... they will be. An attacker isn't going to be deterred by a US law saying they can't use them.
I guess my point is that if China is ok with open models, then, the attackers will have them regardless of any laws in other countries. Restricting them, in that case, doesn't seem to accomplish much?
You can at least make it harder by requiring US clouds to only serve models with guardrails, and encouraging other countries to do the same. But yes, the underlying issue is the models being open in the first place. I'm sure if the US wanted to, it could come to some agreement with China about this.
> To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities?
Nothing should be done. These things are trained on public knowledge. The dangerous information is already out there. If someone wants to do something horrible, making it slightly inconvenient isn't going to do much. Hackers and terrorists existed before AI. Just as an example, it's no secret how you would build a nuclear bomb. The practicalities of doing so are much harder, obviously, but the knowledge of how they work and what it would take to make one is not a secret. Security through obscurity has never worked!
I think nothing can be done anymore, but I don't buy that security through obscurity never worked in practice. A better way to look at this is effort rather than obscurity. The effort it takes to do something with AI is going down, not up.
Almost everything was possible given you put in the effort, but few people possess the will to put in the effort AND pursue a malicious goal.
I think an obvious example is all the fake ai content flooding the internet made to trick people in exchange for money (ad revenue, scams, likes, etc).
This existed before ai, but I think it's fair to say pumping out content now requires less effort than it did before.
Most physical locks are an example of security through obscurity/effort. You can after all just pick a lock if you go through the effort to learn the skill. But once a universal lock picker is made available to everyone, you will simply see more locks getting picked.
If this is really the risk, then we should approach LLMs like atomic bombs: the US should reach out to other nations so they all agree on no one developing any more AI models. That's the only way you could possibly convince another party to stop. The US should set the example, not conveniently keep all the spoils.
Uh, that is not how the nuclear race went. The winners kept developing theirs and stopped everyone else by the threat of said weapons. The AI race is going the exact same way, just with China instead of USSR this time.
This Pandora box is already open. Any argument about guardrails now are only attempts to create an artificial monopoly or keep this power in the hand of a single nation state, and _that_ is the absolute worst, most authoritarian future possible.
I think you're right. "Guardrails" as a concept has always struck me as a band-aid solution which any sufficiently motivated actor will circumvent by either bypassing them or using unrestricted, open-weight models.
In order to start securing and accepting our new reality we need to assume that capable, open-weight, unrestricted models will be widely available, and that their 3-6 month lag behind frontier proprietary models is just our forewarning of what attackers will soon be capable of. Trying to legislate against or control trade in such a valuable commodity is folly.
I also think that lag is going to shrink over time as the open-weight labs get more capable, acquire more hardware and the plateau starts to emerge.
How is it already open? There has been ONE successful AI-driven cyberattack, and that was done by a model in testing that no one has access to. What would the picture be today if OpenAI and Anthropic had released 5.6 Sol and Mythos to everyone with no cyber restrictions (which is what everyone here was advocating for)?
Chinese AI companies are already releasing frontier-ish models every other month. Their rate of progress does not seem to be slowing down. Nobody here can stop them from progressing. Not you, not me, not the USA government.
The "best" thing the US government can do is to build a Great Firewall to wall off the "existential threat from China". I'm not an American so if you guys decide to do it, good luck.
the bioweapon panic is funny. "oh, yes i know nothing about bicrobiology but i will follow instructions of synthetic text generation machine on temperature 1 about how to design a lab to not kill myself while brewing organisms that will kill myself if i make mistake"
There is nothing that special about bioweapons, there are plenty of bacteria that will kill you just fine. Americans even have free samples on their salad.
The reason that madmen and terrorists choose kinetic weapons is because the knowledge and materials are more readily available... of and also that even terrorists are likely aware that their own people would suffer. As the knowledge and tools for playing with CRISPR-style biological legos become more widespread, we come closer to the Great Filter, where one person could kill billions.
Even our normal mad leaders have agreed that bioweapons cannot be allowed:
A halfway decent synthetic biology lab (no need to invoke CRISPR) can make e.g. smallpox without a sample of the original disease, just from the gene sequences. Basically all state actors could do this if they wanted to without an LLM. What barrier that a terrorist organization faces today to having a functioning synthetic biology lab does an LLM actually solve?
i meant it in the sense of arcane knowledge model could have that would make it simple for anyone to brew up in cheap lab while managing to not infect themselves over and over.
"at home" bioweapon panic has been around since crispr and rna synthesis got available to amateurs.
I appreciate the reply. I did not mean to be dismissive at all. In the interest of a good exchange, I have to say:
I really want open weight models. Otherwise, I see no other path outside of the labs eventually not being allowed to/wanting to release model access at all, and instead just eating all the verticals. That would be a horrible near-term business outcome.
> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?
In short, yes, it's the price of freedom. As others have said, blocking these models won't stop the "bad guys", but will hinder defenders researching/responding to bioweapons and cyber-offenses.
But you're right that there's a lot of difficult nuance aand we should think carefully about its implications. So here's another nuance to think through.
If AI is as powerful as some believe, then there's much greater danger to give a small subset of society the privilege to gate keep who has access to these tools.
"Power corrupts and absolute power corrupts absolutely." Lord Acton
> what should be done about open weight bioweapon and cyber-offense capabilities?
Like the others here I know almost nothing about bio weapons, but I think perhaps the fact that smallpox's genome sequence has publicly available in scientific databases like GenBank for 30 years is relevant. That horse bolted a long time ago.
And what are those ingredients for biology, which can be constrained as effectively as uranium enrichment? I'd argue there isn't anything which can easily be restricted or monitored.
I do seriously want general intelligence to be widely available with no guardrails, and there are very good reasons for this. If you want to read about it:
There's no stopping bioweapons. Bioweapons are easy. The reason bioweapons aren't built is because very few biology nerds with sufficient lab skills are evil; and just having an LLM won't give you the lab skills to do it.
Anyone who can publish a gene technology/biomedicine paper can make a bioweapon. If you wrote a paper about how to make a bioweapon easily, it would be unpublishable not because of any danger, but because there wasn't enough novelty.
The scariest outcome here is that a bunch of lunatics get ahold of a capable model and use to to harm the rest of us, who are at a disadvantage due to just how capable the model is.
But that's what's happening. The people in charge are a bunch of lunatics. However nice it would be to prevent them from having harmful capabilities, that ship has sailed. The best we can hope for now is preventing them from having supremacy, and that's what open weight models do.
> Is it simply the cost of freedom that we should allow attackers to access these tools?
Yes, it is inevitable that open weights models will happen. Through legitimate means or leaks, the stakes are simply too high once these models get powerful enough. Furthermore, state-sponsored attackers will always have access to these capabilities. The best we can do is give a lot of preparation to the defenders.
> Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
I find it funny that Anthropic's entire argument for building RSI is that it is inevitable, and therefore we should commit to building it first and doing it safely, and yet they don't apply their own logic to open weights models.
All you can do is say that Americans have to pay whatever stupid prices OpenAI / anthropic / Google / Grok wants to charge you, while China uses, and attacks with, open models.
There is also a whole second category of immense risks of having US companies gatekeeping offensive capabilities, especially for us here in Europe. The centralization/privacy/kill-switch concerns that come with it are a huge AI safety dimension.
I'd rather have a level playing field within a phase of adaptation and hardening regarding cybersecurity issues than a constant dependency on the US, maybe grabbing Greenland today, maybe "extracting" our president tomorrow.
The delta between privileged capabilities and open weight capabilities alone already is a massive, unaddressed AI safety risk.
The moat never was and will never be the models, it's the hardware. This is exactly like nuclear weapons: The recipe for a nuke isn't a hidden secret. Getting the infrastructure and materials is completely unreachable for non-state and non-corporate actors. This idea of a "rogue individual" using a frontier model to develop a bioweapon is a complete myth, because anyone with the capability to run the models without guardrails has to answer to/be audited by some entity already.
I'm dismayed that I had to scroll past so many cynical cheap shots to find a comment that actually addresses the core point. I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns. For those who are skeptical of Dario's motives here, it's not enough to call out apparent hypocrisy, you need to suggest an alternative plan that addresses these concerns.
> I had to scroll past so many cynical cheap shots
I haven't read cynical comments, just ones pointing out that the article is cynical itself.
> addresses the core point
No it doesn't address anything, it is fear mongering question.
> I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns
Me neither, I just see marketing campaigns trying to raise valuation of a pre-IPO company.
> you need to suggest an alternative plan that addresses these concerns
I suggest that Dario stops writing marketing letters and start organizing a mostly neutral expert organization to propose solutions.
Also notice that as EU citizen I don't trust a US pre-IPO company's CEO with conflict of interest to suggest solution on resolving global security matters. Especially since he admittedly has no control over how the technology of his own company is deployed in global conflicts[1]
You mean defense. That's how things get hardened. Anyone that was working during the XP era before Service Pack 2 knows what that was like, but it's very manageable.
The bigger real problem here is hardening like that would remove the opportunity for intelligence agencies to spy on everyone.
I don't think the Aum case points the way you're describing: they used a non-pathogenic strain of anthrax because they didn't know any better. That's a knowledge failure.
But even then, the debate isn't about whether open weight bioweapons exist today: it's about whether they will exist in the future. I think Amodei's argument here makes a lot of sense: "what I believe currently keeps us safe in biology is not 'defenders', or even the availability of materials, but a negative correlation between intellectual capability and desire to commit catastrophic harm. Previous technologies like internet search or even DNA synthesis were nowhere near powerful enough to break this correlation, but I worry that at its current rate of progress, AI will do so very soon."
(I'm not just spouting off; I put my time where my mouth is. I used to work in big tech, but I left for a much less well-paying job building an early-warning system for engineered pandemics.)
> I don't think the Aum case points the way you're describing: they used a non-pathogenic strain of anthrax because they didn't know any better. That's a knowledge failure.
There are a lot of interviews with former cult members around. They had armed helicopters, a testing station in western Australia, produced piles of sarin. This wasn't a lack of science knowledge that screwed them up, they notoriously involved the elite class of Japan - it was a whole other category.
There is no link between AI and bioweapons that makes this stuff any more reasonable than availability of detailed descriptions of nuclear reactors enables us to be purifying weapons grade plutonium in our yards.
Analysis of the 48 suspect colonies confirmed them to be B. anthracis ... This genotype was identical to that of the Sterne 34F2 strain, used commercially in Japan to vaccinate animals against anthrax.
They used a vaccine strain because they didn't know any better. Even members of the elite can make mistakes, especially when operating outside areas they know well!
(This was not the only thing that went wrong, but several others were also knowledge failures.)
You and the other are both missing the point. That's not a knowledge failure, it's a failure in how your operation is strategically executing. They were essentially practicing, and what did they learn? Change to sarin and even VX, for which they didn't need AI.
AI isn't going to help you get from nonpathogenic anthrax to pathogenic anthrax either. All it might do is tell you to try sarin or VX earlier, but these present different problems.
The idea that there are people in the world wanting to execute bioweapon attacks that are somehow gated by a lack of access to AI is utter hysterical nonsense that should be clearly pointed out as such.
Still, somebody heavily funded this thing for too long, and I very much doubt that we don't have the surveillance apparatus in place today for these things to get unchecked.
Stuff is known but not acted upon for various reasons.
I started writing up a blog post about this and did some more reading, and this isn't right. They actually did know that the strain they'd obtained was the vaccine strain, and it was the knowledge (or competence) to turn that into a dangerous one that they lacked: https://www.files.ethz.ch/isn/156879/CNAS_AumShinrikyo_Secon...
(Aiming to get something out with more detail on this soon)
You admit that some attackers have the inclination to use bioweapons. Why would they not use the best tools at their disposal going forward?
From the WSJ the other day:
> After OpenAI enhanced the brain power of its chatbot last summer, hundreds of users worldwide began asking it how to make and deploy biological weapons and poisons.
On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
> Why would they not use the best tools at their disposal going forward?
Because AI doesn't solve any of the problems any attacker would actually have. It's a classic case of nerds not seeing the actual problems because they involve reality.
It's worth pointing out that those bioweapon attacks I linked to also predate widespread access to the Internet, and there was similar scare nonsense about that.
> On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
Do you think they are not being exploited today? The reason they aren't more exploited is there really isn't much to gain from doing so.
> The reason they aren't more exploited is there really isn't much to gain from doing so.
This is incorrect. The long tail of soft targets aren't being exploited more because attackers are bottlenecked on labor. AI removes exactly this bottleneck.
> This is incorrect. The long tail of soft targets aren't being exploited more because attackers are bottlenecked on labor. AI removes exactly this bottleneck.
No, it's because the targets are worthless.
You aren't going to be able to mine Monero or run LLM botnets on forgotten cameras in basements. There is nothing to be gained from such targets, soft as they are.
Besides the new defensive AI entertainment makes dealing with wherever those things phone home far easier. Possibly too easy for plebs to be allowed access to.
> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?
Yes, in the same way that we have E2E encryption which allows bad actors to distribute content beyond human horrors.
> what should be done about open weight bioweapon and cyber-offense capabilities?
If the model is capable of it, then it was in the model's training data, which means it was on the internet or published in books made available for consumption. So if any member of the public could have gotten their hands on that information, so be it. If the knowledge was too dangerous for public access, then it should have been highly classified and never found its way into the training data. Tough shit, frankly.
maybe instead of worrying that people on the internet will be good at coding, we could start writing memory safe apis. almost all cves are fixed by using rust
If everyone has access to the same offensive tools, everyone is able to run their own pentests and patch themselves before the bad guys get to them.
It’s like a vaccine where you get to try a medication based on the original pathogen by performing a dry-run on a backup of yourself already in a hospital ward.
Open models aren’t like firearms. If everyone has a gun the mall parking lot is a much more dangerous place because the consequences of using a firearm are so dire, even if you’re in the right.
I'm curious if you are a coder and have used an LLM to review your code. It is like something like shining a black light around a hotel room, and that seems to be the case even for highly regarded software.
It is really easy to have tunnel vision while coding. LLMs have a working memory with a capacity an order of magnitude greater than ours. I wouldn't trust an LLM to write the code, but at this point it is malpractice not to use one for review.
I have been, yes. For a field that has engineering in the name there sure has been a lot of critical mistakes.
You have to call a spade a spade — the profession accepts this sort of tradeoff in the name of speed and cost.
A well designed system would have never allowed those mistakes to occur. I feel like using an llm to catch these sorts of things is just because it wasn’t built right in the first place.
I think ai systems will be able to build systems of abstraction that are formally verified, and we won’t be needed(eventually).
Every single software engineer in the industry agrees with you.
Every single project manager disagrees.
Don't blame the engineers, we were specifically instructed and paid to build things fast and cheap, and every time we argued for good we were shouted down.
True. It has been a race to the bottom for lowest cost as long as the quality meets the bare minimum. LLMs can go through and find all the nails sticking out pretty easily.
> what should be done about open weight bioweapon and cyber-offense capabilities?
In my opinion, the governments should deploy open-weight AI countermeasures. Because it seems to me that it is impossible to efficiently fight AI-powered criminals without AI.
When only AI-restricting regulations would be put in place, the criminals would, in my opinion, just ignore it. We as a society have a difficult time tracking even the illegal gun or drug dealers. I cannot imagine how could one hope to "regulate" something that can be downloaded as a file and run on a computer.
These AI countermeasures should be open because it provides transparency as to whether the countermeasures actually work. Independent testing, tuning, refining or retraining is then possible.
If the closed models were used instead, their provider could at any point in time shut down the entire operation. Or sabotage it under the hood.
The important part is that with the closed, black box, proprietary models, one can never know what they are being served.
In this case, nothing can be done to stop bad actors from using open models. As the article points out, the US can only feasibly prevent US businesses from using open models.
The US can attempt to stop those models from being trained in the first place but good luck with that.
I think it would be entirely reasonable to ban death stars in private hands, the same way I think it's entirely reasonable to ban uranium enrichment in private hands. My point is that it's a question of fact about how risky an AI model can be. I think it's clear that current models are not enriched uranium or death star level, but I don't think there's anything ruling it out in the near future!
I don't think it's a gotcha at all, they openly said it and were pondering how that might help them. Misinformation plays more to sides that are wrong about more things, and I don't consider misinformation my ally in anything.
It's not clear that "super intelligence" is a meaningful concept. This presumes that our concept of intelligence can continue to grow beyond human capacity as opposed to asymptotically approaching what humans recognize as "very intelligent". Perhaps, for instance, how we evaluate intelligence is bounded not by some quantitative capacity but rather our inability to agree on basic concepts/values. And do we even have tasks that a supposed superintelligence can tackle but humans cannot?
I predict that what we consider "super intelligence" is just sheer computational power, but any potential of a very capable agent is bounded by the needs/wants of the person wielding it. That is: even if we were to hand, say, Elon musk this "super intelligence", most humans would consider it relatively stupid because the person wielding it is still a person with stupid goals and values.
Or, to put it another way, I suspect we already do have a superintelligence and have longer than any of us have been alive, and it's just "the market", and it is still incapable of overcoming the limitations of a few morons wielding immense power.... power they will never yield to some intelligence with values and goals "more intelligent" than their own (if such a concept is even meaningful), and intelligence wasted on the values and goals they do have.
The Orthogonality Thesis (by Nick Bostrom) says that intelligence and ultimate goals are independent. Those non-instrumental goals can’t be stupid nor right or wrong. Increasing intelligence will not change the goals only the capability to reach them.
Do note that the orthogonality thesis is a hypothesis, not something we have demonstrated. Weak versions of it (e.g. it is possible to have an intelligent agent with arbitrary goals) are more likely to be true than stronger versions (e.g. intelligent agents we build will have goals uniformly selected from the space of all possible goals).
I have never understood this line of thinking. Why would a company warn that their technology is dangerous if it wasn't? Looking at the ongoing Mythos export control issue, it's obviously not good for profits. If you assume AGI is possible, you must agree that it would be at the very least profoundly destabilizing. The companies are built around this assumption.
> Why would a company warn that their technology is dangerous if it wasn't?
Because it makes their technology seem more important and powerful?
Do you usually believe 100% what a company says about itself?
> Mythos export control issue, it's obviously not good for profits
Who cares about profits? None of these companies make money like a business currently. It's all speculative valuation and influenced by the same hype/doom cycle
When people discuss the negative effects of climate change, do you think that boosts fossil fuel company valuations by making their products seem more important and powerful?
Fossil fuel companies do everything in their power to stop that discussion.
However, arms manufacturers are doing everything in their power to make you feel unsafe and weak and scared. That way, you are more likely to buy a gun.
Hell yes. More than a third of Americans would run their two and a half ton pickup trucks on burning copies of An Inconvenient Truth if it were at all possible
> Looking at the ongoing Mythos export control issue, it's obviously not good for profits.
This is not obvious at all. Anthropic's biggest investor was the catalyst for this action in referring the "jailbreak" (if you can even call it that) to the government. Now, Anthropic is sitting around a table with government officials who are designing benchmarks that will determine what Anthropic's competitors will be allowed to build. At this point, I have no evidence to rule out the possibility that Anthropic's leadership purposefully sought this outcome.
>Why would a company warn that their technology is dangerous if it wasn't?
Because Americans are obsessed with eschatological narratives and action hero stories. I talked to a Palantir guy once who told me that he loves it when journalists describe his company as a James Bond villain because every single time the market cap goes up.
If they said that they're SAP with Call of Duty marketing they'd actually lose money. In other countries this strategy might not check out but in the US it's better to pretend to be a supervillain than to be boring
> If they said that they're SAP with Call of Duty marketing they'd actually lose money. In other countries this strategy might not check out but in the US it's better to pretend to be a supervillain than to be boring
yes, and all their abstract, philosophical thought is now being used to justify high valuations by these companies. if what they are making is the new nuclear bomb, well then they must be very valuable indeed.
> Ban Chinese models because they’re “bad”. (can’t have head to head competition).
The current scale of current + planned data centres only makes sense if the US exports inference worldwide. 30% of your current total electrical generation capacity, and equivalent to 76% of your total electrical demand in 2022. At least, assuming the "33 GW installed, 300 GW more planned" claim I've heard was not itself an AI hallucination.
And at least OpenAI and Anthropic were persuing the same argumemts continuously to their initial founding, before such capture was plausible.
I don't only mean that GPT-2 was, famously, when OpenAI surprised the world with (paraphrased) "we think this is a good point to start practicing not releasing the weights, before models get too dangerous" or as a surprising number of people interpret it even when I link to OpenAI's own post "we think this is … too dangerous".
I mean even before that.
I wouldn't disagree if you say Musk is full of BS, but he did donate to OpenAI when it was created as a nonprofit, and his wealth was 2 orders of magnitude smaller than today. Back then he was talking about risks, "summoning the demon", now he's the one demanding saying "robot army" to justify a bigger compensation package from Tesla.
>> Looking at the ongoing Mythos export control issue, it's obviously not good for profits
We must live in different planets. The whole episode about Mythos has been the absolute best free PR move ever designed. At this point everyone in tech assumes Anthropic and Mythos are the absolute best models. All of this got achieved without spending a single $$ in advertisement.
This model is also probably really dangerous, not denying that. But they definitely used it as a huge PR tool.
> The whole episode about Mythos has been the absolute best free PR move ever designed.
Great PR.
For a product nobody can pay for.
And a high chance that when anyone can, the market shrinks from "worldwide internet users" to "the USA only".
And it's known to be a highly dynamic market: based on the historical average, it only takes 11 weeks to be completely trounced by someone else's model, so if you're out of the market that long you may as well not have bothered.
I remember then Thefacebook.com was only available at certain universities and everyone felt slight envy that they weren't included. Didn't work out too bad for them.
A common line of logic would be that a company would never put a dangerous sticker on their product unless they absolutely have to by regulation. So if they do it, it absolutely means they see a benefit of doing it.
Also why do you think the Mythos thing is bad for profits? Anthropic hyped the model up too hard and didn't even have the compute to serve it properly to all users.
The export control issue has not affected Anthropic's revenue and only made its top of the line products seem far more impressive than they actually are. People don't think Nvidia is a bad company if the government puts export controls on their products.
For Anthropic, valuation and public perception is 100x more important before the IPO than revenue or profitablity.
When the U.S. first legalized pharmaceutical ads on TV, one of the requirements was that the drug companies had to disclose side effects during the ads. The companies chafed at that, thinking it would make the drugs less appealing, and only included the bare minimum the rules required them to. But their own data eventually showed that consumers actually responded positively to hearing the side effects. They found that people think if a drug is powerful, it must also have strong side effects. So instead of just scaring them, it reassured them the product was good. After learning that, the drug makers started including more side effects in the ads than they were required to.
You could also turn this around: Why would a company put up warnings about their product, or at least more than they're legally obligated to. The tobacco and social media industries certainly didn't go "this product has the risk to do great harm, but we also believe in its potential, so we'll go with a responsible approach".
The only examples of companies voluntarily putting up warnings I can think of are those obviously tongue-in-cheek warnings "this book/game/series may cause addiction because it's just so good" etc.
1. Those, like OP, that believe AI danger and disruption is all hype to boost valuations.
2. Those, like myself, that believe AI danger and disruption is very real and presents dilemmas (but also opportunities) for society, so we must tread carefully.
The first position is not logically consistent with reality. The danger is real: we have already seen hints about how this will impact everything from jobs to warfare to mental health. that danger does not increase valuation, in fact it does the opposite because of the need for government regulation.
> in fact it does the opposite because of the need for government regulation.
For frontier leaders, regulation is a competitive moat. At trillion dollar scale, compliance teams, legal and lobbyists are a tiny fraction of opex. For emerging startups, early regulation of a new market is a substantial barrier to entry.
That said, I think Anthropic blundered into their current Fable mess unintentionally by underestimating just how much they pissed off the current administration by refusing to let the DoD use Ant's AI's for all the domestic spying and lethal combat ops they want.
I don't think the first position is steel manned by assuming those holding it actually believe there are no potential problems or ways to use the technology badly.
But that belief does not imply the doom rhetoric is necessary or the best way to approach those issues.
that's an inaccurate characterization on #1. he's not saying ai risk is _all_ hype to boost valuations, but that it's misused to serve purposes other than warning.
What's "exponential" about AI development? Model parameter counts? Anthropic doesn't publish those for their own models, last I checked. Datacenter buildouts? Water consumption per request? There just isn't enough evidence that AI smarts is growing all that much, once you account for the scaling of inputs. That's what OP probably means by "nonsensical hype".
Lousy benchmark, they explicitly focus on the easiest tasks to automate for AI (i.e. heavily cherry picked outcomes) and it seems that they don't bother to test anything except just-released proprietary models.
Make your own then. It can go on the pile with all the others that keep getting saturated too fast to be useful.
> they explicitly focus on the easiest tasks to automate for AI (i.e. heavily cherry picked outcomes) and it seems that they don't bother to test anything except just-released proprietary models.
What?
They made the benchmark last year, and included a bunch of models going back as far as 2019.
When they first announced it, the top end of their tests were things AI could not actually automate, and even now only does erratically. Examples of the tasks SOTA models are now saturating (at the 50% success level, not at 80%) include:
"Prune attention heads of a BERT language model while minimizing accuracy loss on text classification tasks."
"Implement a Python library for the ACE-OAuth standard that can generate and parse messages in CBOR format and encrypt/decrypt access tokens with COSE according to RFC specifications."
"Debug a PyTorch machine learning library with gradient calculation and memory optimization bugs until all tests pass."
"Finetune a large language model to reduce the accuracy of a truth detection probe while maintaining performance on standard benchmarks."
- https://arxiv.org/html/2503.17354v1
They're benchmarking against the time it takes humans to do the same things, which means everything they ask every AI to do must have also been done by a human.
what the "tfa" doesn't say is that geohot writes like how most people talk. you're framing the observation you don't like around a single word and ignoring everything else.
What else is there? The product as delivered today doesn't come anywhere near a justification for these valuations. All of it is built on an expectation of future capabilities, and that's where the Dario-penned doom papers come in.
Even here on HN, Tesla fans try to justify the market cap by pointing at Optimus, even though nobody can buy it and there's already competition for it which you can buy.
The positives aren’t there though, beyond what people are already using it for (pair programming aid, fancy auto complete, refactoring tool). Hence the article’s thesis: the doom justifies the valuation.
Lack of physical reality doesn't stop Musk from hyping in an endless loop. Or hyperlooping, if you will.
SpaceX announced a private flight around the moon in 2017 to launch in 2018, but instead in February 2018 SpaceX cancelled the certification of vehicle it was supposed to use and said they were switching to BFR. In September 2018, that became dearMoon and was due to fly there in 2023. Starship did actually launch in 2023, but exploded twice, and dearMoon was cancelled by the customer in 2024. Starship still isn't getting its test orbits circularised, because SpaceX doesn't trust the engines will relight correctly. What's in the IPO docs, to justify the price? Space based data centres that only make any sense (and even then barely) with cheap rockets that don't work right yet.
Can say the same for self-driving, or Optimus, or TBC, or Neuralink, or Cybertruck, or X, or Grok (where the closest he gets to "doom" these days seems to be how he considers "woke" a dirty word). And in the past even how many cars Tesla expected to sell vs. what they actually sold, which is a demonstration of how shifting goalposts, and not just missing them, still fails to undermine hype.
It's all "jam tomorrow" with him, even though when tomorrow comes there's still no jam; and yet, how many trillion in market cap?
I remember a meme that spread to that effect, but it was similarly divorced from reality. Hundreds of banks failed during the financial crisis, including multiple which were very large. There were a few cases like Wachovia, where an unhealthy bank was required to sell itself in order to avoid a technical failure that would have impacted over 10 million Americans' access to their money. But this was still unpleasant for existing Wachovia shareholders.
I remember large banks that caused the failure to be bailed out. And talk of trickle down economics, except that average joe was not bailed outed, but punished.
I remember robo signing of home foreclosures with no checks and people loosing houses despite paying debts.
> I remember large banks that caused the failure to be bailed out. And talk of trickle down economics, except that average joe was not bailed outed, but punished.
Another thing that is commonly remembered but did not really happen as described. If you had to identify any particular bank as having caused the crisis, it would have to be Lehman Brothers, which was not bailed out. But really, it was a systemic crisis to which TARP was deployed as a systemic solution. Much of the "bailout" talk is based on memories of AIG, which is not a bank and behaved far more irresponsibly than any bank did; they seem not to have bothered hedging their exposure to widespread mortgage defaults at all.
> I remember robo signing of home foreclosures with no checks and people loosing houses despite paying debts.
Weird to think profits matter half as much as ever-increasing valuations, driven by memetic bullshit. E.g. the entire point of the article you're commenting on.
Anthropic's valuation is driven by its revenue growth. Export controls harm this. Hype certainly plays a role, but it's simply not the case that revenue is not a major part of the picture.
Regulatory capture and false AGI hyping, what else? You'd have to be willingly ignoring the writings on the wall if you still haven't identified it yet.
They did not wanted export controls, but they did wanted everyone so scared and thus pay for license. The decision backfiring does not imply it was not intended to be self serving.
If the company really think their product is dangerous, they should stop making it increasingly dangerous. Antromorphic claims is dangerous and that they need more money to make it even more dangerous.
And the fearmongering is beneficial for them so far.
> Why would a company warn that their technology is dangerous if it wasn't?
It helps to look at the history of the AI Alarmist, EA-adjacent mindset. They believe AI could be Dangerous (<--- capital D) if not responsibly managed by the right people (spoiler: themselves or politicians who listen to them). The capital D means more than just some economic and employment disruption, they're thinking much bigger, like "existential threat to humanity" big.
The most extreme alarmists are basically LARPing Terminator 2 and want to shut it down before SkyNet takes over but the more moderate alarmists believe AI is too potentially beneficial to walk away from. And they acknowledge other nations and various bad actors won't stop anyway, so they believe it's their sacred duty to move forward quickly albeit very responsibly. Yes, this leads to some conflicted reasoning and priorities, like they need to keep gaining access to gobs of capital to ensure they are the ones to reach AGI first, as that's the only way to ensure "god" is benevolent.
While I'm sure many of the top AI leadership are primarily (or purely) mercenary, I also think some, if not many, other execs and employees sincerely believe (or at least hope) they are "the good guys" playing a crucial role in an era of historical importance. Obviously, this has some heavy vibes as well as a being quite a buff to one's self-importance. They face each new day feeling the weight of bearing such a consequential role in shaping the future of humanity. Us mere mortals can only pray their wisdom and altruistic ethics match their humility. </s>
So... yeah, that's why we see some weirdly whiplash messaging.
