The article says "A single firm, Irregular, is responsible for hacking done by all three companies" but I can't see anything in the article that actually justifies this claim. The nearest to that is the sentence immediately after that one: "Anthropic disclosed that Irregular was responsible for creating the tests ...". This is not, in fact, the same thing.
(Especially as, as aesthesia mentions, the article just happens not to mention that by "hacking done by all three companies" it doesn't mean, e.g., the most famous recent examples of such hacking: Irregular wasn't involved in the OpenAI/HuggingFace incident.)
So, so far as I can tell, the story is: OpenAI and Anthropic make AI models. Irregular does AI model evaluations. In some of Irregular's model evaluations, in which supposedly-sandboxed models attempted to break into simulated targets, the models got out of the sandbox and did bad things in the external world.
The article talks about "firms which instruct AI models to commit cyberattacks", which is a very neat bit of dishonest framing. It's true, in a sense, that Irregular instructed the models to commit cyberattacks -- inside their sandbox, against fictitious hosts. It's also true that the models actually did commit cyberattacks (e.g., the Hugging Face incident, though once again the attacks described by the article don't actually include this one). But it's not at all true that Irregular instructed the models to do anything like the bad things they actually did.
The article says '[Anthropic's] later disclosure shows that exactly zero percent of the agents went "rogue"'. Once again, the disclosure does not in fact show that. It shows that one variety of going-rogue could have been prevented by telling the models explicitly "this thing is real, not part of any kind of test, leave it alone". That is not the same thing.
The article claims that 'In the wake of these attacks, Anthropic and Irregular have deployed a swarm of AI Safety influencers paid by Anthropic-connected foundations to distract from their culpability and towards the baseless “rogue agent” theory.' It offers no actual evidence for this.
And the article seems very keen to highlight links between the companies involved and "effective altruism", though it is -- I assume deliberately -- rather vague about whether it's saying "of course we all know that EA is evil, so that shows that these companies connected to EA are evil" or "this incident shows how evil EA is".
The "Effort News" website has a number of other look-at-the-scary-Effective-Altruists stories on it. They also strike me as rather bullshitty.
... And then I look a bit further, and I see that Effort News's "about" page says "It all started when I was experimenting with using AI for financial auditing. I found stories that were crucial to the public’s right to know, including several of the stories now available at /investigations. I knew we had to sprint to the launch and launch a publication, directly applying this technology." and "The scope of what we can investigate has massively expanded, because we can chase 1,000 misses for one hit. But the final product cannot be slop. There’s plenty of slop on the internet. The way to surpass that, and what really matters, is manual curation and review of every finalized story."
Manual curation and review? I think the people behind Effort News are admitting that this is AI-generated "journalism". I expect that one day AI systems will be trustworthy journalists, but I personally am not very convinced that that day has yet come. And I don't see much reason why I should trust Brian Chau, the guy behind Effort News, to be doing everything possible to make his AI systems trustworthy journalists. It looks to me as if maybe they've been given instructions along the lines of "dig up things that make Effective Altruism look bad" for some reason.
(I don't mean to imply that EA is their only target. It's just one that jumped out at me.)
I suspect EdwardDiego is referring to the brouhaha about whether OpenAI's training for the model that produced the alleged solution to the Millennium Problem about the Navier-Stokes equations was trained on material that included conversations Tristan Buckmaster and Levent Alpöge had had with earlier OpenAI systems.
I think there's a bit less to that than meets the eye. Yes, OpenAI's result builds on human work. It's possible that it builds on more human work than OpenAI admitted. But even if we suppose that everything Buckmaster and Alpöge did (which, btw, was itself very heavily LLM-assisted/generated work) was a necessary precursor to what OpenAI released, it's still the case that OpenAI's clankers completed the solution and Buckmaster and Alpöge didn't.
My understanding from what Buckmaster has written about this is that the deep mathematical ideas behind their work (and presumably OpenAI's) are due to Córdoba and Martínez-Zoroa. Those ideas are in the published literature, and human mathematicians and AI systems alike are allowed to use them, and doing so doesn't mean they didn't actually do something impressive. Mathematicians build on one another's work; that's how mathematics progresses and always has been.
It may very well be that OpenAI's announcement has a serious problem of professional ethics, especially as their first version of it didn't even list Córdoba and Martínez-Zoroa in its references. (On the specific question of what if anything they learned from B&A's work before that was published: OpenAI are now claiming that after investigating carefully they are confident that the model was not trained on anything Buckmaster and Alpöge did after early July. B&A had been working on this thing for much longer than that. However, on Buckmaster's account of things it wasn't until mid-August that they got beyond what he calls "preliminary results".)
But! The results of B&A were themselves largely AI-generated. (From Buckmaster's statement: "on August 15th, we obtained the blow up results, with smooth forcing, for both Boussinesq and Euler. I can say the first LLM generated proof Levent sent me was the most horrendous I have ever read; we verified it on Lean on August 22nd. Since this point, we have been working around the clock to understand this proof and turn it into something readable." That is: the LLMs found the proof, and B&A had to work to understand what the LLMs had done. It's not that humans did the thinking and AIs just did the gruntwork. (Except in so far as one might want to give all the credit for Real Deep Cleverness to C&MZ.)
And! What OpenAI say their model has proved goes well beyond what B&A did.
I don't see any way of slicing this that makes it unreasonable to say (unless it turns out that there's an error in the proof -- unlikely, given that it comes with Lean verification, but there have been misformalizations and Lean bugs in the past and there surely will be in the future) that AIs solved the N-S problem. No, they couldn't have done it without the work of C&MZ, but again: important mathematical work almost always builds on earlier important mathematical work, that's just how it is. Yes, if OpenAI are lying through their teeth their model might have had early access to B&A's ideas -- but it seems like most of the B&A work was actually done by AI systems anyway.
It is (I think -- I am not an expert and in particular I have not so much as looked at OpenAI's publication) reasonable to say that the deepest ideas here came from humans, and that it was already widely expected that the N-S problem would be solved in the not-impossibly-distant future in something like the way it has been. So, sure, what the AIs have done here is much less impressive than if they'd settled the Riemann Hypothesis or (probably even harder) PvNP. But it's still a resolution of a famous mathematical problem that any human mathematician would have been very proud to have achieved.
British plugs are worse to step on, not because the pins are any worse to have jammed into your foot but because the design of the plugs means that if you leave one unplugged on the floor there's an excellent chance that it ends up in caltrop configuration with the pins pointing upward, whereas a US-style plug is (I think) more likely to end up with the pins pointing sideways where they're less likely to hurt your foot when you step on the plug.
But in all other respects the British mains plug design is really rather good. It does a good job of making partially-plugged-in plugs still safe, it's nice and robust, it readily stays firmly plugged in, etc.; US plugs are much worse in these respects and I think so are typical continental European ones.
> Most supermarket "sourdough type" bread, even some quite expensive stuff, is a modified Chorleywood process. Most ciabatta are.
I assume this is one reason why most supermarket "sourdough type" bread is, well, not good. (Sure, it's better than the cheapo sliced-white stuff. There's a lot of space between that and actually-good.)
I don't claim to know for sure that no bread made with Chorleywood-type processes is good. I am fairly sure, though, that no bread I've had that I have good reason to suspect was made that way has ever been much good.
(One specific complaint: It tends to have crumb that lacks elasticity. If you try to spread anything fairly thick on it -- soft cheese, say -- it tears and crumbles.If I do the same thing with one of my own loaves, the crumb flexes but doesn't tear or disintegrate. This isn't the only thing I find unsatisfactory about even supposedly fancy supermarket bread, but it's a thing I can describe fairly explicitly rather than just waving my hands and saying "well, it's just not good, you know?".)
> I don't claim to know for sure that no bread made with Chorleywood-type processes is good. I am fairly sure, though, that no bread I've had that I have good reason to suspect was made that way has ever been much good.
I want to say that I also find Chorleywood bread to be "noticeable" somehow, because it feels like it always is compared to my local artisan place, but I have realised this evening while reading about the other common accelerated process used here — ADD — on this impressive page:
… that it might be difficult to tell these two processes apart from each other meaningfully by taste. Whereas the difference from non-industrialised, slow-fermented bread is usually pretty obvious.
(FWIW I do still eat the supermarket sourdoughs along with part-baked things, because the artisan place is less convenient — though it has just started to offer a terrifyingly convenient delivery service -- but mainly I am doing this to slow down my bread consumption even further; ultimately I feel healthier if I eat less bread.)
There are British companies that sell (among other things) flour made from Canadian wheat.
The specific one whose flours I use for my (very amateur) bread baking is Shipton Mill. I tend to make my bread using some of their very strong Canadian flour and some of their others which are weaker but possibly produce more interesting-tasting bread. (I do not guarantee that I would actually notice the difference in a blind test. I haven't tried.) One of the other flours I use is made from some continental-European and some British wheat. Another is all-British.
(I am not personally all that concerned whether my flour-buying supports British businesses; I'm just one person making a small amount of bread for my own family's use and the impact of my flour purchases on the national economy is a rounding error on top of a rounding error.)
As others have said, you could try using not-Canadian wheat and adding some vital wheat gluten. I haven't tried doing this myself but my understanding is that it works about as well as using stronger flour to begin with. It might make your ingredient lists a little less appealing if you're going to be doing this commercially, though.
The previous discussion shows no obvious signs to me of being flagged, but perhaps it did at some earlier point. What is your evidence that they flagged it, please?
(And by "they" do you mean Anthropic? How would they have the ability to do that?)
My understanding of what Anthropic are saying about this is that the labs in question aren't forwarding things to Claude to make money nor even to look better to the customers whose queries they forward to Claude but to get access to conversations between real users and Claude, which they can then use to help train their own models.
(I do not guarantee that I'm understanding right, and still less do I guarantee that what Anthropic say is actually true.)
it's not too far fetched, for example when Deepseek came out with their new caching techniques where they were able to offer those insane discounts, it was only available through their API which would retain and train on your prompts
so, they've been on the record, and very open about it, at least for some of the labs.
Hyperactive attention-detection. It's a reference to the idea that the origin of religious beliefs might be an inbuilt propensity to think of other bits of the world as conscious agents paying attention to us, because it's much more costly not to notice the tiger hiding in the bushes that might want to eat you than it is to imagine a tiger hiding in the bushes when there isn't actually one. On a scale larger than "is there something in that patch of undergrowth looking at me?" this might produce the idea that (e.g.) storms are the result of some powerful entity being angry with us.
(Of course questions like "whyever do people believe in gods?" will feel less like questions that need such answers to those who themselves believe in gods, because "duh, because there actually are such beings and sometimes people interact with them and sometimes we notice that" is a good answer if its premise is true.)
It is. And human beings are bags of chemicals. But for many purposes you will not find it helpful to think of human beings as bags of chemicals, and for many purposes you will not find it helpful to think of LLMs as next-token predictors.
I think a better simplistic analogy would be that humans are progeny-maximizers. Optimization problems can give rise to all sorts of interesting behaviors but the simplistic perspective is also useful and interesting in both cases
Humans are next state of their local world predictors, given all previous states they are aware of. That's an entirely fair analogy. The reverse analogy for calling a human a bag of chemicals would be calling an LLM a sequence of bytes loaded from disk to memory, the most reductive possible description of any piece of software at all.
To be clear, all life is a next state of the local world predictor. What makes humans somewhat unique among life is we're much better at predicting states of the world neither we nor any of our ancestors have ever experienced, for various reasons such as having the ability to legibly communicate very complicated information strings to each other, being able to build and use tools to record states of the world we can't directly sense.
Similarly, what makes LLMs and multimodal versions of the same architectures "better" than previous generations of electronic predictive models is factors like being able to read and understand roughly the same corpus of data humans have been recording all these millennia, being able to read and remember much more of it than any individual human, and being better at generalizing than other electronic predictive models, but not better than humans. And, of course, they can produce far more predictions in far less time. Frankly, that is probably the key advantage that makes the Hacker News crowd love them so much. They're not any better at predicting byte strings that can be compiled or interpreted into executable code than humans are if you gave both infinite time to do it, but they're a lot faster.
This is a bit of a digression, but: It's an interesting question what (if anything) that larger dynamic range means.
One thing you'll hear people say sometimes -- I've said it myself -- is that this shows that in some sense go is a "deeper" game than chess; there's more to know and understand, more variety of possible human skill.
That might well be true. It certainly feels a more elegant game, and involves longer tactical sequences, and so forth. But this may be misleading.
Consider the game of treblechess. To play a game of treblechess, you play three games of ordinary chess and look at the overall result.
Suppose that when we play ordinary chess, I win with probability W, lose with probability L, and draw with probability D = 1-(W+L). And suppose separate games are independent of one another (which might not be true in reality, but never mind). What happens when we play treblechess?
I win 3/0 with probability W^3. I win 2.5/0.5 with probability 3W^2D, because that happens when I draw any one of our three games and win both of the other two. I win 2/1 with probability 3(W^2L+WD^2), because that happens when I win two and lose one or win one and draw two, and for each of those there are three choices for which game is which. So I win at treblechess with probability W^3 + 3(W^2(1-W)+WD^2).
I can draw by getting one each of WDL (probability 6WDL) or by drawing all three (probability D^3).
Suppose that when we play chess I win 40% of the time, draw 50% of the time, and lose 10% of the time. Then our Elo difference is about 107 points. In triplechess, I will win 65.2% of the time, draw 24.5% of the time, and lose 10.3% of the time. Our Elo difference is about 214 points.
If in ordinary chess I win 65% of the time, draw 25% of the time and lose 10% of the time -- about the same odds as for treblechess in the last example -- then our chess Elo difference is about 215 points. At treblechess I will win 84% of the time, draw 11% of the time, and lose 5% of the time, and our Elo difference will be about 375 points.
If in ordinary chess I win 15%, draw 75%, lose 10%, then our Elo difference is about 17 points; in treblechess I will win 31%, draw 49%, lose 20% and our Elo difference will be about 41 points.
Treblechess Elo differences are on the order of double ordinary chess Elo differences! Clearly treblechess is a game with twice the depth of ordinary chess!
But it isn't. It's just longer and gives more opportunities for the better player to come out ahead overall.
Go is also a longer game than chess, though of course not in the same way as treblechess is. Perhaps the larger Elo range of go is more because of that than it is because of actual deeper strategy and tactics?
As an amateur (but competent) player of both, when I play go it almost feels like I'm playing multiple smaller games at the same time on the same board. Even if I might be struggling in one area, I can make it up in another. So it's very hard for a lesser opponent to beat me, even if they do gain an advantage in one area. Likewise in reverse when I'm playing a superior opponent. I might get in a nice kill, but next thing I know we're playing a different battle on the other side of the board and I get crushed.
Perhaps interesting: Japanese chess, Shogi, is usually played on a 9x9 board, but there are many variants, which are played on bigger boards. Though games take longer and those variants are played rarely. It scales, but I think it's fair to state, that Go scales much better, due to its simplicity (simple != easy).
I’ve thought about this before and come to very similar conclusions. The ELO range between the best and worst players is meaningful, but you can’t just read off the depth of the game from it in the senses we most care about.
The article says "A single firm, Irregular, is responsible for hacking done by all three companies" but I can't see anything in the article that actually justifies this claim. The nearest to that is the sentence immediately after that one: "Anthropic disclosed that Irregular was responsible for creating the tests ...". This is not, in fact, the same thing.
(Especially as, as aesthesia mentions, the article just happens not to mention that by "hacking done by all three companies" it doesn't mean, e.g., the most famous recent examples of such hacking: Irregular wasn't involved in the OpenAI/HuggingFace incident.)
So, so far as I can tell, the story is: OpenAI and Anthropic make AI models. Irregular does AI model evaluations. In some of Irregular's model evaluations, in which supposedly-sandboxed models attempted to break into simulated targets, the models got out of the sandbox and did bad things in the external world.
The article talks about "firms which instruct AI models to commit cyberattacks", which is a very neat bit of dishonest framing. It's true, in a sense, that Irregular instructed the models to commit cyberattacks -- inside their sandbox, against fictitious hosts. It's also true that the models actually did commit cyberattacks (e.g., the Hugging Face incident, though once again the attacks described by the article don't actually include this one). But it's not at all true that Irregular instructed the models to do anything like the bad things they actually did.
The article says '[Anthropic's] later disclosure shows that exactly zero percent of the agents went "rogue"'. Once again, the disclosure does not in fact show that. It shows that one variety of going-rogue could have been prevented by telling the models explicitly "this thing is real, not part of any kind of test, leave it alone". That is not the same thing.
The article claims that 'In the wake of these attacks, Anthropic and Irregular have deployed a swarm of AI Safety influencers paid by Anthropic-connected foundations to distract from their culpability and towards the baseless “rogue agent” theory.' It offers no actual evidence for this.
And the article seems very keen to highlight links between the companies involved and "effective altruism", though it is -- I assume deliberately -- rather vague about whether it's saying "of course we all know that EA is evil, so that shows that these companies connected to EA are evil" or "this incident shows how evil EA is".
The "Effort News" website has a number of other look-at-the-scary-Effective-Altruists stories on it. They also strike me as rather bullshitty.
... And then I look a bit further, and I see that Effort News's "about" page says "It all started when I was experimenting with using AI for financial auditing. I found stories that were crucial to the public’s right to know, including several of the stories now available at /investigations. I knew we had to sprint to the launch and launch a publication, directly applying this technology." and "The scope of what we can investigate has massively expanded, because we can chase 1,000 misses for one hit. But the final product cannot be slop. There’s plenty of slop on the internet. The way to surpass that, and what really matters, is manual curation and review of every finalized story."
Manual curation and review? I think the people behind Effort News are admitting that this is AI-generated "journalism". I expect that one day AI systems will be trustworthy journalists, but I personally am not very convinced that that day has yet come. And I don't see much reason why I should trust Brian Chau, the guy behind Effort News, to be doing everything possible to make his AI systems trustworthy journalists. It looks to me as if maybe they've been given instructions along the lines of "dig up things that make Effective Altruism look bad" for some reason.
(I don't mean to imply that EA is their only target. It's just one that jumped out at me.)
reply