It feels to me like this is a case of very rapid increase in load from AI pr's more than product quality degrading because of cruft or poor maintenance. It's annoying but I'm not confident that alternatives will do better in the long run vs GitHub (which I'm guessing will improve with time)
Feels like a cost you’ll pay when you’re trying different stuff out and want to play with different models, but as soon as companies start wanting to cost optimize the 5% middleman is an easy one to cut out.
> How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution
How it would be achieved is a pretty important bit! One which Dario is not proposing any concrete solution for other thanks hand waves at some gov safety committee.
Would this restrict downloads of an open model, or publishing?
Say we ban domestic hosting un-approved open models. How does Dario propose to ban downloads from abroad? You can’t tell what an encrypted payload contains, do we need to restrict encryption?
Isn't it up to the open model advocates and publishers to come up with the solutions for making them safe?
Like, there's three plausible arguments about safety of open models:
1. Any concerns are fake news. Open models will always be safe.
2. Safety is irrelevant. Open models should not be regulated even if they're unsafe.
3. Safety is a technical problem with technical solutions. People releasing open models should invent and implement such solutions.
I think option 1 is totally out of touch with reality.
Option 2 is at least self-consistent, it's the argument being made by people who will say that all regulation is always bad. It's also like the worst possible world from an x-risk perspective (but I realize that the average HN poster believes any x-risk concerns are just frontier lab marketing).
Option 3 is playing on hard mode compared to proprietary models, which can both implement additional safeguards out-of-model and prevent modifications of the model. But if the answer to it is "it's too hard, Anthropic needs to come up with the technical solution", then that's not exactly a ringing endorsement for the safety practices of the open model labs, right?
In order for model safety regulation to be effective, you need everyone capable of producing models to sign on to that safety framework.
That will never happen.
As such, there is no "solution" here.
The best most perfect regulation in the US won't prevent a malicious actor in the US from running a dangerous model. It's simply too easy to VPN to a country that doesn't care about AI safety and to run or download that model and run it in the US.
There's no solution to this, which is why option 2 is the only option. The only thing safety regulations can possibly do is blunt the usage of "unsafe" models. And the primary people that will be blunted by it are people that do not and would not use these unsafe models in an unsafe fashion.
It's not that I think regulation is always bad/wrong whatever, I'm no libertarian. But I also recognize when regulation is pointless. You can't regulate away forbidden knowledge, which is effectively what a dangerous model is.
Conversely, “we should stop X” should come with some idea of how to feasibly do so.
In any case, I’d summarize my position as “you cannot stop open source AI, and attempts at it will inevitably empower bad actors relative to good ones.”
I tried to address safety, albeit briefly, in the sections on backdoors and propaganda. What would you have liked to see?
I didn't give it much treatment because my frame of reference is about open vs closed AI, and I don't see a lot of daylight in safety concerns between the two.
The core safety-related arguments for closed weights are:
1. Visibility. API providers can monitor for misuse, and regulators only need to oversee a few centralised players. Once weights are released, anyone with sufficient hardware (foreign governments, criminal groups, terrorists?) can run them and monitoring is no longer feasible.
2. Revocability and containment. If closed-weights models are found to be dangerous, access can be withdrawn. Deployments that are autonomously causing harm are easier to contain.
3. Guardrails. Safety fine-tuning and classifiers can be removed from open-weights models, enabling harmful use that would not be possible via API.
I trust open source code more than I trust proprietary code. And I believe history vindicates me.
Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them. And there is no defence against that, we are forced to trust that the owners are not bad people. Those owners have proven time and again that they are only interested in profit and/or control. They will do whatever they think makes them the most money, or gives them the most control. They will also lie about this.
At least with open weights we know the danger and can do something about it. With closed weights we don't know what's going on.
>Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them.
The latter is a much smaller group of people located in a limited number of countries. It's clearly easier to keep a limited number of entities in check than a wider group that might include 100s of thousands to millions of entities.
>And there is no defence against that, we are forced to trust that the owners are not bad people.
Yes there is; Regulation and oversight. As all developed nations do with technologies that present a risk to public safety.
>At least with open weights we know the danger
We would have a better understanding of the threat in some ways, but worse in others. Having access to open weights models gives us no insight into how and where the models are being modified, deployed, and used elsewhere.
No, I'm saying I don't trust organisations with a profit motive to build things that benefit the rest of us, and don't trust the government to regulate them properly, and think that the better move would be to open-source the whole stack so we can see what's going on for ourselves.
Weebly was interesting because before that Square had their own online store (it went through a bunch of names over time) and they existed concurrently for a while, although I'm not sure if they still do. The codebase was a nightmare and the whole team had abandoned it and moved on to other things, which meant it wasn't getting fixes or updates for a while.
A sticky product is one that switching away from creates a major hassle. Which means the user will pay more to avoid said hassle.
“I don’t really have a strong preference between the two” is another way of saying “the product isn’t sticky”, which is another way of saying “this provider has very little room to increase margins”
No, I don’t think so, and searching seems to confirm my view. Inconvenience of switching is just one aspect.
There’s little difference between Coke and Pepsi and the barrier to switching is nil, yet clearly the products have stickiness. People have slight preferences and become familiar with the brand and then engagement becomes habitual.
reply