That's what I meant - we are currently use fp4 formats for training, and we cannot quite get away with that, despite dynamic quant and small block size - we still have to use quite a bit of higher precision (fp8 or even fp16) in various model components.
I might still be misunderstanding what you are saying, but bitnet also keeps high precision latent weights during training. The optimizer updates those, while the weights used in the forward pass are quantized to ternary values.
Doesn't really even need to look like it. If you can verify rewards, RLVR will optimize really really well. If you can't... it's a struggle. There are probably fewer fields where you can verify rewards than one might hope.
> There are probably fewer fields where you can verify rewards than one might hope.
2 tasks I've done today that I believe robots are nowhere near being able to do: Cleaning my wardrobe and draining bad fuel out of my generator. As in generic use cases.
Hard to verify that your wardrobe is clean. Also hard to verify that the bad fuel is out without physical sensors. Many, many tasks are quite difficult to verify beyond "you know it when you see it". That doesn't work so well for training a model.
Humans wear a lot of hats when the do work. They don't even realize how many. My experience with building real systems using LLMs is that you have to be very explicit about such hats and you don't realize how many are worn until you see edge case after edge case after edge case. Check this. Check that. Check this. Check that. Check check check.
As a class, they steer things through emergent behavior and overlapping self interest in certain areas. As individuals, they can’t afford to trust each other without leverage.
As a class, they have a set of similar incentives (grow/retain resources and power networks) that lead to similar actions (avoid taxation, buy influence, weaken regulators, etc). They may cooperate at times, but they are not a coherent organization with a roster, votes, and leadership.
As individuals, they are dangerous competitors, and they can’t afford to trust each other too much without some kind of mutual leverage. This could be in the form of mutual equity holdings, blackmail, complex arrangements where each party has exposure, whatever. But even then, trust has limits.
There’s no cabal that runs things with rules and elections at this level. It’s all individuals who are immersed in the systems that made them wealthy. Those systems tend to lead to certain behaviors. That’s it.
> You could make this statement about any group at all.
Yes, that's the point
> And then once you realize people are in multiple groups, the whole thing devolves into chaos.
You have bad priors. That would only happen if groups could only have singular and simple goals. It requires those goals to be mutually exclusive. Because as soon as they overlap, even just a little, cooperation becomes an effective strategy.
Hell, history shows us enemies often cooperate. Mostly to defeat a mutual enemy, so they can go back to fighting each other, but that's still cooperation
Nobody is suggesting nobody or group cooperates on anything ever. This comment could go in the definition section of "strawman". My comment was in response to the idea that there is "a conspiracy of the elite". It's just an absurd idea to anyone who has spent time in the world working with people, both "elite" and non "elite".
In the bay area you'll find thousands and thousands of people who have worked at a company run by a billionaire founder. They will tell you said billionaire founder can't even get everyone in the company to conspire to row in the same direction.
But the comment above wasn't about an explicit conspiracy, with billionaires meeting in dark rooms colluding. It was about implicit, uncoordinated conspiracy, where objectives align. My comment is defending the latter
Exactly, there is no conspiracy, there’s just multiple actors moving predictably in the same direction, and sometimes coordinating weakly. I don’t know how my comments could have been read otherwise.
True, but these are generally limited agreements among a few individuals (at most a small handful at the top). Pulling off a big conspiracy between many powerful competing actors carries too much risk of defection. It may be possible but it’s certainly rare.
I do think that these limited agreements may be copied by other people in similar contexts, for instance there were other tech firms who made similar agreements with their own competitors. And the big tech anti-poaching agreements may have themselves been copied from similar agreements made in other industries, such as finance and fast food. The illegal conduct probably spread organically through word of mouth, as execs and board members informally exchange this sort of “best practice” with each other.
It’s likely that big shareholders or board members could hold private meetings to advise the CEOs of their companies on how to pull off this sort of trick, if they think it would help the stock price. Given how many people sit on multiple boards, this could be seen as a sort of conspiracy. It’s still very informal and of course it’s nearly impossible to prove.
Maybe I'm overindexing on "they steer" and "effective strategy" - it still reads that this group has way more control and deliberate planning than they do.
And, "implicit, uncoordinated conspiracy" is an oxymoron. The word "conspiracy" has a definition.
If all you are saying is "sometimes billionaires are aligned" - sure. Sometimes politicians are aligned. Sometimes teammates are aligned. If one wants a platitude with some predictive power "people are self interested" is going to work more often.
“People are self interested” works fine as long as you extend the analysis to include the class dimension. People in a specific class are generally going to advocate on behalf of that class, because it benefits them to do so. No conspiracy needed.
To a degree, yes, it’s why looking at people on the level of groups is not that useful for predicting individual behavior and beliefs, only aggregate behavior and beliefs.
However, some groups do have identifiable leadership, rules, membership rosters, etc. Political parties for instance, or governmental bodies. Decisions from groups like this often have an identifiable lineage (from proposal to vote) and are considered binding. This contrasts with the sort of emergent decisions I was discussing in the context of the very wealthy. There was no vote, there was no formal coordination, and the “decision” is not binding, yet everyone can see its effects. Opposing something like this is like boxing with a shadow.
Or, someone who knows the books and hot takes on "systems thinking" are pop-science, and the actual good ideas have been around for a long time and taught in operations research, control theory etc.
It doesn't need any intelligence at all. There's mutually beneficial things to any group of people. So it doesn't require explicit coordination. Even fairly unintelligent animals have symbiotic relationships with other species. So your argument is what, that billionaires are less intelligent than an ant?
In the real world it's hard enough to get 5 people on the same page about anything. The idea that dozens/hundreds of people with billions of dollars who are in business against each other in many cases are secretly colluding with any effectiveness is a joke. If they were, zuckerberg, larry and sergey would have been squashed in the 90's/00's by the elite of the time. Sam Altman would have been crushed years ago. Obama or Trump would have been put in their place, probably both. Look at all the billionaires all of a sudden kowtowing to Trump. The billionaires just got beat real bad in NYC.
Billionaires just don't have much power. This isn't a movie.
This seems to contradict itself. You first claim that there's no organization but then you claim billionaires have no power because of the more powerful above them. Hierarchy is organization, so which is it
It really sounds like you're trying hard to define a square circle. Your argument isn't even self consistent because what you described sure sounds like someone has power...
Nope. It's perfectly consistent. No one has much power at all. No one is in charge, there is no greater power sitting above us all calling all (or even most) the shots. To use someone else's term, there is no cabal.
You seem to conflate "no one is powerful" with "no one has any power in any situation at all".
I never said nobody hoped to be powerful, or increase the level of power they have. They just don't have much, and they can't/don't coordinate like some conspiracy theorists believe.
There might not be a good abstraction. I've built a few harnesses for different types of workflows, and the details are so different I struggle to see a good abstraction. It's also not clear there should be - if you look at most complex software systems, it's a collection of smaller abstractions/tools/systems pulled together to achieve X.
There are similar exceptions. For instance, the copy of licensed content that your computer makes in RAM in order to play it is explicitly exempt from being treated as a copyright violation.
isnt that how the modern telephone system works? digitize your voice, encode, transmit, decode, convert to analog audio. with buffering or keeping bits in memory all along the way?
and taken to it's extreme.... isn't the atmosphere secretly taking the vibrations in the air right outside out mouth and propagating them through a physical medium, making them available to anyone to capture?
You've always been allowed to transcribe (ie, write) out a conversation. Recording is a different thing. A person can always claim, easily and believably, that the transcription is made up. It's just text.
From a legal perspective you can't record something in many us states. Using the microphone, keeping the bits in memory while you turn it into text, probably doesn't meet any reasonable definition of "record".
Maybe this will be true legally (we wont really know until it is tested in court), but I think there's a huge difference in practice.
Siri Recap is an electronic record of a conversation that includes timestamps. Imagine for example that during a scheduled meeting with my boss, he threatens to fire me for not doing something illegal. I now have strong evidence that this was said (even if it cant be officially attributed to him) during the time I was in a meeting with him. Not saying this will hold up in court, but it certainly is more damning that having written notes (not to mention the fact that my boss probably wouldn't say this when I am taking written notes).
Making that feature illegal may backfire spectacularly across the whole IT sector. What Apple does is it inputs voice data into an AI program and that outputs some transformed output. If a court would rule that output in any way matches the input (to be classified as a "record") then the whole ethics principle of stealing other people's data and funneling it through AI to make it company's own, would be at risk.
I imagine no single spineless impotent modern court would risk a wrath of our new benevolent AI overlords calling them all criminals they really are.
Maybe. Maybe not. Look at AI drug design - it's not really speeding up the important part - drug trials. There isn't really a coherent plan to use AI for the most complex part of drug discovery at all.
I work in pharma and I've advocated for directly training models that predict drug trial outcomes. I don't think we have the necessary support (data + algorithms) to produce accurate models in this space. I suspect that predicting drug trial outcomes is roughly isomorphic to understanding human biology at a fundamental level (and accepting that our existing models of how drugs work are extremely limited).
reply