It's partly because of disagreement about what "complexity" and "simplicity" actually are. Many simple statements are in fact backed by massively complex, unstated assumptions. In attempting to deal with assumptions, scientists necessarily end up having to deal with the complexity involved in that. But the problem is if you don't engage with that, how do you know what is really more conformant with Occam's razor, as opposed to just satisfying what is readily expressed in common language?
Likewise in software development a C developer will say abstractions are not simple, a java programmer will argue that dealing with low levels details is not simple. They're both kinda right but will resort to framings which back their world view.
I am also wondering the same thing as OP. Somewhat tangential thought, but I recently realised that what I would really love in an IDE is a "first class" Markdown editing experience, just because most of my day-to-day is managing and writing docs rather than editing code. Cursor has Markdown rendering support of course but it seems to have two implementations, each sort of clunky in their own way. And I want something more WYSIWYG but with Latex / flowchart / etc. support built-in.
OpenKnowledge seems to maybe fill that niche somewhat (it also supports annotating Markdown with comments to feed into an agent), and has agent integration within the GUI, but one killer feature that seems to be missing is Remote SSH support (short of hosting its web server on the remote and having the IDE live in the browser).
I was looking at CLI-only solutions a while ago and the best thing I found was to use something like revdiff: https://github.com/umputun/revdiff
For instance, in Claude Code you can have it bring up this TUI overlay after every edit it makes (it literally is an overlay, rather than a separate tab in your terminal) and you can annotate line-by-line anything you want changed or rejected. Then once you exit the overlay those suggestions are automatically piped back into the agent.
I tried a few of these TUI-based code annotators, and this one is by far the best. It also works for files not under version control.
I'm currently not using it as I also prefer something more akin to "text editor with agent built in", but I wanted to mention here in case you were curious.
Interesting to see someone else is in the same boat as I am. Regarding editor review I agree, surely the solution would be to `git diff` at the end, but if you only want to git diff what was literally just changed by the agent it means you'd have to commit as you go so that your git diffs are more localised. And then at the end of your work just squash the commits. But it feels annoying. (EDIT: ok I forgot about git staging, that seems to be better)
I mainly use Cursor for another reason though: all of my work is remote over SSH and I don't like keystroke latency, so I prefer code editing / agent conversations to feel local. (Everything else is done in the terminal.) But I would also prefer to use something less propriety. I may go back to VSCode with plugins to achieve a similar experience.
I'm not sure if this is directly mentioned in the paper, but I didn't see any mention specifically about the conflation between a validation set and test set. When people actually make a distinction between the two (which is seemingly not all that common nowadays), you're meant to perform model selection on the validation set, i.e. find the best HPs such that you minimise `loss(model,valid_set)`. Once you've found your most performant model according to that, you then evaluate it on the test set once, and that's your unbiased measure of generalisation error. Since the ML community (and reviewers) are obsessed with "SOTA", "novelty", and bold numbers, a table of results purely composed of test set numbers is not easily controllable (when you're trying to be ethical) from the point of view of actually "passing" the peer review process. Conversely, what's easily controllable is a table full of validation set numbers: just perform extremely aggressive model validation on your model until your model gets higher numbers than everything else. Even simpler solution, why not just ditch the distinction between the valid and test set to begin with? (I'm joking, btw.) Now you see the problem.
Your description of tanh isn't even correct, it squashes a real number to `(-1, 1)`, not "less than one".
You're curious about whether there is gain in parameterising activation functions and learning them instead, or rather, why it's not used much in practice. That's an interesting and curious academic question, and it seems like you're already experimenting with trying out your own kinds of activation functions. However, people in this thread (including myself) wanted to clarify some perceived misunderstandings you had about nonlinearities and "why" they are used in DNNs. Or how "squashing functions" is a misnomer because `g(x) = x/1000` doesn't introduce any nonlinearities. Yet you continue to fixate and double down on your knowledge of "what" a tanh is, and even that is incorrect.
When discussing `tanh squashing` among other AI experts it's generally assumed that even the most pedantic and uncharitable parsing of words won't be able to misinterpret "smashing to less than one" as an incorrect sentence fragment, because the "one", in that context, obviously refers to distance from zero.
If course they do exist. A parameterized activation function is the most obvious thing to try in NN design, and has certainly been invented/studied by 1000s of researchers.
How was that person derailing the convo? Nothing says an activation function has to "squash" a number to be in some range. Leaky ReLUs for instance do `f(x) = x if x > 0 else ax` (for some coefficient `a != 0`), that doesn't squash `x` to be in any range (unless you want to be peculiar about your precise definition of what it means to squash a number). The function takes a real in `[-inf, inf]` and produces a number in `[-inf, inf]`.
> Sure there's a squashing function on the output to keep it in a range from 0 to 1 but that's done BECAUSE we're just adding up stuff.
It's not because you're "adding up stuff", there is specific mathematical or statistical reason why it is used. For neural networks it's there to stop your multi layer network collapsing to a single layer one (i.e. a linear algebra reason). You can choose whatever function you want, for hidden layers tanh generally isn't used anymore, it's usually some variant of a ReLU. In fact Leaky ReLUs are very commonly used so OP isn't changing the subject.
If you define a "perceptron" (`g(Wx+b)` and `W` is a `Px1` matrix) and train it as a logistic regression model then you want `g` to be sigmoid. Its purpose is to ensure that the output can be interpreted as a probability (given that use the correct statistical loss), which means squashing the number. The inverse isn't true, if I take random numbers from the internet and squash them to `[0,1]` I don't go call them probabilities.
> and not only is it's PRIMARY function to squash a number, that's it's ONLY function.
Squashing the number isn't the reason, it's the side effect. And even then, I just said that not all activation functions squash numbers.
> All the training does is adjust linear weights tho, like I said.
Not sure what your point is. What is a "linear weight"?
We call layers of the form `g(Wx+b)` "linear" layers but that's an abused term, if g() is non-linear then the output is not linear. Who cares if the inner term `Wx + b` is linear? With enough of these layers you can approximate fairly complicated functions. If you're arguing as to whether there is a better fundamental building block then that is another discussion.
In the context of discussing linearity v.s. non-linearity adding the word "linear" in front of "weight" is more clear, which is what my top level post on this thread was all about too.
It's astounding to me (and everyone else who's being honest) that LLMs can accomplish what they do when it's only linear "factors" (i.e. weights) that are all that's required to be adjusted during training, to achieve genuine reasoning. During training we're not [normally] adjusting any parameters or weights on any non-linear functions. I include the caveat "normally", because I'm speaking of the basic Perceptron NN using a squashing-type activation function.
> It's astounding to me (and everyone else who's being honest) that LLMs can accomplish what they do when it's only linear "factors" (i.e. weights) that are all that's required to be adjusted during training, to achieve genuine reasoning.
When such basic perceptrons are scaled enormously, it becomes less surprising that they can achieve some level of 'genuine reasoning' (e.g., accurate next-word prediction), since the goal with such networks at the end of the day is just function approximation. What is more surprising to me is how we found ways to train such models i.e., advances in hardware accelerators, combined with massive data, which are factors just as significant in my opinion.
Yeah, no one is surprised that LLMs do what they're trained to do: predict tokens. The surprise comes from the fact that merely training to predict tokens ends up with model weights that generate emergent reasoning.
If you want to say reasoning and token prediction are just the same thing at scale you can say that, but I don't fall into that camp. I think there's MUCH more to learn, and indeed a new field of math or even physics that we haven't even discovered yet. Like a step change in mathematical understanding analogous to the invention of Calculus.
reply