The more complex the process you use to automate tasks, the more difficult it is to troubleshoot and maintain, and the more impossible it is to inevitably replace parts of it with a new system. https://xkcd.com/1319/ is not just a comic, it's a truism.
I am basically a Perl developer by trade, and have been building and maintaining customized Linux distributions for large clusters of enterprise machines for years. I would still rather use shell scripts to maintain it all than Perl, or Python, or Ruby, or anything else, and would rather use a system of 'stupid' shell scripts than invest more time in another complicated configuration management scheme.
Why use shell? It forces you to think simpler, and it greatly encourages you to extend existing tools rather than create your own. Even when you do create your own tools with it, they can be incredibly simple and yet work together to manage any aspect of a system at all. And of course, anyone can maintain it [especially non-developers].
As an example of how incredibly dumb it can be to reinvent the wheel, i've worked for a company that wanted a tool that could automate any task, and that anyone could use. They ended up writing a large, clunky program with a custom configuration format and lots of specific functions for specific tasks. It came to the point where if I needed to get something done I would avoid it and just write expect scripts, because expect was simpler. Could the proprietary program have been made as simple as expect? Sure! But what the hell would be the point of creating and maintaining something that is already done better in an existing ages-old tool?
That said, there are certain tasks i'd rather leave to a robust configuration management system (of which there are very few in the open source world [if any] that contain all the functionality you need in a large org). But it would be quite begrudgingly. The amount of times i've ripped out my hair trying to get the thing to do what I wanted it to do while in a time and resource crunch is not something i'd like to revisit.
> i've worked for a company that wanted a tool that could automate any task, and that anyone could use. They ended up writing a large, clunky program with a custom configuration format
Lots of people design terrible, overly complicated systems. This is an argument for good design, not for ad hoc shell scripts for everything.
Would you write a build system in shell? Why not? Because "make" is far better suited to the job. (Even "make" has a lot of room for improvement, but that's a story for another day).
As another example, even Debian (known for its conservatism) decided that systemd was a net benefit over sysvinit, where the latter could quite accurately be described as a pile of shell scripts. Abstractions that suit the problem domain are far superior to ad hoc imperative bags of commands.
And my argument was that a simple design is good design, and in my opinion, better than something that isn't simple.
You could definitely write a build system in shell. In my opinion, "the job" is subjective and you shouldn't have to pick one tool for it. But I would prefer a simple one where I can see pretty much everything it's doing up front and not have to download source code or search through docs.
Debian picked up systemd because it was a popular decision, not because it was better. As far as i'm aware there was nothing broken in sysvinit that systemd fixed; there was just a vocal minority that wanted some new features and insisted it be shipped as default.
And in defense of shell scripts, nearly every single operating system you can think of other than Windows uses scripts to initialize (hell, even Windows has some startup scripts), and that is an abstraction that's suited to the problem domain. It's not like for the past 50 years people have been too stupid to come up with something better than shell scripts. The whole point of init scripts is to let anyone be able to modify their operating system's start-up procedures quickly and easily. If we didn't need to modify them to suit our individual cases they'd be statically compiled C programs or assembly (which, by the way, I have done in the past; it's not a good idea)
> Debian picked up systemd because it was a popular decision, not because it was better. As far as i'm aware there was nothing broken in sysvinit that systemd fixed; there was just a vocal minority that wanted some new features and insisted it be shipped as default.
I don't feel that's a fair summary of the lengthy debate had about this. There is a long page here [1] listing the reasons for selecting systemd. There is also [2], a very good summary by Russ Allberry of the different init systems being suggested (including staying with sysvinit). Debian has rarely been accused of taking a major technical decision because it is hip.
Shell scripts are not simple. They rely on mutable state and loads of it. For every bit (in the binary sense) of mutable state you double the number of possible states your system can be in. This rapidly expands to ridiculous levels of complexity, the vast majority of which the programmer is ignorant of. This is not good design.
It may not be as "good design" as functional programming, but functional (in the sense of usable by human beings) it is very good design. It's essentially a flat program that can be interpreted by non-developers, edited on the fly, is incredibly flexible and customizable and is backwards-compatible by 40 years. It's actually pretty damn useful. But I can see how someone might not like just getting things done and would rather design an immutable state functional program to start their ssh daemon.
It's not about getting things done, it's about reliability and repeatability. When you deploy large numbers of nodes in a system you don't want little bits of state causing random failures. You want everything to be as homogeneous and clean as possible.
> You could definitely write a build system in shell. In my opinion, "the job" is subjective and you shouldn't have to pick one tool for it. But I would prefer a simple one where I can see pretty much everything it's doing up front and not have to download source code or search through docs.
You likewise have to read docs or the source to know what your shell is really doing. Or search through sysvinit docs to know which shell scripts it will run and when. I have to look up how to use shell conditionals every time, and it's extremely easy to get it subtly wrong such that it's not doing what you think it's doing. There are subtle distinctions between shell built-ins and programs of the same name, and it's very unclear in some cases which is being called. The shell is far from being a beautifully simple and transparent abstraction. Perhaps you know it too well to be able to appreciate this.
The shell and sysvinit are abstractions over writing C programs that make POSIX library calls directly. It's all abstractions unless you are writing machine code directly, (and even then you have to read docs to know what the instructions really do). So given that we're all using abstractions, we ought to use the best ones; the one that gives us the most advantages and the fewest disadvantages given the problem we are trying to solve. The ones that have the greatest combination of simplicity, transparency, and expressiveness.
If you want to make sure that only one copy of a program is running, is it a better fit to write a shell script that creates a pidfile, checks to make sure there's a running process that matches it, and that handles a million cases about how the two can fall out of sync? Or is it better to be able to express directly "I want one copy of this program running" and then let the abstraction handle the hard cases?
You might say "well the shell script is easier to debug, because I wrote it and I understand it." Well of course, you always understand the things you wrote yourself. But if I had to debug either a pidfile-making shell script or a well-designed babysitter tool, I'll take the specific tool every time. If it's mature, it's also a lot less likely to have bugs and race conditions.
> Debian picked up systemd because it was a popular decision, not because it was better. As far as i'm aware there was nothing broken in sysvinit that systemd fixed; there was just a vocal minority that wanted some new features and insisted it be shipped as default.
I don't think you're very well-informed about this. Your claim of a "vocal minority" is particularly suspect, considering the decision was made by a vote of the technical committee. Though there was disagreement, almost no one was in favor of sysvinit. And the position paper in favor of systemd identifies specific things that are lacking in sysvinit: https://wiki.debian.org/Debate/initsystem/systemd
What you're arguing for is essentially the difference between a shell script and a more complicated shell script [some tool designed to "express directly" what you want to do]. You ask, why wouldn't we use something more complicated if it does exactly what we want to do? Because unless you really need to do some exact thing, it adds unnecessary complexity which leads to many additional problems. And if you need the added functionality, you can always add a feature or use a pre-existing tool.
Your debugging argument is bonkers. You claim shell scripting is too hard, then say it must be easy to troubleshoot a "well-designed babysitter tool", which requires WAY more domain-specific knowledge of debugging tools! If you don't know how to write bash scripts, you sure as hell aren't going to have an easy time figuring out why your package manager's borking on an update of openssl.
Did you even read the executive summary of the position paper? "Systemd is becoming the de facto standard" .. "Systemd represents a leap in terms of functionality" .. "People are starting to expect this functionality [..] and missing it could [..] make Debian lose its purpose." They only want it because it has new features, and people are starting to expect it. It's a popularity contest, and systemd won.
> What you're arguing for is essentially the difference between a shell script and a more complicated shell script
You seem to have a really shell-script-centric view of the world, as if the shell is somehow a fundamental and "pure" tool, and everything else is a more complicated version of a shell script.
What you are missing is that the shell is just another tool, and does an awful lot behind the scenes to achieve the functionality that appears "simple" to you. Bash is tens of thousands of lines of C and its manpage is hundreds of thousands of words. Using the shell is not "free", complexity-wise. The shell is a programming language, and not a particularly well-designed one at that. Shell script programming introduces its own peculiarities, and it is known not to scale in complexity very well.
> Your debugging argument is bonkers. You claim shell scripting is too hard
No, I claim that the shell is not a particularly simple tool. There is a difference.
I write JIT compilers for fun. I don't shy away from things that are hard. But my brain has only so much room for uninteresting and arbitrary facts. The very pecular way that conditionals work in the Bourne Shell is not something my brain can be bothered to remember.
> Did you even read the executive summary of the position paper?
You have edited out precisely the parts that contradict your position: "It replaces the venerable SysV init with a clean and efficient design [...] It is better than existing alternatives for all of Debian’s current use cases." Yes, the momentum of systemd was clearly a factor, but your claim that it is only but a popularity contest is not a conclusion that a dispassionate observer would reach.
Is it really so hard to remember these three forms of conditionals?
if ! grep foo /some/file ; then
run_something
fi
if [ $INTEGER -eq 0 ] ; then
run_something_else
fi
if [ "one" = "two" ] ; then
do_something
fi
Those are the only conditionals I ever use. I don't really use conditionals in any other way, and it's really not that complicated to see how they work. Sure, there are more complicated forms, and forms that other shells use (what you see above are fairly compatible, common forms of conditionals, except for the -eq). But if you go back to the original shells and how they did scripting, that should work for most if not all other shells today.
Also, those parts you quote don't contradict anything. It's saying systemd has a "better design", which means it is a shinier, fancier new toy to play with. But the paper never once points out any flaw in sysvinit. But that's obvious; Debian had been chugging along for over over two decades with sysvinit without any problems. If your argument is that suddenly, after 20 years, someone realized sysvinit was some horribly flawed design that needed to be replaced, and it just so happens that systemd came along right when they realized it, I don't buy it. What the paper does spell out, though, is all the advantages of systemd for things other than system init. Basically it says "Hey, we want all these new features, and we need systemd to replace init for it all to work, so please just go along with it because it's a much better design."
Here are some things that seem reasonable but don't work:
FOO=""
if [ $FOO = bar ] ; then
echo "equal!"
else
echo "not equal!"
fi
This errors out with:
test.sh: line 4: [: =: unary operator expected
This is because the shell is based on text substitution. So once it's replaced "$FOO" with nothing, it ceases to be an actual token, and the expansion of:
if [ = bar ] ; then
...is an error. This is terrible.
One solution you sometimes see to this is:
FOO=""
if [ x$FOO = xbar ] ; then
echo "equal!"
else
echo "not equal!"
fi
This handles the empty string and other simple strings, but once $FOO has any spaces or shell metacharacters in it, it will break also. This is also terrible.
> It's saying systemd has a "better design", which means it is a shinier, fancier new toy to play with.
You seem dismissive of new technology. If you want to keep using CVS while browsing the web with lynx and sending your email with pine, more power to you (after all, graphical web browsing is just a "new feature"). But the rest of us are moving on.
It's not new technology that bothers me. Me having to do more work bothers me. Systemd is going to make my job more difficult in terms of troubleshooting and maintenance - way more difficult than remembering that an operator requires two operands to evaluate.
What's really funny about systemd is I think that all its features have tons of value, and I would definitely use them. But I also think its creators are completely fucking batshit insane for making it mandatory to replace huge chunks of the operating system just to get those features. You should be able to just run systemd as a normal user process and still maintain the same level of functionality, but for some fucked up reason somebody thought it would be a great idea to make it a completely non-backwards-compatible non-portable operating system requirement. It's a stupendously bad idea, and the only reasoning anyone can come up with for why they designed it that way is "It's Advanced!" Of course, I should add the caveat that I don't care at all about boot times, and so people who are obsessed with short boot times will find systemd very refreshing, in the way an Apple user finds replacing their old iPhone with a new iPhone very refreshing.
Who says scripts have to be ad-hoc? They can be very well designed and tested. In fact, in some ways I agree with the parent: simple well tested scripts are a very powerful and often underutilized tool.
Just to be sure I've understood correctly, what you're saying is "Bah, humbug!" Is that right?
Edit: OK, sarcasm aside, I can't see how this comment amounts to more than "I don't need anything fancier than shell scripts." Fine, you don't. But there are lots of people out there that do.
For several years, I ran a cluster that consisted of 10 or so EC2 instances that had to rebuilt and replaced every day. Shell scripts didn't cut it for me. Chef and Puppet were painful. I ended up using Fabric, boto and a pile of custom Python code.
And 10 machines is nothing. How do you maintain a 1000-machine cluster with frequent application deployments, configuration changes and variable load? If you've got a bunch of shell scripts that make that easy, that would be fantastic news. I kind of doubt it, though.
> And 10 machines is nothing. How do you maintain a 1000-machine cluster with frequent application deployments, configuration changes and variable load? If you've got a bunch of shell scripts that make that easy, that would be fantastic news. I kind of doubt it, though.
Four years ago, I was going this with shell scripts, a little bit of Python, and a cluster management tool called Rocks (built in Python and shell scripts). This was for a 5500+ Linux server cluster processing data from the CMS detector at the LHC. I had the ability to do a rolling reinstall of Scientific Linux/CentOS while the cluster was actively processing data with no interruption to the end user jobs.
Everyone keeps reinventing the wheel, but in most cases, shell scripts can do a lot of heavy lifting before you need to start writing one-off python scripts.
...shell scripts can do a lot of heavy lifting before you need to start writing one-off python scripts.
I've written loads of both, and I realize the thread had already started down this path, but how is a shell script really different from a python script? If you're targeting Bourne, your shell isn't a separate installation, but it isn't onerous to install your favorite python version on machines you control. Sure, python has an ecosystem of libraries that shell doesn't have, but you don't have to use those libraries if you don't want to. What terrible consequences does someone who prefers python to shell scripting face?
> What terrible consequences does someone who prefers python to shell scripting face?
Terrible consequences? Probably not. Shell script dependencies are easier to handle, because the commands you're relying on are built right into the OS. They're going to be there because they have to be. How often does sed, grep, awk, and cut change?
It's a matter of complexity. If you're munging files and using regexps, shell is fine, and quicker than Python. If you're starting to think "I need a set" or "I need real error handling", it's about the time I move to Python or Perl.
> called Rocks (built in Python and shell scripts)
Regardless of if it "worked", I think mixing languages in an application is a bad idea, and should be avoided whenever possible. There are a few reasons,
1) maintainability - you're requiring anyone to know more things to maintain the application
2) Attack surface - lots of security bugs come in between boundaries between tools
3) Correctness - it's harder to automate testing
Did you have any sort of automated testing?
Just because shell scripts can do a lot of heavy lifting, doesn't mean they should.
10 or so EC2 instances that had to rebuilt and replaced every day
Just to be understood correctly, you're saying the exception proves the rule? Sidestepping the necessity of the task above, which I'm sure was rendered easier by Fabric or whatever, but it's not like that's a typical use case. You would have to ignore nuance to extrapolate that to generic VM instances from known-state images, etc. that might respond just as well to a bunch (not even a bunch) of scripts.
The troubleshooting issue is one of the biggest benefits of Nix IMO - because the complex automation process which it significantly reduces the need to troubleshoot is the GNU[1] build system. ([1] replace GNU with other build system of your choice.)
The GNU build system (+pkg_config) is really awful to work with when it doesn't magically work the first time. You'll often get errors such as m4 macros not being defined (but which don't hint at what dependency is missing), and more often than not you'll need to build several dependencies for the software you want, which are often poorly specified. It's not uncommon for one to waste hours or whole days trying to build a piece of software (and if you're lucky, he will blog about his experience so that you can spend less time when you inevitably google the problem, because that's always easier than trying to understand and debug the build scripts yourself.)
Nix reduces this by requiring exact dependencies and build instructions - there's no "hidden knowledge" that the software developer/package maintainer had which the user should not also have available, so at most, only one person needs to experience the pain of building the software and others can reproduce based on his precise instructions, rather than his blog post.
I can't honestly say that there will be no problems when using Nix, but a build error in nix is a bug in the distributed nix packages, not just some unknown problem on the users system which he needs to fix.
I agree with you that reinventing the wheel is usually bad, and you get a long way by reusing existing programs with shell scripts, but sometimes you need to throw away your assumptions about how things work because a new model is significant enough to rethink the old. (We didn't build aeroplanes by attaching wings to cars).
i both agree and disagree. you can create rather huge complex incomprehensible systems in bash too.
also, it's not always better to use existing systems. sometimes rolling something new is indeed better.
but i had to work in a place where everyone was a ruby fetishist. don't get me wrong ruby paid my rent for the last couple of years, but i too thought that using it for basic system administration tasks made absolutely no sense. at least what they used it for.
it's used in chef/puppet for one reason only. because it's easy to make a dsl in ruby, and it's good for that purpose actually. although i find chef terribly overengineered
Funny enough, I'm using a combination of small shell scrips that are executed within an internal application depending on the status of the the machine. So I guess I'm in the "why not use both" position.
Currently, I have automated system set up where basically a cluster of vm's in nodes of two communicate with each other to pass over where the other server in a node left off in its operation (or dies from timeouts) running nginx/gunicorn/django.
And those servers on each node are monitored by another server (running apache/php/mysql) that checks the progress/status of the operations and may send requests (reinitialized a node that stopped running) to the nodes where those bash scripts (concatenating files, finding a specific place in one of the files to help reinitialize a process in a node) are executed and piped through back to the monitoring server.
It is way more complex now, but surprisingly, I don't have troubleshoot not even close to as much as before since I automated that.
> As an example of how incredibly dumb it can be to reinvent the wheel
That's a misunderstanding. It's like saying "Java is a reinvention of Haskell". Neither is a reinvention of the other, they are two very different beasts. You don't have any OS whose entire state (package and configuration) is described at a central location and can be recreated by a single command (outside of research projects, possibly). You may disagree with the approach they take, but such a blunt dismissal dismisses at the same time any kind of original approach to a problem.
Speaking of which, you speak in favour of shell as a solution to automation. What you are saying is that you push the complexity on the side of the shell script developer. In my experience, declarative systems tend to have more complex innards, but it also means more maintainability, and a flow much easier to understand. This is the kind of tradeoff you make when forsaking sysvinit in favour of systemd.
I am basically a Perl developer by trade, and have been building and maintaining customized Linux distributions for large clusters of enterprise machines for years. I would still rather use shell scripts to maintain it all than Perl, or Python, or Ruby, or anything else, and would rather use a system of 'stupid' shell scripts than invest more time in another complicated configuration management scheme.
Why use shell? It forces you to think simpler, and it greatly encourages you to extend existing tools rather than create your own. Even when you do create your own tools with it, they can be incredibly simple and yet work together to manage any aspect of a system at all. And of course, anyone can maintain it [especially non-developers].
As an example of how incredibly dumb it can be to reinvent the wheel, i've worked for a company that wanted a tool that could automate any task, and that anyone could use. They ended up writing a large, clunky program with a custom configuration format and lots of specific functions for specific tasks. It came to the point where if I needed to get something done I would avoid it and just write expect scripts, because expect was simpler. Could the proprietary program have been made as simple as expect? Sure! But what the hell would be the point of creating and maintaining something that is already done better in an existing ages-old tool?
That said, there are certain tasks i'd rather leave to a robust configuration management system (of which there are very few in the open source world [if any] that contain all the functionality you need in a large org). But it would be quite begrudgingly. The amount of times i've ripped out my hair trying to get the thing to do what I wanted it to do while in a time and resource crunch is not something i'd like to revisit.