Is Systemd that bad afterall?

danie10 , 11 months ago

We’d probably need to qualify this with “bad compared to what”. I can’t complain, as it does its job, and I’ve been able to tweak what I needed to. As I don’t tinker with it every week, I keep a sticky note rolled up on my desktop, or I quickly use ‘cheat systemd’ to remember some key examples.

I was getting really long start up time earlier this year (like 19 mins before the desktop was fully responding) and after trying everything else I tried ditching BTRFS and reverting my /home drive back to ext4. Turns out BTRFS start and checks was killing my boot times. Now, as fast as anything.

The following have been my saviours though in identifying boot times: journalctl -b -p err systemd-analyze blame --user systemd-analyze blame

argv_minus_one , 11 months ago

SystemD is blamed for long boot times

That is and always was nonsense. Systemd shortens boot times by starting things in parallel. That’s one of its key features.

There are some things to note about that:

Systemd only starts services in parallel when it isn’t told otherwise by Before and/or After settings in the service files. This makes it pretty easy to make systemd slow by misconfiguring it. You can use the systemd-analyze program to see which services held up your boot.

Systemd has a very long default timeout (90 seconds) for starting or stopping a service. It’s appropriate for the big, lumbering servers that systemd was probably designed for, but it might be wise to shorten the timeout on desktops, where a service taking more than 5 seconds to start is almost certainly broken. It’s a setting in /etc/systemd/system.conf.

Is the current SystemD rant derived from years ago (while they’ve improved a lot)?

I’m an early adopter of systemd. I installed it on my Debian desktops pretty much as soon as it was available in Debian, and I later started moving servers to it as well. I had long been jealous of Windows NT’s service manager, and systemd is exactly what I had hoped would come to Linux one day.

Yes, the rant you’re talking about is old, and yes, systemd is better now than it was then, but not in the sense of what the rant was complaining about. The rant was already patent nonsense when it was written, which has given me a very dim view of the anti-systemd crowd.

Besides systemd proper, they also spent a lot of time ranting about the journal system, which redirects syslog entries into a set of binary log files. They complained that this would make logs impossible to read in emergencies. This isn’t even close to being true—any emergency bootable Linux image worth its salt has a copy of journalctl on it—and the binary nature of systemd’s logs has caused me serious problems on exactly zero occasions.

bloodfart , 11 months ago

Not against systemd (although it’s bad and needs replacing), just against pottering.

corsicanguppy , 11 months ago

1. systemd hasn’t become a better project built by better, smarter people to deliver a better set of features. It’s still hot garbage.
2. it’s okay to continue pointing out it’s hot garbage, in the hopes we can go forward or back or just get on something better/else (same thing).

ozoned , 11 months ago

Ok, so I have a very unique background in systemd. I worked at Red Hat supporting it basically as the primary support and I’ve worked with the developers of systemd at Red Hat directly. I no longer work there.

So first off, it’s “systemd” all lower case. I don’t care, but for some reason Lennart Pottering (creator) does.

systemd was a MASSIVE change. And Red Hat did a TERRIBLE job relaying it. To the point where I’m still trying to get my company to understand that it can NOT be treated like the old init systems. You can NOT just drop an init script in place and walk away and hope it works. Because a LOT of times it doesn’t. Due to forks, switch users, etc.

systemd is NOT an init system. RHEL 5 and older had sysvinit as it’s init systemd. RHEL 6 had UpStart as it’s init system and looked exactly like sysvinit that no one even noticed. systemd again is NOT an init system. Init system is 1 part of systemd. systemd does a lot of cool things. It bundles applications together, it manages those applications and can restart them or kill children, it can do resource constraints, it separates out users from the system, and lots more.

Because it is not an init system there is a LOT LOT LOT of bad recommendations out on the internet where someone has X problem and person suggests Y and IT WORKS! … except it doesn’t REALLY work as far as systemd is concerned and you’ll hit other issues or your application takes longer to start or stop and people just blame systemd.

It is systemd’s fault that it has done an ATROCIOUS job of helping people adapt. It’s a great example of RTFM. systemd’s man pages are INCREDIBLE and extensive, but when you drop so much knowledge it becomes more difficult to find what you want/need. systemd.index and systemd.directives are your best bet.

So systemd does a lot of amazing things that sysvinit never attempted to do. It’s never attempted to explain anything it expects everyone just learn magically. it’s INCREDIBLY complex, but once you understand it’s basics you can more easily get an application running, but as soon as there’s a problem it’ll just break your brain.

To give you an example, sshd’s old init script is like 250 lines of bash. systemd’s unit file comparative is like 12. Because systemd handles a LOT of what you manually had to handle before. BUT to get to that 12 you literally have to learn EVERYTHING new.

There is no “is it good or bad” here really imo. It’s a completely different fundamental design. Red Hat made it for themselves. Other distros picked it up. It can be argued that lots of folks followed Debian and Debian had a few Red Hat board members that were pushing it. Whether they pushed it of their own accord or because they were with Red Hat I don’t have a clue.

What I can say is at my current company they’re suffering from a LOT of systemd issues and they don’t even realize it. I’ve been working with Red Hat to try to get Insights to alert people to the failures and we’re making progress.

To see if you have issues just to start run the two following commands:

<pre style="background-color:#ffffff;">
<span style="color:#323232;"># systemctl list-units --failed
</span><span style="color:#323232;"># systemd-cgls
</span>

If you have any units that are failed, investigate those. If you don’t need them, disable them. As for the systemd-cgls this shows HOW systemd is grouping things. ANY application that runs as a service (or daemon or application or runs in the background or however you wanna say it) should be under system.slice. ONLY humans logging into the system (meat bags NOT applications switching to users) should be in user.slice. A LOT of times what happens is an old init script is dropped in place, they start it, it has a switch user and systemd assumes it’s a user and puts it into user.slice. systemd does NOT treat anything in user.slice the same as in system.slice and this WILL eventually cause problems.

So again, is it good or bad? Eh. It does a lot of cool things, but they did a MASSIVE disservice to ALL of us by just expecting to relearn absolutely EVERYTHING.

nyan , 11 months ago

sshd’s init script under OpenRC is 87 lines, of which around half are blanks, comments, closing braces, and other boilerplate. Granted, that still makes the real code maybe three times the size of your systemd unit file, but the difference isn’t as impressive as you’re making out.

95% of people shouldn’t need to poke around in their init scripts or unit files anyway. If you actually need to do that, your use case is already somewhat unusual.

ozoned , 11 months ago

As an end user, unless you’re running a server, then no you shouldn’t have to mess with any of it.

If you’re running a server or a sysadmin you absolutely 100% should be paying attention. Almost every single vendor I’ve seen selling their applications only have initscripts. Which then cause issues. I’ve gone to the vendors and told them and they’ve said go to Red Hat. Well Red Hat doesn’t support that vendor’s init scripts.

Not naming an application, but it was from a BIG BLUE company and they said their only instructions are to call their script from the user. But it won’t remain running if you do that because systemd will close out the slice when the user logs out. SO it’s obvious they haven’t tried what they’re suggesting.

And I’m not attempting to state that systemd is impressive in any way. systemd basically took what had been building over 40 years of init scripting and threw it out the window and said our way is better. I don’t think it is. I’m just saying, with a directive based unit file it’ll be simpler to parse than a bash script.

gnumdk , 11 months ago

Just try to implement user session management on a non systemd distro…

Systemd is way better than others init system. I’m using Alpine Linux on my phone and I really wait for a Fedora/Arch like PMOS project (it’s on the way)

jarfil , 11 months ago

[pi@raspberry]# sudo su

Just saying, not everyone needs session management…

SneakyThunder , 11 months ago

sudo su

Why spawn additional process when you can get into shell directly with sudo -s?

nyan , 11 months ago

Well, sudo itself is a purely optional component—you can run a system quite happily with just su .

wgs , 11 months ago

What do you do with all the process you save with that trick ?

monobot , 11 months ago

Keep in mind that it all started 20 years ago with Pulseaudio. Pottering was not really a nice guy (on mailing lists ofc, I don’t know him personally) whose software I wanted on my machine.

Problem was never speed or even technical, problem was trust on original author and single-mindedness that they were promoting. Acting like it is the only way forward, so anyone believing in freedom part of free software was against it. Additionally, it was looking like tactics used by proprietary software companies to diminish competition.

It looked scary to some of us, and it still does, even worse is that other software started having it as hard dependency.

All of this looks like it was pushed from one place: Portering and RedHat.

While after 20 years I might have gotten a bit softer, you can imagine that 15 years ago some agresive and arogant guy who had quite a bad habbit of writing (IMHO) stupid opinions wanted to take over my init system… no, I will not let him, not for technical reasons but for principal.

I want solutions to come from community and nice people, even if they are inferior, I will not have pottering’s code on my machine so no systemd and no pulseaudio for me, thank you, and for me it is an important choice to have.

argv_minus_one , 11 months ago

Keep in mind that it all started 20 years ago with Pulseaudio. Pottering was not really a nice guy (on mailing lists ofc, I don’t know him personally) whose software I wanted on my machine.

Poettering is like Torvalds: gruff when pressed, but not wrong.

PulseAudio is like systemd: dramatically better than what came before, and the subject of a great deal of criticism with no apparent basis in reality.

PulseAudio did expose a lot of ALSA driver bugs early on. That may be the reason for its bad rap. But it’s still quite undeserved.

Additionally, it was looking like tactics used by proprietary software companies to diminish competition.

This is a nonsensical argument. Systemd is FOSS. It can and will be forked if that becomes necessary.

Which, in light of recent changes at Red Hat, seems likely to happen soon…

Problem was never speed or even technical, problem was trust on original author and single-mindedness that they were promoting.

That’s because fragmentation among fundamental components like sound servers and process supervisors results in a compatibility nightmare. You really want to go back to the bad old days when video games had to support four different sound servers and the user had to select one with an environment variable? Good riddance to that.

I want solutions to come from community and nice people

Then you’d best pack your bags and move to something other than Linux, because Linus Torvalds is infamous for his scathing (albeit almost invariably correct) rants.

monobot , 11 months ago

Poettering is like Torvalds

Lol, not even close. I am not talking about being harsh for writing stupid code. Nor I want to go 20 years back to proove it to some random person, do it yourself.

Systemd is FOSS. It can and will be forked if

Yeah, the same way chrome can be forked. No, software developed like that - in closed room just source being dropped on to community, what happened with PA and SD in the begging no one wants to touch. Gentoo had big problems just maintaing eudev and elogind to enable gnome and some other software to work.

Luckily, it is not important anymore, there is pipewire so I managed to skeep PA completely.

JoYo , 11 months ago

It would be fine if it kept to system init rather than growing like a cancerous tumor.

wiki.gentoo.org/…/Hard_dependencies_on_systemd

lightrush , 11 months ago

1. Is the current SystemD rant derived from years ago (while they’ve improved a lot)?

No it’s almost always been derived from people’s behinds.

2. Should Linux community rant about bigger problems such as Wayland related things not ready for current needs of normies?

Yes.

Systemd is spectacular in many ways. Every modern OS has a process management system that can handle dependencies, schedule, manage restarts via policy and a lot more. Systemd is pretty sophisticated on that front. I’ve been able to get it to manage countless services in many environments with great success and few lines of code.

nyan , 11 months ago

Speaking as someone who uses OpenRC on all my machines . . . no, systemd is not necessarily slow, and personally I don’t care about the speed of my init system anyway. Thing is, systemd also has nothing that makes it more useful to me than OpenRC, so I have no incentive to change. Plus, I dislike the philosophy behind it, the bloat, and the obnoxious behaviour the project showed when interacting with others in its early days. I’m a splitter, not a lumper, and systemd’s attempts to absorb All The Things strike me as rather . . . Windows-like.

So, in a technical sense I have no reason to believe that systemd is inferior to OpenRC + sysv, and it may be superior for some use cases which are not mine. I don’t spend a lot of time ranting about it, and I see no point in trying to convince people not to use it if it fits their needs. But I still won’t use it if I have another option.

chaorace , 11 months ago

I agree. SystemD is a great service daemon (or, sigh, unit daemon in the stupid parlance). I like unit file syntax and I like the ergonomics of systemctl. It’s solid and I appreciate the feeling of consistency that systemd lends to the otherwise chaotic landscape of Linux distrobutions.

It’s for this reason that I’m willing to forgive SystemD overstepping the boundaries of services somewhat. System init/mounting? Sure, that’s a blurry line after all. Logging? Okay – it does make sense to provide a single reliable solution if the alternative is dealing with dozens of different implementations. Network resolution & session management? Fine, I’ll begrudgingly accept that it’s convenient to be able to treat logins/networking as psuedo-services for the sake of dependencies.

If that’s as far as the scope crept, SystemD and I would be cool, but the so-called “component” list just keeps on going. SystemD has no business being a boot manager, nor a credential manager, nor a user manager, nor a container manager, nor an NTP client. I understand why they can’t deprecate most of this junk, but why can’t they just at least make this cruft optional to install?

Atemu , 11 months ago

Systemd (PID1) is not your boot manager, network deamon, resolver, user manager or ntp service.

Those are entirely independent deamons that happen to be developed under the systemd project umbrella but can be exchanged for equivalent components.
Tkey are gully optional.

In many cases, the systemd project’s one is one of the best choices though, especially when used with other systemd-developed components.
In some cases, there is no other viable choice because the systemd-* is just better and nobody wants to deal with something worse.

ikidd , 11 months ago

As a guy that’s been installing Linux since you had to compile network drivers and adjust the init scripts to use them; SystemD rocks.

PlaidDragon , 11 months ago

A lot of the people I see complaining about it are comparing to what was before it.

As someone who has only ever known systemd, I have no issues with it and, dare I say: I like it.

eleitl , 11 months ago

The problem of systemd is that it hasn’t been just a replacement of init as they initially claimed, and now deny they ever did. Things like Mono, Gnome and systemd are bad for the ecosystem long term.

An init done by constructive people wouldn’t be a problem at all.

Fryboyter , 11 months ago

The problem of systemd is that it hasn’t been just a replacement of init as they initially claimed

Apart from the PID 1 part of systemd, almost all tools are optional.

Although I have a positive opinion about the systemd project, I used netctl instead of systemd-networkd for a long time without any problems. And even today I don’t use systemd-resolved because I use a combination of unbound and Pi-Hole in my private LAN. And so on.

So you can’t say that the systemd project has replaced various solutions in such a way that you don’t have a choice anymore.

eleitl , 11 months ago

I was more referring to things like e.g. wiki.gentoo.org/…/Hard_dependencies_on_systemd

Notice that it’s from 2021 and just for Gentoo. This is what people politely describe as invasive.

taladar , 11 months ago

Honestly, that looks like a fairly short list and half of the tools interact closely with useful functionality that didn’t even exist at all before systemd came around.

russjr08 , 11 months ago

I do not think systemd is bad, I (and personal preference here) much prefer it over the older style of init systems.

Quite frankly, one of the things that has always irked me about a portion of the Linux community is that as far as I know, a strength and selling point of Linux has always been the freedom of choice. And yet, people start wars over your choices. For example, I know at least on r/Linux if you were to make a post saying that you liked Snaps over Flatpaks you’d get torn to shreds over it. Wouldn’t matter what reasons you had either.

It is always something. Whether its about Arch vs other distros, Snaps vs Flatpak vs AppImage vs Traditional packaging, X11 vs Wayland, systemd vs Sys V/init.d, pulseaudio vs pipewire, etc.

I never understood why it mattered so much what someone ran on their own computer. Assuming they’re the only one using it, what is the big deal if they choose to run OpenRC, X11, Snaps, and Alsa?

And I get a bad feeling the next one is going to be immutable distros vs non-immutable distros, but I guess we’ll see.

addie , 11 months ago

It’s a massive question, and I think quite a lot of the argument comes from the fact that it depends what direction you’re answering it from.

As a user, do I like being able to just systemctl enable --now whatever.service , and have a nice overview of ‘how’s my computer’ in systemctl status ? Yes, that’s a big step up from symlinking run levels and other nonsense, much easier.

As an administrator, do I like having services, mounts and timers all managed in one way? Yes, that is very nice - can do more with less, and have to spend less time hunting for where things are configured. Do I think that the configuration files for these are a fucking mess of ‘just keep adding new features in’ and the override system is lunacy? Also yes.

As a developer trying to do post-mortem debugging, who just wants all the logs in front of him for some server that’s gone wrong somehow, which I often have to request via an insane daisy-chain of emails and ‘Salesforce nonsense that our tech support use’ from our often fairly non-technical end users, on some server that I’ve no other access to? No, I do not find having logs spread between /var/log and journalctl (and various CloudFormation logs in a web console) makes my life easier. I would be pleased if that got sorted out.

tl:dr; mostly an improvement, some caveats.