hckrnws

I know everyone hates on systemd, but I've generally been happy with it, in no small part because I find the timers to be pretty sensible.

They are, in all ways that I care about, simply better than cron, and especially in NixOS they're really easy to set up.

+1 for NixOS. I used to find systemd a bit intimidating, but working with NixOS has made it all fit together somehow.

As an example, this is a systemd timer I have that periodically runs rclone to sync photos to Backblaze:

    systemd.services.rclone-photos-sync = {
      serviceConfig.Type = "oneshot";
      path = [ pkgs.rclone ];
      script = ''
        rclone \
          --config ${config.age.secrets."rclone.conf".path} \
          --bwlimit 20M --transfers 16 \
          sync /mnt/photos/originals/ photos:
      '';
      unitConfig = {
        RequiresMountsFor = "/mnt/photos";
      };
    };
    systemd.timers.rclone-photos-sync = {
      timerConfig = {
        # Every 2 hours.
        OnCalendar = "00/2:00:00";
        RandomizedDelaySec = "5m";
        Persistent = true;
        Unit = "rclone-photos-sync.service";
      };
      partOf = [ "rclone-photos-sync.service" ];
      wantedBy = [ "timers.target" ];
    };

I had to write a systemd timer to rescan PCIe ports about ten seconds after the computer starts. I was able to add that to my NixOS config and it worked how I wanted, and it was great because if I had broken anything all I would have to do is reboot and choose an older generation.

I like so much about Nix and the Nix philosophy, but I'm a bit stuck on the perceived ugliness of the Nix language. It just feels so ... icky to me. I don't typically feel that way about languages. Even Erlang (while being bizarre) didn't/doesn't bother me that much.

Does anybody else have this problem? How do you get over it? (it seems like such a ridiculous reason to avoid picking up an otherwise good tech)

I think at this point it's a quasi Stockholm Syndrome for me, because I've grown to genuinely kind of like it.

I started making my own packages for Nixpkgs, and I have made dozens (maybe hundreds) of flakes, so I'm just kind of used to the language now. It's an ugly language, but once you write a lot of it, you get used to its quirks.

And there are some niceties, like being able to define a variable and use that variable to access and define fields, kind of like JavaScript:

   {
      blah = {
        ${myVar} = "something";
      };
    }

Otherwise it's just kind of an awkward functional language. You get used to it.

You may wish to give Guix a try. Nix philosophy, but the language is Scheme.

That would all be one line in a crontab.

The article already explains the difference between cron and systemd timers, so I won't rehash that. Yes, systemd is more verbose, but that comes with advantages over cron. And if cron suits your usecases, it's a fine tool as well.

But with NixOS and this example specifically, you also get (at a minimum):

a) A declarative config without requiring an external tool like Ansible.

b) Any dependencies (rclone in this case) will be implicitly installed if not already present on the system.

c) If configuring a remote machine, this will copy over the encrypted rclone.conf file and decrypt it on the target.

d) And of course, it's trivial to version control and track changes to the config over time.

I want to use NixOS but man it’s a mess with its flakes vs no flakes debates. I’ve stuck with learning Ansible for now, but if flakes is ever properly built in I’d consider it.

Flakes are as stable as they can get imo. They are already widely adopted, so I don't see how they can ever completely walk the feature back. The unstable marker is only there because there are some edge cases to work around afaik.

And I agree that NixOS without Flakes is a worse experience overall.

Putting aside that this is all possible with nix and without systemd, the additional complexity strikes me as having hamstrung the linux community for the last decade. I hope reliable service management arises but I have yet to see the benefit.

In the meantime I'm just going to keep using void because I know how to fix it when it breaks. Systemd i have to google how to even find the service logs.... they certainly aren't easy to find via the filesystem any more.

Like I said, it is a bit intimidating at the start. But you soon realize it’s quite intuitive.

Finding logs for a specific service is a one-liner:

  journalctl -u rclone-photos-sync.service

Here is a nice overview: https://www.digitalocean.com/community/tutorials/how-to-use-...

Well, above a level tools can't be designed to be discoverable without any form of help/learning, and I believe that's fine. They should rely on a "common knowledge base" as much as possible, and I believe systemd commands are done in a POSIX-y style, but you do have to occasionally read a man page or Google.

Do we expect an average dude to just pop into a cockpit and land a plane with no previous training?

That isn't inherently better.

The really nice thing about systemd timers is that as units you can get status of them and check the journal output the same as any other unit.

How is ‘persist’ realised with cron? What about ‘randomized delay’?

I'm not sure what 'persist' means here but at least some cron implementations have support for randomization in times or time intervals.

    the time when the service unit was last triggered is stored on disk. When the timer is activated, the service unit is triggered immediately if it would have been triggered at least once during the time when the timer was inactive. Such triggering is nonetheless subject to the delay imposed by RandomizedDelaySec=. This is useful to catch up on missed runs of the service when the system was powered down.

I'm convinced that the people who hate systemd are a small minority. They also happen to be fairly vocal, which I think makes them seem outsized.

I think in general it's a shrinking minority. I was outspokenly critical of systemd in the past, but much about it has improved, and I have to admit it has made my life significantly easier in many ways. I'm still not a big fan of some things being systemd projects that don't really need to be under the umbrella (why is resolved even a systemd project? Or systemd-timesyncd?) but service definition and management, transient services, sockets, timers, and many many more things are pretty great. At this point, I pretty much love it. Even things that initially annoyed me, like timers, I've grown to love, not least because my cronjob entries are manageable files that I can commit and backup, and I can individually enable and disable them without having to deal with stupid commenting and such.

Checking service status and managing services is also significantly better with systemd than any other init system I've used, and I've used a lot over 15 years of systems administration.

I still find journald a bit annoying, though, and I still am not completely sold on binary log files. I understand the benefits, but working with them is still much more opaque than text logs.

>I still find journald a bit annoying

I've come to like it. At first, probably around 15 years ago, I was kind of annoyed at it and just wanted my old /var/log/syslog back. But it has some really nice features, especially for log processing and tooling, like "show me logs from the last minute" (--since) or "show me logs from where I left off" (--cursor).

I'd be excellent to see their complaints collected and elaborated. Because God Fucking Damn, these seem like some incredible whining Advanced Persistent Threats against anything getting better. If I were a pro-discord disruptor, I'd pay these people to keep at their endless grudges against every single person having to figure out ever single fucking capability de-novo.

Systemd makes so so so many incredibly good service administration capabilities easy & on tap. The main thing I want from systemd haters is to show how how how folks did things well before. Mostly I think this is a rebellion of the losers, people who hate having to do a good job, people who hate learning the many many many many ways we could run services better, who valorize each service figuring out their own unique special bespoke ways of running their stuff. Its fallen as fuck. We need need need the systemd socialization of running stuff well, controlling permissions & access. There's near zero precedent from the past of anyone else being responsible. That's just the situation. Systemd has drastically elevated what sysops can do, orthogonally to the service by service init launching of the past that got us no where.

> I'd be excellent to see their complaints collected and elaborated.

https://web.archive.org/web/20250310201357/https://nosystemd...

I challenge the authors of this website to show me a software written in the last 10 years that never had a bug. I have scanned the first 20-ish items and it seems those are either fixed or not considered important (unsure about the latter; didn't give them too much reading as of the time this comment was written).

In general it seems it's just a collection of GitHub issues. The FQDNS thing seem to be people outraged at systemd for something that the Linux kernel does not (or did not?) do well with certain hostnames -- those ending with a dot.

And there are some security issues / CVEs which is regrettable and I hate it but again, show me software that does not have them.

Seems like a very petty way to just attack something from a cursory glance. The author(s) might have as well made a ClickHouse remote query directed at systemd's GitHub issue tracker. ¯\_(ツ)_/¯

This is ultimately such a weirdly aggressive comment. I don't feel like systemd or its alternatives drastically elevated what ops can do.

Most goals achievable in 2025 are those achievable in 1995.Its a small problem space without 30-60 years of headroom to explore and by simple logic most interesting things are going to be outside this fairly basic space.

It provided greater standardization and more power like 5 other options many of which existed prior with a simple syntax which is not without virtue.

Its just weird to refer to proponents of alternatives as "losers" who hate learning things when most proponents know systemd and other options.

One might even go so far as to say they're a vocal minority.

One might even go so far as to say all Linux users are a "vocal minority" to begin with.

I'm familiar, but I broke it up that way for a reason.

Systemd hater here. My stance is and always was: if you love systemd, go nuts with it. I'm not looking to reach into your bathroom to set your shower temperature.

But I don't want to have to run it or even think about it. Lennart has other plans, hence his campaign to get distros to require it and software up the stack to hard-depend on it. I don't like its design, I want to opt out of it in favor of something better, but one of the development effort's goals is making that extremely difficult for modern Linux systems. That's my issue with it.

But really it macht nichs for me right now because I run Void, btw.

I was feeling a bit Kafka-ish with all the gushing systemd support until I saw this comment. And same here; I don't care what other folks are enamored with, but them "other plans" you mention, they concern me and I wish it concerned others too.

>I don't like its design, I want to opt out of it in favor of something better

Others design were already tested extensively, and the one that stuck was systemd, don't you already stopped to think that the "better design" is the one that systemd are currently using?

Why are you even using Linux to begin with? Other OS designs have already been tested extensively, and the one that stuck was Windows. Didn't you consider that Windows is the better OS for your needs?

My criteria for a good init system design do not align with Lennart's, or with Red Hat's. It's not winner-take-all in principle, even if it approaches being so in practice.

> distros to require it

Distros requires a lot of stuff, e.g. libc. This does not seem inappropriate given what a distro is.

> software up the stack to hard-depend on it

What does this refer to?

I can see this for session/user/service management programs. But not normal single-user programs.

I was initially against systemd (around the time Debian started discussing using it) for non-technical reasons: the project seemed to have some very negative aspects in terms of how feedback and concerns are addressed. I don't know for sure but I strongly suspect this was the core issue a good chunk of those "against" were concerned about: the way the project is run.

Having used it for several major releases of Debian technically I'm happy to have it - it's an improvement over the sysv init & crontabs. I'm not completely in love with journal compared to regular logs but I can see some advantages.

I actually love systemd like 96% of the time.

The only time I don't is when it just cannot do something, and then you need to layer the old Unix way on top of it using the wrong semantics, i.e., services that are actually complicated bash scripts which are used to do something for which systemd has specialised units, like an (auto)mount. Then it's all the pain of brittle Unix scripts, with misleading semantics, and an extra layer of abstraction and set of limitations to keep in your head.

But no, I genuinely think that systemd is great these days.

I do understand why people become irked by it becoming a dependency of programs which really should be able to function without it though.

One thing that makes it difficult is how scattered between all the different manpages the documentation is. It took me quite a lot of time and frustration to build up a decent mental model of what everything is for, when it needn't have if (ironically) the docs were just written in a less Unixy way.

But making my system feel more event-driven and less imperative has been a pleasure.

When I started using systemd timers, I really liked the systemd-analyze calendar facility, to calculate n trigger times for a given calendar expression.

For example, show the next five trigger times for the end of the last day when the month has 31 days :

systemd-analyze calendar --iterations=5 '*-*-31 23:59:59'

There is one huge advantage of cron that is usually missed with such comparisons. cron is dead-easy to create and maintain. Systemd timers make sense in certain situations, but in 90%+ of my use cases so far, the added complexity really adds up the amount of time I'm administering my systems. I've been in a situation where I needed to troubleshoot why a systemd timer wasn't triggering, and I didn't like that experience at all. To me, it is something I would use if I actually needed it, it isn't a 1-to-1 replacement for cron.

I actually disagree with simplicity. cron does weird things with the environment and it can be hard to correctly reproduce that environment when creating a job and debugging. Conversely, systemd timers are very easy to manually trigger and otherwise behave as a normal oneshot service.

I’m the developer of crontab.guru and Cronitor.

We have a free tool called CronitorCLI that includes a cron-like shell. You can run and test your scripts in an environment that matches how they will be run by cron itself.

Dude! you're my hero! This is why I like HN :)

Thank you for many years of reassuring me when I doubted my cron-expressions.

As somebody who has made cron jobs my career for the last 11 years, who left a good gig as an engineering director to build a bootstrapped business — that means a lot. It keeps me chugging. Thanks so much.

My approach to cron is to assume that you have an empty environment. Just set everything you will need.

Yes this is correct. Something most developers don’t know is that you can set env vars directly in your crontab file.

So you find troubleshooting cron jobs easier?

Cron has no diagnostics whatsoever. Most asked question "How do I force run cron job for debugging" and answer "You don't"

You just add script and pray it works. If it doesn't work there is nothing to work with. No logs, no checks. Nothing.

It not 1-to-1 replacement. It is superior replacement. Cron is just unusable.

> You just add script and pray it works.

All the cron issues I've had boiled down to a difference in env vars. So testing the cronjob means reproducing the environment like this SO answer explains.

https://stackoverflow.com/questions/2135478/how-to-simulate-...

I found the implicit column-based configuration layout of cron rather cubersome. I prefer the key-value configuration layout of systemd timers.

There is also one huge disadvantage of cron: it requires root. Users can create and run systemd timers without root/sudo.

And quite frankly, your experience is likely due to your personal knowledge bias. Systemd timers are quite easy to debug, in fact easier than cron in my opinion. systemctl list-timers lists all the timers, when they last ran, when they will run next, and you can use other commands to inspect each timer in detail and all associated logs.

In contrast, I find cron much harder to debug. Starting with the first problem, you must first figure out what cron is running, as different implementations have different behavior!

One can set up cron jobs under non-root.

You cannot edit /etc/crontab without root. I'm not talking about dropping privileges for the job.

Although speaking of which, cron dropping privileges is not as secure as systemd running as the user before parsing the user's timers, from a defense-in-depth perspective.

But non-root users still have their own crontab, which you don't need sudo to edit.

That depends on which cron you're using, not all of them support user crontabs (and the ones that do probably have subtle differences in behavior).

I was unaware of that, so thank you for the education. But for what it's worth, I have never seen a Linux system which didn't have user crontabs. So even if not universal, it is at least very widespread to the point that you can bet on its presence.

Give me an example of cron, which doesn't support user crontabs? Let me list three that support user crontabs: (a) cronie is controlled by /etc/cron.{allow, deny}; when the latter files are not present, every user can submit cron jobs to crond. (b) One can set up Anacron within the user's home directory[1]. (c) systemd timer is another kid on the block; any user can create a systemd.timer unit.

[1] https://opensource.com/article/21/2/linux-automation

It’s true that root can disable user crontabs but the cron that’s shipped with all popular Linux distributions does have built-in support for them.

Minor nitpick - shouldn't you first define the service and only then a timer for it? Otherwise since you enabled timer and are still trying to figure out how to write service, systemd won't have anything to run when timer triggers. Maybe I am wrong, but that just feels like logical order. Anyways, after years on hating on systemd I also started to embrace it and porting my cron jobs to systemd timers and I must admit it's really nice, the overall overview with list-timers, next execution timestamp, total execution time, ordering of services so one can run after another is completed and of course the logging in journal so I can filter output and keep track of everything it's just wonderful experience.

EDIT: yea, the email reporting is certainly missing, but it was hard to control it since whole STDOUT was shipped, which is not what I wanted most of the time anyways. It would be good to come up with some way to still have small overview emails sent about important jobs done, maybe a dependency service which starts when important job finished and just sends an email about that

Really liked this read. Is anyone able to explain how the backup.timer runs the backup.service? It wasn't obvious to me where the trigger was defined. I guess it's just inherent to what that unit type does, and the fact that both units are named backup? What is the name for that "package" of units that make up the whole backup program?

> I guess it's just inherent to what that unit type does, and the fact that both units are named backup?

Correct. If there's no `Unit=` specified in the timer unit, it defaults to the service unit of the same name. See https://www.freedesktop.org/software/systemd/man/latest/syst...

This is also a common pattern for the socket units of socket activated services

It took a minute to setup, but using a combination of rsync and timers to backup system files has done wonders to decrease my anxiety around upgrades on arch

Using a modern CoW filesystem lets you level this up to just a snapshot command in a pacman pre-upgrade hook.

I used systemd user timers, a shell script, and an HDHomerun to make a simple VCR.

I like systemd's timers when they're appropriate, but I really think the author's use case here is better suited to cron. The "issues" he listed for cron aren't very good either.

> If you want to execute pre/post commands you have to do it inside the script itself

So?

> There are no built-in logs

Every cron implementation I can remember using logs each run to syslog and emails me the output of the run by default

> There is no built-in status monitoring

I can't think of any built-in status monitoring that systemd has for timers that's materially different from cron's logging/emailing

> If the system is down when the cron needs to run, the cron will be missed

Some cron implementations support this and some don't. Most modern ones that I'm aware of do.

Much more significantly, the amount of setup involved in a systemd timer is way higher than putting a line in a crontab, especially for the author's case of just running a backup script.

Cron only involves running `crontab -e` and adding the line "@daily /path/to/script.sh" (which also handles the author's issue of cron "skipping" runs if the system was powered off, assuming the cron implementation uses something modern like anacron)

Systemd involves writing a 7 line timer unit file, an additional 5 line service file, running a daemon-reload, then enabling the timer. It turns what's usually a 10 second mindless task into a much more involved procedure. That can be worth it if there are material benefits from it, but I'm not really seeing them here.

IMO Systemd Timers provide much better control over how the cron job is run.

Need to distribute lots of cron jobs evenly to avoid overloading the system? Use RandomizedDelaySec.

Some cron jobs are flaky and you want to re-run if it fails? Add Restart=on-failure to corresponding service.

Some cron jobs conflict with each other? Set Conflicts=foo.service or maybe Before & After.

Sure, all above are possible with shell scripts. But systemd provides a standard, reliable way to define these properties.

> Systemd Timers provide much better control over how the cron job is run

Yes, they do. And as the very first sentence in my comment says, I like systemd timers when they're appropriate. None of the features you've mentioned were used in the author's solution, which is the sole thing I'm arguing against.

> Much more significantly, the amount of setup involved in a systemd timer is way higher than putting a line in a crontab

Counterpoint: this requires root, while you can edit and run systemd timers without root. Thus it's also generally more secure, both when creating the job and every time it runs.

> Systemd involves writing a 7 line timer unit file, an additional 5 line service file, running a daemon-reload, then enabling the timer. It turns what's usually a 10 second mindless task into a much more involved procedure. That can be worth it if there are material benefits from it, but I'm not really seeing them here.

You could write a script to automate it, if it's such a big deal. Creating timers/cronjobs isn't something that needs to be done often enough for this to matter.

If it is, another counterpoint: systemd supports creating transient timers, which you could do programmatically.

And final counterpoint: you can make systemd understand crontab and convert it into timers (systemd-crontab-generator)

> Counterpoint: this requires root

Counter-counterpoint: No it doesn't? Running `crontab -e` as a non-root user will edit that user's crontab, and running it as root will edit the system crontab. Cron can be configured to deny users their own crontabs, but every common distro I'm aware of defaults to allowing user crontabs.

> You could write a script to automate it, if it's such a big deal

Or I could not bother with that and just use cron?

> another counterpoint: systemd supports creating transient timers

What is this a counterpoint to? That the author's particular use case of running a backup script once a day is a task better suited to cron than systemd timers? I'm not sure how transient timers are even relevant here, much less a counterpoint.

I think support for per-user crontab's is implementation dependent. There are implementations (AFAIK) that do not support per-user crontabs.

Per Gentoo's wiki, both fcron and cronie have their own (different!) ways of whitelisting non-root users to run cronjobs.

Everything is implementation dependent. There's no ISO cron standard or anything.

> Per Gentoo's wiki, both fcron and cronie have their own (different!) ways of whitelisting non-root users to run cronjobs.

Both allow any user in the cron group to have their own crontabs by default, and both support optional .allow/.deny files

> There's no ISO cron standard or anything.

It's part of the Single Unix Specification.

https://pubs.opengroup.org/onlinepubs/9799919799/utilities/c...

That's a light spec for a subset of the interface and behavior of the crontab command line frontend, not a standard for cron in general. It's not really relevant to a discussion about how distros set up their cron defaults for per-user access.

Thanks for sharing it though. I wasn't aware it was in the posix spec and it explains why pretty much every implementation supports .allow/.deny files even when most already implement better access control mechanisms.

> Running `crontab -e` as a non-root user will edit that user's crontab, and running it as root will edit the system crontab.

Running it as root will edit the root user's crontab (in /var/spool/cron), which is separate from the system crontab (/etc/crontab), which has a slightly different format.

Alpine executes system cron jobs (daily, weekly etc) from root's crontab. That simple way of taking care of it makes me smile every time I see it.

It would be cool if someone made a crontab interface to systemd timers. Use crontab syntax but generate systemd boilerplate with sane defaults.

https://wiki.debian.org/systemd-cron

I pair crons with healthchecks.io

I use them all the time, my only wish is they get rid of the .service file and make it possible to define the service inside the .timer file.

I initially had a similar reaction, but there are so many cases like this for unit files. I think a higher level DSL for generating them would be useful.

The separation of concerns is legitimate.

I would love to see something like https://github.com/isd-project/isd but for systemd timers.

This is cool and after learning of this I'm tempted to convert my crons. I currently use systemd for running a redis queue and it's worked great for years.