hckrnws

I know everyone hates on systemd, but I've generally been happy with it, in no small part because I find the timers to be pretty sensible.

They are, in all ways that I care about, simply better than cron, and especially in NixOS they're really easy to set up.

+1 for NixOS. I used to find systemd a bit intimidating, but working with NixOS has made it all fit together somehow.

As an example, this is a systemd timer I have that periodically runs rclone to sync photos to Backblaze:

    systemd.services.rclone-photos-sync = {
      serviceConfig.Type = "oneshot";
      path = [ pkgs.rclone ];
      script = ''
        rclone \
          --config ${config.age.secrets."rclone.conf".path} \
          --bwlimit 20M --transfers 16 \
          sync /mnt/photos/originals/ photos:
      '';
      unitConfig = {
        RequiresMountsFor = "/mnt/photos";
      };
    };
    systemd.timers.rclone-photos-sync = {
      timerConfig = {
        # Every 2 hours.
        OnCalendar = "00/2:00:00";
        RandomizedDelaySec = "5m";
        Persistent = true;
        Unit = "rclone-photos-sync.service";
      };
      partOf = [ "rclone-photos-sync.service" ];
      wantedBy = [ "timers.target" ];
    };

I had to write a systemd timer to rescan PCIe ports about ten seconds after the computer starts. I was able to add that to my NixOS config and it worked how I wanted, and it was great because if I had broken anything all I would have to do is reboot and choose an older generation.

That would all be one line in a crontab.

The article already explains the difference between cron and systemd timers, so I won't rehash that. Yes, systemd is more verbose, but that comes with advantages over cron. And if cron suits your usecases, it's a fine tool as well.

But with NixOS and this example specifically, you also get (at a minimum):

a) A declarative config without requiring an external tool like Ansible.

b) Any dependencies (rclone in this case) will be implicitly installed if not already present on the system.

c) If configuring a remote machine, this will copy over the encrypted rclone.conf file and decrypt it on the target.

d) And of course, it's trivial to version control and track changes to the config over time.

Putting aside that this is all possible with nix and without systemd, the additional complexity strikes me as having hamstrung the linux community for the last decade. I hope reliable service management arises but I have yet to see the benefit.

In the meantime I'm just going to keep using void because I know how to fix it when it breaks. Systemd i have to google how to even find the service logs.... they certainly aren't easy to find via the filesystem any more.

How is ‘persist’ realised with cron? What about ‘randomized delay’?

I'm not sure what 'persist' means here but at least some cron implementations have support for randomization in times or time intervals.

    the time when the service unit was last triggered is stored on disk. When the timer is activated, the service unit is triggered immediately if it would have been triggered at least once during the time when the timer was inactive. Such triggering is nonetheless subject to the delay imposed by RandomizedDelaySec=. This is useful to catch up on missed runs of the service when the system was powered down.

That isn't inherently better.

The really nice thing about systemd timers is that as units you can get status of them and check the journal output the same as any other unit.

I'm convinced that the people who hate systemd are a small minority. They also happen to be fairly vocal, which I think makes them seem outsized.

I think in general it's a shrinking minority. I was outspokenly critical of systemd in the past, but much about it has improved, and I have to admit it has made my life significantly easier in many ways. I'm still not a big fan of some things being systemd projects that don't really need to be under the umbrella (why is resolved even a systemd project? Or systemd-timesyncd?) but service definition and management, transient services, sockets, timers, and many many more things are pretty great. At this point, I pretty much love it. Even things that initially annoyed me, like timers, I've grown to love, not least because my cronjob entries are manageable files that I can commit and backup, and I can individually enable and disable them without having to deal with stupid commenting and such.

Checking service status and managing services is also significantly better with systemd than any other init system I've used, and I've used a lot over 15 years of systems administration.

I still find journald a bit annoying, though, and I still am not completely sold on binary log files. I understand the benefits, but working with them is still much more opaque than text logs.

I'd be excellent to see their complaints collected and elaborated. Because God Fucking Damn, these seem like some incredible whining Advanced Persistent Threats against anything getting better. If I were a pro-discord disruptor, I'd pay these people to keep at their endless grudges against every single person having to figure out ever single fucking capability de-novo.

Systemd makes so so so many incredibly good service administration capabilities easy & on tap. The main thing I want from systemd haters is to show how how how folks did things well before. Mostly I think this is a rebellion of the losers, people who hate having to do a good job, people who hate learning the many many many many ways we could run services better, who valorize each service figuring out their own unique special bespoke ways of running their stuff. Its fallen as fuck. We need need need the systemd socialization of running stuff well, controlling permissions & access. There's near zero precedent from the past of anyone else being responsible. That's just the situation. Systemd has drastically elevated what sysops can do, orthogonally to the service by service init launching of the past that got us no where.

One might even go so far as to say they're a vocal minority.

I was initially against systemd (around the time Debian started discussing using it) for non-technical reasons: the project seemed to have some very negative aspects in terms of how feedback and concerns are addressed. I don't know for sure but I strongly suspect this was the core issue a good chunk of those "against" were concerned about: the way the project is run.

Having used it for several major releases of Debian technically I'm happy to have it - it's an improvement over the sysv init & crontabs. I'm not completely in love with journal compared to regular logs but I can see some advantages.

Systemd hater here. My stance is and always was: if you love systemd, go nuts with it. I'm not looking to reach into your bathroom to set your shower temperature.

But I don't want to have to run it or even think about it. Lennart has other plans, hence his campaign to get distros to require it and software up the stack to hard-depend on it. I don't like its design, I want to opt out of it in favor of something better, but one of the development effort's goals is making that extremely difficult for modern Linux systems. That's my issue with it.

But really it macht nichs for me right now because I run Void, btw.

When I started using systemd timers, I really liked the systemd-analyze calendar facility, to calculate n trigger times for a given calendar expression.

For example, show the next five trigger times for the end of the last day when the month has 31 days :

systemd-analyze calendar --iterations=5 '*-*-31 23:59:59'

There is one huge advantage of cron that is usually missed with such comparisons. cron is dead-easy to create and maintain. Systemd timers make sense in certain situations, but in 90%+ of my use cases so far, the added complexity really adds up the amount of time I'm administering my systems. I've been in a situation where I needed to troubleshoot why a systemd timer wasn't triggering, and I didn't like that experience at all. To me, it is something I would use if I actually needed it, it isn't a 1-to-1 replacement for cron.

I actually disagree with simplicity. cron does weird things with the environment and it can be hard to correctly reproduce that environment when creating a job and debugging. Conversely, systemd timers are very easy to manually trigger and otherwise behave as a normal oneshot service.

My approach to cron is to assume that you have an empty environment. Just set everything you will need.

There is also one huge disadvantage of cron: it requires root. Users can create and run systemd timers without root/sudo.

And quite frankly, your experience is likely due to your personal knowledge bias. Systemd timers are quite easy to debug, in fact easier than cron in my opinion. systemctl list-timers lists all the timers, when they last ran, when they will run next, and you can use other commands to inspect each timer in detail and all associated logs.

In contrast, I find cron much harder to debug. Starting with the first problem, you must first figure out what cron is running, as different implementations have different behavior!

One can set up cron jobs under non-root.

You cannot edit /etc/crontab without root. I'm not talking about dropping privileges for the job.

Although speaking of which, cron dropping privileges is not as secure as systemd running as the user before parsing the user's timers, from a defense-in-depth perspective.

But non-root users still have their own crontab, which you don't need sudo to edit.

That depends on which cron you're using, not all of them support user crontabs (and the ones that do probably have subtle differences in behavior).

I was unaware of that, so thank you for the education. But for what it's worth, I have never seen a Linux system which didn't have user crontabs. So even if not universal, it is at least very widespread to the point that you can bet on its presence.

Minor nitpick - shouldn't you first define the service and only then a timer for it? Otherwise since you enabled timer and are still trying to figure out how to write service, systemd won't have anything to run when timer triggers. Maybe I am wrong, but that just feels like logical order. Anyways, after years on hating on systemd I also started to embrace it and porting my cron jobs to systemd timers and I must admit it's really nice, the overall overview with list-timers, next execution timestamp, total execution time, ordering of services so one can run after another is completed and of course the logging in journal so I can filter output and keep track of everything it's just wonderful experience.

EDIT: yea, the email reporting is certainly missing, but it was hard to control it since whole STDOUT was shipped, which is not what I wanted most of the time anyways. It would be good to come up with some way to still have small overview emails sent about important jobs done, maybe a dependency service which starts when important job finished and just sends an email about that

Really liked this read. Is anyone able to explain how the backup.timer runs the backup.service? It wasn't obvious to me where the trigger was defined. I guess it's just inherent to what that unit type does, and the fact that both units are named backup? What is the name for that "package" of units that make up the whole backup program?

> I guess it's just inherent to what that unit type does, and the fact that both units are named backup?

Correct. If there's no `Unit=` specified in the timer unit, it defaults to the service unit of the same name. See https://www.freedesktop.org/software/systemd/man/latest/syst...

This is also a common pattern for the socket units of socket activated services

I used systemd user timers, a shell script, and an HDHomerun to make a simple VCR.

It took a minute to setup, but using a combination of rsync and timers to backup system files has done wonders to decrease my anxiety around upgrades on arch

Using a modern CoW filesystem lets you level this up to just a snapshot command in a pacman pre-upgrade hook.

I like systemd's timers when they're appropriate, but I really think the author's use case here is better suited to cron. The "issues" he listed for cron aren't very good either.

> If you want to execute pre/post commands you have to do it inside the script itself

So?

> There are no built-in logs

Every cron implementation I can remember using logs each run to syslog and emails me the output of the run by default

> There is no built-in status monitoring

I can't think of any built-in status monitoring that systemd has for timers that's materially different from cron's logging/emailing

> If the system is down when the cron needs to run, the cron will be missed

Some cron implementations support this and some don't. Most modern ones that I'm aware of do.

Much more significantly, the amount of setup involved in a systemd timer is way higher than putting a line in a crontab, especially for the author's case of just running a backup script.

Cron only involves running `crontab -e` and adding the line "@daily /path/to/script.sh" (which also handles the author's issue of cron "skipping" runs if the system was powered off, assuming the cron implementation uses something modern like anacron)

Systemd involves writing a 7 line timer unit file, an additional 5 line service file, running a daemon-reload, then enabling the timer. It turns what's usually a 10 second mindless task into a much more involved procedure. That can be worth it if there are material benefits from it, but I'm not really seeing them here.

IMO Systemd Timers provide much better control over how the cron job is run.

Need to distribute lots of cron jobs evenly to avoid overloading the system? Use RandomizedDelaySec.

Some cron jobs are flaky and you want to re-run if it fails? Add Restart=on-failure to corresponding service.

Some cron jobs conflict with each other? Set Conflicts=foo.service or maybe Before & After.

Sure, all above are possible with shell scripts. But systemd provides a standard, reliable way to define these properties.

> Systemd Timers provide much better control over how the cron job is run

Yes, they do. And as the very first sentence in my comment says, I like systemd timers when they're appropriate. None of the features you've mentioned were used in the author's solution, which is the sole thing I'm arguing against.

> Much more significantly, the amount of setup involved in a systemd timer is way higher than putting a line in a crontab

Counterpoint: this requires root, while you can edit and run systemd timers without root. Thus it's also generally more secure, both when creating the job and every time it runs.

> Systemd involves writing a 7 line timer unit file, an additional 5 line service file, running a daemon-reload, then enabling the timer. It turns what's usually a 10 second mindless task into a much more involved procedure. That can be worth it if there are material benefits from it, but I'm not really seeing them here.

You could write a script to automate it, if it's such a big deal. Creating timers/cronjobs isn't something that needs to be done often enough for this to matter.

If it is, another counterpoint: systemd supports creating transient timers, which you could do programmatically.

And final counterpoint: you can make systemd understand crontab and convert it into timers (systemd-crontab-generator)

> Counterpoint: this requires root

Counter-counterpoint: No it doesn't? Running `crontab -e` as a non-root user will edit that user's crontab, and running it as root will edit the system crontab. Cron can be configured to deny users their own crontabs, but every common distro I'm aware of defaults to allowing user crontabs.

> You could write a script to automate it, if it's such a big deal

Or I could not bother with that and just use cron?

> another counterpoint: systemd supports creating transient timers

What is this a counterpoint to? That the author's particular use case of running a backup script once a day is a task better suited to cron than systemd timers? I'm not sure how transient timers are even relevant here, much less a counterpoint.

I think support for per-user crontab's is implementation dependent. There are implementations (AFAIK) that do not support per-user crontabs.

Per Gentoo's wiki, both fcron and cronie have their own (different!) ways of whitelisting non-root users to run cronjobs.

Everything is implementation dependent. There's no ISO cron standard or anything.

> Per Gentoo's wiki, both fcron and cronie have their own (different!) ways of whitelisting non-root users to run cronjobs.

Both allow any user in the cron group to have their own crontabs by default, and both support optional .allow/.deny files

> There's no ISO cron standard or anything.

It's part of the Single Unix Specification.

https://pubs.opengroup.org/onlinepubs/9799919799/utilities/c...

That's a light spec for a subset of the interface and behavior of the crontab command line frontend, not a standard for cron in general. It's not really relevant to a discussion about how distros set up their cron defaults for per-user access.

Thanks for sharing it though. I wasn't aware it was in the posix spec and it explains why pretty much every implementation supports .allow/.deny files even when most already implement better access control mechanisms.

> Running `crontab -e` as a non-root user will edit that user's crontab, and running it as root will edit the system crontab.

Running it as root will edit the root user's crontab (in /var/spool/cron), which is separate from the system crontab (/etc/crontab), which has a slightly different format.

Alpine executes system cron jobs (daily, weekly etc) from root's crontab. That simple way of taking care of it makes me smile every time I see it.

It would be cool if someone made a crontab interface to systemd timers. Use crontab syntax but generate systemd boilerplate with sane defaults.

https://wiki.debian.org/systemd-cron

I pair crons with healthchecks.io

This is cool and after learning of this I'm tempted to convert my crons. I currently use systemd for running a redis queue and it's worked great for years.

I use them all the time, my only wish is they get rid of the .service file and make it possible to define the service inside the .timer file.