144

I am having a service that depends on Cassandra coming up gracefully and the cluster being up and ready.

To ensure that the dependency order is met, I have the following unit file 

[Unit]
Requires=cassandra.service
After=cassandra.service

[Service]
Environment=JAVA_HOME=/usr/java/jre
[email protected]@/webapps/bringup-app/bin/bringup
TimeoutStartSec=0
ExecStop=
[email protected]@/logs/bringup.pid
Restart=always

[Install]
WantedBy=multi-user.target

How do I ensure that the bringup-app process waits for 30 seconds before it attempts to start up? Currently although it is started after Cassandra, I have noticed that the Cassandra cluster is not up yet and hence any attempt from the bringup-app to connect to Cassandra as part of startup fails.

I therefore want to add a delay. Is that possible via the unit file?

Improve this question
2
  • cassandra-service should only return once fully started. That is, the launcher should wait until the service is ready, then exit. Also, cassandra-service could make use of socket activation.
    – André Werlang
    Commented Jul 28, 2020 at 22:43
  • You shoud use add 2 lines with "ExecStop" in this order: ExecStop=kill -TERM $MAINPID and ExecStop=timeout 15 tail --pid=$MAINPID -f /dev/null. The first line sends SIGTERM signal to JVM so it requests it to exit. The second line waits the process to actually stop but no longer than 15 seconds (so waiting with time limit).
    – Krzysztof Tomaszewski
    Commented Jan 13 at 18:59

7 Answers 7

Reset to default
237

You can run the sleep command before your ExecStart with ExecStartPre :

[Service]
ExecStartPre=/bin/sleep 30
Improve this answer
6
  • 34
    how do you solve this without having a sleep on service restart?
    – rubo77
    Commented Aug 14, 2018 at 19:47
  • 12
    The fact that this works on a service restart is a bonus for my use case
    – sergtech
    Commented Dec 8, 2018 at 22:52
  • Don't know why but it makes a 30sec loop for the service and ExecStart not going to be started.
    – Mohi Rostami
    Commented Feb 23, 2020 at 6:32
  • 7
    I feel like systemd doesn't really want me to do this. With a longer delay this causes timeouts and e.g. systemctl start x doesn't return until the sleep has completed
    – phiresky
    Commented Feb 23, 2020 at 23:14
  • 11
    When using the above method with delays longer than 90 seconds, the service’s start timeout will have to be increased (e.g. TimeoutStartSec=120)
    – SomeDude
    Commented Jul 3, 2021 at 6:23
86

You can create a .timer systemd unit file to control the execution of your .service unit file.

So for example, to wait for 1 minute after boot-up before starting your foo.service, create a foo.timer file in the same directory with the contents:

[Timer]
OnBootSec=1min

It is important that the service is disabled (so it doesn't start at boot), and the timer enabled, for all this to work (thanks to user tride for this):

systemctl disable foo.service
systemctl enable foo.timer

You can find quite a few more options and all information needed here: https://wiki.archlinux.org/index.php/Systemd/Timers

Improve this answer
6
  • 3
    While this will work, it is an overkill for a simple delay. For simple delay use ExecStartPre and use timers for more complex schedules.
    – mikijov
    Commented May 22, 2019 at 17:44
  • 2
    OnBootSec is apparently relative to when the box "first started" so if the service you are trying to "start after" doesn't happen to startup within 1m, it could still fire too early, I wonder? freedesktop.org/software/systemd/man/systemd.timer.html But I guess if you set a long enough number it could work...
    – rogerdpack
    Commented Jul 9, 2019 at 16:31
  • @rogerdpack That is true. However with Cassandra, even after the service starts, starting a session (with cqlsh or otherwise) is not instantly possible, which is why this approach is more convenient for this specific case. But yeah you are basically "guessing" the time needed for it to become available.
    – mj3c
    Commented Jul 10, 2019 at 8:22
  • 1
    IIRC you also need to start the timer systemctl start foo.timer.
    – Robert Klemme
    Commented Jan 16, 2020 at 13:17
  • 6
    IMO This is the best answer because systemd considers a unit to be 'starting' when the ExecStartPre directives are executed.
    – Paul Back
    Commented Mar 30, 2020 at 21:56
41

Combining the answers from @Ortomala Lokni and @rogerdpack, another alternative is to have the dependent service monitor when the first one has started / done the thing you're waiting for.

For example, here's how I am making the fail2ban service wait for Docker to open port 443 (so that fail2ban's iptables entries take priority over Docker's):

[Service]
ExecStartPre=/bin/bash -c '(while ! nc -z -v -w1 localhost 443 2>/dev/null; do echo "Waiting for port 443 to open..."; sleep 2; done); sleep 2'

Simply replace nc -z -v -w1 localhost 443 with a command that fails (non-zero exit code) while the first service is starting and succeeds once it is up.

For the Cassandra case, the ideal would be a command that only returns 0 when the cluster is available.

(May want to increase TimeoutStartSec from the default 90s as well, or set TimeoutStartSec=0 to disable startup timeouts)

Improve this answer
4
  • 5
    Best answer on here and completely underrated! I added the note from rogerdpack's answer about increasing TimeoutStartSec, or you can set it to 0 to disable startup timeouts.
    – PolyTekPatrick
    Commented Mar 18, 2021 at 21:20
  • 1
    The brilliance of this is that it can be adapted to suit virtually any need, and since it doesn't have to wait for a fixed amount of time before the boot sequence can continue, it's a good method to delay the boot process, especially if the length of the delay is unknown.
    – Robidu
    Commented Oct 8, 2022 at 18:41
  • 1
    This was an inspirational answer! I added this to my service; I hate arbitrarily long delays. I want the thing to get rolling as soon as it can! This was for mounting a Veritas filesystem: ExecStartPre=/bin/bash -c 'delay=60; i=0; while [ $i -lt $delay ]; do printf "$i "; [ -e /dev/vx/dsk/veritas0 ] && break; let i=$i+1; sleep 1; done; if [ $i = $delay ] ; then echo "ERROR: Timeout; exited."; exit 1; else exit 0; fi'
    – Mike S
    Commented Nov 2, 2022 at 20:16
  • Note that you shouldn't have to put the parers around the while loop. I haven't tested it, but my example doesn't use them. They'll just add an additional subshell which is not necessary in this case.
    – Mike S
    Commented Nov 3, 2022 at 13:36
39

Instead of editing the bringup service, add a post-start delay to the service which it depends on. Edit cassandra.service like so:

ExecStartPost=/bin/sleep 30

This way the added sleep shouldn't slow down restarts of starting services that depend on it (though does slow down its own start, maybe that's desirable?).

Improve this answer
10

The systemd way to do this is to have the process "talk back" when it's setup somehow, like by opening a socket or sending a notification (or a parent script exiting). Which is of course not always straight-forward especially with third party stuff :|

You might be able to do something inline like

ExecStart=/bin/bash -c '/bin/start_cassandra &; do_bash_loop_waiting_for_it_to_come_up_here'

or a script that does the same. Or put do_bash_loop_waiting_for_it_to_come_up_here in an ExecStartPost

Or create a helper .service that waits for it to come up, so the helper service depends on cassandra, and waits for it to come up, then your other process can depend on the helper service.

(May want to increase TimeoutStartSec from the default 90s as well)

Improve this answer
2
  • do_bash_loop_waiting_for_it_to_come_up_here ? wait for several services
    – Kiquenet
    Commented Jun 25 at 18:44
  • @Kiquenet It's just any bash "waiting" code (either a loop or serially wait for one thing after another) so yeah you could do several services in it, whichever way...
    – rogerdpack
    Commented Jun 25 at 19:44
10

I have used systemd timer for delaying a service and it works very well.

# /lib/systemd/system/foo.timer 
[Unit]
Description=Wait some second before run foo

[Timer]
OnActiveSec=5sec
AccuracySec=1s

[Install]
WantedBy=timers.target

see timers : systemctl list-timers

log:

journalctl -f -u foo.timer
journalctl -f -u foo
Improve this answer
3
  • 1
    Best way to delay executing a program on boot but not on restarts
    – Grumoll
    Commented Mar 29, 2023 at 9:05
  • All I know is the above, for me, ran when foo started, not before foo started. I.e., it did not delay foo one single bit.
    – Dan Jacobson
    Commented Jun 30, 2023 at 13:40
  • github.com/systemd/systemd/issues/24026#issuecomment-1615867761 says to use the "edit" command instead. That creates a /etc/systemd/system/foo.service.d/override.conf . Nifty!
    – Dan Jacobson
    Commented Jul 1, 2023 at 12:59
9

This answer on super user I think is a better answer. From https://superuser.com/a/573761/67952

"But since you asked for a way without using Before and After, you can use:

Type=idle

which as man systemd.service explains

Behavior of idle is very similar to simple; however, actual execution of the service program is delayed until all active jobs are dispatched. This may be used to avoid interleaving of output of shell services with the status output on the console. Note that this type is useful only to improve console output, it is not useful as a general unit ordering tool, and the effect of this service type is subject to a 5s time-out, after which the service program is invoked anyway. "

Improve this answer
5
  • It is very similar to oneshot too, simple won't run ExecStop for commands that exit shell after execution like vncservers (of course that can be tweaked to not run in background) but i mean that it runs also for background processes.
    – m3nda
    Commented Mar 31, 2018 at 1:18
  • 5
    You conveniently left out the most important part from the quote: "this type is useful only to improve console output, it is not useful as a general unit ordering tool, and the effect of this service type is subject to a 5s timeout, after which the service program is invoked anyway [...] it is generally not recommended to use idle or oneshot for long-running services."
    – bviktor
    Commented Feb 18, 2019 at 19:06
  • 2
    @bviktor, the documentation has updated. You were correct, it did not explain its purpose correctly The documentation does say however, actual execution of the service program is delayed until all active jobs are dispatched. The documentation seams to contradict itself.
    – nelaaro
    Commented Feb 19, 2019 at 13:08
  • This still might not be enough if your service starts and then immediately the boot sequence "ends and enters idle" there might not be a long enough delay... :|
    – rogerdpack
    Commented Jul 9, 2019 at 16:25
  • I doubt that this is going to work, because you are provoking a race condition. Expect any jobs that have already been scheduled to be completed before the one that you have marked as idle so there's a high chance that whatever you are attempting to perform after your checks to be executed before them. Your best bet still would be to directly check for the required service's availability, e. g. by looking out for their control sockets or whatever may be feasible.
    – Robidu
    Commented Oct 8, 2022 at 18:37

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.