Scale to zero with systemd

2024-08-02

I self-host many services. Some of them need to be up 24/7, other are used only occasionally. I tend to choose lightweight services so I could keeping them running all the time without too much trouble, but if I just did that I would never had written this post. Instead, let's see how we can setup a service to start when it receives a request and to have it shutdown after a period of inactivity.

"Scale to zero" is often highlighted as one of the benefits one can get from adopting a serverless approach but that's not the only way. We don't even need to move our workloads to the cloud nor leverage any particular framework. All we need is systemd and a few lines of configuration.

systemd supports socket activation, which takes care of starting up the service for us as soon as a request hits the socket. Shutdown is a bit more convoluted and it either requires the service to be aware it needs to stop when inactive or we will need to leverage something like systemd-socket-proxyd.

Configuration

Let's assume you already have a service that wasn't designed to shutdown after a period of inactivity. Let's also assume you already have a systemd unit configured for it. We need to tweak a few things on the service unit:

  1. to prevent the service from starting automatically, let's ensure its WantedBy= is not set;
  2. to allow the service to be stopped when inactive, let's add StopWhenUnneeded=true;
  3. to ensure we are not going to drop any request, let's add an ExecStartPost= entry to wait for the service to be fully up and ready to accept connections. There might be a better way of doing this, if you know one please let me know;
  4. to avoid waiting forever let's also add a TimeoutStartSec=. That will be useful if for some reason the service fails to start.

The resulting example.service file should look like this:

[Unit]
Description=Example service
StopWhenUnneeded=true

[Service]
Type=simple
ExecStart=nc -l 1234
ExecStartPost=/bin/bash -c 'while ! nc -z 127.0.0.1 1234; do sleep 1; done'
TimeoutStatSec=10

Now we're ready to define a socket (example-proxy.socket):

[Unit]
Description=Example service activation socket

[Socket]
ListenStream=12345

[Install]
WantedBy=sockets.target

This will tell systemd to listen for connection on port 12345. Note that the socket needs its own port (or a Unix Domain Socket), distinct from the one the service is listening on.

Finally, we neede to configure the proxy that will serve the connection accepted on the systemd-managed socket, call our own service and manage it's lifetime. Let's write another unit (example-proxy.serivce, the name has to match the one used for the socket unless you explicitly set the Service= option in the socket file) to configure the proxy, tell it where to find the target service and give it an idle timeout:

[Unit]
Requires=example.service example-proxy.socket
After=example.service example-proxy.socket

[Service]
Type=notify
ExecStart=/usr/lib/systemd/systemd-socket-proxyd --exit-idle-time="10 min" 127.0.0.1:1234

That's all, let's and start the socket (with systemctl start example-proxy.socket, also enable it if you want to start when the system boots) and your service is configured to startup when a request comes it and to shutdown after 10 minutes of inactivity.

Conclusion

Now that I've shown you how this works let's get to the real question: is it worth it? The answer is "it depends". With this approach you can save some resources, which is good. Startup time might be affected but I haven't noticed a significant delay for the couple or services I'm using it with. So that's a win for me but your mileage may vary. Upon inactivity the application shuts down cleanly.

Overall, I think this is a cool technique that can be useful in some scenarios, especially when running services on devices with limited resources. If this is better than ordering more hardware or replacing applications with more light-weight alternative it's up to you.