Home Posts Notes Now

Homelab, at the right scale with NixOS

2024-02-15

About a month ago I explained what wasn’t working well with my homelab. The root cause is that my scale is as small as possible: a handful of users at most, with everything running on a single machine. Container orchestration tools like Kubernetes or Docker Swarmd are designed for a much bigger scale and were mostly just adding complexity to my setup with very limited practical benefits. The services I’m running are monolithic. The traffic they see is so low I’m not going to scale them to more than a single instance. I don’t need High-Availability and I wouldn’t really be able to achieve it with a single machine anyway.

What I really need is:

Virtualisation

I’ve been really happy with Proxmox VE so far and I didn’t have any reason to look for any alternative. Virtualisation enabled all the experiments I’ve done. The possibility of bringing up a new VM to try something out while keeping the old setup running is priceless.

In the spirit of getting the leanest setup possible I decided to use LXC containers instead of VMs. For my use case, the main difference is that containers share the kernel with the hypervisor instead of having their own. I only need to run Linux guests, so. thats' fine and it has the side benefit of having lower resources utilisation, which is always welcome.

I decided to bring up three different containers:

Containers are really cheap and keeping these three different workloads isolated is definitely a good idea.

Infrastructure as code

Again, I’ve been happy with my Terraform setup and I’m currently using it to bring up the LXC containers (using bpg/proxmox), to configure DNS and to setup my Grafana dashboards. All of this has always been rock solid.

NixOS

Being a fan of defining everything as code I’ve always been interested in NixOS. This time I gave it a serious try and it’s definitely clicking. The approach is what I’ve always wanted when it comes to configure a system and I also went full on with Home Manager to declaratively manage the configuration of all the tools I use.

Nix the language is sadly living up to its reputation of being pretty obscure. The good news is that to get started and configure a system there is no need to fully understand it. It’s not ideal but a very basics understanding of the language can stretch pretty far as long as one is using pre-packaged software. As an example, have a look at how little config is required to install and configure Grafana. Note how we have a single language to configure everything and how configuration lives all in the same place. That's powerful!

All my three LXC containers are based on NixOS and share part of their configuration to configure my user and to install some basic programs I want to have available everywhere. On top of that, each one has their own specific configuration. I use Colmena from my operations LXC to rebuild the system of all the three containers whenever I change something and to push the new version of the system via ssh to the target hosts. So far this has worked really well. I have some raw notes on how to get started with this setup.

Finally, NixOS is also proving to be really useful when it comes to my development LXC. I have a couple projects I am working on and for each one of them I defined a Nix Flake. That gives me two really cool pieces of functionality: devShell and packages.

With a devShell each project knows precisely what tool I need in its development environment. The integration with direnv is particularly cool: all I need to do is clone the repository, cd into it and, after some downloading of what's needed, the development enviroment is ready. The era of manually installing packages is over!

packages tell Nix how to build the source code into the final executable and supporting files. This post covers the topic and compares it to the equivalent process of creating a Docker image. I like how this approach is more declarative and composable compared to writing a Dockerfile.

State management

Moving to long running LXC containers kind of solved the issue I was talking about in my previous post. I’m no longer in a world where I’m supposed to be able to destroy VMs on a whim and that makes state management way easier. Each machine has their own state (actually very little since everything I can is expressed “as code” and safely stored in a Git repository) that I can easily backup using Restic running periodically.

Conclusion

Container orchestration technologies are really valuable when multiple teams are writing software that needs to be deployed on a huge number of machines. That's cool, but my homelab is at the opposite end of the spectrum. I think NixOS is a really good fit for my use case and it's living up to the high expectation I had. I get all the benefits of declaratively defining every aspects of my systems and at the same time I don't get to care about many problems that don't really make sense at my scale. Having a system running a well defined set of services is more than enough to fulfill my needs and NixOS does that well. Goodies like Home Manager and dev shells are the cherries on top that are making me happy to use it on all the three LXC containers I am running.

Let's see how long this setup lasts. Even though NixOS is not flawless by any means so far my setup doesn't have any known issue which makes me hopeful I can keep it for the long run.

Thanks for reading. Feel free to reach out for any comment or question.