NixOS in a Proxmox LXC

2024-07-05

I run a few NixOS LXC containers in Proxmox. This setup works very well and it's pretty resource efficient. I've seen a few people struggle getting everything setup, hopefully this post helps.

Generate the CT template

Let's start by generating the most minimal CT template which we can use to start the container. I assume you want to keep a long lived LXC and run nixos-rebuild (or some other way of updating the system) on it. If all you want is a container and you're happy with treating it as immutable just add everything you need to your config.

Let's define a configuration.nix file with the following content:

{ modulesPath, ... }:
{
  imports = [
    (modulesPath + "/virtualisation/proxmox-lxc.nix")
  ];
  boot.isContainer = true;
  # Supress systemd units that don't work because of LXC
  systemd.suppressedSystemUnits = [
    "dev-mqueue.mount"
    "sys-kernel-debug.mount"
    "sys-fs-fuse-connections.mount"
  ];
  system.stateVersion = "24.11"; # Read the docs and set this to an appropriate value
  nix.settings.trusted-users = [ "nixos" ];
  users.users.nixos =
    {
      isNormalUser = true;
      extraGroups = [ "wheel" ];
      openssh.authorizedKeys.keys = [
        "ssh-ed25519 <INSERT YOUR OWN>"
      ];
    };
  services.openssh = {
    enable = true;
    settings.PasswordAuthentication = false;
    settings.KbdInteractiveAuthentication = false;
    settings.PermitRootLogin = "no";
  };
  security.sudo.wheelNeedsPassword = false;
}

This gives you a system with a nixos user and a way to SSH into it using your own SSH key. It's hard to get more basic than this.

With that in place, generate the template by running this command:

nix run github:nix-community/nixos-generators -- \
 -f proxmox-lxc \
 -c ./configuration.nix \
 -I nixpkgs=channel:nixos-unstable  

This will give you the path to the image, a file called nixos-system-x86_64-linux.tar.xz in your Nix store. Upload it to Proxmox CT Templates.

Booting the LXC

Create a CT container making sure the following options are selected:

• unprivileged container
• enable nesting
• console mode set to /dev/console

Boot the container, which will acquire an IP address via DHCP and it's ready to accept SSH connections. Unfortunately Proxmox doesn't expose IP addresses for LXC containers in the UI. I just check in my router DHCP status page.

Updating configuration

I usually build configurations on a remote machine and push them to the LXC using colmena but that's out of scope for this post. Let's just use nix-rebuild on the container.

SSH into the machine and let's get everything ready. NixOS wiki says:

The template built above without any options does not come with /etc/nixos/configuration.nix. A minimal working example is presented below. Be sure to run nix-channel --update, reboot the container running before nixos-rebuild switch.

Let's do the nix-channel --update and reboot. Copy the exact configuration.nix we used to generate the template and apply the changes you need to it.

nix-rebuild switch \
 -c ./configuration.nix \
 -I nixpkgs=channel:nixos-unstable  

We're almost there. For some reason I yet have to fully understand this command will put everything in place but switch will fail due to some permissions issue. Something like:

Failed to start transient service unit: Access denied
stat: cannot read file system information for '/boot': No such file or directory
WARNING: /boot being on a different filesystem not supported by init-script-builder.sh
'/nix/store/6544q8n5bj89vsschrfg8dnd0k7lhmkr-system-path/bin/busctl --json=short call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager ListUnitsByPatterns asas 0 0' exited with value 1 at /nix/store/577xpzvwiiim58n841vzccslzy6qrgga-nixos-system-unnamed-24.11pre646099.00d80d13810d/bin/switch-to-configuration line 145.     warning: error(s) occurred while switching to the new configuration

One last reboot with sudo reboot -f and when the LXC will come back up it will have all the new configuration applied. From now everything will just work as normal. Enjoy your new NixOS LXC.

Thanks for reading. Feel free to reach out for any comment or question.