towerops/docs/gitlab-runner-nix-setup.md

GitLab Runner with Nix Support

This guide explains how to set up a GitLab Runner with Nix support for building towerops Docker images using Nix flakes.

Prerequisites

• A server with Nix installed (can be NixOS or any Linux with Nix package manager)
• GitLab Runner installed
• Docker installed (for loading and pushing images)
• Network access to GitLab and Docker registries

Option 1: NixOS GitLab Runner (Recommended)

If you're running NixOS, this is the cleanest approach.

1. Configure GitLab Runner in NixOS

Add to /etc/nixos/configuration.nix:

{ config, pkgs, ... }:

{
  # Enable Nix flakes
  nix.settings.experimental-features = [ "nix-command" "flakes" ];

  # Install Docker
  virtualisation.docker.enable = true;

  # Configure gitlab-runner user and group
  users.users.gitlab-runner = {
    isSystemUser = true;
    group = "gitlab-runner";
    extraGroups = [ "docker" ];
  };

  users.groups.gitlab-runner = {};

  # Configure GitLab Runner
  services.gitlab-runner = {
    enable = true;
    services = {
      # Nix builder
      nix-builder = {
        registrationConfigFile = "/etc/gitlab-runner/nix-builder-registration";
        dockerImage = "nixos/nix:latest";
        dockerPrivileged = true;  # Required for docker-in-docker
        dockerVolumes = [
          "/nix/store:/nix/store:ro"
          "/nix/var/nix/db:/nix/var/nix/db:ro"
          "/nix/var/nix/daemon-socket:/nix/var/nix/daemon-socket:ro"
          "/var/run/docker.sock:/var/run/docker.sock"  # For docker load/push
        ];
        tagList = [ "nix" ];
      };
    };
  };
}

2. Create Registration Config

Create /etc/gitlab-runner/nix-builder-registration:

[[runners]]
  url = "https://gitlab.com/"
  token = "YOUR_RUNNER_REGISTRATION_TOKEN"
  executor = "docker"

Getting the registration token:

New Token Workflow (Recommended - GitLab 16.0+):

1. Go to: Settings → CI/CD → Runners
2. Click: New project runner
3. Configure runner settings:
   • Tags: nix
   • Run untagged jobs: Uncheck
   • Description: "Nix Builder"
4. Click Create runner
5. Copy the token (starts with glrt-)

Use simplified registration command:

sudo gitlab-runner register \
  --url https://gitlab.com/ \
  --token glrt-YOUR_TOKEN_HERE \
  --executor docker \
  --docker-image nixos/nix:latest \
  --docker-privileged \
  --docker-volumes /nix/store:/nix/store:ro \
  --docker-volumes /nix/var/nix/db:/nix/var/nix/db:ro \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock

Important: With new tokens, tags are configured in GitLab UI, not in the registration command.

Legacy Token Workflow (GitLab <16.0):

Get token from: Settings → CI/CD → Runners → Project runners → Registration token

Registration config format is the same as above.

3. Apply Configuration

sudo nixos-rebuild switch

This will:

• Enable Nix flakes
• Install and start Docker
• Create gitlab-runner user and group
• Configure and start GitLab Runner service
• Add gitlab-runner to docker group

4. Verify Setup

# Check services are running
sudo systemctl status gitlab-runner
sudo systemctl status docker

# Verify gitlab-runner is in docker group
id gitlab-runner
# Should show: groups=... docker ...

# Check runner registration
sudo gitlab-runner list

Option 2: Standard Linux with Nix

If you're using a standard Linux distribution with Nix installed:

1. Install Prerequisites

# Install Nix (if not already installed)
curl -L https://nixos.org/nix/install | sh -s -- --daemon

# Enable flakes
mkdir -p ~/.config/nix
echo "experimental-features = nix-command flakes" >> ~/.config/nix/nix.conf

# Install Docker
# (distribution-specific - see Docker docs)

# Install GitLab Runner
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt-get install gitlab-runner

2. Register Runner

With new GitLab tokens (16.0+):

sudo gitlab-runner register \
  --url https://gitlab.com/ \
  --token glrt-YOUR_TOKEN_HERE \
  --executor docker \
  --docker-image nixos/nix:latest \
  --docker-privileged \
  --docker-volumes /nix/store:/nix/store:ro \
  --docker-volumes /nix/var/nix/db:/nix/var/nix/db:ro \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock

Then configure tags in GitLab UI: Settings → CI/CD → Runners → Edit → Add tag: nix

With legacy tokens (<16.0):

sudo gitlab-runner register \
  --url https://gitlab.com/ \
  --token YOUR_REGISTRATION_TOKEN \
  --executor docker \
  --docker-image nixos/nix:latest \
  --docker-privileged \
  --docker-volumes /nix/store:/nix/store:ro \
  --docker-volumes /nix/var/nix/db:/nix/var/nix/db:ro \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock \
  --tag-list nix \
  --run-untagged=false

3. Add gitlab-runner to Docker Group

sudo usermod -aG docker gitlab-runner
sudo systemctl restart gitlab-runner

Setting Up Cachix (Optional but Recommended)

Cachix provides binary caching to speed up builds dramatically.

1. Create Cachix Account and Cache

# Install cachix
nix-env -iA cachix -f https://cachix.org/api/v1/install

# Login to cachix
cachix authtoken

# Create cache
cachix create towerops

# Generate keypair
cachix generate-keypair towerops

2. Get Public Key

cachix get towerops

This will output something like:

towerops.cachix.org-1:AbCdEfGhIjKlMnOpQrStUvWxYz1234567890ABCDEFG=

3. Update Configuration

In flake.nix, replace the placeholder:

extra-trusted-public-keys = [
  "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
  "towerops.cachix.org-1:YOUR_ACTUAL_PUBLIC_KEY_HERE"  # Replace this
];

In .gitlab-ci.yml, replace:

trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= towerops.cachix.org-1:YOUR_ACTUAL_PUBLIC_KEY_HERE

4. Add Auth Token to GitLab

Get your auth token:

cachix authtoken

Add to GitLab:

1. Go to: Project → Settings → CI/CD → Variables
2. Add variable:
   • Key: CACHIX_AUTH_TOKEN
   • Value: (paste token from above)
   • Type: Variable
   • Protected: Yes
   • Masked: Yes
   • Expand variable reference: No

Verifying the Setup

Test Runner Connection

Check that the runner is connected:

# On the runner host
sudo gitlab-runner verify

# In GitLab UI
# Go to: Settings → CI/CD → Runners
# You should see your runner with the "nix" tag listed

Test Build

Push a commit to the main branch and watch the pipeline:

1. Go to: CI/CD → Pipelines
2. Click on the latest pipeline
3. Watch the build job

Expected output:

Building Docker image with Nix...
[Building all dependencies...]
Creating layer 1 from paths: [...]
Creating layer 2 from paths: [...]
...
Done.
Loading Docker image...
Pushing to GitLab Container Registry...

Troubleshooting

NixOS: User configuration errors

Error: users.users.gitlab-runner.isSystemUser must be set

Fix: Add user configuration to /etc/nixos/configuration.nix:

users.users.gitlab-runner = {
  isSystemUser = true;
  group = "gitlab-runner";
  extraGroups = [ "docker" ];
};

users.groups.gitlab-runner = {};

Runner registration with new tokens fails

Error: Runner configuration other than name and executor... is reserved

Fix: With new GitLab tokens (glrt-*), you cannot specify --tag-list, --run-untagged, etc. in the registration command. Configure these in GitLab UI instead:

1. Register with minimal command (without --tag-list, --run-untagged)
2. Go to GitLab UI: Settings → CI/CD → Runners → Edit
3. Add tags and configure settings there

Runner shows offline in GitLab

Check runner status:

sudo gitlab-runner status
sudo systemctl status gitlab-runner

Check logs:

sudo journalctl -u gitlab-runner -f

Build fails with "experimental feature 'flakes' is not enabled"

Fix: Ensure NIX_CONFIG in .gitlab-ci.yml includes experimental features:

NIX_CONFIG: |
  experimental-features = nix-command flakes

Docker load fails with permission denied

Fix: Ensure gitlab-runner user is in docker group:

sudo usermod -aG docker gitlab-runner
sudo systemctl restart gitlab-runner

Cachix push fails

Check auth token:

• Verify CACHIX_AUTH_TOKEN is set in GitLab CI/CD variables
• Verify token is correct: run cachix authtoken locally

Check network:

• Ensure runner can reach cachix.org
• Check firewall rules

Build is very slow

Without Cachix: First build will compile everything from source (~20-30 minutes)

With Cachix: Subsequent builds should be much faster (~2-5 minutes) as dependencies are cached

Improve speed:

1. Set up Cachix (see above)
2. Use a more powerful runner instance
3. Increase runner's concurrent setting in /etc/gitlab-runner/config.toml

Rollback to Docker Builds

If you need to roll back to the old Docker-based builds:

# The old config is in git history
git log --all -- .gitlab-ci.yml

# Restore the old version (find the commit hash from log)
git show COMMIT_HASH:.gitlab-ci.yml > .gitlab-ci.yml

# Commit and push
git add .gitlab-ci.yml
git commit -m "Revert to Docker-based builds"
git push

The old Dockerfile is still present at k8s/Dockerfile and can be used.

Performance Comparison

Docker-based builds (before Nix):

• First build: ~15 minutes
• Incremental: ~10 minutes (with layer caching)
• Image size: ~500 MB

Nix-based builds (with Cachix):

• First build: ~25 minutes (everything from source)
• Subsequent builds: ~2-5 minutes (binary cache hits)
• Image size: ~507 MB compressed (~960 MB uncompressed)

Benefits of Nix:

• Reproducible: Identical builds across all environments
• Cacheable: Binary caching eliminates redundant compilation
• Declarative: All dependencies pinned in flake.lock
• Fast incremental builds: Only changed layers rebuild

Additional Resources

• Nix Flakes Manual
• GitLab Runner Docker Executor
• Cachix Documentation
• NixOS GitLab Runner