towerops/docs/gitlab-runner-nix-setup.md

# GitLab Runner with Nix Support

This guide explains how to set up a GitLab Runner with Nix support for building towerops Docker images using Nix flakes.

## Prerequisites

- A server with Nix installed (can be NixOS or any Linux with Nix package manager)
- GitLab Runner installed
- Docker installed (for loading and pushing images)
- Network access to GitLab and Docker registries

## Option 1: NixOS GitLab Runner (Recommended)

If you're running NixOS, this is the cleanest approach.

### 1. Configure GitLab Runner in NixOS

Add to `/etc/nixos/configuration.nix`:

```nix
{ config, pkgs, ... }:

{
  # Enable Nix flakes
  nix.settings.experimental-features = [ "nix-command" "flakes" ];

  # Install Docker
  virtualisation.docker.enable = true;

  # Configure gitlab-runner user and group
  users.users.gitlab-runner = {
    isSystemUser = true;
    group = "gitlab-runner";
    extraGroups = [ "docker" ];
  };

  users.groups.gitlab-runner = {};

  # Configure GitLab Runner
  services.gitlab-runner = {
    enable = true;
    services = {
      # Nix builder
      nix-builder = {
        registrationConfigFile = "/etc/gitlab-runner/nix-builder-registration";
        dockerImage = "nixos/nix:latest";
        dockerPrivileged = true;  # Required for docker-in-docker
        dockerVolumes = [
          "/nix/store:/nix/store:ro"
          "/nix/var/nix/db:/nix/var/nix/db:ro"
          "/nix/var/nix/daemon-socket:/nix/var/nix/daemon-socket:ro"
          "/var/run/docker.sock:/var/run/docker.sock"  # For docker load/push
        ];
        tagList = [ "nix" ];
      };
    };
  };
}
```

### 2. Create Registration Config

Create `/etc/gitlab-runner/nix-builder-registration`:

```toml
[[runners]]
  url = "https://gitlab.com/"
  token = "YOUR_RUNNER_REGISTRATION_TOKEN"
  executor = "docker"
```

**Getting the registration token**:

#### New Token Workflow (Recommended - GitLab 16.0+):

1. Go to: **Settings → CI/CD → Runners**
2. Click: **New project runner**
3. Configure runner settings:
   - Tags: `nix`
   - Run untagged jobs: **Uncheck**
   - Description: "Nix Builder"
4. Click **Create runner**
5. Copy the token (starts with `glrt-`)

Use simplified registration command:

```bash
sudo gitlab-runner register \
  --url https://gitlab.com/ \
  --token glrt-YOUR_TOKEN_HERE \
  --executor docker \
  --docker-image nixos/nix:latest \
  --docker-privileged \
  --docker-volumes /nix/store:/nix/store:ro \
  --docker-volumes /nix/var/nix/db:/nix/var/nix/db:ro \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock
```

**Important**: With new tokens, tags are configured in GitLab UI, not in the registration command.

#### Legacy Token Workflow (GitLab <16.0):

Get token from: **Settings → CI/CD → Runners → Project runners → Registration token**

Registration config format is the same as above.

### 3. Apply Configuration

```bash
sudo nixos-rebuild switch
```

This will:
- Enable Nix flakes
- Install and start Docker
- Create gitlab-runner user and group
- Configure and start GitLab Runner service
- Add gitlab-runner to docker group

### 4. Verify Setup

```bash
# Check services are running
sudo systemctl status gitlab-runner
sudo systemctl status docker

# Verify gitlab-runner is in docker group
id gitlab-runner
# Should show: groups=... docker ...

# Check runner registration
sudo gitlab-runner list
```

## Option 2: Standard Linux with Nix

If you're using a standard Linux distribution with Nix installed:

### 1. Install Prerequisites

```bash
# Install Nix (if not already installed)
curl -L https://nixos.org/nix/install | sh -s -- --daemon

# Enable flakes
mkdir -p ~/.config/nix
echo "experimental-features = nix-command flakes" >> ~/.config/nix/nix.conf

# Install Docker
# (distribution-specific - see Docker docs)

# Install GitLab Runner
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt-get install gitlab-runner
```

### 2. Register Runner

**With new GitLab tokens (16.0+)**:

```bash
sudo gitlab-runner register \
  --url https://gitlab.com/ \
  --token glrt-YOUR_TOKEN_HERE \
  --executor docker \
  --docker-image nixos/nix:latest \
  --docker-privileged \
  --docker-volumes /nix/store:/nix/store:ro \
  --docker-volumes /nix/var/nix/db:/nix/var/nix/db:ro \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock
```

Then configure tags in GitLab UI: **Settings → CI/CD → Runners → Edit → Add tag: `nix`**

**With legacy tokens (<16.0)**:

```bash
sudo gitlab-runner register \
  --url https://gitlab.com/ \
  --token YOUR_REGISTRATION_TOKEN \
  --executor docker \
  --docker-image nixos/nix:latest \
  --docker-privileged \
  --docker-volumes /nix/store:/nix/store:ro \
  --docker-volumes /nix/var/nix/db:/nix/var/nix/db:ro \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock \
  --tag-list nix \
  --run-untagged=false
```

### 3. Add gitlab-runner to Docker Group

```bash
sudo usermod -aG docker gitlab-runner
sudo systemctl restart gitlab-runner
```

## Setting Up Cachix (Optional but Recommended)

Cachix provides binary caching to speed up builds dramatically.

### 1. Create Cachix Account and Cache

```bash
# Install cachix
nix-env -iA cachix -f https://cachix.org/api/v1/install

# Login to cachix
cachix authtoken

# Create cache
cachix create towerops

# Generate keypair
cachix generate-keypair towerops
```

### 2. Get Public Key

```bash
cachix get towerops
```

This will output something like:
```
towerops.cachix.org-1:AbCdEfGhIjKlMnOpQrStUvWxYz1234567890ABCDEFG=
```

### 3. Update Configuration

**In flake.nix**, replace the placeholder:

```nix
extra-trusted-public-keys = [
  "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
  "towerops.cachix.org-1:YOUR_ACTUAL_PUBLIC_KEY_HERE"  # Replace this
];
```

**In .gitlab-ci.yml**, replace:

```yaml
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= towerops.cachix.org-1:YOUR_ACTUAL_PUBLIC_KEY_HERE
```

### 4. Add Auth Token to GitLab

Get your auth token:

```bash
cachix authtoken
```

Add to GitLab:
1. Go to: Project → Settings → CI/CD → Variables
2. Add variable:
   - **Key**: `CACHIX_AUTH_TOKEN`
   - **Value**: (paste token from above)
   - **Type**: Variable
   - **Protected**: Yes
   - **Masked**: Yes
   - **Expand variable reference**: No

## Verifying the Setup

### Test Runner Connection

Check that the runner is connected:

```bash
# On the runner host
sudo gitlab-runner verify

# In GitLab UI
# Go to: Settings → CI/CD → Runners
# You should see your runner with the "nix" tag listed
```

### Test Build

Push a commit to the `main` branch and watch the pipeline:

1. Go to: CI/CD → Pipelines
2. Click on the latest pipeline
3. Watch the build job

Expected output:
```
Building Docker image with Nix...
[Building all dependencies...]
Creating layer 1 from paths: [...]
Creating layer 2 from paths: [...]
...
Done.
Loading Docker image...
Pushing to GitLab Container Registry...
```

## Troubleshooting

### NixOS: User configuration errors

**Error**: `users.users.gitlab-runner.isSystemUser` must be set

**Fix**: Add user configuration to `/etc/nixos/configuration.nix`:

```nix
users.users.gitlab-runner = {
  isSystemUser = true;
  group = "gitlab-runner";
  extraGroups = [ "docker" ];
};

users.groups.gitlab-runner = {};
```

### Runner registration with new tokens fails

**Error**: `Runner configuration other than name and executor... is reserved`

**Fix**: With new GitLab tokens (`glrt-*`), you cannot specify `--tag-list`, `--run-untagged`, etc. in the registration command. Configure these in GitLab UI instead:

1. Register with minimal command (without `--tag-list`, `--run-untagged`)
2. Go to GitLab UI: **Settings → CI/CD → Runners → Edit**
3. Add tags and configure settings there

### Runner shows offline in GitLab

**Check runner status**:
```bash
sudo gitlab-runner status
sudo systemctl status gitlab-runner
```

**Check logs**:
```bash
sudo journalctl -u gitlab-runner -f
```

### Build fails with "experimental feature 'flakes' is not enabled"

**Fix**: Ensure NIX_CONFIG in `.gitlab-ci.yml` includes experimental features:
```yaml
NIX_CONFIG: |
  experimental-features = nix-command flakes
```

### Docker load fails with permission denied

**Fix**: Ensure gitlab-runner user is in docker group:
```bash
sudo usermod -aG docker gitlab-runner
sudo systemctl restart gitlab-runner
```

### Cachix push fails

**Check auth token**:
- Verify `CACHIX_AUTH_TOKEN` is set in GitLab CI/CD variables
- Verify token is correct: run `cachix authtoken` locally

**Check network**:
- Ensure runner can reach cachix.org
- Check firewall rules

### Build is very slow

**Without Cachix**: First build will compile everything from source (~20-30 minutes)

**With Cachix**: Subsequent builds should be much faster (~2-5 minutes) as dependencies are cached

**Improve speed**:
1. Set up Cachix (see above)
2. Use a more powerful runner instance
3. Increase runner's `concurrent` setting in `/etc/gitlab-runner/config.toml`

## Rollback to Docker Builds

If you need to roll back to the old Docker-based builds:

```bash
# The old config is in git history
git log --all -- .gitlab-ci.yml

# Restore the old version (find the commit hash from log)
git show COMMIT_HASH:.gitlab-ci.yml > .gitlab-ci.yml

# Commit and push
git add .gitlab-ci.yml
git commit -m "Revert to Docker-based builds"
git push
```

The old Dockerfile is still present at `k8s/Dockerfile` and can be used.

## Performance Comparison

### Docker-based builds (before Nix):
- First build: ~15 minutes
- Incremental: ~10 minutes (with layer caching)
- Image size: ~500 MB

### Nix-based builds (with Cachix):
- First build: ~25 minutes (everything from source)
- Subsequent builds: ~2-5 minutes (binary cache hits)
- Image size: ~507 MB compressed (~960 MB uncompressed)

### Benefits of Nix:
- **Reproducible**: Identical builds across all environments
- **Cacheable**: Binary caching eliminates redundant compilation
- **Declarative**: All dependencies pinned in flake.lock
- **Fast incremental builds**: Only changed layers rebuild

## Additional Resources

- [Nix Flakes Manual](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake.html)
- [GitLab Runner Docker Executor](https://docs.gitlab.com/runner/executors/docker.html)
- [Cachix Documentation](https://docs.cachix.org/)
- [NixOS GitLab Runner](https://nixos.wiki/wiki/Gitlab_runner)