The CI runner already has a working Docker daemon and the runner user
is in the docker group. All previous failures were caused by the workflow
trying to start a NEW dockerd instance with vfs storage driver.
Solution: Remove all daemon startup code and just use the existing Docker
daemon directly. The runner user already has access via group membership.
Verified on ci host:
- Docker service running: overlayfs storage driver
- runner user in docker group: uid=1001(runner) groups=1001(runner),989(docker)
- docker info works as runner user
This is the simplest solution and should have been tried first.
Reviewed-on: graham/towerops-web#226
Podman fails even with chroot isolation on this runner. The environment
is too restricted for standard container tools (Docker, Buildx, Podman).
Solution: Use 'img' by Jess Frazelle, designed specifically for truly
unprivileged environments:
- No root/sudo required for builds
- No user namespaces required
- No special capabilities required
- Pure userspace implementation
- Built for restricted CI environments
This is the last resort tool for environments where Docker/Podman
cannot run at all.
Reviewed-on: graham/towerops-web#225
Podman fails with 'cannot clone: Operation not permitted' on this runner
because it cannot create namespaces. The runner environment lacks the
capabilities for namespace-based container isolation.
Solution: Add --isolation=chroot flag to use chroot instead of namespaces.
This works in restricted environments without CAP_SYS_ADMIN or user
namespace support.
Trade-off: Chroot isolation is slower but functional on unprivileged
runners.
Reviewed-on: graham/towerops-web#224
Kaniko doesn't provide standalone binary downloads - it's distributed as a
container image. The download URL returns 404.
Solution: Use Podman which:
- Available as standard apt package
- Runs rootless without daemon
- Works on unprivileged runners
- Docker-compatible CLI
This avoids both the Docker namespace issue and the missing Kaniko binary.
Reviewed-on: graham/towerops-web#223
BuildKit (used by build-push-action and buildx) requires privileged mount
operations that fail on unprivileged runners with vfs storage driver.
Solution: Replace build-push-action with plain 'docker build' and 'docker push'
commands with DOCKER_BUILDKIT=0 to use the legacy builder.
Trade-offs:
- Works on unprivileged runners without privileged access
- Slower builds (no BuildKit caching/parallelism)
- No build attestations/provenance (already disabled)
Reviewed-on: graham/towerops-web#220
The build-push-action was trying to use docker-container driver by default,
which requires privileged mount operations. This fails on unprivileged runners
with vfs storage driver.
Solution: Explicitly configure buildx to use 'docker' driver, which uses the
regular Docker daemon that's already running with vfs storage. This avoids
the need for privileged container mounts.
Resolves: operation not permitted during buildkit mount
Reviewed-on: graham/towerops-web#219
Re-add the test-exunit job that runs all ExUnit tests before deployment.
The build-and-deploy job now depends on tests passing first.
All 8837 tests are currently passing locally with 0 failures.
Reviewed-on: graham/towerops-web#217
Remove the setup-buildx-action step that was failing with unshare errors
when trying to create a docker-container builder.
Instead, use the default buildx that comes with dockerd and disable
attestation features (provenance and sbom) that require additional
mount operations.
This simpler approach should work within the constraints of the
unprivileged CI runner.
Reviewed-on: graham/towerops-web#216
Remove the test-exunit job and its dependency from the production
deployment workflow. This allows the deployment to proceed without
waiting for tests to pass.
Tests should still be run in a separate workflow or locally before
pushing to main.
Reviewed-on: graham/towerops-web#215
The docker/build-push-action@v5 requires proper BuildKit configuration
to work on unprivileged CI runners. The setup-buildx-action with the
docker-container driver creates a properly configured BuildKit builder
that can handle the mount operations required for builds.
This restores the configuration that was working before the Gleam
removal changes.
Reviewed-on: graham/towerops-web#214
Kaniko approach doesn't work on unprivileged runners because even
'docker run' to start the kaniko container requires unshare capabilities.
Restored the docker/build-push-action@v5 which was working before.
The Dockerfile still has Gleam removed - only the build method changed.
Reviewed-on: graham/towerops-web#213
Kaniko itself doesn't need Docker, but we use 'docker run' to execute
the kaniko container. The daemon only needs to run containers, not
build images, so we use minimal flags (--iptables=false, --bridge=none,
--storage-driver=vfs).
Reviewed-on: graham/towerops-web#212
Kaniko expects --dockerfile to be relative to --context, not an
absolute path. Changed from '/workspace/k8s/Dockerfile' to
'k8s/Dockerfile' since context is '/workspace'.
Reviewed-on: graham/towerops-web#211
- Replace Docker build with kaniko executor
- Kaniko doesn't require Docker daemon or unshare capabilities
- Works in unprivileged CI environments
- Builds and pushes directly to registry
Reviewed-on: graham/towerops-web#209
Phase 1: Foundation Types (100% complete)
- query_helpers: SQL LIKE sanitization with pipe operators
- numeric: Integer parsing with pattern matching guards
- result: Pure Elixir Result monad (map, and_then, unwrap_or)
Phase 2: Ecto Domain Types (100% complete)
- ip_address: IPv4/IPv6 validation using :inet directly
- mac_address: Multi-format MAC parsing (colon/hyphen/dot/compact)
- snmp_oid: OID parsing/manipulation with recursive pattern matching
All 198 tests passing across converted modules.
API changed from Gleam-style {:some/:none to idiomatic {:ok/:error.
Refactored parse_numeric_oid to use with statement, reducing nesting depth.
Reviewed-on: graham/towerops-web#196
The production deployment was failing with "operation not permitted" errors
when BuildKit tried to perform bind mounts during Docker image builds.
Root cause:
- The workflow starts Docker with --storage-driver=vfs for unprivileged runners
- VFS storage driver doesn't support the advanced mount operations needed by BuildKit
- The Dockerfile uses --mount=type=cache for build performance
- Using buildx driver=docker tried to use the daemon's VFS storage directly
Solution:
- Switch to driver=docker-container which runs BuildKit in an isolated container
- This container has its own storage layer that properly supports BuildKit features
- BuildKit container uses overlay2 internally regardless of host storage driver
This is the recommended approach for CI/CD environments with storage limitations.
Reviewed-on: graham/towerops-web#191
Dependency update PRs from Renovate don't need to be deployed to
staging. Uses job-level condition to check head branch prefix.
Reviewed-on: graham/towerops-web#124
Replace the protobuf hex package (agent.pb.ex) and hand-written validator
(validator.ex) with a pure Gleam implementation: wire format primitives,
all 23 message types, encode/decode functions, and validation merged into
decode. Thin Elixir wrappers in agent.ex preserve the existing struct API.
- Wire format: varint, length-delimited, double (IEEE 754 via FFI), tag encode/decode
- Types: all proto3 message types as Gleam custom types with enums and oneof
- Decode: field-level parsing with integrated validation (UUID, IP, hostname, limits)
- Encode: proto3 default omission, bytes_tree concatenation, map/repeated fields
- Wrappers: Elixir structs with encode/decode delegating to Gleam, enum atom conversion
- Remove {:protobuf, "~> 0.12"} dependency
Reviewed-on: graham/towerops-web#104
Set up mix_gleam archive, gleam_stdlib, and gleeunit deps. Add Gleam
compiler to all Dockerfiles, CI workflows, nix shell, and .tool-versions.
Rewrite Numeric, QueryHelpers, and URLValidator from Elixir to Gleam with
full ExUnit test coverage. Update all callers to use the Gleam modules
directly via :towerops@module syntax. Remove duplicate sanitize_like/1
private functions in snmp.ex and trace.ex.
Reviewed-on: graham/towerops-web#98
Moves e2e tests from production workflow into a separate workflow that requires manual triggering. This allows e2e tests to be run on-demand without blocking production deployments.
Changes:
- Created new .forgejo/workflows/e2e-tests.yaml with workflow_dispatch trigger
- Removed disabled e2e job from production.yaml workflow
- E2E tests can now be triggered manually from Actions tab with browser selection (all, chromium, firefox, webkit)
- Test results and videos uploaded as artifacts on all runs
Reviewed-on: graham/towerops-web#66
The docker buildx driver (required on unprivileged ci runner) does not
support registry cache export/import. Remove cache-from and cache-to
options to unblock builds.
Reviewed-on: graham/towerops-web#62
overlayfs requires mount/unmount capabilities not available on the
unprivileged ci runner. Switch to vfs storage driver and use buildx
docker driver (not docker-container) to avoid nested mount issues.
Reviewed-on: graham/towerops-web#61
The new CI server doesn't have Docker daemon running by default,
causing buildx setup to fail with:
ERROR: failed to connect to the docker API at unix:///var/run/docker.sock
This fix:
- Adds explicit Docker daemon startup check and initialization
- Configures buildx with docker-container driver
- Adds install: true to ensure proper buildx installation
This was already fixed in build-deploy.yml (commit 54280fd7) but
was missing from production.yaml.
Reviewed-on: graham/towerops-web#57
The styfle/cancel-workflow-action is not compatible with Forgejo Actions,
causing failures with 'Cannot read properties of undefined (reading id)'.
The pr-tests.yaml workflow already has concurrency control that cancels
in-progress runs when new commits are pushed, which handles most cases.
Removing this workflow to eliminate CI errors.
Reviewed-on: graham/towerops-web#38
The cancel-workflow-action requires an access_token to query and cancel
running workflows. Without it, the action fails with 'Cannot read properties
of undefined' error when trying to access workflow information.
Reviewed-on: graham/towerops-web#36
- Updated .tool-versions to use Node 22.17.1
- Updated CI workflow to use Node 22
- Rebuilt package-lock.json with Node 22
- Fixes engine compatibility warnings from @noble/hashes
The if: success() might be causing issues with Forgejo's secret handling.
Jobs with 'needs' automatically wait for dependencies to succeed, making
the condition redundant.
Playwright config:
- Added chromium-unauthenticated project for registration/login tests
- These tests run without saved auth state (as intended)
- Other tests continue using authenticated state
Workflow:
- build-and-deploy now only requires test-exunit to pass
- test-e2e runs in parallel but doesn't block deployment
- Allows faster deployments while e2e tests provide additional validation
Created seeds_e2e.exs to automatically create a test user with TOTP enabled
for Playwright e2e tests. This ensures tests can authenticate in CI without
requiring manual user setup.
Test credentials (for CI only):
- Email: test@example.com
- Password: TestPassword123!
- TOTP Secret: JBSWY3DPEHPK3PXP (fixed for reproducibility)
The workflow now:
1. Loads database structure
2. Seeds test user with TOTP
3. Passes credentials to e2e tests via environment variables
