The previous commit bumped CI from OTP 27.2/Elixir 1.18.1 to
OTP 28.5/Elixir 1.19.5. The broad mix cache restore-keys was
restoring a stale ~/.mix cache with hex compiled for the old
OTP, causing beam_load errors on new bs_add instructions.
Adding explicit mix local.hex --force ensures the correct hex
version for the running OTP is installed before deps.get.
Splits the runtime apt installs (gdal-bin, snmp, libsnmp40, locales,
BEAM runtime libs) into k8s/Dockerfile.base, hosted at
codeberg.org/gmcintire/towerops-base:latest. The app Dockerfile now
does FROM that base instead of re-installing gdal (~500 MB) on every
push.
The new build-base workflow rebuilds the base image only when
k8s/Dockerfile.base or the workflow itself changes, weekly via cron
(Sundays 06:00 UTC, with CACHE_BUST=<ISO week> to force apt-get update
on a week boundary), or via workflow_dispatch.
Production workflow now uses buildx + does docker login before the
build so it can pull the private base image.
The repo got renamed towerops-web → towerops on the Codeberg side,
so the published image follows: image is now
codeberg.org/gmcintire/towerops with the same main-<ts>-<sha> tag
pattern. Login URL is the hardcoded env.REGISTRY rather than
secrets.REGISTRY_URL so a stale URL secret can't push to the wrong
registry.
CI: switch postgres service image to timescale/timescaledb-ha:pg17-all
which bundles PostGIS, fixing the lidar migration failure in PR and
production workflows.
Tests: ~30 new test files / extensions, ~600+ new tests, taking total
coverage from 70.42% to 73.4%. Targets included GraphQL resolver
unauthenticated paths, SNMP context queries (mempools, processors,
ip_addresses, neighbors, wireless_clients, arp_entries), worker
routing/lifecycle (alert notification, alert digest, weather sync,
job cleanup, cn_maestro, uisp, mikrotik backup, report worker),
SNMP monitoring executors (storage, interface, processor) end-to-end
via stub adapter, NetBox + Sonar sync integration paths, ConfigChanges
listing and correlator, organizations membership query, and live-view
smoke tests for org-scoped settings, integrations, maintenance, agent,
device deep paths, schedules. Property tests used for query
composition and pure helpers where natural.
Bookworm runner images use /etc/apt/sources.list.d/*.sources (deb822),
which the legacy sed didn't touch — non-free stayed off and
snmp-mibs-downloader was unavailable. Patch the Components: line in
deb822 sources too.
Runner doesn't have docker pre-installed; explicitly install docker.io
matching the pattern used in microwaveprop.
Reviewed-on: graham/towerops-web#237
The CI runner already has a working Docker daemon and the runner user
is in the docker group. All previous failures were caused by the workflow
trying to start a NEW dockerd instance with vfs storage driver.
Solution: Remove all daemon startup code and just use the existing Docker
daemon directly. The runner user already has access via group membership.
Verified on ci host:
- Docker service running: overlayfs storage driver
- runner user in docker group: uid=1001(runner) groups=1001(runner),989(docker)
- docker info works as runner user
This is the simplest solution and should have been tried first.
Reviewed-on: graham/towerops-web#226
Podman fails even with chroot isolation on this runner. The environment
is too restricted for standard container tools (Docker, Buildx, Podman).
Solution: Use 'img' by Jess Frazelle, designed specifically for truly
unprivileged environments:
- No root/sudo required for builds
- No user namespaces required
- No special capabilities required
- Pure userspace implementation
- Built for restricted CI environments
This is the last resort tool for environments where Docker/Podman
cannot run at all.
Reviewed-on: graham/towerops-web#225
Podman fails with 'cannot clone: Operation not permitted' on this runner
because it cannot create namespaces. The runner environment lacks the
capabilities for namespace-based container isolation.
Solution: Add --isolation=chroot flag to use chroot instead of namespaces.
This works in restricted environments without CAP_SYS_ADMIN or user
namespace support.
Trade-off: Chroot isolation is slower but functional on unprivileged
runners.
Reviewed-on: graham/towerops-web#224
Kaniko doesn't provide standalone binary downloads - it's distributed as a
container image. The download URL returns 404.
Solution: Use Podman which:
- Available as standard apt package
- Runs rootless without daemon
- Works on unprivileged runners
- Docker-compatible CLI
This avoids both the Docker namespace issue and the missing Kaniko binary.
Reviewed-on: graham/towerops-web#223
BuildKit (used by build-push-action and buildx) requires privileged mount
operations that fail on unprivileged runners with vfs storage driver.
Solution: Replace build-push-action with plain 'docker build' and 'docker push'
commands with DOCKER_BUILDKIT=0 to use the legacy builder.
Trade-offs:
- Works on unprivileged runners without privileged access
- Slower builds (no BuildKit caching/parallelism)
- No build attestations/provenance (already disabled)
Reviewed-on: graham/towerops-web#220
The build-push-action was trying to use docker-container driver by default,
which requires privileged mount operations. This fails on unprivileged runners
with vfs storage driver.
Solution: Explicitly configure buildx to use 'docker' driver, which uses the
regular Docker daemon that's already running with vfs storage. This avoids
the need for privileged container mounts.
Resolves: operation not permitted during buildkit mount
Reviewed-on: graham/towerops-web#219
Re-add the test-exunit job that runs all ExUnit tests before deployment.
The build-and-deploy job now depends on tests passing first.
All 8837 tests are currently passing locally with 0 failures.
Reviewed-on: graham/towerops-web#217
Remove the setup-buildx-action step that was failing with unshare errors
when trying to create a docker-container builder.
Instead, use the default buildx that comes with dockerd and disable
attestation features (provenance and sbom) that require additional
mount operations.
This simpler approach should work within the constraints of the
unprivileged CI runner.
Reviewed-on: graham/towerops-web#216
Remove the test-exunit job and its dependency from the production
deployment workflow. This allows the deployment to proceed without
waiting for tests to pass.
Tests should still be run in a separate workflow or locally before
pushing to main.
Reviewed-on: graham/towerops-web#215
The docker/build-push-action@v5 requires proper BuildKit configuration
to work on unprivileged CI runners. The setup-buildx-action with the
docker-container driver creates a properly configured BuildKit builder
that can handle the mount operations required for builds.
This restores the configuration that was working before the Gleam
removal changes.
Reviewed-on: graham/towerops-web#214
Kaniko approach doesn't work on unprivileged runners because even
'docker run' to start the kaniko container requires unshare capabilities.
Restored the docker/build-push-action@v5 which was working before.
The Dockerfile still has Gleam removed - only the build method changed.
Reviewed-on: graham/towerops-web#213
Kaniko itself doesn't need Docker, but we use 'docker run' to execute
the kaniko container. The daemon only needs to run containers, not
build images, so we use minimal flags (--iptables=false, --bridge=none,
--storage-driver=vfs).
Reviewed-on: graham/towerops-web#212
Kaniko expects --dockerfile to be relative to --context, not an
absolute path. Changed from '/workspace/k8s/Dockerfile' to
'k8s/Dockerfile' since context is '/workspace'.
Reviewed-on: graham/towerops-web#211
- Replace Docker build with kaniko executor
- Kaniko doesn't require Docker daemon or unshare capabilities
- Works in unprivileged CI environments
- Builds and pushes directly to registry
Reviewed-on: graham/towerops-web#209
Phase 1: Foundation Types (100% complete)
- query_helpers: SQL LIKE sanitization with pipe operators
- numeric: Integer parsing with pattern matching guards
- result: Pure Elixir Result monad (map, and_then, unwrap_or)
Phase 2: Ecto Domain Types (100% complete)
- ip_address: IPv4/IPv6 validation using :inet directly
- mac_address: Multi-format MAC parsing (colon/hyphen/dot/compact)
- snmp_oid: OID parsing/manipulation with recursive pattern matching
All 198 tests passing across converted modules.
API changed from Gleam-style {:some/:none to idiomatic {:ok/:error.
Refactored parse_numeric_oid to use with statement, reducing nesting depth.
Reviewed-on: graham/towerops-web#196
The production deployment was failing with "operation not permitted" errors
when BuildKit tried to perform bind mounts during Docker image builds.
Root cause:
- The workflow starts Docker with --storage-driver=vfs for unprivileged runners
- VFS storage driver doesn't support the advanced mount operations needed by BuildKit
- The Dockerfile uses --mount=type=cache for build performance
- Using buildx driver=docker tried to use the daemon's VFS storage directly
Solution:
- Switch to driver=docker-container which runs BuildKit in an isolated container
- This container has its own storage layer that properly supports BuildKit features
- BuildKit container uses overlay2 internally regardless of host storage driver
This is the recommended approach for CI/CD environments with storage limitations.
Reviewed-on: graham/towerops-web#191
Dependency update PRs from Renovate don't need to be deployed to
staging. Uses job-level condition to check head branch prefix.
Reviewed-on: graham/towerops-web#124
Replace the protobuf hex package (agent.pb.ex) and hand-written validator
(validator.ex) with a pure Gleam implementation: wire format primitives,
all 23 message types, encode/decode functions, and validation merged into
decode. Thin Elixir wrappers in agent.ex preserve the existing struct API.
- Wire format: varint, length-delimited, double (IEEE 754 via FFI), tag encode/decode
- Types: all proto3 message types as Gleam custom types with enums and oneof
- Decode: field-level parsing with integrated validation (UUID, IP, hostname, limits)
- Encode: proto3 default omission, bytes_tree concatenation, map/repeated fields
- Wrappers: Elixir structs with encode/decode delegating to Gleam, enum atom conversion
- Remove {:protobuf, "~> 0.12"} dependency
Reviewed-on: graham/towerops-web#104
Set up mix_gleam archive, gleam_stdlib, and gleeunit deps. Add Gleam
compiler to all Dockerfiles, CI workflows, nix shell, and .tool-versions.
Rewrite Numeric, QueryHelpers, and URLValidator from Elixir to Gleam with
full ExUnit test coverage. Update all callers to use the Gleam modules
directly via :towerops@module syntax. Remove duplicate sanitize_like/1
private functions in snmp.ex and trace.ex.
Reviewed-on: graham/towerops-web#98
Moves e2e tests from production workflow into a separate workflow that requires manual triggering. This allows e2e tests to be run on-demand without blocking production deployments.
Changes:
- Created new .forgejo/workflows/e2e-tests.yaml with workflow_dispatch trigger
- Removed disabled e2e job from production.yaml workflow
- E2E tests can now be triggered manually from Actions tab with browser selection (all, chromium, firefox, webkit)
- Test results and videos uploaded as artifacts on all runs
Reviewed-on: graham/towerops-web#66
The docker buildx driver (required on unprivileged ci runner) does not
support registry cache export/import. Remove cache-from and cache-to
options to unblock builds.
Reviewed-on: graham/towerops-web#62
overlayfs requires mount/unmount capabilities not available on the
unprivileged ci runner. Switch to vfs storage driver and use buildx
docker driver (not docker-container) to avoid nested mount issues.
Reviewed-on: graham/towerops-web#61
