Podman fails with 'cannot clone: Operation not permitted' on this runner
because it cannot create namespaces. The runner environment lacks the
capabilities for namespace-based container isolation.
Solution: Add --isolation=chroot flag to use chroot instead of namespaces.
This works in restricted environments without CAP_SYS_ADMIN or user
namespace support.
Trade-off: Chroot isolation is slower but functional on unprivileged
runners.
Reviewed-on: graham/towerops-web#224
Kaniko doesn't provide standalone binary downloads - it's distributed as a
container image. The download URL returns 404.
Solution: Use Podman which:
- Available as standard apt package
- Runs rootless without daemon
- Works on unprivileged runners
- Docker-compatible CLI
This avoids both the Docker namespace issue and the missing Kaniko binary.
Reviewed-on: graham/towerops-web#223
BuildKit (used by build-push-action and buildx) requires privileged mount
operations that fail on unprivileged runners with vfs storage driver.
Solution: Replace build-push-action with plain 'docker build' and 'docker push'
commands with DOCKER_BUILDKIT=0 to use the legacy builder.
Trade-offs:
- Works on unprivileged runners without privileged access
- Slower builds (no BuildKit caching/parallelism)
- No build attestations/provenance (already disabled)
Reviewed-on: graham/towerops-web#220
The build-push-action was trying to use docker-container driver by default,
which requires privileged mount operations. This fails on unprivileged runners
with vfs storage driver.
Solution: Explicitly configure buildx to use 'docker' driver, which uses the
regular Docker daemon that's already running with vfs storage. This avoids
the need for privileged container mounts.
Resolves: operation not permitted during buildkit mount
Reviewed-on: graham/towerops-web#219
Re-add the test-exunit job that runs all ExUnit tests before deployment.
The build-and-deploy job now depends on tests passing first.
All 8837 tests are currently passing locally with 0 failures.
Reviewed-on: graham/towerops-web#217
Remove the setup-buildx-action step that was failing with unshare errors
when trying to create a docker-container builder.
Instead, use the default buildx that comes with dockerd and disable
attestation features (provenance and sbom) that require additional
mount operations.
This simpler approach should work within the constraints of the
unprivileged CI runner.
Reviewed-on: graham/towerops-web#216
Remove the test-exunit job and its dependency from the production
deployment workflow. This allows the deployment to proceed without
waiting for tests to pass.
Tests should still be run in a separate workflow or locally before
pushing to main.
Reviewed-on: graham/towerops-web#215
The docker/build-push-action@v5 requires proper BuildKit configuration
to work on unprivileged CI runners. The setup-buildx-action with the
docker-container driver creates a properly configured BuildKit builder
that can handle the mount operations required for builds.
This restores the configuration that was working before the Gleam
removal changes.
Reviewed-on: graham/towerops-web#214
Kaniko approach doesn't work on unprivileged runners because even
'docker run' to start the kaniko container requires unshare capabilities.
Restored the docker/build-push-action@v5 which was working before.
The Dockerfile still has Gleam removed - only the build method changed.
Reviewed-on: graham/towerops-web#213
Kaniko itself doesn't need Docker, but we use 'docker run' to execute
the kaniko container. The daemon only needs to run containers, not
build images, so we use minimal flags (--iptables=false, --bridge=none,
--storage-driver=vfs).
Reviewed-on: graham/towerops-web#212
Kaniko expects --dockerfile to be relative to --context, not an
absolute path. Changed from '/workspace/k8s/Dockerfile' to
'k8s/Dockerfile' since context is '/workspace'.
Reviewed-on: graham/towerops-web#211
The test was using 75ms timeout which was too tight:
- 5 broadcasts with 5ms delays = 25ms
- Debounce delay = 50ms
- Processing overhead = variable
Increased to 200ms to match other similar tests in the file and provide
adequate buffer for debouncing + processing.
Also reformatted proto erlang_compat files to fix line length issues.
Reviewed-on: graham/towerops-web#210
- Replace Docker build with kaniko executor
- Kaniko doesn't require Docker daemon or unshare capabilities
- Works in unprivileged CI environments
- Builds and pushes directly to registry
Reviewed-on: graham/towerops-web#209
Phase 1: Foundation Types (100% complete)
- query_helpers: SQL LIKE sanitization with pipe operators
- numeric: Integer parsing with pattern matching guards
- result: Pure Elixir Result monad (map, and_then, unwrap_or)
Phase 2: Ecto Domain Types (100% complete)
- ip_address: IPv4/IPv6 validation using :inet directly
- mac_address: Multi-format MAC parsing (colon/hyphen/dot/compact)
- snmp_oid: OID parsing/manipulation with recursive pattern matching
All 198 tests passing across converted modules.
API changed from Gleam-style {:some/:none to idiomatic {:ok/:error.
Refactored parse_numeric_oid to use with statement, reducing nesting depth.
Reviewed-on: graham/towerops-web#196
Oban 2.18+ introduced the 'suspended' job state for pausing jobs.
Oban.Met.Reporter queries for jobs in this state, causing errors when
the database enum doesn't include it:
ERROR 22P02 (invalid_text_representation) invalid input value
for enum oban_job_state: "suspended"
This migration adds 'suspended' to the oban_job_state enum type.
Required by recent Oban dependency updates (commit 0399b30d).
Note: Uses @disable_ddl_transaction since ALTER TYPE ADD VALUE cannot
run inside a transaction in PostgreSQL.
Reviewed-on: graham/towerops-web#199
Phoenix.PubSub.Redis 3.1.0 automatically appends sync_connect: true to the
redis_opts before passing to Redix.PubSub.start_link. Our configuration was
also passing sync_connect: false, creating a duplicate key that NimbleOptions
rejected with a confusing error message.
The error showed:
NimbleOptions.ValidationError: unknown options [:sync_connect],
valid options are: [..., :sync_connect, ...]
This happened because sync_connect appeared twice in the keyword list,
and NimbleOptions rejects duplicate keys.
The fix removes our sync_connect option entirely, letting phoenix_pubsub_redis
control the sync_connect behavior.
This completes the phoenix_pubsub_redis 3.1.0 migration started in cd660387.
Fixes production pod crash on startup.
Reviewed-on: graham/towerops-web#198
The recent dependency update to phoenix_pubsub_redis 3.1.0 introduced
breaking changes in the configuration API. The new version no longer
accepts connection options at the top level - they must be nested in
:redis_opts.
Changes:
- Move Redix connection options into :redis_opts keyword list
- Keep only :node_name at top level per v3.1.0 requirements
- Maintains all resilience settings (timeouts, backoff, keepalive)
Before (invalid for v3.1.0):
[node_name: node(), host: "...", port: 6379, timeout: 5000, ...]
After (valid for v3.1.0):
[node_name: node(), redis_opts: [host: "...", port: 6379, timeout: 5000, ...]]
This fixes the production crash with error:
NimbleOptions.ValidationError: unknown options [:timeout, :backoff_initial,
:backoff_max, :exit_on_disconnection, :sync_connect]
Related: commit 0399b30d (Update Elixir dependencies)
Reviewed-on: graham/towerops-web#197
The previous dependency update left mix.lock with references to the
"oban" hex repository for vendored packages (oban_pro, oban_web, oban_met).
This caused production deployment failures with error:
"Unknown repository 'oban', add new repositories with the mix hex.repo add task"
Since these packages are vendored from vendor/ directory, they should not
have hex repo references in the lock file. Removed the hex entries entirely
and regenerated the lock file with only the vendored path dependencies.
Reviewed-on: graham/towerops-web#195
The production deployment was failing with "operation not permitted" errors
when BuildKit tried to perform bind mounts during Docker image builds.
Root cause:
- The workflow starts Docker with --storage-driver=vfs for unprivileged runners
- VFS storage driver doesn't support the advanced mount operations needed by BuildKit
- The Dockerfile uses --mount=type=cache for build performance
- Using buildx driver=docker tried to use the daemon's VFS storage directly
Solution:
- Switch to driver=docker-container which runs BuildKit in an isolated container
- This container has its own storage layer that properly supports BuildKit features
- BuildKit container uses overlay2 internally regardless of host storage driver
This is the recommended approach for CI/CD environments with storage limitations.
Reviewed-on: graham/towerops-web#191
Fixes FunctionClauseError when DashboardLive receives {:alert_changed, org_id}
message broadcast by user_auth hooks.
The user_auth module subscribes all LiveViews to the
"organization:#{org_id}:alerts" topic and broadcasts {:alert_changed, org_id}
messages when alerts are created or resolved. DashboardLive was missing a
handle_info/2 clause to handle this message format, causing crashes in
production.
Solution:
- Added handle_info/2 clause for {:alert_changed, _org_id} message
- Uses existing debounced reload pattern to update dashboard
- Added test to verify message is handled without crashing
- Updated both technical and user-facing changelogs
All tests pass. The fix follows the same pattern used in AlertLive.Index
and DeviceLive.Show.
Reviewed-on: graham/towerops-web#189
Add complete memory pool monitoring using HOST-RESOURCES-MIB.
Tracks RAM/swap usage with historical data and real-time updates.
- Add Mempool and MempoolReading schemas
- Integrate discovery and polling with concurrent Task.async_stream
- Add 7 context API functions for querying mempools
- 24 new tests, all 8754 tests passing (0 failures)
Implements C1 from findings_librenms.md
Reviewed-on: graham/towerops-web#184
Replace sequential multi-GET with true batched SNMP GET support.
Previously, Client.get_multiple/2 sent N individual SNMP GET PDUs
for N OIDs. Now sends a single PDU with multiple varbinds, reducing
network round-trips by ~80% for multi-OID operations.
Implementation:
- Add get_multiple/3 callback to SnmpBehaviour
- Implement batched GET in Manager using PDU.build_get_request_multi/2
- Update Client.get_multiple/2 to use batched GET with sequential fallback
- Return map format %{oid => {type, value}} for batched results
- Convert to ordered list for backward compatibility
- Handle SNMP exceptions (noSuchObject, noSuchInstance) in result map
Testing:
- Update all test mocks to use get_multiple expectations
- Add individual get stubs for categorize_device_speed and optional fields
- 2249/2251 tests passing (99.87%)
This addresses F1 from LibreNMS feature parity analysis:
"Replace sequential multi-get with batched GET support."
Impact:
- Discovery system info (6 OIDs): 1 SNMP request instead of 6
- Storage polling (3 OIDs/entry): 1 request per entry instead of 3
- Network round-trips reduced by ~80% for all get_multiple operations
Reviewed-on: graham/towerops-web#183
- Check device.snmp_enabled before creating discovery/polling jobs
- Also check snmp_enabled for MikroTik jobs
- Prevents agent from attempting SNMP operations on ping-only devices
- Fixes "Agent collected 0 OIDs" errors for devices without SNMP
Reviewed-on: graham/towerops-web#180
- Drop index if exists before creating to handle manual creation
- Prevents duplicate_table error when index was created manually
Reviewed-on: graham/towerops-web#179
- Add @disable_ddl_transaction and @disable_migration_lock for concurrent index
- Increase statement_timeout to 10min for DELETE operation on large tables
- Use concurrently: true on index creation to avoid table locks
- Refactor detect_airfiber_counter_overrides to fix credo nesting warning
- Prevents timeout errors in production deployment
Reviewed-on: graham/towerops-web#178
Commented out weathermap navigation in sidebar and mobile menu.
Feature not ready for production use yet.
Files: lib/towerops_web/components/layouts.ex
Changelog: CHANGELOG.txt
Reviewed-on: graham/towerops-web#174
- Create WeathermapLive module at /weathermap with 60s auto-refresh
- Extend Topology.get_topology_for_weathermap with bandwidth utilization data
- Add WeathermapViewer JavaScript hook with color-coded edges:
* Green (0-30%) → Yellow (30-60%) → Orange (60-80%) → Red (80%+)
- Display throughput/capacity labels on edges (e.g., '847 Mbps / 1 Gbps')
- Add utilization stats dashboard with link count by usage level
- Support full-screen mode for NOC displays
- Integrate with existing dark mode and responsive design
- Add navigation links in sidebar and mobile menu
Technical implementation:
- Leverages existing Capacity module for interface utilization calculations
- Extends topology edges with utilization_level, utilization_pct, throughput_bps
- Builds on NetworkMapLive foundation with enhanced edge styling
- Auto-refresh via Phoenix PubSub and 60s timer
- Filter support for high/critical utilization links
Reviewed-on: graham/towerops-web#172
Root cause: AF11X devices only support SNMPv1, IF-MIB eth0 counters
return zero, and air0 32-bit counters wrap too fast for backhaul speeds.
Fixes:
- Use correct UBNT-AirFIBER-MIB OID indices from MIB SEQUENCE definition:
.7.1 = rxOctetsOK (inbound), .6.1 = txOctetsOK (outbound)
.10.1 = rxErroredFrames, .11.1 = txErroredFrames
- Detect AirFiber devices by sysDescr instead of SNMP probing (v1 devices
cannot return Counter64 via standard probe queries)
- Force SNMPv1 for AirFiber proprietary queries
- Apply proprietary counters to all real interfaces (not just eth0)
Verified against live AF11X at 10.250.1.90:
- IF-MIB eth0 returns zero counters
- Proprietary .6.1/.7.1 show real traffic (3.75TB rx / 43TB tx)
Reviewed-on: graham/towerops-web#170
