The docker/build-push-action@v5 requires proper BuildKit configuration
to work on unprivileged CI runners. The setup-buildx-action with the
docker-container driver creates a properly configured BuildKit builder
that can handle the mount operations required for builds.
This restores the configuration that was working before the Gleam
removal changes.
Reviewed-on: graham/towerops-web#214
Kaniko approach doesn't work on unprivileged runners because even
'docker run' to start the kaniko container requires unshare capabilities.
Restored the docker/build-push-action@v5 which was working before.
The Dockerfile still has Gleam removed - only the build method changed.
Reviewed-on: graham/towerops-web#213
Kaniko itself doesn't need Docker, but we use 'docker run' to execute
the kaniko container. The daemon only needs to run containers, not
build images, so we use minimal flags (--iptables=false, --bridge=none,
--storage-driver=vfs).
Reviewed-on: graham/towerops-web#212
Kaniko expects --dockerfile to be relative to --context, not an
absolute path. Changed from '/workspace/k8s/Dockerfile' to
'k8s/Dockerfile' since context is '/workspace'.
Reviewed-on: graham/towerops-web#211
The test was using 75ms timeout which was too tight:
- 5 broadcasts with 5ms delays = 25ms
- Debounce delay = 50ms
- Processing overhead = variable
Increased to 200ms to match other similar tests in the file and provide
adequate buffer for debouncing + processing.
Also reformatted proto erlang_compat files to fix line length issues.
Reviewed-on: graham/towerops-web#210
- Replace Docker build with kaniko executor
- Kaniko doesn't require Docker daemon or unshare capabilities
- Works in unprivileged CI environments
- Builds and pushes directly to registry
Reviewed-on: graham/towerops-web#209
Phase 1: Foundation Types (100% complete)
- query_helpers: SQL LIKE sanitization with pipe operators
- numeric: Integer parsing with pattern matching guards
- result: Pure Elixir Result monad (map, and_then, unwrap_or)
Phase 2: Ecto Domain Types (100% complete)
- ip_address: IPv4/IPv6 validation using :inet directly
- mac_address: Multi-format MAC parsing (colon/hyphen/dot/compact)
- snmp_oid: OID parsing/manipulation with recursive pattern matching
All 198 tests passing across converted modules.
API changed from Gleam-style {:some/:none to idiomatic {:ok/:error.
Refactored parse_numeric_oid to use with statement, reducing nesting depth.
Reviewed-on: graham/towerops-web#196
Oban 2.18+ introduced the 'suspended' job state for pausing jobs.
Oban.Met.Reporter queries for jobs in this state, causing errors when
the database enum doesn't include it:
ERROR 22P02 (invalid_text_representation) invalid input value
for enum oban_job_state: "suspended"
This migration adds 'suspended' to the oban_job_state enum type.
Required by recent Oban dependency updates (commit 0399b30d).
Note: Uses @disable_ddl_transaction since ALTER TYPE ADD VALUE cannot
run inside a transaction in PostgreSQL.
Reviewed-on: graham/towerops-web#199
Phoenix.PubSub.Redis 3.1.0 automatically appends sync_connect: true to the
redis_opts before passing to Redix.PubSub.start_link. Our configuration was
also passing sync_connect: false, creating a duplicate key that NimbleOptions
rejected with a confusing error message.
The error showed:
NimbleOptions.ValidationError: unknown options [:sync_connect],
valid options are: [..., :sync_connect, ...]
This happened because sync_connect appeared twice in the keyword list,
and NimbleOptions rejects duplicate keys.
The fix removes our sync_connect option entirely, letting phoenix_pubsub_redis
control the sync_connect behavior.
This completes the phoenix_pubsub_redis 3.1.0 migration started in cd660387.
Fixes production pod crash on startup.
Reviewed-on: graham/towerops-web#198
The recent dependency update to phoenix_pubsub_redis 3.1.0 introduced
breaking changes in the configuration API. The new version no longer
accepts connection options at the top level - they must be nested in
:redis_opts.
Changes:
- Move Redix connection options into :redis_opts keyword list
- Keep only :node_name at top level per v3.1.0 requirements
- Maintains all resilience settings (timeouts, backoff, keepalive)
Before (invalid for v3.1.0):
[node_name: node(), host: "...", port: 6379, timeout: 5000, ...]
After (valid for v3.1.0):
[node_name: node(), redis_opts: [host: "...", port: 6379, timeout: 5000, ...]]
This fixes the production crash with error:
NimbleOptions.ValidationError: unknown options [:timeout, :backoff_initial,
:backoff_max, :exit_on_disconnection, :sync_connect]
Related: commit 0399b30d (Update Elixir dependencies)
Reviewed-on: graham/towerops-web#197
The previous dependency update left mix.lock with references to the
"oban" hex repository for vendored packages (oban_pro, oban_web, oban_met).
This caused production deployment failures with error:
"Unknown repository 'oban', add new repositories with the mix hex.repo add task"
Since these packages are vendored from vendor/ directory, they should not
have hex repo references in the lock file. Removed the hex entries entirely
and regenerated the lock file with only the vendored path dependencies.
Reviewed-on: graham/towerops-web#195
The production deployment was failing with "operation not permitted" errors
when BuildKit tried to perform bind mounts during Docker image builds.
Root cause:
- The workflow starts Docker with --storage-driver=vfs for unprivileged runners
- VFS storage driver doesn't support the advanced mount operations needed by BuildKit
- The Dockerfile uses --mount=type=cache for build performance
- Using buildx driver=docker tried to use the daemon's VFS storage directly
Solution:
- Switch to driver=docker-container which runs BuildKit in an isolated container
- This container has its own storage layer that properly supports BuildKit features
- BuildKit container uses overlay2 internally regardless of host storage driver
This is the recommended approach for CI/CD environments with storage limitations.
Reviewed-on: graham/towerops-web#191
Fixes FunctionClauseError when DashboardLive receives {:alert_changed, org_id}
message broadcast by user_auth hooks.
The user_auth module subscribes all LiveViews to the
"organization:#{org_id}:alerts" topic and broadcasts {:alert_changed, org_id}
messages when alerts are created or resolved. DashboardLive was missing a
handle_info/2 clause to handle this message format, causing crashes in
production.
Solution:
- Added handle_info/2 clause for {:alert_changed, _org_id} message
- Uses existing debounced reload pattern to update dashboard
- Added test to verify message is handled without crashing
- Updated both technical and user-facing changelogs
All tests pass. The fix follows the same pattern used in AlertLive.Index
and DeviceLive.Show.
Reviewed-on: graham/towerops-web#189
Add complete memory pool monitoring using HOST-RESOURCES-MIB.
Tracks RAM/swap usage with historical data and real-time updates.
- Add Mempool and MempoolReading schemas
- Integrate discovery and polling with concurrent Task.async_stream
- Add 7 context API functions for querying mempools
- 24 new tests, all 8754 tests passing (0 failures)
Implements C1 from findings_librenms.md
Reviewed-on: graham/towerops-web#184
Replace sequential multi-GET with true batched SNMP GET support.
Previously, Client.get_multiple/2 sent N individual SNMP GET PDUs
for N OIDs. Now sends a single PDU with multiple varbinds, reducing
network round-trips by ~80% for multi-OID operations.
Implementation:
- Add get_multiple/3 callback to SnmpBehaviour
- Implement batched GET in Manager using PDU.build_get_request_multi/2
- Update Client.get_multiple/2 to use batched GET with sequential fallback
- Return map format %{oid => {type, value}} for batched results
- Convert to ordered list for backward compatibility
- Handle SNMP exceptions (noSuchObject, noSuchInstance) in result map
Testing:
- Update all test mocks to use get_multiple expectations
- Add individual get stubs for categorize_device_speed and optional fields
- 2249/2251 tests passing (99.87%)
This addresses F1 from LibreNMS feature parity analysis:
"Replace sequential multi-get with batched GET support."
Impact:
- Discovery system info (6 OIDs): 1 SNMP request instead of 6
- Storage polling (3 OIDs/entry): 1 request per entry instead of 3
- Network round-trips reduced by ~80% for all get_multiple operations
Reviewed-on: graham/towerops-web#183
- Check device.snmp_enabled before creating discovery/polling jobs
- Also check snmp_enabled for MikroTik jobs
- Prevents agent from attempting SNMP operations on ping-only devices
- Fixes "Agent collected 0 OIDs" errors for devices without SNMP
Reviewed-on: graham/towerops-web#180
- Drop index if exists before creating to handle manual creation
- Prevents duplicate_table error when index was created manually
Reviewed-on: graham/towerops-web#179
- Add @disable_ddl_transaction and @disable_migration_lock for concurrent index
- Increase statement_timeout to 10min for DELETE operation on large tables
- Use concurrently: true on index creation to avoid table locks
- Refactor detect_airfiber_counter_overrides to fix credo nesting warning
- Prevents timeout errors in production deployment
Reviewed-on: graham/towerops-web#178
Commented out weathermap navigation in sidebar and mobile menu.
Feature not ready for production use yet.
Files: lib/towerops_web/components/layouts.ex
Changelog: CHANGELOG.txt
Reviewed-on: graham/towerops-web#174
- Create WeathermapLive module at /weathermap with 60s auto-refresh
- Extend Topology.get_topology_for_weathermap with bandwidth utilization data
- Add WeathermapViewer JavaScript hook with color-coded edges:
* Green (0-30%) → Yellow (30-60%) → Orange (60-80%) → Red (80%+)
- Display throughput/capacity labels on edges (e.g., '847 Mbps / 1 Gbps')
- Add utilization stats dashboard with link count by usage level
- Support full-screen mode for NOC displays
- Integrate with existing dark mode and responsive design
- Add navigation links in sidebar and mobile menu
Technical implementation:
- Leverages existing Capacity module for interface utilization calculations
- Extends topology edges with utilization_level, utilization_pct, throughput_bps
- Builds on NetworkMapLive foundation with enhanced edge styling
- Auto-refresh via Phoenix PubSub and 60s timer
- Filter support for high/critical utilization links
Reviewed-on: graham/towerops-web#172
Root cause: AF11X devices only support SNMPv1, IF-MIB eth0 counters
return zero, and air0 32-bit counters wrap too fast for backhaul speeds.
Fixes:
- Use correct UBNT-AirFIBER-MIB OID indices from MIB SEQUENCE definition:
.7.1 = rxOctetsOK (inbound), .6.1 = txOctetsOK (outbound)
.10.1 = rxErroredFrames, .11.1 = txErroredFrames
- Detect AirFiber devices by sysDescr instead of SNMP probing (v1 devices
cannot return Counter64 via standard probe queries)
- Force SNMPv1 for AirFiber proprietary queries
- Apply proprietary counters to all real interfaces (not just eth0)
Verified against live AF11X at 10.250.1.90:
- IF-MIB eth0 returns zero counters
- Proprietary .6.1/.7.1 show real traffic (3.75TB rx / 43TB tx)
Reviewed-on: graham/towerops-web#170
Root cause: AF11X devices only support SNMPv1, IF-MIB eth0 counters
return zero, and air0 32-bit counters wrap too fast for backhaul speeds.
Fixes:
- Use correct UBNT-AirFIBER-MIB OID indices from MIB SEQUENCE definition:
.7.1 = rxOctetsOK (inbound), .6.1 = txOctetsOK (outbound)
.10.1 = rxErroredFrames, .11.1 = txErroredFrames
- Detect AirFiber devices by sysDescr instead of SNMP probing (v1 devices
cannot return Counter64 via standard probe queries)
- Force SNMPv1 for AirFiber proprietary queries
- Apply proprietary counters to all real interfaces (not just eth0)
Verified against live AF11X at 10.250.1.90:
- IF-MIB eth0 returns zero counters
- Proprietary .6.1/.7.1 show real traffic (3.75TB rx / 43TB tx)
Reviewed-on: graham/towerops-web#169
- Clamp negative counter deltas to zero instead of attempting 32-bit wrap
correction (matches LibreNMS behavior — wraps are indistinguishable from
HC counter resets, so discard the ambiguous data point)
- Add rate sanity check: discard values exceeding 400 Gbps as counter anomalies
- Add 4-byte binary decoding for 32-bit SNMP counters in decode_snmp_value
- Track HC vs standard counter usage per interface stat (is_hc field)
- Add migration for is_hc column on snmp_interface_stats
- Update tests to match new clamping behavior
Reviewed-on: graham/towerops-web#168
Missing indexes detected on 7 foreign key columns:
- on_call_overrides.user_id
- on_call_layer_members.user_id
- site_outages.site_id
- escalation_targets.user_id
- notification_digests.organization_id
- on_call_notifications.user_id
- alerts.site_outage_id (already existed, skipped)
Without indexes, PostgreSQL must perform sequential scans for:
- Referential integrity checks on DELETE/UPDATE of parent rows
- JOIN operations on these columns
- Can cause lock contention and slow queries
Indexes created with CONCURRENTLY option to avoid blocking production
tables during deployment.
Migration uses @disable_ddl_transaction and @disable_migration_lock
to ensure indexes are built without holding locks.
Reviewed-on: graham/towerops-web#167
Devices configured as SNMPv1 were being force-upgraded to v2c for agent
polling, which breaks devices that don't support v2c (returning empty
OID responses and no throughput/capacity data).
Reviewed-on: graham/towerops-web#166
Severe index bloat discovered on oban_jobs:
- oban_jobs_pkey: 82.2 bloat ratio, 3.2GB waste
- oban_jobs_state_queue_priority_scheduled_at_id_index: 216.8 bloat, 7.9GB waste
Root cause: High DELETE churn from aggressive Pruner config (10k jobs/sec)
causes PostgreSQL VACUUM to leave dead index entries. VACUUM only marks space
as reusable but cannot shrink indexes.
The Oban Reindexer plugin was only reindexing args_index and meta_index by
default, missing the primary key and main scheduling index.
Solution: Explicitly configure Reindexer to include all high-churn indexes:
- oban_jobs_pkey (primary key, used by all queries)
- oban_jobs_state_queue_priority_scheduled_at_id_index (job scheduling)
- oban_jobs_args_index (default, GIN index)
- oban_jobs_meta_index (default, GIN index)
Daily REINDEX CONCURRENTLY at 2 AM will prevent future bloat accumulation.
Manual immediate reindex required to reclaim 11GB:
REINDEX INDEX CONCURRENTLY oban_jobs_pkey;
REINDEX INDEX CONCURRENTLY oban_jobs_state_queue_priority_scheduled_at_id_index;
Reviewed-on: graham/towerops-web#165
The wrap detection only applied the 2^32 correction when the previous
counter was in the upper half of the 32-bit range (> 2^31). On fast
links, counters wrap regardless of where in the range they sit. Three
graph rendering paths had no wrap detection at all, just max(0).
Consolidated all bps calculations into a single public
Capacity.calculate_bps/3 that always tries the 2^32 correction for
negative deltas. 64-bit HC counter resets remain clamped to zero since
the corrected delta stays negative.
Reviewed-on: graham/towerops-web#163
This PR contains the following updates:
| Package | Update | Change |
|---|---|---|
| [node](https://nodejs.org) ([source](https://github.com/nodejs/node)) | minor | `22.17.1` → `22.22.2` |
---
> ⚠️ **Warning**
>
> Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/9) for more information.
> ❗ **Important**
>
> Release Notes retrieval for this PR were skipped because no github.com credentials were available.
> If you are self-hosted, please see [this instruction](https://github.com/renovatebot/renovate/blob/master/docs/usage/examples/self-hosting.md#githubcom-token-for-release-notes).
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My43Ny44IiwidXBkYXRlZEluVmVyIjoiNDMuNzcuOCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiZGVwZW5kZW5jaWVzIl19-->
Co-authored-by: Renovate Bot <bot@renovateapp.com>
Reviewed-on: graham/towerops-web#147
Removed duplicate function definitions that were causing compile warnings:
- handle_info(:cleanup_expired, state) was defined twice (lines 94 and 110)
- schedule_cleanup/0 was defined twice (lines 105 and 121)
The duplicate definitions at lines 110-124 were exact copies and have been removed.
Reviewed-on: graham/towerops-web#162
Standard ifInOctets/ifOutOctets are 32-bit counters that wrap around
4,294,967,296 bytes (~4.3 GB). At 1 Gbps this wraps every ~34 seconds,
causing brief drops to 0 bps in traffic charts and capacity reports.
When prev > 2^31 and delta is negative, add 2^32 to recover the correct
rate. If the result is still negative (HC 64-bit counter after reboot),
clamp to 0. Small-value decreases (prev < 2^31) continue to clamp to 0
since they cannot be real 32-bit wraps.
Fixes both capacity.ex (utilization reports) and device_live/show.ex
(per-device traffic chart).
Reviewed-on: graham/towerops-web#161
