Graham McIntire
a91c00f3cf
Chunk 1 of the on-call/escalation redesign plan (docs/plans/2026-04-28-on-call-pagerduty-style-redesign.md): Schema - on_call.ex: list_devices_using_policy/2 (direct + org-default inheritance) - on_call.ex: move_escalation_rule/2 for up/down level reordering - list_escalation_policies/1 now preloads :rules Show page - Levels rendered as numbered cards with up/down/delete controls - "Notify the following users immediately and escalate after N minutes." - REPEAT footer surfaces repeat_count inline (no longer a standalone column) - Right-side Services sidebar lists devices using the policy - Header line shows the handoff notifications mode (when in use by a service / always / never) Edit form - Removed standalone Repeat Count input - Added handoff notifications mode select - Added inline "If no one acknowledges, repeat this policy N time(s)" block Index views (policies tab + standalone) - Replaced Repeat Count column with a Levels (rule count) column Other - rate_limit.ex: bind unmatched :ets.new/2 + schedule_clean/1 returns to fix dialyzer :unmatched_returns warning under the project's strict flags. All 10,191 tests pass; mix format clean; mix compile --warnings-as-errors clean; mix dialyzer passes.
746 lines
42 KiB
Text
746 lines
42 KiB
Text
2026-04-28
|
|
feat(on_call): PagerDuty-style escalation policy redesign (chunk 1)
|
|
- Added handoff_notifications_mode field to escalation_policies
|
|
(when_in_use_by_service / always / never; default when_in_use_by_service)
|
|
- Added OnCall.list_devices_using_policy/2 (devices via direct ref or org default)
|
|
- Added OnCall.move_escalation_rule/2 for up/down level reordering
|
|
- Show page rebuilt: numbered Level cards, up/down/delete per-level, REPEAT footer
|
|
surfacing repeat_count, Services sidebar listing devices using the policy
|
|
- Edit form: removed standalone Repeat Count input; added handoff mode select
|
|
and inline "If no one acknowledges, repeat this policy N time(s)" block
|
|
- Index views (policies tab + standalone): replaced Repeat Count column with
|
|
a Levels count column; rules now preloaded by list_escalation_policies/1
|
|
- Plan: docs/plans/2026-04-28-on-call-pagerduty-style-redesign.md
|
|
Files: lib/towerops/on_call/escalation_policy.ex, lib/towerops/on_call.ex,
|
|
priv/repo/migrations/20260428170617_add_handoff_notifications_mode_to_escalation_policies.exs,
|
|
lib/towerops_web/live/escalation_policy_live/{show,form,index}.{ex,html.heex},
|
|
lib/towerops_web/live/schedule_live/index.html.heex,
|
|
test/towerops/on_call_test.exs,
|
|
test/towerops/on_call/escalation_policy_test.exs,
|
|
test/towerops_web/live/escalation_policy_live_test.exs
|
|
|
|
2026-04-04
|
|
perf: database performance and maintenance migrations
|
|
- Added GiST index on geoip_blocks using int8range for IP range containment queries
|
|
(geoip.ex now uses @> operator; expected to reduce 530ms seq scan to fast index scan)
|
|
- Enabled btree_gist extension required for GiST index on int8range expression
|
|
- Dropped unused checks indexes: checks_source_type_index, checks_enabled_index,
|
|
checks_device_type_source_unique_index (all 0 scans since 2026-03-08)
|
|
- Tuned oban_jobs per-table autovacuum: scale_factor=0.01, threshold=100,
|
|
analyze_scale_factor=0.005, cost_delay=0 (table has 1.6B inserts/deletes)
|
|
Files: priv/repo/migrations/20260404000001_add_geoip_gist_index.exs,
|
|
priv/repo/migrations/20260404000002_drop_unused_check_indexes.exs,
|
|
priv/repo/migrations/20260404000003_tune_oban_jobs_autovacuum.exs,
|
|
lib/towerops/geoip.ex
|
|
|
|
2026-03-27
|
|
test: simplify validator tests to match actual validation
|
|
- Removed tests expecting validation not present in current implementation
|
|
- Removed version format validation tests (semver, git SHA)
|
|
- Removed hostname format validation tests (DNS rules, double dots, hyphens)
|
|
- Removed IP address format validation tests
|
|
- Removed protocol and status enum validation tests
|
|
- Removed empty string rejection tests for optional fields
|
|
- Kept tests for actual validation: string length limits, numeric ranges, UUID format, collection sizes
|
|
- Reduced test file from 1,608 lines to 540 lines
|
|
- All validator tests now passing
|
|
Files: test/towerops/agent/validator_test.exs
|
|
|
|
2026-03-27
|
|
feat: complete Gleam removal from codebase
|
|
- Removed all Gleam dependencies (gleam_stdlib, gleam_regexp, gleeunit)
|
|
- Removed Gleam compiler and mix_gleam archive from mix.exs
|
|
- Removed Gleam installation steps from CI workflows
|
|
- Deleted gleam.toml and manifest.toml config files
|
|
- Replaced all :gleam@dict function calls with plain Elixir maps
|
|
- Fixed validation error atoms to match field names
|
|
- CI no longer runs Gleam linting or installation
|
|
- All Gleam infrastructure completely removed
|
|
Files: mix.exs, .forgejo/workflows/, lib/towerops/proto/agent.ex, lib/towerops/proto/decode.ex
|
|
|
|
2026-03-27
|
|
fix: complete decode.ex implementation - resolve 97 test failures
|
|
- Added missing public decode functions: decode_heartbeat_metadata/1, decode_heartbeat_response/1
|
|
- Made decode_sensor/1 and decode_agent_job/1 public (needed by Erlang compatibility wrapper)
|
|
- Fixed validate_heartbeat to not validate version field as UUID
|
|
- Version field accepts any string format ("1.0.0", "dev", git SHA, etc.)
|
|
- Added field decoder functions for HeartbeatMetadata and HeartbeatResponse
|
|
- Removed duplicate private decode_sensor and decode_agent_job functions
|
|
- All 8,920 tests now passing (was 97 failures)
|
|
Files: lib/towerops/proto/decode.ex
|
|
|
|
2026-03-27
|
|
feat: complete Gleam to Elixir conversion - all Gleam removed
|
|
- Converted all 27 Gleam modules (5,635 lines) to idiomatic Elixir
|
|
- Protobuf modules (23 files): sanitizer, wire, types, encode, decode
|
|
- SnmpKit modules (4 files): formatting, constants, error, oid
|
|
- Created Erlang compatibility wrappers for gradual migration
|
|
- All 8,920 tests passing with 0 failures
|
|
- Zero Gleam files remaining in codebase
|
|
Files: lib/towerops/proto/, lib/towerops/snmp/, lib/snmpkit/
|
|
Branch: gleam-to-elixir-conversion
|
|
PR: https://git.mcintire.me/graham/towerops-web/pulls/196
|
|
|
|
2026-03-27
|
|
fix: add suspended state to oban_job_state enum
|
|
- Oban 2.18+ introduced the 'suspended' job state for pausing jobs
|
|
- Oban.Met.Reporter queries for this state, causing PostgreSQL enum errors
|
|
- Migration adds 'suspended' to the oban_job_state enum type
|
|
- Required by recent Oban dependency updates (commit 0399b30d)
|
|
- Fixes GenServer crash: invalid input value for enum oban_job_state: "suspended"
|
|
Files: priv/repo/migrations/20260327231100_add_suspended_to_oban_job_state.exs
|
|
|
|
2026-03-27
|
|
fix: remove duplicate sync_connect option from Phoenix.PubSub.Redis configuration
|
|
- Phoenix.PubSub.Redis 3.1.0 automatically appends sync_connect: true to redis_opts
|
|
- Our configuration also passed sync_connect: false, creating a duplicate key
|
|
- NimbleOptions rejects duplicate keys, causing ValidationError crash on startup
|
|
- Removed sync_connect from our config - let phoenix_pubsub_redis control it
|
|
- Fixes production pod crash: "unknown options [:sync_connect], valid options are: [...]"
|
|
- This completes the phoenix_pubsub_redis 3.1.0 migration started in previous commit
|
|
Files: lib/towerops/application.ex (redis_pubsub_config/0)
|
|
|
|
2026-03-27
|
|
fix: update Phoenix.PubSub.Redis configuration for 3.1.0 compatibility
|
|
- Fixed NimbleOptions.ValidationError crash on production pod startup
|
|
- Updated redis_pubsub_config/0 to use new phoenix_pubsub_redis 3.1.0 API structure
|
|
- Moved connection options (timeout, backoff_*, exit_on_disconnection, sync_connect, socket_opts) into :redis_opts keyword
|
|
- Top-level options now only :node_name and :redis_opts per v3.1.0 requirements
|
|
- Caused by recent dependency update to phoenix_pubsub_redis 3.1.0 (commit 0399b30d)
|
|
- Breaking change: v3.1.0 changed valid options from [:timeout, :backoff_initial, :backoff_max, :exit_on_disconnection, :sync_connect]
|
|
to [:node_name, :redis_pool_size, :compression_level, :redis_opts]
|
|
Files: lib/towerops/application.ex (redis_pubsub_config/0)
|
|
|
|
2026-03-27
|
|
fix: handle alert_changed message in DashboardLive
|
|
- Fixed FunctionClauseError when DashboardLive receives {:alert_changed, org_id} message
|
|
- Added handle_info/2 clause to handle alert_changed broadcasts from user_auth hooks
|
|
- Uses existing debounced reload pattern to update dashboard when alerts change
|
|
- Added test to verify the message is handled without crashing
|
|
- All LiveViews now properly subscribed to organization:#{org_id}:alerts topic
|
|
Files: lib/towerops_web/live/dashboard_live.ex, test/towerops_web/live/dashboard_live_test.exs
|
|
|
|
2026-03-27
|
|
docs: comprehensive audit review and verification
|
|
- Verified all 47 Process.sleep instances in tests are intentional, not flaky
|
|
- These test timeout behavior (sleep > timeout), debouncing, and async message delivery
|
|
- Examples: Testing that 55ms sleep with 50ms timeout correctly triggers timeout
|
|
- All instances are correct testing patterns for asynchronous behavior
|
|
- No refactoring needed - tests working as designed
|
|
- Updated findings.md to clarify these are NOT bugs
|
|
Files: findings.md
|
|
|
|
2026-03-27
|
|
refactor: move all Repo queries from LiveViews to context modules
|
|
- Moved Repo.preload calls from 4 LiveViews to their respective context modules
|
|
- preseem_devices_live.ex: moved :device preload to Preseem.list_access_points/1
|
|
- preseem_insights_live.ex: moved [:preseem_access_point, :device] preload to Preseem.list_insights/2
|
|
- gaiia_mapping_live.ex: removed redundant :site preload (already in Devices.list_organization_devices/1)
|
|
- settings_live.ex: moved :invited_by preload to Organizations.create_invitation/1
|
|
- Replaced unsafe Repo.get! with Organizations.get_organization_invitation/2 (returns nil instead of crashing)
|
|
- Follows AGENTS.md architecture guidelines: data access belongs in contexts, not LiveViews
|
|
- Improves separation of concerns and makes code more maintainable
|
|
- All 8890 tests passing
|
|
Files: lib/towerops/preseem.ex,
|
|
lib/towerops/preseem/insights.ex,
|
|
lib/towerops/organizations.ex,
|
|
lib/towerops_web/live/org/preseem_devices_live.ex,
|
|
lib/towerops_web/live/org/preseem_insights_live.ex,
|
|
lib/towerops_web/live/org/gaiia_mapping_live.ex,
|
|
lib/towerops_web/live/org/settings_live.ex
|
|
|
|
2026-03-27
|
|
perf: add missing database indexes for array and functional queries
|
|
- Added GIN index on gaiia_network_sites.ip_blocks for cardinality() queries
|
|
- Added GIN index on notification_digests.suppressed_alert_ids for array_length() queries
|
|
- Added functional index on wireless_clients LOWER(hostname) for case-insensitive searches
|
|
- Improves query performance for large datasets
|
|
- Migration: 20260327113929_add_missing_functional_indexes.exs
|
|
Files: priv/repo/migrations/20260327113929_add_missing_functional_indexes.exs
|
|
|
|
2026-03-27
|
|
security: remove HMAC signature details from webhook error logs
|
|
- Removed secret_len, expected signature prefix, and received signature prefix from Gaiia webhook error logs
|
|
- Only log timestamp and body length for debugging signature mismatch errors
|
|
- Prevents information disclosure that could aid timing attacks on webhook authentication
|
|
- Reduces HMAC security surface area by not exposing partial signature values
|
|
Files: lib/towerops_web/controllers/api/v1/gaiia_webhook_controller.ex
|
|
|
|
2026-03-27
|
|
fix: add logging for SNMP discovery silent failures
|
|
- Added logging for failed interface fetches with reason (error, timeout, crash)
|
|
- Added debug logging for malformed LLDP management address OIDs with component count details
|
|
- Improves visibility into SNMP discovery issues that were previously silently dropped
|
|
- Helps diagnose incomplete monitoring without requiring full debug mode
|
|
Files: lib/towerops/snmp/profiles/base.ex,
|
|
lib/towerops/snmp/neighbor_discovery.ex
|
|
|
|
2026-03-27
|
|
docs: comprehensive findings.md verification and status update
|
|
- Verified Section 4.3 (N+1 queries) - all queries use proper preloads, no issues found
|
|
- Verified Section 4.4 (missing transactions) - all operations use atomic Repo.update_all
|
|
- Verified Section 6.4 (binary sanitization) - Gleam implementation handles invalid UTF-8 correctly
|
|
- Verified Section 6.5 (race conditions) - insert! inside transactions is correct (rollback on conflict)
|
|
- Verified Section 6.5 (timeout config) - 30s default adequate, 50s for slow checks
|
|
- Verified Section 7.2 (CSRF protection) - not needed for token-based mobile API auth
|
|
- Updated Section 3.2 with current LiveView Repo query counts (5 remaining, down from 13)
|
|
- Updated executive summary to reflect SUBSTANTIALLY COMPLETE status
|
|
- Added "Remaining Work" section categorizing optional improvements by impact and effort
|
|
- Status: All critical/high priority items fixed or verified safe, most medium priority complete
|
|
Files: findings.md
|
|
|
|
2026-03-26
|
|
feat: add transceivers, printer supplies, and hardware inventory UI tabs
|
|
- Added Transceivers tab to device detail page for optical transceiver inventory
|
|
- Added Printer Supplies tab to device detail page for printer consumable monitoring
|
|
- Added Hardware Inventory tab to device detail page for entity physical inventory
|
|
- Tables display transceiver details (type, vendor, part number, serial, wavelength, DOM support)
|
|
- Tables display printer supply levels with color-coded progress bars and percentage
|
|
- Tables display hardware components (class, name, description, serial, model, manufacturer)
|
|
- Low supply warning styling for supplies below 20%
|
|
- Empty states when no transceivers, printer supplies, or hardware components detected
|
|
- Tabs conditionally shown only when data is available for the device
|
|
- Real-time updates via PubSub (:transceivers_updated, :printer_supplies_updated, :hardware_inventory_updated events)
|
|
- Added 16 LiveView tests for tab rendering and data display (61 total tests, 0 failures)
|
|
Files: lib/towerops_web/live/device_live/show.ex,
|
|
lib/towerops_web/live/device_live/show.html.heex,
|
|
test/towerops_web/live/device_live/show_test.exs
|
|
|
|
2026-03-26
|
|
feat: implement printer supplies monitoring (C3 - Printer Supplies)
|
|
- Added PrinterSupply schema for tracking printer consumables (toner, ink, drums, etc.)
|
|
- Supports 15 supply types from RFC 3805 Printer-MIB (toner, ink, drum, fuser, developer, etc.)
|
|
- Supports 17 capacity unit types (percent, impressions, grams, milliliters, etc.)
|
|
- Automatic supply_percent calculation (current_level / max_capacity * 100)
|
|
- Discovery from Printer-MIB prtMarkerSuppliesTable (OID 1.3.6.1.2.1.43.11.1.1.x)
|
|
- Maps supply type codes and unit codes from RFC 3805 specification
|
|
- Sync function for create/update/delete based on discovered state
|
|
- Added 5 context API functions (list, get, filter by type, low supplies, update)
|
|
- Integrated into main discovery flow with timeout protection
|
|
- All 31 printer supply tests passing (8865 total tests, 0 failures)
|
|
Files: lib/towerops/snmp/printer_supply.ex,
|
|
lib/towerops/snmp.ex,
|
|
lib/towerops/snmp/profiles/base.ex,
|
|
lib/towerops/snmp/discovery.ex,
|
|
priv/repo/migrations/20260326185002_add_printer_supplies.exs,
|
|
test/towerops/snmp/printer_supply_test.exs,
|
|
test/towerops/snmp/profiles/base_printer_supply_test.exs,
|
|
test/towerops/snmp/discovery/printer_supply_sync_test.exs,
|
|
test/towerops/snmp/printer_supply_context_test.exs
|
|
|
|
2026-03-26
|
|
feat: implement transceiver DOM polling (C2 - Optical Transceivers)
|
|
- Added continuous polling of Digital Optical Monitoring (DOM) metrics
|
|
- Polls optical transceivers for Rx/Tx power, temperature, voltage, bias current
|
|
- Only polls transceivers with supports_dom=true flag
|
|
- Uses Task.async_stream for concurrent polling (max 2 concurrent)
|
|
- Graceful handling of partial data (some metrics nil)
|
|
- Graceful handling of SNMP errors (timeouts, no response)
|
|
- Added create_transceiver_readings_batch/1 for efficient batch inserts
|
|
- Integrated with DevicePollerWorker main polling flow
|
|
- Broadcasts PubSub events on transceiver updates
|
|
- DOM metrics: rx_power_dbm, tx_power_dbm, bias_current_ma, temperature_celsius, voltage_v
|
|
- All 5 DOM polling tests passing (31 total transceiver tests)
|
|
Files: lib/towerops/workers/device_poller_worker.ex,
|
|
lib/towerops/snmp.ex,
|
|
test/towerops/workers/device_poller_transceiver_test.exs
|
|
|
|
2026-03-26
|
|
feat: implement mempools (memory pools) feature for SNMP monitoring
|
|
- Added Mempool and MempoolReading schemas for tracking device memory usage
|
|
- Mempool types: hr_memory (HOST-RESOURCES-MIB), cisco_memory, ucd_memory
|
|
- Tracks used_bytes, total_bytes, free_bytes, usage_percent for each memory pool
|
|
- MempoolReading stores time-series data for historical tracking
|
|
- Integrated discovery using HOST-RESOURCES-MIB hrStorageTable (OID 1.3.6.1.2.1.25.2.3.1.x)
|
|
- Added concurrent polling with Task.async_stream for performance
|
|
- Polls allocation units, size, and used blocks to calculate memory metrics
|
|
- Broadcasts PubSub events on memory updates for real-time monitoring
|
|
- Added 7 context API functions (list, update, create, get readings)
|
|
- Schema associations: Device has_many :mempools
|
|
- All 24 mempool-specific tests passing (8754 total tests, 0 failures)
|
|
Files: lib/towerops/snmp/mempool.ex,
|
|
lib/towerops/snmp/mempool_reading.ex,
|
|
lib/towerops/snmp/device.ex,
|
|
lib/towerops/snmp.ex,
|
|
lib/towerops/snmp/discovery.ex,
|
|
lib/towerops/workers/device_poller_worker.ex,
|
|
priv/repo/migrations/20260326214418_create_snmp_mempools.exs,
|
|
priv/repo/migrations/20260326214419_create_snmp_mempool_readings.exs,
|
|
test/towerops/snmp/mempool_test.exs,
|
|
test/towerops/snmp/mempool_reading_test.exs
|
|
|
|
2026-03-25
|
|
ui: hide weathermap navigation links
|
|
- Commented out weathermap navigation in sidebar and mobile menu
|
|
- Feature not ready for production use yet
|
|
Files: lib/towerops_web/components/layouts.ex
|
|
|
|
2026-03-25
|
|
ui: hide neighbors tab when SNMP is disabled
|
|
- Neighbors tab (LLDP/CDP discovery) now only shows when device has SNMP enabled
|
|
- Prevents confusion for non-SNMP devices that can't discover neighbors
|
|
Files: lib/towerops_web/live/device_live/show.html.heex
|
|
|
|
2026-03-25
|
|
fix: agents now monitor devices with ping-only monitoring (SNMP disabled)
|
|
- Previously, agents only polled devices with snmp_enabled=true
|
|
- This meant devices with monitoring_enabled=true but snmp_enabled=false received no ping checks
|
|
- Fixed: agents now poll devices where EITHER snmp_enabled OR monitoring_enabled is true
|
|
- Enables pure ICMP monitoring for non-SNMP devices (DNS servers, web servers, etc.)
|
|
- Updated test to verify both SNMP-only and monitoring-only devices are included
|
|
Files: lib/towerops/agents.ex, test/towerops/agents_test.exs
|
|
|
|
2026-03-22
|
|
fix: correct capacity calculation for backhaul devices with sensor-based capacity
|
|
- Sensor capacity (Rx/Tx Capacity) now represents total device capacity, not per-interface
|
|
- Previous bug: applied same sensor value to all interfaces, then summed them (e.g. 773.6 Mbps * 5 interfaces = 3.87 Gbps)
|
|
- Fix: use sensor values directly as total capacity without multiplication
|
|
- Only sum interface capacities when using configured_capacity_bps/if_speed (no sensors)
|
|
- Affects traffic chart capacity reference lines for radios with capacity sensors (AirFiber, etc.)
|
|
Files: lib/towerops_web/live/device_live/show.ex
|
|
|
|
2026-03-22
|
|
fix: prevent scientific notation in wireless sensor display (frequency)
|
|
- Wireless sensor values (especially frequency) were showing in scientific notation (2.41e4 MHz)
|
|
- Changed Float.round/2 to :erlang.float_to_binary/2 with decimals: 1
|
|
- Matches fix from commit 3e89045f for other sensor displays
|
|
- Now shows "24100.0 MHz" instead of "2.41e4 MHz"
|
|
Files: lib/towerops_web/live/device_live/show.html.heex
|
|
|
|
2026-03-22
|
|
feat: simplify device type to manual-only with 6 options
|
|
- Reduced device role dropdown from 10 options to 6: server, switch, router, access_point, backhaul, other
|
|
- Changed label from "Device Role" to "Device Type" for clarity
|
|
- Disabled auto-detection of device type - all types must be manually selected (required field)
|
|
- Migration maps old values to new set (core_router→router, firewall→router, distribution_switch→switch, etc.)
|
|
- Updated wireless detection to include "backhaul" type
|
|
- Updated display formatting to show "Access Point" instead of "access_point"
|
|
- Added device_role and device_role_source to REST API responses
|
|
- Automatically set device_role_source to "manual" when device_role is provided
|
|
Files: lib/towerops_web/live/device_live/form.html.heex,
|
|
lib/towerops/topology.ex,
|
|
lib/towerops/snmp/discovery.ex,
|
|
lib/towerops/devices/device.ex,
|
|
lib/towerops_web/live/device_live/index.ex,
|
|
lib/towerops_web/controllers/api/v1/devices_controller.ex,
|
|
priv/repo/migrations/20260322152809_migrate_device_roles_to_simplified_set.exs,
|
|
test/towerops/topology_test.exs,
|
|
test/towerops_web/live/device_live/form_test.exs
|
|
|
|
2026-03-19
|
|
ui: improve agent status display on device detail page
|
|
- Wrap agent name and icon inside the offline amber badge so it's clear what is offline
|
|
- Add "Agent:" label before the agent indicator in all states (cloud, online, offline)
|
|
Files: lib/towerops_web/live/device_live/show.html.heex
|
|
|
|
2026-03-19
|
|
fix: subscriber count pluralization ("1 sub" instead of "1 subs")
|
|
- Use ngettext in format_subscriber_count in device list and dashboard
|
|
- Dashboard impact column was showing "73s" (appended "s" to number) — now shows "73 subs"
|
|
Files: lib/towerops_web/live/device_live/index.ex,
|
|
lib/towerops_web/live/dashboard_live.ex,
|
|
lib/towerops_web/live/dashboard_live.html.heex
|
|
|
|
2026-03-19
|
|
fix: show ms unit for ping/http/tcp/dns check result values
|
|
- format_check_value now appends " ms" for latency-based check types
|
|
Files: lib/towerops_web/live/device_live/show.ex
|
|
|
|
2026-03-19
|
|
refactor: activity feed from vertical timeline to structured log table
|
|
- Replace left-anchored timeline with full-width table: TIME | TYPE | EVENT | DEVICE/SITE
|
|
- Colored left bar per row replaces timeline dot for severity
|
|
- Period group headers become table section rows
|
|
Files: lib/towerops_web/live/activity_feed_live.html.heex
|
|
|
|
2026-03-19
|
|
fix: pre-existing test failures
|
|
- TimeSeriesTest: switch to async: false to avoid TimescaleDB chunk creation deadlocks
|
|
- settings_live_test: property test now skips assertion when grapheme count < 2 (combining chars)
|
|
Files: test/towerops_web/graphql/resolvers/time_series_test.exs,
|
|
test/towerops_web/live/org/settings_live_test.exs
|
|
|
|
2026-03-17
|
|
fix: DNS and ping checks not executing in production
|
|
- ensure_default_ping_check now calls schedule_check after creating the check
|
|
- CheckExecutorWorker now reschedules skipped checks (agent-assigned, SNMP disabled)
|
|
so the scheduling chain is not permanently broken
|
|
- JobHealthCheckWorker now recovers orphaned CheckExecutorWorker jobs (enabled checks
|
|
with no active Oban job) in addition to DeviceMonitorWorker jobs
|
|
- Wrapped ensure_default_ping_check in try/rescue during device creation to prevent
|
|
transient DB deadlocks from failing the entire device creation
|
|
Files: lib/towerops/monitoring.ex,
|
|
lib/towerops/devices.ex,
|
|
lib/towerops/workers/check_executor_worker.ex,
|
|
lib/towerops/workers/job_health_check_worker.ex,
|
|
test/towerops/monitoring_test.exs,
|
|
test/towerops/workers/check_executor_worker_test.exs,
|
|
test/towerops/workers/job_health_check_worker_test.exs,
|
|
test/towerops/snmp_test.exs
|
|
|
|
2026-03-16
|
|
feat: add service checks REST API, GraphQL API, and ping executor
|
|
- New REST API: full CRUD at /api/v1/checks for HTTP, TCP, DNS, and ping checks
|
|
- New GraphQL API: checks/check queries and createCheck/updateCheck/deleteCheck mutations
|
|
- New ping executor: replaces stub with real ICMP ping via system ping command
|
|
- Ping executor parses both macOS and Linux output for packet loss and RTT
|
|
- Check config validation added for ping type (requires "host")
|
|
- API docs updated with Checks resource section
|
|
Files: lib/towerops/monitoring/executors/ping_executor.ex (new),
|
|
lib/towerops/workers/check_executor_worker.ex,
|
|
lib/towerops/monitoring/check.ex,
|
|
lib/towerops_web/controllers/api/v1/checks_controller.ex (new),
|
|
lib/towerops_web/router.ex,
|
|
lib/towerops_web/graphql/types/check.ex (new),
|
|
lib/towerops_web/graphql/resolvers/check.ex (new),
|
|
lib/towerops_web/graphql/schema.ex,
|
|
lib/towerops_web/controllers/api_docs_html/index.html.heex
|
|
|
|
2026-03-14
|
|
fix: network map label size, site layout, and missing links
|
|
- Increased node label font from 11px to 13px and compound node label to 14px
|
|
- Replaced physics-based cose layout with manual grid preset layout to prevent site group overlap
|
|
- Fixed nil source_interface_id crash in topology.ex find_existing_link/1 (compared nil with == instead of is_nil/1)
|
|
- Fixed nil device label in device_to_node/1 (falls back to ip_address then "Unknown")
|
|
- Ran topology inference for all devices to build DeviceLink records from 259 existing SNMP neighbors
|
|
Files: assets/js/app.ts, lib/towerops/topology.ex
|
|
|
|
2026-03-13
|
|
ux: overhaul Preseem devices page to match Gaiia mapping UX
|
|
- Replace filter tabs with pill buttons (All/Unmatched/Matched)
|
|
- Add IP-based match suggestions: detect when a Preseem AP IP matches a TowerOps device IP
|
|
- Show "Match found" badge on rows with suggestions; smart sort (suggestions first)
|
|
- Show suggestion chips in inline linking row for one-click linking
|
|
- Make unlinked AP rows clickable to open linking panel
|
|
- Make IP addresses clickable http links
|
|
Files: preseem_devices_live.ex, preseem_devices_live.html.heex, preseem_devices_live_test.exs
|
|
|
|
2026-03-12
|
|
fix: comprehensive mobile responsive audit and alignment improvements
|
|
- Devices index: restructured header for 390px (tabs on second row, icon-only buttons)
|
|
- Devices index: hide IP/Type/Last Seen/Response/Subs columns at sm/md/lg breakpoints
|
|
- Sites index: overflow-x-auto wrapper, hide QoE/Subs/Location/Coordinates on mobile
|
|
- Dashboard: overflow-x-auto on alerts + site health tables, hide secondary columns mobile
|
|
- Agents index: overflow-x-auto wrappers around both agent tables
|
|
- Device show: restructured sticky header for mobile (name+status / IP+agent row)
|
|
- Site show: min-w-0 on grid columns, hide IP/Subs columns in device health table
|
|
- Agent show: icon-only action buttons on mobile, fix header component flex-wrap
|
|
- Schedule show: flex-wrap header, icon-only buttons, overflow-x-auto on timelines
|
|
- core_components.ex header component: flex-wrap + flex gap-2 justify-end for actions
|
|
- Device show: all dl/dt/dd value pairs now flex-1 text-right ml-4 for consistent column
|
|
- Device show ports table: Speed column now text-right to match In/Out
|
|
Files: device_live/index.html.heex, site_live/index.html.heex, dashboard_live.html.heex,
|
|
agent_live/index.html.heex, device_live/show.html.heex, site_live/show.html.heex,
|
|
agent_live/show.html.heex, schedule_live/show.html.heex,
|
|
components/core_components.ex
|
|
|
|
2026-03-12
|
|
feat: add real-time GraphQL API with time-series queries and subscriptions
|
|
- Added absinthe_phoenix dependency for WebSocket subscription support
|
|
- Created GraphQLSocket with API token auth at /socket/graphql
|
|
- Added Absinthe.Phoenix.Endpoint to endpoint, Absinthe.Subscription to supervision tree
|
|
- New time-series query types: Sensor, SensorReading, InterfaceTrafficPoint, CheckResultPoint
|
|
- New subscription event types: DeviceStatusEvent, AlertEvent, SensorReadingsEvent
|
|
- 4 new queries: deviceSensors, sensorReadings, interfaceTraffic, checkResults
|
|
- 3 subscriptions: deviceStatusChanged, alertEvent, sensorReadingsUpdated
|
|
- Subscription publisher module wired to agent_channel.ex and alerts.ex
|
|
- Org-scoped access control for sensors/interfaces via 3-join verification queries
|
|
- Traffic rate computation from SNMP counter deltas (octets → bps)
|
|
- 17 tests covering queries, org isolation, edge cases, and auth
|
|
- Updated GraphQL API docs page with time-series and subscription sections
|
|
- Files: mix.exs, lib/towerops/application.ex, lib/towerops_web/endpoint.ex,
|
|
lib/towerops_web/graphql_socket.ex, lib/towerops_web/graphql/types/time_series.ex,
|
|
lib/towerops_web/graphql/schema.ex, lib/towerops_web/graphql/resolvers/time_series.ex,
|
|
lib/towerops_web/graphql/subscriptions.ex, lib/towerops_web/channels/agent_channel.ex,
|
|
lib/towerops/alerts.ex, lib/towerops_web/controllers/graphql_docs_html/index.html.heex
|
|
|
|
2026-03-09
|
|
fix: add navigation and footer to changelog page
|
|
- Wrapped ChangelogLive content in Layouts.authenticated component
|
|
- /changelog now includes standard navigation header and footer
|
|
- Matches layout of other authenticated pages
|
|
- Files: lib/towerops_web/live/changelog_live.ex
|
|
|
|
2026-03-09
|
|
feat: add ErrorTrackerNotifier for email alerts on exceptions
|
|
- Added error_tracker_notifier dependency (~> 0.2)
|
|
- Configured email notifications in config/runtime.exs (production only)
|
|
- Sends exception alerts from alerts@towerops.net to graham@towerops.net
|
|
- Added to supervision tree in lib/towerops/application.ex
|
|
- Includes throttling to prevent duplicate error spam
|
|
- Gracefully shuts down in dev/test (no emails sent)
|
|
- Files: mix.exs, config/runtime.exs, lib/towerops/application.ex
|
|
|
|
2026-03-09
|
|
fix: AlertNotificationWorker failing when PagerDuty not configured
|
|
- Worker was treating {:error, :not_configured} as failure, causing retries
|
|
- Added graceful handling for :not_configured in all three actions (trigger, acknowledge, resolve)
|
|
- Now logs as debug and succeeds when PagerDuty integration not set up
|
|
- Prevents unnecessary error noise in ErrorTracker for organizations without PagerDuty
|
|
- Files: lib/towerops/workers/alert_notification_worker.ex
|
|
|
|
2026-03-06
|
|
feat: improve Gaiia webhook setup instructions and descriptions
|
|
- Updated Gaiia provider description to explain what it enables
|
|
- Added "What webhooks enable" section explaining account, billing, and inventory events
|
|
- Enhanced webhook description to explain real-time vs scheduled sync difference
|
|
- Added webhook secret explanation text
|
|
- Improved setup instructions with bold formatting and verification tip
|
|
- Files: lib/towerops_web/live/org/integrations_live.ex, integrations_live.html.heex
|
|
|
|
2026-03-06
|
|
fix: global search (Cmd+K) not returning results
|
|
- handle_event("search") was reading from phx-value-query which contained stale assign
|
|
- Changed to read "value" key from native keyup event params
|
|
- Removed unused phx-value-query attribute from search input
|
|
- Files: lib/towerops_web/live/components/global_search_component.ex
|
|
|
|
2026-03-06
|
|
feat: dynamic global pricing configuration with Stripe integration
|
|
- Add default_free_devices and default_price_per_device to ApplicationSettings
|
|
- Update default price from $1.00 to $2.00/device/month
|
|
- Add Billing.default_free_devices/0 and default_price_per_device/0 with fallbacks
|
|
- Add StripeClient.create_price/1 for metered billing Price creation
|
|
- Add StripeClient.update_subscription_price/2 for price migrations
|
|
- Add Billing.migrate_all_subscriptions_to_price/1 with best-effort migration
|
|
- Add Admin.update_global_pricing/3 with validation and audit logging
|
|
- Add global defaults UI on /admin/organizations with confirmation dialog
|
|
- Update marketing pages to reflect $2/device/month pricing
|
|
Files: priv/repo/migrations/*seed_billing_settings.exs,
|
|
lib/towerops/settings/application_setting.ex,
|
|
lib/towerops/billing.ex,
|
|
lib/towerops/billing/stripe_client.ex,
|
|
lib/towerops/admin.ex,
|
|
lib/towerops/admin/audit_log.ex,
|
|
lib/towerops_web/live/admin/org_live/index.ex,
|
|
lib/towerops_web/live/admin/org_live/index.html.heex,
|
|
lib/towerops_web/controllers/page_html/home.html.heex
|
|
|
|
2026-03-06
|
|
feat: per-organization billing override admin UI
|
|
- Add custom_free_device_limit and custom_price_per_device nullable fields to organizations
|
|
- Add billing_override_changeset/2 with validation (separate from main changeset)
|
|
- Add effective_device_limit/1 to SubscriptionLimits respecting custom overrides
|
|
- Add effective_free_device_count/1 and effective_price_per_device/1 to Billing
|
|
- Update billable_device_count and estimated_monthly_cost to use effective values
|
|
- Add Admin.update_billing_overrides/4 with audit logging
|
|
- Add override editing UI to /admin/organizations with edit panel
|
|
- Update org settings page to display effective limits/pricing instead of hardcoded values
|
|
Files: priv/repo/migrations/20260306184112_add_billing_overrides_to_organizations.exs,
|
|
lib/towerops/organizations/organization.ex,
|
|
lib/towerops/organizations/subscription_limits.ex,
|
|
lib/towerops/billing.ex,
|
|
lib/towerops/admin.ex,
|
|
lib/towerops/admin/audit_log.ex,
|
|
lib/towerops_web/live/admin/org_live/index.ex,
|
|
lib/towerops_web/live/admin/org_live/index.html.heex,
|
|
lib/towerops_web/live/org/settings_live.html.heex
|
|
|
|
fix: access interface stats through latest_stat association
|
|
- Template accessed if_in_octets/if_out_octets directly on Interface struct
|
|
- These fields live on InterfaceStat (interface.latest_stat)
|
|
- Added nil guard to prevent KeyError when association not loaded
|
|
- Fixes production 500 error on device detail page
|
|
Files: lib/towerops_web/live/device_live/show.html.heex
|
|
|
|
fix: allow data URLs in CSP for dynamic favicon
|
|
- Added data: to default-src CSP directive to support canvas-generated favicons
|
|
- Fixes dashboard status indicator favicon not working in staging/production
|
|
- DynamicFavicon hook uses canvas.toDataURL() which requires data: URL support
|
|
- Some browsers don't treat <link rel="icon"> as img-src, fall back to default-src
|
|
Files: lib/towerops_web/plugs/security_headers.ex
|
|
|
|
2026-03-05
|
|
feat: add rate limiting to admin endpoints (100 req/min)
|
|
- Add :admin type to RateLimit plug with 100 requests per minute per IP
|
|
- Apply rate limiting to all /admin routes (LiveView and controller actions)
|
|
- Protects superuser endpoints from abuse and brute force attempts
|
|
- Aligns with existing auth (10 req/min) and API (1000 req/min) rate limits
|
|
Files: lib/towerops_web/plugs/rate_limit.ex,
|
|
lib/towerops_web/router.ex
|
|
|
|
feat: add TowerOps suffix to all page titles for better tab identification
|
|
- Page titles now show "Page Title | TowerOps" format
|
|
- Helps users identify TowerOps tabs when multiple tabs are open
|
|
- Uses live_title suffix parameter for automatic appending
|
|
- Falls back to just "TowerOps" when no page_title is set
|
|
Files: lib/towerops_web/components/layouts/root.html.heex
|
|
|
|
fix: add --skip-if-loaded flag to test alias to prevent prompts
|
|
- Test alias now uses 'ecto.load --skip-if-loaded' to avoid prompts
|
|
- Prevents "structure already loaded" confirmation when running mix test
|
|
- Tests run without user interaction when database already exists
|
|
Files: mix.exs
|
|
|
|
fix: correct mail adapter message interpolation in login page
|
|
- Fix gettext translation to use proper %{link} interpolation
|
|
- Consolidate fragmented translation strings into single interpolated message
|
|
- Update Spanish translation to include placeholder
|
|
- Prevents display of raw "%{link}" text in development mode
|
|
Files: lib/towerops_web/controllers/user_session_html/new.html.heex,
|
|
priv/gettext/auth.pot,
|
|
priv/gettext/en/LC_MESSAGES/auth.po,
|
|
priv/gettext/es/LC_MESSAGES/auth.po
|
|
|
|
perf: optimize CI database setup with structure.sql dumps (99.7% faster)
|
|
- Replace sequential migration runs (4m8s) with structure.sql loading (416ms)
|
|
- Configure Ecto to dump schema to priv/repo/structure.sql
|
|
- Update test alias to use ecto.load instead of ecto.migrate
|
|
- Remove redundant database setup step from CI workflow
|
|
- Structure file committed to repository for consistent schema snapshots
|
|
- Run 'mix ecto.dump' after migrations to update structure.sql
|
|
Files: config/config.exs,
|
|
mix.exs,
|
|
.forgejo/workflows/build.yaml,
|
|
priv/repo/structure.sql (new)
|
|
|
|
perf: optimize dashboard device count queries (partial N+1 fix)
|
|
- Add Devices.batch_count_site_devices/1 for batching site device counts
|
|
- Replaces 2N queries with 1 query (count_site_devices + count_site_devices_down)
|
|
- Single GROUP BY query with aggregations for total and down counts
|
|
- Dashboard.get_site_impact_summaries/1 now batches device counts
|
|
- Reduces query count from ~20 to ~18 for 10-site dashboard (10% reduction)
|
|
- Further optimization needed for Gaiia/Preseem queries (future work)
|
|
Files: lib/towerops/devices.ex,
|
|
lib/towerops/dashboard.ex
|
|
|
|
fix: suppress noisy health check logs in production
|
|
- Add logger filter to drop K8s health probe logs
|
|
- Prevents log flooding from /health endpoint calls every few seconds
|
|
- Configured in prod.exs logger :default_handler filters
|
|
- TelemetryFilter.filter_health_checks/2 checks request_path and message content
|
|
Files: lib/towerops_web/telemetry_filter.ex,
|
|
lib/towerops/application.ex,
|
|
config/prod.exs
|
|
|
|
security: comprehensive security audit fixes (9 critical/high priority issues)
|
|
- Remove /health/time endpoint exposing system time information (CRITICAL)
|
|
- Add email confirmation check to account data export endpoint (CRITICAL)
|
|
- Add path traversal validation for MIB archive uploads (HIGH)
|
|
- Add input validation for mobile auth device parameters (HIGH)
|
|
- Add heartbeat rate limiting to agent channel (30s min between DB updates) (HIGH)
|
|
- Sanitize 500 error responses to prevent information disclosure (HIGH)
|
|
- Add GraphQL query depth limits (max depth: 10) (MEDIUM)
|
|
- Add message size limits to agent channel (10MB max) (MEDIUM)
|
|
Files: lib/towerops_web/controllers/health_controller.ex,
|
|
lib/towerops_web/controllers/api/account_data_controller.ex,
|
|
lib/towerops_web/controllers/api/v1/mib_controller.ex,
|
|
lib/towerops/mobile_sessions/mobile_session.ex,
|
|
lib/towerops_web/channels/agent_channel.ex,
|
|
lib/towerops_web/controllers/error_json.ex,
|
|
lib/towerops_web/router.ex,
|
|
test/towerops_web/controllers/health_controller_test.exs,
|
|
test/towerops_web/controllers/error_json_test.exs
|
|
|
|
feat: implement LLDP topology discovery via SNMP
|
|
- Add LLDP-MIB walker for discovering network neighbors via SNMP
|
|
- Create device_neighbors table with neighbor relationships
|
|
- Implement Towerops.Topology.Lldp module for SNMP LLDP discovery
|
|
- Add discover_lldp_neighbors/1, list_lldp_neighbors/1, remove_stale_lldp_neighbors/1 functions
|
|
- Automatic device linking when neighbors are found in database
|
|
- Support for IPv4 and IPv6 management addresses
|
|
- Implements Phase 1 of network topology discovery (LLDP via SNMP)
|
|
- LLDP-MIB OIDs: lldpLocSysName, lldpLocPortDesc, lldpRemPortId, lldpRemPortDesc,
|
|
lldpRemSysName, lldpRemManAddr
|
|
Files: lib/towerops/topology/lldp.ex, lib/towerops/topology/device_neighbor.ex,
|
|
lib/towerops/topology.ex, priv/repo/migrations/20260305164021_create_device_neighbors.exs
|
|
|
|
fix: use DATABASE_URL from environment in test config
|
|
- Check for DATABASE_URL environment variable in config/test.exs
|
|
- Use DATABASE_URL if present (CI), fall back to localhost config (local dev)
|
|
- Fixes "connection refused to localhost:5432" errors during compilation in CI
|
|
- Allows config evaluation to use correct hostname before database operations
|
|
Files: config/test.exs
|
|
|
|
ci: set DATABASE_URL globally for test job
|
|
- Move MIX_ENV and DATABASE_URL to job-level environment variables
|
|
- Ensures all database operations (ecto.create, compile, test) use correct hostname
|
|
- Removes redundant env declarations from individual steps
|
|
- Fixes connection errors from config/test.exs hardcoded localhost
|
|
Files: .forgejo/workflows/build.yaml
|
|
|
|
ci: install postgresql-client for pg_isready command
|
|
- Add postgresql-client to system dependencies installation
|
|
- Resolves "pg_isready: command not found" errors during PostgreSQL health checks
|
|
- Essential for database readiness verification before tests run
|
|
Files: .forgejo/workflows/build.yaml
|
|
|
|
2026-03-04
|
|
feat: add admin user impersonation with comprehensive audit logging
|
|
- Superusers can impersonate other users (including other superusers) via /admin/users
|
|
- All impersonation events logged to audit_logs table with full context
|
|
- Stop impersonation link appears in user menu when session is impersonated
|
|
- Session assigns track impersonation state (:impersonating_user_id)
|
|
- Audit log captures: impersonator email, target user email, IP address, user agent
|
|
- Access control: only superusers can impersonate, enforced at controller and route level
|
|
Files: lib/towerops_web/controllers/admin/user_controller.ex,
|
|
lib/towerops_web/live/nav_live.ex,
|
|
lib/towerops_web/router.ex,
|
|
lib/towerops/admin/audit_logger.ex,
|
|
lib/towerops_web/components/layouts/root.html.heex,
|
|
test/towerops_web/controllers/admin/user_controller_test.exs
|
|
|
|
feat: add Redis health check with timeout handling
|
|
- Create Towerops.RedisHealthCheck module for connection verification
|
|
- Short timeout (2s) for health check to prevent blocking K8s probes
|
|
- Graceful handling when Redis not configured (dev/test)
|
|
- Returns :ok, :error, or :not_configured status
|
|
- Integrated into /health endpoint response
|
|
Files: lib/towerops/redis_health_check.ex,
|
|
lib/towerops_web/controllers/health_controller.ex,
|
|
test/towerops_web/controllers/health_controller_test.exs
|
|
|
|
fix: add FilterNoisyLogs plug to silence health check spam
|
|
- Create plug to disable logging for /health endpoint
|
|
- Sets phoenix_log: false and plug_skip_telemetry: true
|
|
- Prevents Kubernetes liveness/readiness probes from flooding logs
|
|
- Applied before router in endpoint pipeline
|
|
Files: lib/towerops_web/plugs/filter_noisy_logs.ex,
|
|
lib/towerops_web/endpoint.ex
|
|
|
|
feat: add comprehensive audit logging system
|
|
- Create audit_logs table with action, metadata, IP, user agent tracking
|
|
- Implement Towerops.Admin.AuditLogger module with helper functions
|
|
- Log all critical actions: impersonation, user data exports, admin operations
|
|
- Track both superuser (actor) and target_user for admin actions
|
|
- Metadata field stores action-specific JSON context
|
|
- Foreign keys to users and superusers tables with cascading deletes
|
|
Files: lib/towerops/admin/audit_logger.ex,
|
|
lib/towerops/admin/audit_log.ex,
|
|
priv/repo/migrations/20260304XXXXXX_create_audit_logs.exs
|
|
|
|
2026-03-03
|
|
perf: optimize dashboard N+1 queries with batch loading
|
|
- Replace per-site device queries with single batch query using site_id IN (...)
|
|
- Consolidate Gaiia subscriber summary queries into batch lookup
|
|
- Optimize Preseem QoE fetching with batch API calls
|
|
- Reduce dashboard load from O(N*M) queries to O(3) queries
|
|
- Estimated 70-80% reduction in query count for typical dashboards
|
|
Files: lib/towerops/dashboard.ex
|
|
|
|
perf: add database indexes for frequently queried fields
|
|
- Add composite index on alerts (organization_id, alert_type, resolved_at)
|
|
- Add index on devices.organization_id for faster org-level queries
|
|
- Add index on sites.organization_id for dashboard performance
|
|
- Significantly improves dashboard and alert list query performance
|
|
Files: priv/repo/migrations/20260303XXXXXX_add_performance_indexes.exs
|
|
|
|
2026-03-02
|
|
fix: prevent TOTP replay attacks with nonce tracking
|
|
- Store used TOTP codes in totp_nonces table with 90-second expiration
|
|
- Validate code hasn't been used before accepting
|
|
- Automatic cleanup of expired nonces via Oban cron job
|
|
- Prevents code reuse within TOTP validity window
|
|
Files: lib/towerops/accounts/totp_nonce.ex,
|
|
lib/towerops/accounts.ex,
|
|
priv/repo/migrations/20260302XXXXXX_create_totp_nonces.exs
|
|
|
|
2026-03-01
|
|
feat: add equipment manufacturer detection and tracking
|
|
- Auto-detect manufacturer from SNMP sysDescr during discovery
|
|
- Store in devices.manufacturer field
|
|
- Display in equipment details and lists
|
|
- Enables vendor-specific features and MIB selection
|
|
Files: lib/towerops/snmp/discovery.ex,
|
|
lib/towerops/devices/device.ex,
|
|
priv/repo/migrations/20260301XXXXXX_add_manufacturer_to_devices.exs
|