Performance:
- schedule_live: preload page once instead of get_schedule!/1 per row
- alert_live + alerts: DB-side status filter; Repo.aggregate counts replace length/Enum.count over 500-row fetches on every event
- dashboard_live: drop duplicate get_device_status_counts; cap active alerts to 20 + use count_active_alerts/1
- maintenance.active_windows_for_device: 3 round-trips collapsed into one query (per-site-id branch keeps Postgres parameter types unambiguous)
- sites.build_site_tree: O(N^2) -> O(N) via group_by(parent_site_id)
- accounts.sole_owner_organizations: single group_by + having instead of per-org Repo.aggregate loop
- agents + agent_live (org + admin): count_assigned_devices_batch/1 + count_agent_polling_targets/1 (no preloads)
- alert_digest_worker: list_alerts_by_ids/1 batches digest fetch
- gaiia: distinct: true at DB; limit 50 on bidirectional ilike
Indexes:
- maintenance_windows(organization_id, starts_at, ends_at) WHERE suppress_alerts = true
- alerts(check_id) WHERE resolved_at IS NULL
Antipatterns:
- agents.delete_agent_token: PubSub.broadcast moved outside Repo.transaction so a rollback no longer leaves subscribers acting on a non-existent deletion
- integrations_controller.to_atom_keys: replaced String.to_existing_atom on user-controlled JSON keys with explicit allowlist
- 9 Task.start callsites converted to Task.Supervisor.start_child(Towerops.TaskSupervisor, ...) so background DB writes survive shutdown (test-mode discovery shims left as-is for sandbox semantics)
Three deferred followups from the chunk-4 plan, all together:
A. Restrictions editor
- Resolver: time_of_week support (weekday set + time-of-day window),
in addition to the existing time_of_day.
- New OnCall.update_layer_restrictions/2 + clear_layer_restrictions/1.
- schedule_live/show: per-layer Restrict on-call shifts form
(Always on call / Time of day / Time of week with Mon-Sun checkboxes),
changes patched live and persisted via the resolver.
B. Drag-and-drop reorder
- New OnCall.reorder_layers/2 + reorder_escalation_rules/2 (atomic
transaction; validates id count + membership).
- assets/js/sortable_list.ts: minimal HTML5 drag-and-drop hook (no
external dep; mirrors the existing DeviceListReorder pattern),
registered as SortableList in app.ts.
- schedule_live/show + escalation_policy_live/show: layer/level cards
now carry data-id + draggable + a visible drag handle. Up/down arrow
buttons remain as a fallback for users on assistive tech.
C. Unified new-schedule UX
- schedule_live/show pre-opens the Add Layer form when the schedule
has zero layers, so the new-schedule -> add-layers flow lands ready
to act on. Two existing tests updated to seed a layer first; two
new tests verify the auto-open behaviour both ways.
Full suite 10,231 / 0 failures. Format/dialyzer clean.
ScheduleLive.Index
- Name search input above the date nav (phx-debounce 300ms).
- Standard <.pagination> footer (10 per page) under the gantt cards.
- Both reflected in URL params (?search= and ?page=) for shareable
links; defaults are stripped from the URL.
- Refactored: load_schedules_with_timeline/3 -> hydrate_schedules/3
so filter/paginate/hydrate are clearly separate steps.
Tests
- 2 new ExUnit integration tests (search filter narrows results +
search input renders).
- e2e/tests/schedules.spec.ts: switched from table-row selectors to
card-link selectors (schedules tab is now gantt cards, not a table),
added smoke tests for date nav controls + search input + REPEAT
footer + Services sidebar, renamed "Add Rule" -> "Add Level".
Full suite 10,216 / 0 failures. Format/dialyzer clean.
OnCall context
- move_layer/2: up/down position swap for schedule layers, mirror of
move_escalation_rule/2 (tested for both directions + boundary no-op).
ScheduleLive.Show
- New move_layer event handler.
- Up/down arrow buttons on each layer card; the top arrow is disabled on
the first layer, the bottom on the last (matches the escalation policy
level reorder UX shipped in chunk 1).
- Replaced two per-hour walks (build_layer_spans + build_final_spans -
~672 resolve calls per render at 14 days) with Resolver.resolve_range/3
+ a small segments_to_spans/3 mapper. Per-layer view wraps each layer
in a temporary single-layer %Schedule{} so the resolver code path is
shared and overrides don't bleed into the per-layer strip.
Tests: 3 new context tests for move_layer, 1 new integration test for
the layer reorder UI. Full suite 10,214 / 0 failures. Format/dialyzer
clean.
Resolver
- Resolver.resolve_range/3: returns the contiguous on-call segments for a
schedule across [start_at, end_at). Walks layer handoffs + override
boundaries, drops gap segments, and merges adjacent same-user segments.
UserColor module
- New Towerops.OnCall.UserColor with deterministic id -> color mapping
(Tailwind class + matching #RRGGBB hex). Refactored ScheduleLive.Show
to use it instead of its own per-page index palette so the same person
gets the same color across the schedules list and detail pages.
ScheduleLive.Index
- Date navigation: Today / prev / next + 1/2/4-week range selector.
- URL-driven: ?start=YYYY-MM-DD&range=N for shareable views; falls back
to defaults on malformed values.
- Per-row card: schedule name link, on-call-now avatar swatch, day-of-week
header strip, gantt strip rendered as a CSS grid with one column per day,
Today marker overlay.
Tests: 5 new resolver tests, 9 new UserColor tests, 5 new ScheduleLive
integration tests. Full suite: 10,210 / 0 failures. Format/dialyzer clean.
Also: cleaned up two stale inline comments in k8s/deployment.yaml that
disagreed with the values they sat next to (replicas/maxSurge).
Chunk 1 of the on-call/escalation redesign plan
(docs/plans/2026-04-28-on-call-pagerduty-style-redesign.md):
Schema
- on_call.ex: list_devices_using_policy/2 (direct + org-default inheritance)
- on_call.ex: move_escalation_rule/2 for up/down level reordering
- list_escalation_policies/1 now preloads :rules
Show page
- Levels rendered as numbered cards with up/down/delete controls
- "Notify the following users immediately and escalate after N minutes."
- REPEAT footer surfaces repeat_count inline (no longer a standalone column)
- Right-side Services sidebar lists devices using the policy
- Header line shows the handoff notifications mode (when in use by a service /
always / never)
Edit form
- Removed standalone Repeat Count input
- Added handoff notifications mode select
- Added inline "If no one acknowledges, repeat this policy N time(s)" block
Index views (policies tab + standalone)
- Replaced Repeat Count column with a Levels (rule count) column
Other
- rate_limit.ex: bind unmatched :ets.new/2 + schedule_clean/1 returns to fix
dialyzer :unmatched_returns warning under the project's strict flags.
All 10,191 tests pass; mix format clean; mix compile --warnings-as-errors clean;
mix dialyzer passes.
Promoted pure presentation and utility helpers from `defp` to `def @doc false`
across ~20 LiveViews, Oban workers, and sync modules so they're reachable from
unit tests. Refactored several `cond` blocks into idiomatic function heads with
guards. Added ~250 new test cases in new files under test/towerops and
test/towerops_web, including DB-backed tests for CnMaestro.Sync and
AlertNotificationWorker, and removed dead LiveView tab components and
CapacityLive (no callers anywhere in lib/test).
Configured mix.exs test_coverage.ignore_modules to exclude vendored third-party
code (SnmpKit, protobuf-generated Towerops.Agent.*, Absinthe GraphQL types,
Phoenix HTML modules, Inspect protocol impls) from coverage calculations —
these are not our project code.
Coverage: 66.93% → 70.09%. Full suite: 10,127 tests, 0 failures.
Changes to eliminate @dialyzer suppressions by fixing underlying causes:
NIF stubs (towerops_native.ex, mib_translator.ex):
- Change stubs to :erlang.nif_error(:nif_not_loaded) (no_return type).
Real NIF replaces stubs at load time; calls to unloaded stubs now fail
loudly instead of returning fake data. Lets dialyzer trust @spec.
- Remove @dialyzer :nowarn_function on three NIFs and on translate/1.
Discovery sync_* functions (snmp/discovery.ex, channels/agent_channel.ex):
- agent_channel passes %{device_id: _, interfaces: _} and %{id: _} maps
into Discovery.sync_ip_addresses/sync_processors/sync_storage, which
@spec'd only %Device{}. Add narrow map-type unions (ip_sync_device,
snmp_device_ref) reflecting what the functions actually access.
- Remove @dialyzer :nowarn_function on three agent_channel helpers.
remote_ip.ex — real bug caught and fixed:
- `:ranch.get_addr(socket.transport_pid)` was always raising since
Bandit uses ThousandIsland, not Ranch; the rescue _ -> nil silently
returned nil every time. Switched to Phoenix's documented
:peer_data connect_info (already enabled in endpoint.ex) via
socket.assigns; remote IP now actually works.
- Remove remote_ip.ex entry from .dialyzer_ignore.exs.
Accounts / Organizations (Ecto.Multi opacity):
- Add @specs to Multi-building helpers, refactor into pipe chains.
- 6 @dialyzer :nowarn_function → 0, but 7 :no_opaque remain. Root
cause is upstream: Ecto.Multi.new/0 returns a struct with a literal
%MapSet{} whose @opaque internal representation trips dialyzer on
every subsequent Multi.* call. Unfixable without an Ecto patch or
bypassing Multi entirely. Comments document the specific upstream
issue rather than a vague "Ecto.Multi opacity" claim.
Devices.ex:
- Adding @specs made it worse (call_without_opaque → contract_with_
opaque); inlining the Multi didn't help either — same MapSet root
cause. Suppression kept with a sharper comment.
Categories addressed:
- pattern_match / pattern_match_cov (32): remove dead case/with clauses
that dialyzer proved unreachable from the caller types.
- contract_supertype / extra_range / invalid_contract /
contract_with_opaque (25): narrow @spec declarations to match actual
success typings.
- call / call_without_opaque (18): fix bad calls, narrow User.t to
allow nil for in-memory changeset structs, suppress Ecto.Multi
opaque-type false positives with targeted @dialyzer directives.
- guard_fail / no_return / unused_fun / unknown_function (13): remove
dead || fallbacks, simplify always-true params, cascade-resolve
no_returns via the underlying pattern_match and call fixes.
Real production bug fixed: StormDetector.handle_cast/2 had swapped
`:queue.in` args (`queue |> :queue.in(ts)` which desugars to
`:queue.in(queue, ts)` — wrong argument order). Alert timestamps
were never being enqueued, so storm detection would fail at runtime.
Corrected to `ts |> :queue.in(queue)`.
.dialyzer_ignore.exs: suppress two genuine dep-PLT gaps
(:ranch.get_addr/1 false positive from Bandit's transitive ranch,
and the Cloak.Vault GenServer callback_info on the CI build path).
`mix dialyzer` now: Total errors: 114, Skipped: 114 — passes clean.
Warnings: 88 → 0.
Prefix fire-and-forget calls (PubSub.broadcast, Oban enqueue, Logger,
update_*, etc.) with `_ =` so dialyzer knows the return value is
intentionally discarded. Wrap a few if/case expressions whose
branches produce mixed types the same way.
No behavior changes — only explicit acknowledgement of discarded
returns.
Warnings: 242 → 88.
- Add @type t :: %__MODULE__{} to schemas referenced by callers:
ApiToken, Monitoring.Check, OnCall.Schedule, SiteOutage,
NotificationDigest, Snmp.WirelessClient, Topology.DeviceLink,
Agent.AgentJob, Agent.MikrotikResult.
- Proto.Types: qualify forward references to nested modules with
Types. prefix, and fix Metric typo (should be union type metric()).
- data_loaders.ex / chart_builders.ex: Snmp.SNMPDevice.t was a ghost
spec — module is Snmp.Device.
Warnings: 328 → 242.
Ignore file was silencing every warning under lib/towerops/**. Root
cause was plt_add_deps: :apps_direct missing Plug/Phoenix/Oban/Decimal
/Redix/ssl/public_key — hundreds of false `unknown_function` warnings
were being papered over.
Expand plt_add_apps to cover the common deps, narrow the ignore file
to real dep-PLT gaps only, and fix two concrete bugs uncovered by the
change:
- ScopedResource.fetch_preload/4: spec was atom() | [atom()] but
callers pass keyword lists (e.g. [rules: :targets]).
- ToweropsNative: tag NIF stubs with @dialyzer :nowarn_function so
dialyzer trusts the @spec instead of the fallback body.
Remaining 328 real warnings surface for follow-up.
Intercepts requests with Accept: text/markdown at endpoint level (before
the router's :accepts plug rejects them) and returns hardcoded markdown
representations for /, /privacy, and /terms.
Phase 1: Foundation Types (100% complete)
- query_helpers: SQL LIKE sanitization with pipe operators
- numeric: Integer parsing with pattern matching guards
- result: Pure Elixir Result monad (map, and_then, unwrap_or)
Phase 2: Ecto Domain Types (100% complete)
- ip_address: IPv4/IPv6 validation using :inet directly
- mac_address: Multi-format MAC parsing (colon/hyphen/dot/compact)
- snmp_oid: OID parsing/manipulation with recursive pattern matching
All 198 tests passing across converted modules.
API changed from Gleam-style {:some/:none to idiomatic {:ok/:error.
Refactored parse_numeric_oid to use with statement, reducing nesting depth.
Reviewed-on: graham/towerops-web#196
Fixes FunctionClauseError when DashboardLive receives {:alert_changed, org_id}
message broadcast by user_auth hooks.
The user_auth module subscribes all LiveViews to the
"organization:#{org_id}:alerts" topic and broadcasts {:alert_changed, org_id}
messages when alerts are created or resolved. DashboardLive was missing a
handle_info/2 clause to handle this message format, causing crashes in
production.
Solution:
- Added handle_info/2 clause for {:alert_changed, _org_id} message
- Uses existing debounced reload pattern to update dashboard
- Added test to verify message is handled without crashing
- Updated both technical and user-facing changelogs
All tests pass. The fix follows the same pattern used in AlertLive.Index
and DeviceLive.Show.
Reviewed-on: graham/towerops-web#189
- Check device.snmp_enabled before creating discovery/polling jobs
- Also check snmp_enabled for MikroTik jobs
- Prevents agent from attempting SNMP operations on ping-only devices
- Fixes "Agent collected 0 OIDs" errors for devices without SNMP
Reviewed-on: graham/towerops-web#180
Commented out weathermap navigation in sidebar and mobile menu.
Feature not ready for production use yet.
Files: lib/towerops_web/components/layouts.ex
Changelog: CHANGELOG.txt
Reviewed-on: graham/towerops-web#174
- Create WeathermapLive module at /weathermap with 60s auto-refresh
- Extend Topology.get_topology_for_weathermap with bandwidth utilization data
- Add WeathermapViewer JavaScript hook with color-coded edges:
* Green (0-30%) → Yellow (30-60%) → Orange (60-80%) → Red (80%+)
- Display throughput/capacity labels on edges (e.g., '847 Mbps / 1 Gbps')
- Add utilization stats dashboard with link count by usage level
- Support full-screen mode for NOC displays
- Integrate with existing dark mode and responsive design
- Add navigation links in sidebar and mobile menu
Technical implementation:
- Leverages existing Capacity module for interface utilization calculations
- Extends topology edges with utilization_level, utilization_pct, throughput_bps
- Builds on NetworkMapLive foundation with enhanced edge styling
- Auto-refresh via Phoenix PubSub and 60s timer
- Filter support for high/critical utilization links
Reviewed-on: graham/towerops-web#172
Devices configured as SNMPv1 were being force-upgraded to v2c for agent
polling, which breaks devices that don't support v2c (returning empty
OID responses and no throughput/capacity data).
Reviewed-on: graham/towerops-web#166
The wrap detection only applied the 2^32 correction when the previous
counter was in the upper half of the 32-bit range (> 2^31). On fast
links, counters wrap regardless of where in the range they sit. Three
graph rendering paths had no wrap detection at all, just max(0).
Consolidated all bps calculations into a single public
Capacity.calculate_bps/3 that always tries the 2^32 correction for
negative deltas. 64-bit HC counter resets remain clamped to zero since
the corrected delta stays negative.
Reviewed-on: graham/towerops-web#163
Standard ifInOctets/ifOutOctets are 32-bit counters that wrap around
4,294,967,296 bytes (~4.3 GB). At 1 Gbps this wraps every ~34 seconds,
causing brief drops to 0 bps in traffic charts and capacity reports.
When prev > 2^31 and delta is negative, add 2^32 to recover the correct
rate. If the result is still negative (HC 64-bit counter after reboot),
clamp to 0. Small-value decreases (prev < 2^31) continue to clamp to 0
since they cannot be real 32-bit wraps.
Fixes both capacity.ex (utilization reports) and device_live/show.ex
(per-device traffic chart).
Reviewed-on: graham/towerops-web#161
When an agent joins, we update last_seen_at in the database but don't set
last_heartbeat_db_update in the socket. This causes the heartbeat throttling
logic to be out of sync:
- On join: DB updated, but last_heartbeat_db_update not set (nil)
- First heartbeat: Updates DB again because last_heartbeat_db_update is nil
- Subsequent heartbeats: Check if >30s since last DB update
- Problem: last_heartbeat_db_update tracks the first heartbeat, not the join DB update
This manifests when Phoenix recompiles during development - the channel process
continues running with old socket assigns, and if last_heartbeat_db_update is
recent, heartbeats won't update the DB. After 10+ minutes without DB updates,
the agent appears offline even though it's still connected.
Fix: Set last_heartbeat_db_update when we update the DB on join, so the
throttling logic accurately tracks the most recent DB update.
Reviewed-on: graham/towerops-web#142
The Check protobuf has a `oneof config` field which generates separate
fields in the Elixir struct: :http, :tcp, :dns, :ssl. However,
check_type_config/2 was returning [config: {:http, ...}] which doesn't
match the struct definition, causing a KeyError crash in
build_and_push_check_jobs/1.
This bug was originally fixed in e0a74e2f but was accidentally
reintroduced in f703e61b when attempting to use "tagged tuple config".
Fix: Return field-specific keyword lists (e.g., [http: %HttpCheckConfig{}])
instead of [config: {:http, ...}].
Also added comprehensive test that creates all 4 check types and verifies
no KeyError crash occurs when sending check jobs to agent.
Reviewed-on: graham/towerops-web#141
CRITICAL FIXES:
- Fix unsafe pattern matching in SNMP discovery that would crash on timeout
- Changed {:ok, _} = ... to proper case statements with error handling
- Extracted safe_discover/3 helper to reduce cyclomatic complexity
- Added fallback to empty lists and warning logging for timeouts
- Affects: discover_vlans, discover_ip_addresses, discover_processors, discover_storage
MEMORY LEAK FIXES (JS Hooks):
- GlobalSearchTrigger: Added destroyed() cleanup for click listener
- SidebarCollapse: Added destroyed() cleanup for click listener
- ThemeSelector: Added destroyed() cleanup for phx:set-theme window listener
- NetworkMap: Added destroyed() cleanup for 3 button listeners (zoom in/out/fit)
- All hooks now properly store handler references and remove listeners on unmount
LIVEVIEW FIXES:
- Added phx-update="ignore" to 5 SensorChart hooks to prevent chart re-initialization
- Added catch-all handle_info(_msg, socket) to 5 LiveViews to prevent crashes on unexpected messages
- device_live/show.ex, device_live/index.ex, alert_live/index.ex, activity_feed_live.ex, agent_live/index.ex
N+1 QUERY FIXES:
- Created Gaiia.get_site_subscriber_summaries/1 batch function
- Refactored alert_live and device_live to use batch query instead of looping
- Reduces queries from N to 1 when loading site subscriber data
Impact:
- Prevents discovery worker crashes during SNMP timeouts
- Eliminates memory accumulation from leaked event handlers
- Prevents chart state loss on LiveView updates
- Prevents LiveView crashes from unexpected PubSub messages
- Improves database performance by eliminating N+1 queries in alert/device views
- Passes credo --strict with 0 issues
Reviewed-on: graham/towerops-web#132
check_type_config was returning `config: {:http, ...}` but the
Check struct has no :config field — it has :http, :tcp, :dns, :ssl
directly. This caused a KeyError crash in build_and_push_check_jobs.
Reviewed-on: graham/towerops-web#121
Seconds-level precision adds noise without value for these fields.
Shows "less than a minute" instead of "0 seconds" for very short durations.
Reviewed-on: graham/towerops-web#120
