unpkg.com fetches were intermittently failing in production, leaving
the /sites-map page without a working map. Vendoring Leaflet (JS, CSS,
marker/layer images) under priv/static/vendor/leaflet/ removes the
runtime CDN dependency, the matching CSP relaxation, and the SRI hash
churn on future updates.
Add 'vendor' to ToweropsWeb.static_paths/0 so Plug.Static serves the
new tree, and revert script-src/style-src to no longer list unpkg.com.
connect-src still allows *.tile.openstreetmap.org for tile fetches.
The /sites-map page loads leaflet from unpkg.com (CSS + JS) and fetches
tiles from *.tile.openstreetmap.org. Update Content-Security-Policy to
permit those origins so the map can render.
Adds:
- script-src https://unpkg.com (leaflet.js)
- style-src https://unpkg.com (leaflet.css)
- connect-src https://*.tile.openstreetmap.org (tile fetches)
Mix isn't loaded in mix releases, so Mix.env() raised
UndefinedFunctionError inside YamlProfiles.init/1, crashing the
GenServer at boot. That made Supervisor.start_link return {:error, ...}
in production, which the application-start callback then masked by
exiting on the unrelated Task.Supervisor.start_child noproc — leaving
the pod in CrashLoopBackoff with no clear log of the real failure.
Switch to Application.get_env(:towerops, :env), which is set in all
three environments (dev/test/runtime).
Also harden the post-startup callback so a transient TaskSupervisor
noproc cannot mask a real Supervisor.start_link error in the future.
The 1.7 upgrade migration added the new oban_workflows tracking but left
behind legacy state from earlier versions that the upgrade guide
strongly recommends removing
(https://oban.pro/docs/pro/v1-7.html):
- The uniq_key / partition_key GENERATED ALWAYS columns. The Smart
engine no longer reads them; uniqueness and partition lookups now
use expression indexes on meta. Keeping the columns imposes write
overhead on every oban_jobs insert/update and was contributing to
intermittent transaction-rollback errors.
- The oban_jobs_*_old workflow/chain indexes. v1.7 renamed the
originals and built expression-based replacements; the _old indexes
are dead weight that still slow down writes.
Also extend ErrorTrackerIgnorer to drop transient
DBConnection.ConnectionError noise ("connection is closed",
"transaction rolling back"). These come from the Repo's intentional
queue-target / disconnect-on-error cycling that recovers from stale SSL
connections — Ecto retries automatically, so logging adds noise without
value.
The deferred load via send(self(), :load_profiles) is fine in dev/prod
where SNMP polling starts well after boot, but it races with tests that
exercise discovery immediately after the GenServer starts. With an empty
ETS table, match_profile/2 returns nil and discovery falls back to the
Base profile, breaking tests that rely on YAML-profile-derived
manufacturer strings.
The original equipment->device rename migration's ALTER INDEX RENAME was a
no-op (the index didn't exist by the legacy name at that point in the
migration sequence on existing databases), leaving the unique index named
agent_assignments_equipment_id_index. The schema's unique_constraint now
expects agent_assignments_device_id_index, causing a constraint violation
to surface as Ecto.ConstraintError instead of a changeset error.
Bookworm runner images use /etc/apt/sources.list.d/*.sources (deb822),
which the legacy sed didn't touch — non-free stayed off and
snmp-mibs-downloader was unavailable. Patch the Components: line in
deb822 sources too.
Wire up Towerops.Sites.SiteQuery and Towerops.Monitoring.CheckQuery,
which were created in the recent context split but had no callers.
Sites: 5 callsites now compose via SiteQuery.for_organization/roots/
order_by_display.
Monitoring: 6 callsites compose via CheckQuery.for_organization/
for_device/of_type/with_enabled/order_by_name.
Switches lib/towerops/snmp.ex over to defdelegate calls into the new
submodules introduced in the prior commit. Drops the inline definitions
and unused aliases. Final size: 450 lines (was 2742).
Splits the rest of lib/towerops/snmp.ex into dedicated submodules:
- Towerops.Snmp.WirelessClients
- Towerops.Snmp.EntityPhysicals
- Towerops.Snmp.Transceivers
- Towerops.Snmp.PrinterSupplies
- Towerops.Snmp.ArpEntries
- Towerops.Snmp.MacAddresses
- Towerops.Snmp.DiscoveredDevices (cross-source aggregation + OUI lookup)
- Towerops.Snmp.Topology (graph nodes/edges/subnets)
DiscoveredDevices and Topology use Towerops.Devices.DeviceQuery.for_organization/1
to remove inline 'site.organization_id' joins. Public API preserved via
defdelegate in Towerops.Snmp.
snmp.ex shrinks from 2159 to 450 lines (originally 2742).
Splits cohesive sub-domains out of lib/towerops/snmp.ex into dedicated
submodules under lib/towerops/snmp/. Public API preserved via defdelegate
in Towerops.Snmp.
- Towerops.Snmp.Vlans, IpAddresses, Processors, StorageQueries
- Towerops.Snmp.Mempools, Sensors, Interfaces, Neighbors
Neighbors uses Towerops.Devices.DeviceQuery.for_organization/1 to remove
inline org_id filters. snmp.ex shrinks from 2742 to 2159 lines.
Replaces the last inline 'where: w.organization_id == ^...' clause in
maintenance.ex with composable filters from the existing MaintenanceWindowQuery
module.
Adds 7 new composable Query modules following the existing
DeviceQuery/AlertQuery pattern. Each exposes base/0 and for_organization/1,2
plus filters used by 2+ callers.
New modules:
- Towerops.OnCall.ScheduleQuery
- Towerops.OnCall.EscalationPolicyQuery
- Towerops.Maintenance.MaintenanceWindowQuery
- Towerops.Organizations.MembershipQuery
- Towerops.Organizations.InvitationQuery
- Towerops.Agents.AgentTokenQuery
- Towerops.Gaiia.DeviceSubscriberLinkQuery
Refactors callsites in alerts, agents, devices, gaiia, maintenance, on_call,
organizations, search, and subscription_limits to use existing or new Query
modules instead of inline 'where: x.organization_id == ^...' clauses.
Continues the extraction begun with Accounts.TOTP and Accounts.Sessions.
The context module shrinks from ~1198 to 577 lines by moving cohesive
sub-domains into dedicated submodules under lib/towerops/accounts/:
* Consents - grant/revoke/list user consents (+ private
grant_consents_sequential helper used by
register_user/1)
* PolicyVersions - policy version CRUD + reconsent detection
* LoginHistory - record/list/count login attempts, GDPR anonymize
* MagicLinks - magic-link login flow
* Passwords - change/update/reset + reset-instruction email
* Emails - change/update + update-instruction email
The Accounts module retains a defdelegate facade so existing callers
across lib/towerops_web and other contexts continue to work unchanged.
Two shared transaction helpers (update_user_and_delete_all_tokens/1 and
unwrap_transaction_result/1) are used by multiple extracted modules; they
remain in Accounts but are now public with @doc false rather than
duplicated.
Applies the 11 recommendations from the codebase optimization review.
Module decomposition:
- Extract Towerops.Topology.InferenceEngine: pulls device-role inference
(capability rules, vendor matching, evidence merging) out of the topology
context. Topology now delegates merge_confidence/2,
infer_role_from_capabilities/1, and infer_device_role/1.
- Extract Towerops.Accounts.TOTP: TOTP secret/QR generation, multi-device
verification, recovery code lifecycle, and sudo MFA logic. Accounts
delegates the public TOTP API for backwards compatibility.
- Extract Towerops.Accounts.Sessions: browser session create/list/touch/
revoke/anonymize/expire. Accounts delegates the public session API.
- Extract Towerops.Snmp.Queries: time-series read paths
(sensor readings, interface stats, latest-per-id batches).
- Extract Towerops.Snmp.Monitoring: time-series write paths
(sensor/interface/processor reading inserts, batch inserts via
Repo.insert_all). Snmp.ex shrinks from 2985 to 2742 lines.
DRY refactoring:
- Add per-schema Query modules (Devices.DeviceQuery, Alerts.AlertQuery)
with composable for_organization/for_site/with_status/etc filters;
refactor list_site_devices, count_organization_devices,
count_site_devices, count_site_devices_down, count_organization_alerts
to compose them.
- Introduce create_discovered_checks/6 helper in Snmp; collapses 4 nearly
identical Enum.reduce blocks for sensors/interfaces/processors/storage.
- Unify MAC and ARP evidence collectors behind collect_lookup_evidence/3
(lldp/cdp were already factored).
Pattern matching:
- Replace cond block in infer_role_from_capabilities/1 with a declarative
@role_rules table consulted via Enum.find_value (Elixir 1.19 forbids `in`
with runtime lists in guards, so multi-clause functions weren't an option).
- Replace inline `if since` in get_sensor_readings/2 with a maybe_filter_since
pipe helper.
Performance:
- Replace Enum.each + Repo.insert per evidence row in upsert_link with a
single Repo.insert_all batch (truncates :observed_at to seconds).
Human-centric:
- Decompose build_device_lookup/1 into fetch_lookup_devices, map_ips_to_ids,
map_names_to_ids, map_macs_to_ids.
- Adopt Towerops.Result.map/2 in Snmp.Client.do_get_multiple_sequential
(kept narrow; `with` is preferred over Result.and_then for chained
operations and is already used widely).
CLAUDE.md updates:
- Replace stale main→staging / production-branch deployment notes with the
current flow: PRs deploy to Dokku staging, push to main runs the
ExUnit gate then builds an image; argocd-image-updater rolls production.
- Refresh data-model diagram and references from "Equipment" to "Device"
(post equipment→devices rename migration).
- Correct stale architecture facts: Hammer → in-tree Towerops.RateLimit,
Rust NIF → C NIF (libnetsnmp), gitlab-registry → forgejo-registry,
expand the Oban queue and cron lists, list actual implemented Ecto types.
All 10,228 tests pass.
Bumps vendored Oban Pro from 1.6.13 to 1.7.0, which transitively bumps
oban core to 2.22.1 (requires migrations v14) and db_connection to 2.10.0.
- mix.exs constraint: ~> 1.6 → ~> 1.7
- vendor/oban_pro/ replaced with v1.7.0 hex package
- migration 20260430142200: Oban core v12 → v14 (must run first)
- migration 20260430142241: Oban Pro 1.7.0 schema additions
Also corrects a pre-existing constraint name in AgentAssignment that was
left stale by the equipment→devices rename migration (20260117190134).
The unique index is named agent_assignments_device_id_index, but the
schema's unique_constraint/3 still pointed at the old equipment_id name,
causing the test suite to surface Ecto.ConstraintError instead of a
changeset error. Required to get the suite green for this upgrade.
With 1 replica, the old pod was torn down too quickly after the new pod
passed its first health check. Increase successThreshold to 3 (15s of
consecutive checks) and minReadySeconds to 30 to ensure Traefik has fully
propagated the new endpoint before the old pod is terminated.
Apply patterns from testingliveview.com — replace render_click/submit/change
event-name calls with element/form-targeted equivalents that validate the
actual DOM wiring, not just handler logic. Adds stable test selectors
(id="...") to interactive elements across 10 LiveView templates.
Surfaces (and fixes) several real issues that the bypass-style tests masked:
- Removes dead regenerate_token handler in agent_live/index.ex (no UI ever
triggered it) and its corresponding test.
- Removes dead refresh_topology test in network_map_live_test (event was
referenced only in the test — no handler, no button, no JS anywhere).
- Corrects "shows notification tab with add device modal" — modal lives on
the security tab; original test mounted ?tab=notifications and only
worked because direct event calls bypassed tab gating.
- Corrects form input shape mismatches in agent_live/index_test that the
handler's defensive case clauses had been masking.
- Expands setup data for org/settings_live_test (snmp_community,
default_agent_token_id, multi-org membership) so the gated buttons
actually render.
11 direct render_click/submit/change calls remain — all intentional,
documented server-side guard tests (tampered POSTs, race conditions,
defensive idempotent handlers).
Full LiveView test suite: 1300 tests, 0 failures.
The HelmRelease/HelmRepository CRs targeted flux-system; with flux uninstalled
they were never being applied. The gitlab-agent workload is not deployed on
the cluster.
ArgoCD refuses to clone repos containing tracked symlinks that point outside
the working tree. The .pre-commit-config.yaml symlink resolved to /nix/store/...
on the local dev machine and broke remote checkouts. Generate it locally via
direnv/nix instead.
Performance:
- schedule_live: preload page once instead of get_schedule!/1 per row
- alert_live + alerts: DB-side status filter; Repo.aggregate counts replace length/Enum.count over 500-row fetches on every event
- dashboard_live: drop duplicate get_device_status_counts; cap active alerts to 20 + use count_active_alerts/1
- maintenance.active_windows_for_device: 3 round-trips collapsed into one query (per-site-id branch keeps Postgres parameter types unambiguous)
- sites.build_site_tree: O(N^2) -> O(N) via group_by(parent_site_id)
- accounts.sole_owner_organizations: single group_by + having instead of per-org Repo.aggregate loop
- agents + agent_live (org + admin): count_assigned_devices_batch/1 + count_agent_polling_targets/1 (no preloads)
- alert_digest_worker: list_alerts_by_ids/1 batches digest fetch
- gaiia: distinct: true at DB; limit 50 on bidirectional ilike
Indexes:
- maintenance_windows(organization_id, starts_at, ends_at) WHERE suppress_alerts = true
- alerts(check_id) WHERE resolved_at IS NULL
Antipatterns:
- agents.delete_agent_token: PubSub.broadcast moved outside Repo.transaction so a rollback no longer leaves subscribers acting on a non-existent deletion
- integrations_controller.to_atom_keys: replaced String.to_existing_atom on user-controlled JSON keys with explicit allowlist
- 9 Task.start callsites converted to Task.Supervisor.start_child(Towerops.TaskSupervisor, ...) so background DB writes survive shutdown (test-mode discovery shims left as-is for sandbox semantics)
Captures a high-level review of module decomposition, context-layer
boundaries, query patterns, error handling, and test coverage gaps to
inform future refactor work.
Three deferred followups from the chunk-4 plan, all together:
A. Restrictions editor
- Resolver: time_of_week support (weekday set + time-of-day window),
in addition to the existing time_of_day.
- New OnCall.update_layer_restrictions/2 + clear_layer_restrictions/1.
- schedule_live/show: per-layer Restrict on-call shifts form
(Always on call / Time of day / Time of week with Mon-Sun checkboxes),
changes patched live and persisted via the resolver.
B. Drag-and-drop reorder
- New OnCall.reorder_layers/2 + reorder_escalation_rules/2 (atomic
transaction; validates id count + membership).
- assets/js/sortable_list.ts: minimal HTML5 drag-and-drop hook (no
external dep; mirrors the existing DeviceListReorder pattern),
registered as SortableList in app.ts.
- schedule_live/show + escalation_policy_live/show: layer/level cards
now carry data-id + draggable + a visible drag handle. Up/down arrow
buttons remain as a fallback for users on assistive tech.
C. Unified new-schedule UX
- schedule_live/show pre-opens the Add Layer form when the schedule
has zero layers, so the new-schedule -> add-layers flow lands ready
to act on. Two existing tests updated to seed a layer first; two
new tests verify the auto-open behaviour both ways.
Full suite 10,231 / 0 failures. Format/dialyzer clean.
ScheduleLive.Index
- Name search input above the date nav (phx-debounce 300ms).
- Standard <.pagination> footer (10 per page) under the gantt cards.
- Both reflected in URL params (?search= and ?page=) for shareable
links; defaults are stripped from the URL.
- Refactored: load_schedules_with_timeline/3 -> hydrate_schedules/3
so filter/paginate/hydrate are clearly separate steps.
Tests
- 2 new ExUnit integration tests (search filter narrows results +
search input renders).
- e2e/tests/schedules.spec.ts: switched from table-row selectors to
card-link selectors (schedules tab is now gantt cards, not a table),
added smoke tests for date nav controls + search input + REPEAT
footer + Services sidebar, renamed "Add Rule" -> "Add Level".
Full suite 10,216 / 0 failures. Format/dialyzer clean.
OnCall context
- move_layer/2: up/down position swap for schedule layers, mirror of
move_escalation_rule/2 (tested for both directions + boundary no-op).
ScheduleLive.Show
- New move_layer event handler.
- Up/down arrow buttons on each layer card; the top arrow is disabled on
the first layer, the bottom on the last (matches the escalation policy
level reorder UX shipped in chunk 1).
- Replaced two per-hour walks (build_layer_spans + build_final_spans -
~672 resolve calls per render at 14 days) with Resolver.resolve_range/3
+ a small segments_to_spans/3 mapper. Per-layer view wraps each layer
in a temporary single-layer %Schedule{} so the resolver code path is
shared and overrides don't bleed into the per-layer strip.
Tests: 3 new context tests for move_layer, 1 new integration test for
the layer reorder UI. Full suite 10,214 / 0 failures. Format/dialyzer
clean.
Resolver
- Resolver.resolve_range/3: returns the contiguous on-call segments for a
schedule across [start_at, end_at). Walks layer handoffs + override
boundaries, drops gap segments, and merges adjacent same-user segments.
UserColor module
- New Towerops.OnCall.UserColor with deterministic id -> color mapping
(Tailwind class + matching #RRGGBB hex). Refactored ScheduleLive.Show
to use it instead of its own per-page index palette so the same person
gets the same color across the schedules list and detail pages.
ScheduleLive.Index
- Date navigation: Today / prev / next + 1/2/4-week range selector.
- URL-driven: ?start=YYYY-MM-DD&range=N for shareable views; falls back
to defaults on malformed values.
- Per-row card: schedule name link, on-call-now avatar swatch, day-of-week
header strip, gantt strip rendered as a CSS grid with one column per day,
Today marker overlay.
Tests: 5 new resolver tests, 9 new UserColor tests, 5 new ScheduleLive
integration tests. Full suite: 10,210 / 0 failures. Format/dialyzer clean.
Also: cleaned up two stale inline comments in k8s/deployment.yaml that
disagreed with the values they sat next to (replicas/maxSurge).
- k8s/deployment.yaml: bump startup-probe failureThreshold from 12 to 24
(70s -> 130s max startup time) to give the app more headroom on cold start.
- lib/towerops/profiles/yaml_profiles.ex: defer YAML profile loading from
init/1 to a handle_info(:load_profiles, ...) so the supervisor (and Bandit)
come up immediately and the profiles populate in the background within ~60s.
- e2e/Makefile: convenience target wrappers around the existing npm scripts.
Chunk 1 of the on-call/escalation redesign plan
(docs/plans/2026-04-28-on-call-pagerduty-style-redesign.md):
Schema
- on_call.ex: list_devices_using_policy/2 (direct + org-default inheritance)
- on_call.ex: move_escalation_rule/2 for up/down level reordering
- list_escalation_policies/1 now preloads :rules
Show page
- Levels rendered as numbered cards with up/down/delete controls
- "Notify the following users immediately and escalate after N minutes."
- REPEAT footer surfaces repeat_count inline (no longer a standalone column)
- Right-side Services sidebar lists devices using the policy
- Header line shows the handoff notifications mode (when in use by a service /
always / never)
Edit form
- Removed standalone Repeat Count input
- Added handoff notifications mode select
- Added inline "If no one acknowledges, repeat this policy N time(s)" block
Index views (policies tab + standalone)
- Replaced Repeat Count column with a Levels (rule count) column
Other
- rate_limit.ex: bind unmatched :ets.new/2 + schedule_clean/1 returns to fix
dialyzer :unmatched_returns warning under the project's strict flags.
All 10,191 tests pass; mix format clean; mix compile --warnings-as-errors clean;
mix dialyzer passes.
* UISP diff_snapshots test: backdate first snapshot via UPDATE instead of
Process.sleep(1_100) to get distinct second-precision snapshot_at values.
* PagerDuty.Client.send_event_with_retry/2: check max_retries BEFORE sleeping
on 429. Previously a 429 response always slept once (~1s) before checking
the retry cap, so even with @max_retries = 0 in test env every 429 test
paid that cost.
Suite time: 65.3s → 33.9s (-48%). No test is >400ms now.
