Commit graph

931 commits

Author SHA1 Message Date
ea91dae0e6 fix: 6 medium-severity bugs (M2, M8, M9, M10, M11, M16)
- M2: get_user_by_email/1 uses lower(email) = lower(?) so User@Example.com
  and user@example.com resolve to the same account; closes a lookalike-
  registration / password-reset confusion risk.
- M8: webhook auth plug no longer echoes "Webhook authentication not
  configured" — the misconfiguration is logged server-side and the caller
  gets a generic Internal server error so endpoints can't be probed.
- M9: coverages controller logs the underlying KMZ build error and
  returns a generic "Failed to build KMZ" string — filesystem paths and
  zip internals no longer leak.
- M10: account-data controller no longer crashes with MatchError when an
  org has zero or multiple memberships; defaults to "member" and treats
  any owner membership as owner.
- M11: ActivityController switches to a hard whitelist
  (Atom.to_string/1 comparison) instead of String.to_existing_atom/1;
  unknown filter types are dropped silently and the atom table can't be
  grown from API input.
- M16: title-tracking MutationObserver is stored on `this` so destroyed()
  actually disconnects it — fixes a memory leak per LV navigation.
2026-05-12 11:38:13 -05:00
abfd44fd75 fix: 3 more high-severity bugs (H19, H20, H25)
- H19: /api/v1/mobile/auth/qr/verify no longer returns user_email. Knowing
  a QR token now only tells the caller the token is valid; the email is
  only revealed by /complete which consumes the token.
- H20: CoverageWorker max_attempts dropped from 3 to 1. The fail/2 path
  already returns :ok and writes the failure onto the coverage record,
  so the 3-attempt retry policy was never reachable and was misleading.
- H25: ChecksController.create rejects device_ids that don't belong to
  the caller's organization (via ScopedResource.fetch). Without this an
  API token could attach service checks to devices in another tenant.

Also strip bug ID references from in-code comments (per feedback that
H/M/L numbers don't survive past their bugs.md removal); commit history
keeps the audit trail.
2026-05-12 11:32:20 -05:00
97232117f5 fix: H12 cookie hardening + 5 low/medium bugs (L2, L5, L6, L8, L10, L11)
- H12: session and remember-me cookies get http_only + secure (prod-only).
  Cookie can no longer be read via document.cookie (XSS exfil defense)
  and the Secure flag is set in production via config/prod.exs.
- L2: 404 tracker uses EXPIRE … NX so a sustained probe can't keep
  refreshing the 60s window and dodge the threshold ban.
- L5: vault only reads CLOAK_KEY when :env == :prod — a developer with
  a prod env var set in their shell won't accidentally encrypt local
  data with the production key.
- L6: health endpoint no longer leaks the app version.
- L8: Preseem.dismiss_insight/2 + InsightsLive uses it — dismissing now
  requires the insight to belong to the user's org (closes IDOR).
- L10: SidebarCollapse JS hook stores its click handler and removes it
  in destroyed(); listeners no longer accumulate across LV navigation.
- L11: WebMCP navigate tool rejects anything that isn't a same-origin
  absolute path (blocks javascript:, data:, off-site URLs).
2026-05-12 11:22:47 -05:00
6a9b471d1f fix: 4 more high-severity bugs (H23, H26, H28, H29)
- H23: introduce Towerops.Workers.SyncErrors.transient?/1 classifier;
  uisp/preseem/cn_maestro sync workers now retry transient failures (5xx,
  timeouts, rate limits) via Oban while still discarding permanent errors
  (401/403/404) as :ok. Stale monitoring data is the bigger risk than an
  extra retry on a known-bad credential.
- H26: add Monitoring.get_latency_data_for_devices/2 batch counterpart
  (currently a stub map but called from SiteLive.Show so any future ping
  implementation lands in a single query, not one-per-device).
- H28: Nominatim search popups in the coverage map now bind via a text
  node instead of an HTML string, so a compromised/poisoned response
  can't execute as JS in the popup.
- H29: yaml_profiles dot-boundary OID prefix match (handles optional
  trailing dot from YAML profile keys). Stops 1.3.6.1.4.1.9 from
  incorrectly matching 1.3.6.1.4.1.99.
2026-05-12 11:14:03 -05:00
b252cb81b9 fix: 5 more high-severity bugs (H6, H8, H17, H18)
- H6: batch interface stats query in get_site_capacity_summary, replacing
  per-interface SELECT with a single grouped query
- H8: explicit expires_at check in MobileAuth and GraphQLAuth plugs as
  defense in depth — the query already filters expired rows, but a future
  refactor dropping the filter can't quietly re-enable revoked sessions
- H17: Reports LiveView (toggle/delete/run_now) now uses
  Reports.get_organization_report/2 to scope by organization_id, fixing
  IDOR where a user could manipulate other tenants' reports by ID
- H18: ToweropsWeb.Api.ParamFilter strips identity fields (id,
  organization_id, user_id, created_by_id, inserted_at, updated_at) from
  user-supplied API params; applied to devices, sites, checks, and
  escalation_policies create/update paths to prevent mass-assignment of
  tenant ownership fields if a changeset cast list ever drifts
2026-05-12 11:04:31 -05:00
7e6ec7098c fix: 10 high-severity bugs from code audit (H1-H5, H14-H16, H27, H30)
- H1: batch count queries in MobileController (org sites/devices/alerts, site
  device counts) — reduces 1+3N+2M queries to 4 + 2
- H2: batch Oban cancel for stop_device_checks — single ANY(?) query instead
  of one per check
- H3: resolve_active_alerts_for_device uses update_all + deduped PubSub
  broadcasts (was N updates + N broadcasts)
- H4: atomic ON CONFLICT upsert for auto-discovery checks; restore partial
  unique index dropped in 20260404000002 (dedup migration included)
- H5: wrap apply_agent_to_all_equipment delete+insert in transaction so a
  failed insert_all rolls back the prior delete
- H14: replace String.to_integer/1 on untrusted SNMP OID components with
  Integer.parse/1 + validation (neighbor_discovery, discovery storage index,
  printer supply index)
- H15: ActivityFeedLive whitelists activity types against @all_types instead
  of String.to_existing_atom/1
- H16: defensive page param parsing in user_settings and admin dashboard
  LiveViews
- H27: search sanitize/1 delegates to QueryHelpers.sanitize_like/1 which
  escapes backslash first
- H30: JobCleanupTask no longer cancels jobs in "executing" state so
  in-flight polls complete naturally
2026-05-12 10:55:01 -05:00
ca7bb75472 fix: 11 critical security/correctness bugs from code audit
- C1: Replace innerHTML template literals with DOM textContent in sites_map.ts
- C2: Derive user_id from current_scope instead of client params in policy consent
- C3: Fix broken halt() in GDPR data export (orphaned _ = ... halt())
- C4: Add owner/admin authorization check to MembersController update/delete
- C5: Require HMAC signature on agent release webhook (was optional)
- C6: Fix Repo.all_by/2 (not a real Ecto function) → Repo.all with query
- C7: Move auth exemption to before_send callback in BruteForceProtection
- C8: Block </style>/<script injection in status page custom_css validation
- C9: Create secrets.example.yaml with placeholders; secrets.yaml already gitignored
- C10: Load Stripe key from STRIPE_SECRET_KEY env var instead of hardcoded value
- C13: Remove credentials from PubSub backup broadcast; channel resolves them

C11 (no force_ssl): by design — Cloudflared terminates TLS
C12 (device quota race): false positive — FOR UPDATE serializes correctly
2026-05-12 10:20:52 -05:00
39e588c686 fix: API token access control, admin form crash, MIB validation, CSP dedup
- Add organization membership check before API token creation
- Fix admin security allowlist form reading current_user instead of current_scope.user
- Use recursive File.ls instead of Path.wildcard to include hidden files in MIB validation
- Add upload size check before ZIP extraction in MIB controller
- Centralize CSP in SecurityHeaders plug with per-request nonces instead of 'unsafe-inline'
2026-05-12 09:08:57 -05:00
320ced2d23 fix: MIB archive bomb protection, atom exhaustion guard, and vendor caching
- Enforce upload size limit (100 MB), max extracted files (2000), max
  uncompressed size (500 MB), and reject symlinks in MIB archives
- Replace unbounded String.to_atom calls in SNMP tokenizer with
  safe_atom/1 that warns when approaching the atom table limit
- Cache MIB vendor listing with persistent_term, invalidated on
  upload/delete to avoid blocking synchronous Path.wildcard scans
2026-05-12 08:31:01 -05:00
8338d0af1e fix: remaining bugs.md findings — Repo calls, process dict, CSS, rate limits, Oban, debug route
- Extract Repo calls from LiveViews into Agents.get_agent_token_for_org/2
  and Accounts.verify_new_totp_device/2 context functions
- Remove Process dictionary usage for cookie consent; use explicit assigns
- Reject url() references in status page custom CSS changeset
- Add per-organization rate limit check alongside per-IP
- Document Oban flush as emergency-only with audit logging
- Gate /admin/headers behind dev_routes (debug endpoint)
2026-05-11 19:34:18 -05:00
3632580d1b fix: address security and reliability findings from bugs.md review
- Client IP: only trust X-Forwarded-For from RFC 1918 proxy IPs
- Webhook auth: handle nil/blank secret with controlled error, not 500 crash
- Sudo redirect: reuse validated return_path? from login to prevent open redirect
- Map live: remove redundant inline script (ensureLeaflet hook handles loading)
- Bang calls: convert crash-prone exact matches to case in QR live and API controllers
2026-05-11 18:54:12 -05:00
2ff6a61bd1 fix(gaiia): add ghost devices tab, manual link, and refresh to reconciliation
Fix auto_match return tuple order (mac/ip/site were swapped). Add ghost
devices tab for cleaning up stale mappings to deleted devices. Add "Link"
button on untracked devices to manually link to existing Gaiia items.
Add manual refresh button for on-demand reconciliation.
2026-05-11 18:35:34 -05:00
ecec8ba845 feat(mikrotik): webhook receiver for MikroTik RouterOS events
Receives webhooks from MikroTik RouterOS when devices come online,
extracts MAC/IP data, and reconciles with Gaiia inventory.
2026-05-11 18:02:21 -05:00
97910aa178 fix(gaiia): correct mac_address to if_phys_address in inventory_creator
Also guard against empty model_id form submission.
2026-05-11 16:42:18 -05:00
7067bc2fa9 feat(gaiia): hyperlink IP addresses in untracked devices table 2026-05-11 16:39:53 -05:00
fb2d4b9b1a fix(gaiia): correct Interface MAC field name, relabel auto-match button
- Interface MAC field is `if_phys_address`, not `mac_address`
- Button now says "Auto-Match All" since it matches by MAC, IP, and site
2026-05-11 16:20:35 -05:00
3483d9d5f0 feat(gaiia): MAC-based auto-matching (phase 1, highest confidence)
Adds MAC address matching as the primary matching phase:
- Normalizes MACs (strips colons/dots/dashes) from SNMP interface data
- Matches against gaiia_inventory_items.mac_address (synced from Gaiia)
- Phase 1: MAC → Phase 2: IP → Phase 3: Site

Flash now shows breakdown: "Auto-matched 47 devices (12 by MAC, 34 by
IP, 1 by site). 17 remaining."
2026-05-11 16:09:41 -05:00
40dadededd fix(gaiia): wrap manufacturer select in form for phx-change to pick up name attr
A bare <select phx-change="..."> sends %{"value" => ...}, not the name
attribute. Wrapping in <form phx-change="..."> fixes this so the handler
receives %{"manufacturer_id" => value}.
2026-05-11 15:58:51 -05:00
8db9fb4fcd fix(gaiia): add error handling and logging for model fetch
- Handle empty manufacturer_id selection (select placeholder)
- Log and flash when model fetch fails or returns empty
- Add require Logger to reconciliation live view
2026-05-11 15:25:34 -05:00
2695e8a0eb refactor(gaiia): manufacturer/model dropdowns for inventory creation
Replaces auto-matching with explicit two-step selection:
1. Select manufacturer from Gaiia's inventory model library (dropdown)
2. Select model for that manufacturer (dropdown, filtered live)
3. Click Create

Manufacturers and models are fetched from Gaiia on demand and cached
in socket assigns. Only one device form is open at a time.
2026-05-11 15:12:52 -05:00
31231d9fad fix(insights): correct ngettext usage in auto-match flash
ngettext is a macro that already calls dngettext internally. The
previous code was calling dngettext a second time on the result string,
passing nil as the count — causing a FunctionClauseError.
2026-05-11 15:07:39 -05:00
3267bb134f feat(gaiia): create inventory items for unmatched devices
Adds `Towerops.Gaiia.InventoryCreator` which:
- Matches a device's SNMP-discovered manufacturer/model against Gaiia's
  inventory model library to find the right modelId
- Calls `startAddInventoryItemsJob` to create the item, assigned to the
  correct Gaiia network site with MAC address from SNMP interfaces

A "Create in Gaiia" button now appears next to each untracked device on
the reconciliation page.
2026-05-11 14:58:37 -05:00
1f9fe2ee29 chore: add gaiia library dep, remove stale honeybadger/tidewave/cbor
- Add vendored Gaiia library (path: vendor/gaiia) with auto-generated GraphQL
  queries and mutations from the Gaiia API schema
- Add startAddInventoryItemsJob + updateInventoryItem mutations
- Remove `:cbor` (was unused — no references anywhere)
- Remove stale Honeybadger filter module, tests, router plug, and config
- Remove stale Tidewave plug from endpoint (dev-only, dep already gone)
- Unlock leftover deps from lock file
2026-05-11 14:46:27 -05:00
485748d253 feat(gaiia): site-based auto-matching + automatic site mapping during sync
Two improvements to Gaiia reconciliation:

1. Auto-match now uses two-phase matching:
   - Phase 1: IP match (high confidence, same as before)
   - Phase 2: Site match — if a device is at a Towerops site linked to a
     Gaiia network site, match it to unmapped inventory assigned to that site

2. Gaiia sync now automatically maps network sites to Towerops sites by
   name during sync. When a Gaiia network site name matches a Towerops
   site name, `site_id` is set on the `gaiia_network_sites` row. This
   enables the site-based matching in phase 2.

Flash message now shows breakdown: "Auto-matched 47 devices (34 by IP,
13 by site). 17 remaining."
2026-05-11 14:17:41 -05:00
80837b8b26 feat(gaiia): one-click auto-match untracked devices by IP
Adds `Gaiia.auto_match_untracked/1` which finds unmapped Gaiia inventory
items whose IP matches an untracked Towerops device and links them.

Button on the reconciliation page's Untracked tab runs the match
inline and shows a flash with matched/remaining counts.
2026-05-11 13:51:13 -05:00
3b0961dfc0 refactor(webhooks): enqueue Oban jobs immediately, ACK fast
All 4 webhook controllers now verify the signature, insert an Oban job,
and return immediately. Previously, Stripe and PagerDuty did DB queries +
writes inline; Gaiia ran upserts; Agent Release broadcast to all connected
WebSockets — all in the request path.

New workers:
- StripeWebhookWorker — delegates to WebhookProcessor.process/1
- GaiiaWebhookWorker — delegates to Webhooks.process_event/3
- PagerdutyWebhookWorker — resolve/acknowledge alerts by dedup key
- AgentReleaseWebhookWorker — broadcast mass update to agents

Also fix: log error when resolve_alert fails in agent_channel (was silently
discarded, causing "Resolving stuck device_down alert" to repeat in logs).
2026-05-10 17:18:23 -05:00
a8f119261e fix(insights): surface all LLM/insight errors instead of silently swallowing them
- Regenerate button now logs Oban.insert failures and shows per-worker counts
- AI network insight worker: log errors for LLM failures, insert failures,
  Usage.record failures, and transaction failures (was discarding all)
- Enrichment worker: handle apply_llm_enrichment errors instead of crashing,
  log LLM and Usage.record failures
- Wire up build_gaiia in network snapshot to actually query reconciliation
  finding counts instead of hardcoding zeros
2026-05-10 15:31:26 -05:00
fd235d05b7 feat(insights): AI surfaces its own observations from full network state
Adds a parallel LLM path that reviews the whole org snapshot — Preseem
APs, SNMP signals, backhaul, agents, and the insight types the rules
already covered — and emits novel observations as ai_observation
insights (source=ai). Existing rule-based insights and per-insight
enrichment continue unchanged.

- lib/towerops/llm/network_snapshot.ex builds the bounded snapshot
- lib/towerops/llm/network_insight_prompt.ex builds the prompt and
  parses the LLM's JSON observations array
- lib/towerops/workers/ai_network_insight_worker.ex runs nightly at
  03:30 UTC (after the rule pass + per-insight enrichment) and on
  the superuser regenerate button
- preseem/insight.ex extends @valid_types with "ai_observation" and
  @valid_sources with "ai"
- insights_live/index.ex includes the new worker in the regen list
  and adds an "AI" source badge style

Bumps the SnmpKit Manager timeout-options test threshold from 500ms
to 3000ms (was flaking on loaded dev machines), and scopes
list_unenriched_insights/1 assertion in insights_test:573 to the
test's org so cross-suite leaks don't fail the count.
2026-05-10 14:29:56 -05:00
6338556945 fix(snmp): scale agent-polled temperatures + quiet two info logs
- ToweropsWeb.AgentChannel now runs sensor readings through
  Towerops.Snmp.SensorScale.normalize/2 in both resolve_sensor_value/2
  and the GraphQL publisher. Without this, MikroTik mtxrHl* deci-degree
  readings (e.g. Culleoka reporting 294 -> 29.4°C) were stored and
  alerted on at 294°C. The DevicePollerWorker path already did this;
  only the agent path was missing it.
- Demote "Received SNMP result from agent" and "Processing polling
  result for X" from info to debug — they fire every poll cycle on
  every device and were dominating prod logs.
2026-05-10 14:00:18 -05:00
bdfb20efdf feat(insights): superuser regenerate button + dialyzer cleanup
- /insights now shows a "Regenerate insights" button for superusers
  that enqueues all seven insight workers (Preseem baselines,
  capacity, device-health, Gaiia, system, wireless, LLM enrichment).
  Non-superusers neither see the button nor can trigger the event.
- Dialyzer is clean: added :tools to plt_add_apps, gave
  Preseem.Insight a @type t, pinned a few unmatched returns, and
  suppressed a known MapSet opaque false-positive in Towerops.Unused.
- e2e: NODE_OPTIONS=--disable-warning=DEP0205 silences the Node 25
  deprecation noise from Playwright's internal ESM loader.
2026-05-10 13:41:25 -05:00
dbd4183f37 fix(llm): drop model name from footer, harden JSON parser
UI: the AI-generated footer dropped " · deepseek-v4-pro" — model
identity is irrelevant to operators reviewing the recommendation.

Parser: stop letting raw JSON leak into the summary text when the LLM
returns an almost-but-not-quite-valid response.

- Strip ```json fences (already done) PLUS try a balanced-brace
  extraction so a "Here's the JSON: {…}" preamble doesn't break
  Jason.decode
- When all decode paths fail, regex-extract `"summary"` and `"action"`
  fields from the text rather than dumping the whole blob with braces
  intact
- Last-ditch sanitiser strips remaining brace/quote noise so the
  rendered summary never shows literal `"summary"` keys

Maintenance: `Towerops.Maintenance.RepairLlmSummaries.run()` clears
`llm_enriched_at` on already-stored rows whose summaries contain JSON
artifacts so the next worker cycle re-enriches them with the new
parser.

Test: insights_live_events_test now asserts the AI-generated label is
present *and* that the model name does not leak into the rendered HTML.
2026-05-10 13:03:25 -05:00
d051968ae6 fix(insights): render acronym sources uppercase (AI, SNMP)
The CSS `capitalize` class only capitalises the first letter, so the
source filter pills and per-card badges showed "Ai" and "Snmp". Switch
to a `source_label/1` helper that returns "AI" / "SNMP" verbatim and
falls back to `String.capitalize/1` for the rest (Preseem, Gaiia,
System).
2026-05-10 12:41:57 -05:00
f6cd35d2d3 feat(insights): add 'AI' source filter for LLM-enriched insights
Adds an "ai" pill to the source-filter strip on /insights and a small
blue "AI" badge on every card whose `llm_enriched_at` is set. Selecting
the pill returns every LLM-enriched insight regardless of underlying
source (preseem / snmp / gaiia / system).

`source: "ai"` is a virtual filter — `list_insights/2` translates it to
`WHERE llm_enriched_at IS NOT NULL` rather than equality on the source
column, so existing real source semantics stay intact.
2026-05-10 12:11:45 -05:00
c9202744a5 fix(insights): UI contrast + DeepSeek empty-string env-var fallback
UI:
- Type-specific evidence cards (red/amber/orange/rose/purple/cyan/
  indigo/blue) had `dark:text-{color}-300` labels and
  `dark:bg-{color}-800 dark:text-{color}-100` badges that washed out on
  the dark background. Bumped 31 label shades to -200 and switched the
  three problem badges (Above manufacturer spec, Investigate upstream
  first, own fleet, crit) to solid -600 with white text.
- upstream_degradation card got brighter device-link text and a
  hover state in dark mode.

DeepSeek:
- Empty-string DEEPSEEK_MODEL / DEEPSEEK_BASE_URL env vars were not
  treated as missing — `||` only short-circuits on nil/false — so a
  blank secret would override the @default_model and the API would
  reject `model: ""`. Both runtime.exs and the client itself now treat
  blank values as missing.
2026-05-10 12:07:53 -05:00
b1c803e1d2 test: cut runtime on slow tests, fix time-sensitive flake
Performance:
- HealthController: honor `:timeout` from `:redis` config (was hardcoded
  2s) so the 503-when-unreachable test exits immediately
- DevicePollerBranches: drop `collect_events` window 1500ms → 100ms
  (PubSub broadcasts are synchronous from `run_perform`)
- Towerops.Unused: cache xref analysis in `:persistent_term` keyed by
  BEAM mtimes so repeated `mix unused` invocations skip re-analysis
- DiscoveryWorker: make wait_loop interval configurable via
  `:discovery_wait_interval_ms` (25ms in tests vs 500ms in prod)
- ProfileWatcher: configurable `:profile_reloader` so tests can stub
  the YAML reload instead of paying the full disk-load cost
- PingExecutor: tag the loopback test `:network` (it shells out to
  `ping`); coverage already provided by 4 other `:network`-tagged tests

Flake fix:
- FrequencyChange tests used hardcoded `~U[2026-05-09 12:00:00Z]`,
  which now falls outside the rule's 24h lookback window. Switched to
  `DateTime.utc_now()` so scans stay inside the window regardless of
  clock progression.
2026-05-10 10:39:36 -05:00
b6513ec301 feat(insights): OpticalRxLow rule for fiber transceivers
Fiber operators care a lot about optical Rx power because connectors
get dirty, splices degrade, and patch panels develop bend-radius
problems. The data is already in snmp_transceiver_readings — this
rule turns it into actionable insights.

- Towerops.Recommendations.Rules.OpticalRxLow — single query selects
  the latest reading per transceiver (last hour), filters to those
  with rx_power_dbm <= -28, joins back to the device. Critical at
  <= -32 dBm. One insight per (device, port) via metadata.dedup_key.
- Insight.@valid_types extended with optical_rx_low.
- LLM prompt hint walks the operator through the standard ladder:
  clean SC/LC connector with a one-click cleaner, inspect bend
  radius, verify splices, then replace the transceiver.
- UI: cyan card with port, Rx power, Tx power, transceiver type, and
  wavelength.
- Wired into RecommendationsRunWorker.

8 new tests.
2026-05-09 17:56:38 -05:00
4804dfea5b feat(insights): UpstreamDegradation multi-AP correlation rule
When 50%+ of the access points at one site simultaneously have
Preseem QoE score <60, root cause is almost always a shared upstream
problem (backhaul, transit, DNS, DHCP) — not per-AP RF. This rule
emits one site-level insight that points the operator at the right
investigation layer instead of letting them chase per-AP RF
symptoms across many APs at the same site.

- Towerops.Recommendations.Rules.UpstreamDegradation — single query
  pulls all matched Preseem APs grouped by site_id. Critical when
  100% of APs at the site are degraded; warning otherwise. Lists
  every degraded AP with its QoE score in metadata.
- Insight.@valid_types extended with upstream_degradation.
- LLM prompt hint tells the model to instruct the operator to
  investigate upstream FIRST and to NOT chase per-AP RF symptoms.
- UI: indigo card with degraded/total AP counts, an "Investigate
  upstream first" badge, and a clickable list of affected APs.
- Wired into RecommendationsRunWorker FIRST in the @rules list so
  the upstream signal surfaces before per-AP RF rules.

Also fixes two flakes:
- device_live/show_test.exs is now async: false. It inserts to
  InterfaceStat which is a TimescaleDB hypertable; concurrent inserts
  deadlock on chunk-table locks under parallel SQL-sandbox tests
  (ERROR 40P01 deadlock_detected in setup).
- Cleared an unused-variable warning in device_overheating_test.exs.

8 new tests for UpstreamDegradation.
2026-05-09 17:51:41 -05:00
5eb1acf9d4 feat(insights): device-overheating recommendation
Operators frequently misdiagnose thermal throttling as RF problems
and waste truck rolls. This rule reads the hottest temperature
sensor per device and, when the device is linked to a Preseem AP
with degraded QoE, makes the thermal-correlation explicit so the
fix is ventilation/shading rather than antenna alignment.

- Towerops.Recommendations.Rules.DeviceOverheating — single query
  pulls the hottest temperature sensor per device above 65 °C; one
  insight per device. Critical at 75 °C+. Includes the linked
  Preseem qoe_score in metadata when present.
- Insight.@valid_types extended with device_overheating.
- LLM prompt hint tells the model to recommend ventilation/fan/
  shading inspection BEFORE any RF action when QoE is also low.
- UI: red card with sensor name, temperature, linked QoE score,
  and an "Above manufacturer spec" badge for >=75 °C.

Wired into RecommendationsRunWorker. 8 new tests.
2026-05-09 17:40:22 -05:00
b735adc067 feat(insights): RF-aware channel-conflict — multi-source noise + Preseem
Same-channel conflicts at one site are not all equal. A pair of APs
sharing a channel in a clean RF environment is a planning issue;
the same pair in a contaminated environment is an outage waiting to
happen. This commit makes the rule reason across sources before
deciding urgency.

- Towerops.Snmp.RfHealth — cross-source helper that combines:
  * Most recent noise-floor / noise sensor on the AP (dBm)
  * Preseem rf_score and qoe_score from the linked AccessPoint row
  * Average CPE SNR and client count from wireless_clients
  Returns a snapshot map and an interference_severity/1 classifier
  (:critical | :warning | :ok). Critical when noise floor >= -80 dBm
  OR Preseem rf_score < 4.0; warning when avg CPE SNR < 15 with
  active clients.
- OwnFleetChannelConflict now enriches each conflicting AP with its
  noise floor, rf_score, qoe_score, avg CPE SNR, CPE count, and a
  per-AP rf_severity. Urgency promotes to critical when 3+ APs share
  a channel OR any conflicting AP shows critical RF health.
- UI: amber card now renders an evidence table with width, noise,
  rf_score, CPE SNR, and CPE count per AP, plus a "Critical RF
  environment detected" badge when has_critical_rf is true.

Also strengthens the previous flake fix on agent_live_test.exs:567
by switching to Req.Test shared mode (the test is sync), so the
ReleaseChecker stub is visible to the LiveView process regardless of
when it spawns relative to setup. The earlier Req.Test.allow approach
relied on stub timing that didn't always hold under full-suite load.

11 new tests across RfHealth + the enriched rule. LLM enrichment
automatically picks up the richer metadata via the existing Phase 1
worker.
2026-05-09 17:25:26 -05:00
18f62022f5 feat(snmp): canonical AP radio state + own-fleet channel conflict rule
Adds a vendor-agnostic radio-state extractor that lets the frequency-
aware rules actually fire on production data without per-vendor
plumbing for each profile.

- Towerops.Snmp.RadioState — pure module that maps a frequency in MHz
  to the corresponding 802.11 channel across 2.4 GHz, 5 GHz, and 6 GHz
  bands; reads the freshest "frequency" sensor (already emitted by
  Cambium/MikroTik/AirOS/Mimosa/Exalt/etc. profiles) and persists the
  AP-level current_frequency_mhz, current_channel, last_radio_seen_at
  fields onto snmp_devices.
- DevicePollerWorker now calls RadioState.update_from_sensors/1 after
  each sensor poll so frequency-aware rules read a single canonical
  source instead of joining sensors at query time.
- Towerops.Recommendations.Rules.OwnFleetChannelConflict — new rule
  that needs no neighbor-scan collection: detects when 2+ of our own
  APs at the same site share a channel (self-interference). One
  insight per (site, channel) with a clickable list of conflicting
  APs and their channel widths. Critical when 3+ APs share a channel.
- Insight.@valid_types extended with own_fleet_channel_conflict.
- UI: amber card showing channel, frequency, AP count, and the linked
  list of conflicting devices.

Wired into the existing hourly RecommendationsRunWorker. LLM
enrichment automatically applies via the Phase 1 worker.
2026-05-09 17:14:48 -05:00
91f23416e4 feat(insights): SectorOverload + CpeRealign recommendation rules
Two new multi-source rules using data we already collect, both wired
into the existing hourly RecommendationsRunWorker.

SectorOverload — fires when a Preseem-monitored AP has <25% free
airtime AND active subscribers. Critical urgency when free airtime
drops below 15%, OR when free airtime <25% AND QoE <50. Reuses the
airtime/subscriber/QoE data already pulled by PreseemSyncWorker so
no new collection is needed. UI: orange evidence card with free
airtime, subscriber count, QoE score, and AP model.

CpeRealign — fires when a wireless client has BOTH signal_strength
<=-78 dBm AND SNR <=18 dB seen in the last 2 hours. This is the
classic alignment / obstruction signature, distinct from the
existing single-metric wireless_signal_weak / wireless_snr_low
alerts produced by WirelessInsightWorker. One insight per (AP, CPE)
pair via a metadata.dedup_key. Critical when signal <=-88 OR SNR
<=10. UI: rose evidence card with signal, SNR, TX/RX rate, distance,
hostname.

Insights.insert_insight_if_new/1 dedup logic now prefers a
metadata.dedup_key over device_id when explicitly set, allowing
per-CPE insights without collapsing multiple CPEs on the same AP
into a single insight. Existing rules without dedup_key are
unaffected.

Insight.@valid_types extended with sector_overload and cpe_realign.

LLM enrichment automatically applies to both new types via the
existing Phase 1 worker — no extra wiring.

Also adds k8s/secrets.yaml to .gitignore so operators can drop a
local Secret manifest with real values, kubectl apply manually, and
never accidentally commit it. Documented in k8s/README.md.
2026-05-09 17:07:15 -05:00
f41f5fa1aa feat(insights): AP frequency-change recommendation rule
When a wireless AP has a strong same-channel neighbor (RSSI >= -75 dBm)
and a measurably cleaner alternative channel exists in the same band,
the system now emits an ap_frequency_change insight with a specific
recommended channel and the top interfering BSSIDs as evidence.

- New schema wireless_neighbor_scans (append-only) storing observed
  neighboring APs per device: bssid, ssid, channel, frequency_mhz,
  channel_width_mhz, rssi_dbm, is_own_fleet, seen_at
- New columns on snmp_devices: current_channel, current_frequency_mhz,
  current_channel_width_mhz, last_radio_seen_at — populated by vendor
  SNMP profiles for APs that expose wireless OIDs
- Towerops.Recommendations.Rules.FrequencyChange — pure rule that scores
  each candidate channel using sum-of-linear-power over the last 24h of
  neighbor scans, recommends the cleanest if it beats the current by at
  least 6 dB. Critical urgency when current score >= 100.
- Towerops.Workers.RecommendationsRunWorker — hourly Oban cron, iterates
  organizations and runs all rules, uses insert_insight_if_new/1 for
  idempotent upsert
- LiveView /insights renders a purple frequency-change card with current
  vs recommended channel, dB-cleaner badge, and a top-offender list
  (BSSID, SSID, RSSI, own-fleet flag)
- LLM enrichment automatically applies to the new insight type

Also fixes two pre-existing flaky tests that surfaced in CI:

- test/towerops_web/live/agent_live_test.exs:567 — now sets an explicit
  Req.Test transport_error stub for ReleaseChecker and uses Req.Test.allow
  so the LiveView process inherits it. Previously relied on absence of a
  stub, which leaked across tests and produced different behavior under
  full-suite parallel runs.
- test/towerops_web/telemetry_test.exs:140 — relaxes the "no log" check
  to refute the specific messages the function under test would emit,
  rather than asserting an empty capture_log/1 buffer (which catches
  stray logs from concurrent tests, e.g. Postgrex sandbox disconnects).
2026-05-09 16:56:19 -05:00
03f364a956 feat(insights): LLM-powered insight enrichment
Adds an optional plain-language summary and recommended action to every
active insight. A new Oban cron worker runs every 5 minutes, picks up
unenriched insights, and asks the configured LLM to restate the finding
and suggest one specific action grounded in the structured metadata.

- Migration: adds llm_summary, recommended_action, llm_model,
  llm_enriched_at columns to preseem_insights
- Towerops.LLM context with swappable behaviour; real client uses Req
  with the standard Req.Test plug for test isolation
- MockClient + InsightPrompt module (builds chat messages, parses JSON
  with code-fence stripping and a raw-text fallback)
- Worker logs and skips on rate limits / missing key / parse errors so
  insights still render without an AI summary when the LLM is unavailable
- Optional towerops-llm k8s secret with example template; deployment.yaml
  references it as optional in both init and main containers
- UI renders summary + recommended action callout on /insights and on
  the org-level insights page
2026-05-09 16:41:48 -05:00
2269f38fc5 test: lift coverage 78.42% → 79.59% with focused unit + integration tests
Adds tests across previously-uncovered or under-covered modules:

- Mix tasks: backfill_checks, copy_mibs, import_mibs (new test files)
- Status pages: StatusIncident changeset (new file)
- Webhook signature verification on AgentReleaseWebhookController
- CloudflareBanWorker: HTTP-stubbed success / 5xx / transport-error paths
- FirmwareVersionFetcherWorker: stubbed perform/0 success + non-200 + tx error
- Geoip.Import: production --production code path (HTTP-stubbed)
- AgentLive.Show: superuser restart_agent + update_agent + non-superuser paths
- NetworkMapLive: node_clicked, deep-link node, topology_updated PubSub
- PreseemInsightsLive: toggle_select, deselect_all, filter, dismiss-twice
- AlertQuery: 1-arity defaults variants
- ProfileWatcher: yaml + reload-trigger event passthrough
- MobileAuthController: verify_qr_token success + get/revoke session
- HealthController: redis-configured branch
- RedisHealthCheck: un-tag :integration tests now that Redis is local
- FourOhFourTracker: un-skip module (Redis available)
- PingExecutor: un-tag local-only tests + KeyError surface + clamp branch
- CheckExecutorWorker: dispatch tcp/dns/ssl/ping branches
- UserResetPasswordController: drop dead edit/update + their template

Also removes dead code: edit/update actions on UserResetPasswordController
and the unused edit.html.heex template — both routed via LiveView.
2026-05-07 18:40:43 -05:00
22f5c3ed63 test: lift coverage to 78.42% and link Ansible collection in docs
- Add Codeberg Ansible collection link to API/GraphQL docs and Terraform guide
- Un-skip the previously-disabled DevicePollerWorker tests that still pass
- Expand MibController, CnMaestro.Sync, CheckWorker, ActivityFeed,
  MobileController, JobCleanupTask tests
- New UploadMibs Mix task test covering arg validation and upload paths
- Set :mib_dir in test config so MibController upload/delete flows can
  exercise the real on-disk paths against a writable tmp directory
2026-05-07 16:48:53 -05:00
973c08bc5c docs(api): add Coverages, Maintenance Windows, Schedules, Escalation Policies
The /docs/api and /docs/graphql pages were missing four resource
families that the API has shipped:

- Coverages (RF coverage prediction with async compute, recompute,
  KMZ download)
- Maintenance Windows (one-off + recurring with RRULE)
- On-Call Schedules (top-level CRUD + layers, members, overrides,
  on_call lookup)
- Escalation Policies (CRUD + rules + targets)

Both REST and GraphQL pages now have sidebar entries and example
requests for each.
2026-05-07 16:08:09 -05:00
0e2cc6b7ce chore(coverage,types): clean dialyzer + new tests + ETH canopy provider
Dialyzer:
* Add Lidar.Tile @type, narrow several @specs to match success-typing,
  prefix discarded function returns with `_ =`, fix the broken
  File.stream! call in the ms-buildings worker (it was causing dialyzer
  to mark every private fun unreachable), broaden Antenna.spec key
  types from `float()` to `number()` to match catalog int literals.
* Add :geo, :geo_postgis, :db_connection to plt_add_apps so the dep
  PLT actually contains the modules we use.
* Result: 0 unsuppressed dialyzer warnings (was 50+ in project code).

Tree canopy:
* Replace the dead landfire.gov URL with the ETH Zurich Global
  Canopy Height 10 m product. ETH publishes proper COGs with
  overviews on libdrive.ethz.ch — confirmed working with
  /vsicurl/ byte-range reads. The module now picks the 3°×3° tile
  containing the bbox centroid, with both a `:metres` decoder
  (ETH Byte raster, NoData=255) and a `:landfire_evh` decoder for
  operators self-hosting LANDFIRE EVH COGs.
* Re-enable canopy by default now that there's a working source.

Tests (+105 tests, 0 failures):
* CoverageLive Index/Show/Form/Map — empty/loaded/auth states,
  LiveView event handlers, formatters.
* Coverages.Building changeset + helpers.
* Coverages.TreeCanopy provider tile-URL builder and decoders.
* Workers.MsBuildingsImportWorker (full import flow incl. blank
  lines, MultiPolygon, idempotency, hash fallback) — also fixed
  the partial-index ON CONFLICT bug that prevented imports from
  ever working: the unique index has WHERE ms_footprint_id IS
  NOT NULL, so the worker now uses an :unsafe_fragment conflict
  target that includes the predicate.
* GraphQLDocsController smoke test (was 0%).

Coverage rose from 73.19% → 74% with the major project modules I
touched all moved from 0% / <50% to >75%.
2026-05-07 09:11:17 -05:00
4e391b3ecf feat(coverage): cnHeat-style full-bleed map on the show page
Rewire the single-coverage show page to mount MultiCoverageMap (the
same hook the org map uses) with one coverage in the array, and lift
the cnHeat-style overlay panels — LOS/NLOS toggle + height slider on
the left, opacity + RSSI filter on the bottom, distance probe on
click. Parameters move into a togglable right-side drawer so the map
dominates the viewport.

Show LiveView gains toggle_params / probe_point / close_probe /
set_probe_units handlers and delegates the formatters to
CoverageLive.Map. Site is now preloaded so coverages_payload/1 can
read site.name without an Ecto.AssociationNotLoaded crash that was
silently nuking the data-coverages JSON encode.

Also fixes a real bug in coverage_hooks.ts: pngForCoverage was
returning before the antenna-marker creation block, leaving the
marker code as unreachable dead code — towers never rendered. Moved
the marker creation into addCoverageLayer where it belongs and added
a no-PNG guard so a missing png_path logs instead of silently
failing the L.imageOverlay constructor.
2026-05-07 08:27:27 -05:00
38375dcebb chore: migrate agent repo references from GitHub to Codeberg
Updates all docs, UI copy, release checker, proto go_package,
container image refs, and tests to point at
codeberg.org/towerops-agent/towerops-agent. Also deletes stray
empty package.json/package-lock.json (Phoenix uses esbuild — no
npm) and an unused test_encode_decode.exs scratch file.
2026-05-07 07:44:48 -05:00
4a8d8cb208 feat(coverage): write rasters to NFS-backed dir, serve via dedicated static plug
Coverage rasters were being written to priv/static/coverage on the pod's
ephemeral filesystem — fine on a single-pod dev box, broken under k8s
where pods share NFS at /data and a restart wipes computed predictions.

* New :coverage_storage_dir config (defaults to priv/static/coverage for
  dev/test, "/data/coverage" in production).
* Raster.output_dir/absolute_raster_path read this config and resolve
  {otp_app, rel} or absolute paths.
* Endpoint now mounts a dedicated Plug.Static at /coverage from that
  same config — replicas all serve the same NFS-backed file regardless
  of which one ran the compute.
* Drop "coverage" from static_paths/0 since the dedicated plug owns
  that prefix now.
2026-05-07 07:38:05 -05:00