Same-channel conflicts at one site are not all equal. A pair of APs
sharing a channel in a clean RF environment is a planning issue;
the same pair in a contaminated environment is an outage waiting to
happen. This commit makes the rule reason across sources before
deciding urgency.
- Towerops.Snmp.RfHealth — cross-source helper that combines:
* Most recent noise-floor / noise sensor on the AP (dBm)
* Preseem rf_score and qoe_score from the linked AccessPoint row
* Average CPE SNR and client count from wireless_clients
Returns a snapshot map and an interference_severity/1 classifier
(:critical | :warning | :ok). Critical when noise floor >= -80 dBm
OR Preseem rf_score < 4.0; warning when avg CPE SNR < 15 with
active clients.
- OwnFleetChannelConflict now enriches each conflicting AP with its
noise floor, rf_score, qoe_score, avg CPE SNR, CPE count, and a
per-AP rf_severity. Urgency promotes to critical when 3+ APs share
a channel OR any conflicting AP shows critical RF health.
- UI: amber card now renders an evidence table with width, noise,
rf_score, CPE SNR, and CPE count per AP, plus a "Critical RF
environment detected" badge when has_critical_rf is true.
Also strengthens the previous flake fix on agent_live_test.exs:567
by switching to Req.Test shared mode (the test is sync), so the
ReleaseChecker stub is visible to the LiveView process regardless of
when it spawns relative to setup. The earlier Req.Test.allow approach
relied on stub timing that didn't always hold under full-suite load.
11 new tests across RfHealth + the enriched rule. LLM enrichment
automatically picks up the richer metadata via the existing Phase 1
worker.
Two new multi-source rules using data we already collect, both wired
into the existing hourly RecommendationsRunWorker.
SectorOverload — fires when a Preseem-monitored AP has <25% free
airtime AND active subscribers. Critical urgency when free airtime
drops below 15%, OR when free airtime <25% AND QoE <50. Reuses the
airtime/subscriber/QoE data already pulled by PreseemSyncWorker so
no new collection is needed. UI: orange evidence card with free
airtime, subscriber count, QoE score, and AP model.
CpeRealign — fires when a wireless client has BOTH signal_strength
<=-78 dBm AND SNR <=18 dB seen in the last 2 hours. This is the
classic alignment / obstruction signature, distinct from the
existing single-metric wireless_signal_weak / wireless_snr_low
alerts produced by WirelessInsightWorker. One insight per (AP, CPE)
pair via a metadata.dedup_key. Critical when signal <=-88 OR SNR
<=10. UI: rose evidence card with signal, SNR, TX/RX rate, distance,
hostname.
Insights.insert_insight_if_new/1 dedup logic now prefers a
metadata.dedup_key over device_id when explicitly set, allowing
per-CPE insights without collapsing multiple CPEs on the same AP
into a single insight. Existing rules without dedup_key are
unaffected.
Insight.@valid_types extended with sector_overload and cpe_realign.
LLM enrichment automatically applies to both new types via the
existing Phase 1 worker — no extra wiring.
Also adds k8s/secrets.yaml to .gitignore so operators can drop a
local Secret manifest with real values, kubectl apply manually, and
never accidentally commit it. Documented in k8s/README.md.
When a wireless AP has a strong same-channel neighbor (RSSI >= -75 dBm)
and a measurably cleaner alternative channel exists in the same band,
the system now emits an ap_frequency_change insight with a specific
recommended channel and the top interfering BSSIDs as evidence.
- New schema wireless_neighbor_scans (append-only) storing observed
neighboring APs per device: bssid, ssid, channel, frequency_mhz,
channel_width_mhz, rssi_dbm, is_own_fleet, seen_at
- New columns on snmp_devices: current_channel, current_frequency_mhz,
current_channel_width_mhz, last_radio_seen_at — populated by vendor
SNMP profiles for APs that expose wireless OIDs
- Towerops.Recommendations.Rules.FrequencyChange — pure rule that scores
each candidate channel using sum-of-linear-power over the last 24h of
neighbor scans, recommends the cleanest if it beats the current by at
least 6 dB. Critical urgency when current score >= 100.
- Towerops.Workers.RecommendationsRunWorker — hourly Oban cron, iterates
organizations and runs all rules, uses insert_insight_if_new/1 for
idempotent upsert
- LiveView /insights renders a purple frequency-change card with current
vs recommended channel, dB-cleaner badge, and a top-offender list
(BSSID, SSID, RSSI, own-fleet flag)
- LLM enrichment automatically applies to the new insight type
Also fixes two pre-existing flaky tests that surfaced in CI:
- test/towerops_web/live/agent_live_test.exs:567 — now sets an explicit
Req.Test transport_error stub for ReleaseChecker and uses Req.Test.allow
so the LiveView process inherits it. Previously relied on absence of a
stub, which leaked across tests and produced different behavior under
full-suite parallel runs.
- test/towerops_web/telemetry_test.exs:140 — relaxes the "no log" check
to refute the specific messages the function under test would emit,
rather than asserting an empty capture_log/1 buffer (which catches
stray logs from concurrent tests, e.g. Postgrex sandbox disconnects).
Adds an optional plain-language summary and recommended action to every
active insight. A new Oban cron worker runs every 5 minutes, picks up
unenriched insights, and asks the configured LLM to restate the finding
and suggest one specific action grounded in the structured metadata.
- Migration: adds llm_summary, recommended_action, llm_model,
llm_enriched_at columns to preseem_insights
- Towerops.LLM context with swappable behaviour; real client uses Req
with the standard Req.Test plug for test isolation
- MockClient + InsightPrompt module (builds chat messages, parses JSON
with code-fence stripping and a raw-text fallback)
- Worker logs and skips on rate limits / missing key / parse errors so
insights still render without an AI summary when the LLM is unavailable
- Optional towerops-llm k8s secret with example template; deployment.yaml
references it as optional in both init and main containers
- UI renders summary + recommended action callout on /insights and on
the org-level insights page
Adds 8 tests targeting AgentChannel branches missed by existing tests:
- resolve_any_stuck_down_alerts: device_down and device_up stuck alerts
resolved on next successful poll
- handle_in result with :PING job_type (deprecated SnmpResult path used
by older agents) creates a monitoring check without crashing
- handle_live_poll_result with malformed job_id (missing reply_topic)
hits the warning catch-all branch
- build_mikrotik_job is included in the initial dispatch when the device
has a discovered SNMP profile whose manufacturer is MikroTik or whose
sys_descr mentions RouterOS, and excluded when mikrotik_enabled is false
- process_additional_polling_data path with extra OIDs runs without
crashing the channel
- Adds TcpExecutor receive timeout test using a server that accepts
but never responds, exercising recv timeout branch
- Adds Preseem.get_device_qoe_data tests covering both no-APs and
with-metrics paths through the AP query + metric formatting
Adds 4 tests that drive the nil → nil fallbacks in
load_single_sensor_chart_data, load_storage_chart_data,
load_interface_errors_chart_data, and load_interface_traffic_chart_data
by navigating to graph routes with non-existent IDs.
Adds direct call to MobileController.get_device with a snmp_device that
has interfaces — drives the Enum.map over interfaces in
format_equipment_details that previously had no fixture coverage.
Adds tests across five files:
- Towerops.Monitoring.CheckQuery (full coverage of all composable filter
fragments via inspect+Repo.all)
- ToweropsWeb.Admin.SecurityLive.Index change_filter event for both
permanent and temporary filter values
- Towerops.Agents.AgentCache cleanup_expired handle_info path that
prunes stale device entries while preserving :global_default
- ToweropsWeb.Api.V1.AgentReleaseWebhookController happy path
(notified/skipped/version) and error path (Codeberg 503)
- ToweropsWeb.DeviceLive.Index reorder_device / reorder_site
unauthorized branches via cross-org device/site fixtures
- New `sync_worker_perform_test.exs` drives every cron sync worker's
`perform/1` body via `Req.Test` stubs, exercising the {:error, _}
branch of `sync_integration/1` for Sonar, Splynx, Visp, and NetBox,
plus the should_sync?-skip path.
- OnboardingLive: covers `finish` event, `skip` from :agent (both
trigger complete_onboarding), and `go_to_step :agent` (which calls
the previously-untested `load_agent_token/1`).
- GaiiaInsightWorker: adds a test that drives `generate_mismatch_insights/2`
by inserting an inventory item whose ip_address disagrees with its
mapped device.
Coverage: 86.52% → 86.61%.
Adds focused tests across multiple modules to push toward the 87% threshold:
- CoreComponents: pagination/1 (multi-page, ellipsis, params),
list/1 (slot rendering), and signal_badge/1 across all RSSI bands.
- ApiDocs/GraphQLDocs controllers: bypass-the-router pattern so the
"scope-with-organization" branches are actually exercised (the public
routes are unscoped, so the org branch is unreachable via get/2).
- Mix.Tasks.Unused: text + JSON output, --only / --skip filters.
- Mix.Tasks.JobCleanupTask: prod path with phoenix snmp enabled vs
disabled (uses :job_cleanup_settle_ms = 0 to avoid the 1s sleep).
- Member resolver: success path for remove/update_role on non-owner
membership, and the changeset-error branch for an invalid role.
- Check resolver: unauthenticated fallback clauses for list / get /
create / update / delete.
- RfLinks: nil-signal sort ordering, :unknown classification for
nil/nil signal+snr, :degraded filter, capacity_utilization edge
cases (max_rate 0/missing, both rates nil).
- StatusHelpers: status_emoji + title_with_status across green / red
branches (warning is unreachable given the active-alerts query).
- ApiTokens: spawn_async_update/2 path with env != :test.
- ImpactAnalysis: to_json/1 across nil and populated levels.
- Maintenance: query helpers (default-arity wrappers, active range)
and Show LiveView delete event redirecting to /maintenance.
- Coverage map: probe_point event handler.
- InvitationQuery: default-arity pending/1.
- Remove Mox.set_mox_global() in DPW extra test; rely on $callers
instead so concurrent vendor SNMP tests stop intermittently failing
with VerificationError.
- Make ToweropsWeb.Plugs.GraphQLIntrospectionTest async: false. It
mutates Application.put_env(:towerops, :env, :prod), which polluted
every async vendor test that reads :env via
Towerops.Snmp.Client.phoenix_snmp_disabled/0.
- Bump test pool queue_target/queue_interval. AgentChannelTest spawns
many `:proc_lib` channel processes that hold sandbox connections;
the default 50ms queue caused intermittent
'could not checkout the connection' errors.
- Tag real-System.cmd("ping") tests as :network so they're excluded
by default. Saves ~20s on every full run.
- Make JobCleanupTask's internal 1s settle sleep configurable via
:job_cleanup_settle_ms; the test overrides it to 0.
Plus the pending CoverageLive.Form edit-save / EIRP recompute and
TraceLive deep-link tests from before the compact.
