Adds a parallel LLM path that reviews the whole org snapshot — Preseem
APs, SNMP signals, backhaul, agents, and the insight types the rules
already covered — and emits novel observations as ai_observation
insights (source=ai). Existing rule-based insights and per-insight
enrichment continue unchanged.
- lib/towerops/llm/network_snapshot.ex builds the bounded snapshot
- lib/towerops/llm/network_insight_prompt.ex builds the prompt and
parses the LLM's JSON observations array
- lib/towerops/workers/ai_network_insight_worker.ex runs nightly at
03:30 UTC (after the rule pass + per-insight enrichment) and on
the superuser regenerate button
- preseem/insight.ex extends @valid_types with "ai_observation" and
@valid_sources with "ai"
- insights_live/index.ex includes the new worker in the regen list
and adds an "AI" source badge style
Bumps the SnmpKit Manager timeout-options test threshold from 500ms
to 3000ms (was flaking on loaded dev machines), and scopes
list_unenriched_insights/1 assertion in insights_test:573 to the
test's org so cross-suite leaks don't fail the count.
- ToweropsWeb.AgentChannel now runs sensor readings through
Towerops.Snmp.SensorScale.normalize/2 in both resolve_sensor_value/2
and the GraphQL publisher. Without this, MikroTik mtxrHl* deci-degree
readings (e.g. Culleoka reporting 294 -> 29.4°C) were stored and
alerted on at 294°C. The DevicePollerWorker path already did this;
only the agent path was missing it.
- Demote "Received SNMP result from agent" and "Processing polling
result for X" from info to debug — they fire every poll cycle on
every device and were dominating prod logs.
- /insights now shows a "Regenerate insights" button for superusers
that enqueues all seven insight workers (Preseem baselines,
capacity, device-health, Gaiia, system, wireless, LLM enrichment).
Non-superusers neither see the button nor can trigger the event.
- Dialyzer is clean: added :tools to plt_add_apps, gave
Preseem.Insight a @type t, pinned a few unmatched returns, and
suppressed a known MapSet opaque false-positive in Towerops.Unused.
- e2e: NODE_OPTIONS=--disable-warning=DEP0205 silences the Node 25
deprecation noise from Playwright's internal ESM loader.
Token-usage doesn't belong on the per-insight row — every consumer of
Towerops.LLM (insight enrichment today; billing-anomaly summaries,
monitoring narratives, fleet weekly reports tomorrow) needs to record
into one queryable place. Per-insight columns also made it impossible
to track LLM calls that aren't tied to an insight.
Schema: `llm_usage` keyed (organization_id, day, model, purpose) with a
unique index. organization_id is nullable for system-level analyses.
Each row carries running totals plus a request_count — callers UPSERT
and increment atomically.
API: `Towerops.LLM.Usage.record/1` for write, `summarize/2` for
read-side rollups. The worker now calls `Usage.record(%{... purpose:
"insight_enrichment"})` after each successful enrichment.
Removed: `llm_prompt_tokens` / `llm_completion_tokens` from
preseem_insights — same migration drops the columns and creates
llm_usage in one go. `apply_llm_enrichment` reverts to /3 (no usage
arg).
If a rule keeps re-firing the underlying condition, the dedup query
suppresses duplicate inserts but `inserted_at` of the live row stays
old — so a real 6-month-old issue and a one-off blip both linger
forever. Auto-resolving any active insight older than 7 days clears
the dashboard. Still-real issues will be re-created by the next rule
pass (dedup only checks `status: "active"`), so the operator sees
fresh evidence rather than stale text.
- Towerops.Workers.InsightExpiryWorker (queue: :maintenance) flips
`status` to "resolved" on rows where `inserted_at < now - 7d`.
- Window is configurable via `:max_age_days` for ops who want a
different cadence per env.
- Oban cron: 04:00 UTC nightly, after the 02:00 rule pass and 03:00
LLM enrichment.
UI: the AI-generated footer dropped " · deepseek-v4-pro" — model
identity is irrelevant to operators reviewing the recommendation.
Parser: stop letting raw JSON leak into the summary text when the LLM
returns an almost-but-not-quite-valid response.
- Strip ```json fences (already done) PLUS try a balanced-brace
extraction so a "Here's the JSON: {…}" preamble doesn't break
Jason.decode
- When all decode paths fail, regex-extract `"summary"` and `"action"`
fields from the text rather than dumping the whole blob with braces
intact
- Last-ditch sanitiser strips remaining brace/quote noise so the
rendered summary never shows literal `"summary"` keys
Maintenance: `Towerops.Maintenance.RepairLlmSummaries.run()` clears
`llm_enriched_at` on already-stored rows whose summaries contain JSON
artifacts so the next worker cycle re-enriches them with the new
parser.
Test: insights_live_events_test now asserts the AI-generated label is
present *and* that the model name does not leak into the rendered HTML.
Mix tasks aren't usable from the prod release image — `/app/bin/towerops`
ships without `mix`, and `Mix.shell()` raises at runtime. Move the work
to `Towerops.Maintenance.FixMikrotikTemperatureScaling` (plain module
with `run/0`, IO.puts + Logger.info) and keep the Mix.Task as a thin
dev-only delegator.
Run in prod via:
/app/bin/towerops eval 'Towerops.Maintenance.FixMikrotikTemperatureScaling.run()'
The CSS `capitalize` class only capitalises the first letter, so the
source filter pills and per-card badges showed "Ai" and "Snmp". Switch
to a `source_label/1` helper that returns "AI" / "SNMP" verbatim and
falls back to `String.capitalize/1` for the rest (Preseem, Gaiia,
System).
Some MikroTik models report `mtxrGaugeValue` (mtxrGaugeUnit=1, °C) in
deci-degrees, while others — like the CRS317 in upstream test data —
return whole degrees. The MIB doesn't disclose which, so the YAML
profile can't pick one literal divisor. Result: a sensor that should
read 56.92 °C was being stored and alerted on as 5,692 °C.
- Towerops.Snmp.SensorScale.normalize/2: divide-by-10 a temperature
reading until it falls below 150 °C, capped at 3 iterations. Other
sensor types pass through untouched.
- DevicePollerWorker.poll_simple_sensor wires the normaliser in after
the YAML divisor is applied, so live polling stops storing bogus
values immediately.
- DeviceOverheating rule now bounds the query at 150 °C as defense in
depth so a future scaling glitch never produces a 5,000 °C alert.
- mix towerops.fix_mikrotik_temperature_scaling backfills any rows
already in the DB by re-running normalize/2 over them.
- routeros.yaml profile gets a comment pointing at the runtime
normaliser instead of a wrong unconditional divisor.
Adds an "ai" pill to the source-filter strip on /insights and a small
blue "AI" badge on every card whose `llm_enriched_at` is set. Selecting
the pill returns every LLM-enriched insight regardless of underlying
source (preseem / snmp / gaiia / system).
`source: "ai"` is a virtual filter — `list_insights/2` translates it to
`WHERE llm_enriched_at IS NOT NULL` rather than equality on the source
column, so existing real source semantics stay intact.
UI:
- Type-specific evidence cards (red/amber/orange/rose/purple/cyan/
indigo/blue) had `dark:text-{color}-300` labels and
`dark:bg-{color}-800 dark:text-{color}-100` badges that washed out on
the dark background. Bumped 31 label shades to -200 and switched the
three problem badges (Above manufacturer spec, Investigate upstream
first, own fleet, crit) to solid -600 with white text.
- upstream_degradation card got brighter device-link text and a
hover state in dark mode.
DeepSeek:
- Empty-string DEEPSEEK_MODEL / DEEPSEEK_BASE_URL env vars were not
treated as missing — `||` only short-circuits on nil/false — so a
blank secret would override the @default_model and the API would
reject `model: ""`. Both runtime.exs and the client itself now treat
blank values as missing.
Track DeepSeek token spend per enriched insight so cost can be rolled
up by SQL — `SELECT date_trunc('day', llm_enriched_at), organization_id,
SUM(llm_prompt_tokens), SUM(llm_completion_tokens) FROM preseem_insights
GROUP BY 1, 2`.
- Migration adds llm_prompt_tokens + llm_completion_tokens to
preseem_insights
- DeepSeek client surfaces the API's `usage` block (prompt/completion/
total tokens) on the success response
- apply_llm_enrichment/4 takes a usage map and persists the two int
columns
- Worker plumbs the usage from the response through to the context
Replaces the previous :requires_redis skip with a real test double so
the suite stays self-contained — no external Redis service needed in
CI, but the call paths through RedisHealthCheck and FourOhFourTracker
still get exercised end-to-end.
- Towerops.Redix.Fake: GenServer that speaks PING / SADD / SCARD /
SISMEMBER / SMEMBERS / EXPIRE / DEL / FLUSHDB. start_link refuses
anything other than localhost:6379 so failure-path tests still fail
the way they did against real Redix.
- Application supervisor starts the fake under name Towerops.Redix in
test env (instead of the dummy Agent stub).
- RedisHealthCheck and FourOhFourTracker now route Redix calls through
Application.get_env(:towerops, :redix_module, Redix). Test config
points it at the fake.
- Drops the test_helper :requires_redis exclusion and the @moduletag
on the two test files; FourOhFourTrackerTest setup just resets the
fake between runs.
When a user flips monitoring_enabled true → false on a device, any
unresolved alerts on that device are now resolved silently — the
operator has deliberately stopped monitoring it, so leaving stale
"device down" pages around is noise.
- Alerts.resolve_active_alerts_for_device/1: bulk-resolves via
resolve_alert_silent so PagerDuty/email don't fire
- Devices.handle_monitoring_changes hooks it on the disable transition
Performance:
- HealthController: honor `:timeout` from `:redis` config (was hardcoded
2s) so the 503-when-unreachable test exits immediately
- DevicePollerBranches: drop `collect_events` window 1500ms → 100ms
(PubSub broadcasts are synchronous from `run_perform`)
- Towerops.Unused: cache xref analysis in `:persistent_term` keyed by
BEAM mtimes so repeated `mix unused` invocations skip re-analysis
- DiscoveryWorker: make wait_loop interval configurable via
`:discovery_wait_interval_ms` (25ms in tests vs 500ms in prod)
- ProfileWatcher: configurable `:profile_reloader` so tests can stub
the YAML reload instead of paying the full disk-load cost
- PingExecutor: tag the loopback test `:network` (it shells out to
`ping`); coverage already provided by 4 other `:network`-tagged tests
Flake fix:
- FrequencyChange tests used hardcoded `~U[2026-05-09 12:00:00Z]`,
which now falls outside the rule's 24h lookback window. Switched to
`DateTime.utc_now()` so scans stay inside the window regardless of
clock progression.
Fiber operators care a lot about optical Rx power because connectors
get dirty, splices degrade, and patch panels develop bend-radius
problems. The data is already in snmp_transceiver_readings — this
rule turns it into actionable insights.
- Towerops.Recommendations.Rules.OpticalRxLow — single query selects
the latest reading per transceiver (last hour), filters to those
with rx_power_dbm <= -28, joins back to the device. Critical at
<= -32 dBm. One insight per (device, port) via metadata.dedup_key.
- Insight.@valid_types extended with optical_rx_low.
- LLM prompt hint walks the operator through the standard ladder:
clean SC/LC connector with a one-click cleaner, inspect bend
radius, verify splices, then replace the transceiver.
- UI: cyan card with port, Rx power, Tx power, transceiver type, and
wavelength.
- Wired into RecommendationsRunWorker.
8 new tests.
When 50%+ of the access points at one site simultaneously have
Preseem QoE score <60, root cause is almost always a shared upstream
problem (backhaul, transit, DNS, DHCP) — not per-AP RF. This rule
emits one site-level insight that points the operator at the right
investigation layer instead of letting them chase per-AP RF
symptoms across many APs at the same site.
- Towerops.Recommendations.Rules.UpstreamDegradation — single query
pulls all matched Preseem APs grouped by site_id. Critical when
100% of APs at the site are degraded; warning otherwise. Lists
every degraded AP with its QoE score in metadata.
- Insight.@valid_types extended with upstream_degradation.
- LLM prompt hint tells the model to instruct the operator to
investigate upstream FIRST and to NOT chase per-AP RF symptoms.
- UI: indigo card with degraded/total AP counts, an "Investigate
upstream first" badge, and a clickable list of affected APs.
- Wired into RecommendationsRunWorker FIRST in the @rules list so
the upstream signal surfaces before per-AP RF rules.
Also fixes two flakes:
- device_live/show_test.exs is now async: false. It inserts to
InterfaceStat which is a TimescaleDB hypertable; concurrent inserts
deadlock on chunk-table locks under parallel SQL-sandbox tests
(ERROR 40P01 deadlock_detected in setup).
- Cleared an unused-variable warning in device_overheating_test.exs.
8 new tests for UpstreamDegradation.
Operators frequently misdiagnose thermal throttling as RF problems
and waste truck rolls. This rule reads the hottest temperature
sensor per device and, when the device is linked to a Preseem AP
with degraded QoE, makes the thermal-correlation explicit so the
fix is ventilation/shading rather than antenna alignment.
- Towerops.Recommendations.Rules.DeviceOverheating — single query
pulls the hottest temperature sensor per device above 65 °C; one
insight per device. Critical at 75 °C+. Includes the linked
Preseem qoe_score in metadata when present.
- Insight.@valid_types extended with device_overheating.
- LLM prompt hint tells the model to recommend ventilation/fan/
shading inspection BEFORE any RF action when QoE is also low.
- UI: red card with sensor name, temperature, linked QoE score,
and an "Above manufacturer spec" badge for >=75 °C.
Wired into RecommendationsRunWorker. 8 new tests.
Generic prompts produce generic advice. This commit adds per-type
expert hints appended to the base system prompt, listing the metadata
field names the rule provides plus short domain heuristics so the
LLM has the context it needs to write a tight, specific recommendation.
Hints cover: ap_frequency_change, own_fleet_channel_conflict,
sector_overload, cpe_realign, wireless_signal_weak, wireless_snr_low,
backhaul_over_capacity, backhaul_near_capacity, qoe_degradation,
capacity_saturation. Unknown types fall back to the base prompt.
No change to rules, schema, or worker — this lifts every existing
insight's enrichment quality on the next cron run.
Also fixes two recurring flaky tests:
- agent_live_test.exs:567 now mirrors the companion show_test.exs
pattern of accepting any of the valid update-flow flashes (Failed
/ Update command sent / No binary available). The previous
Req.Test stub-and-allow approach was itself flaky because both
test files stub the same module name and ownership tracking can
leak under full-suite parallelism.
- telemetry_test.exs is now async: false. Its handle_endpoint_stop
describe block mutates global Logger.level which races with other
async tests; running this single module sync removes the race
without sacrificing the per-test setup/restore pattern.
Same-channel conflicts at one site are not all equal. A pair of APs
sharing a channel in a clean RF environment is a planning issue;
the same pair in a contaminated environment is an outage waiting to
happen. This commit makes the rule reason across sources before
deciding urgency.
- Towerops.Snmp.RfHealth — cross-source helper that combines:
* Most recent noise-floor / noise sensor on the AP (dBm)
* Preseem rf_score and qoe_score from the linked AccessPoint row
* Average CPE SNR and client count from wireless_clients
Returns a snapshot map and an interference_severity/1 classifier
(:critical | :warning | :ok). Critical when noise floor >= -80 dBm
OR Preseem rf_score < 4.0; warning when avg CPE SNR < 15 with
active clients.
- OwnFleetChannelConflict now enriches each conflicting AP with its
noise floor, rf_score, qoe_score, avg CPE SNR, CPE count, and a
per-AP rf_severity. Urgency promotes to critical when 3+ APs share
a channel OR any conflicting AP shows critical RF health.
- UI: amber card now renders an evidence table with width, noise,
rf_score, CPE SNR, and CPE count per AP, plus a "Critical RF
environment detected" badge when has_critical_rf is true.
Also strengthens the previous flake fix on agent_live_test.exs:567
by switching to Req.Test shared mode (the test is sync), so the
ReleaseChecker stub is visible to the LiveView process regardless of
when it spawns relative to setup. The earlier Req.Test.allow approach
relied on stub timing that didn't always hold under full-suite load.
11 new tests across RfHealth + the enriched rule. LLM enrichment
automatically picks up the richer metadata via the existing Phase 1
worker.
Adds a vendor-agnostic radio-state extractor that lets the frequency-
aware rules actually fire on production data without per-vendor
plumbing for each profile.
- Towerops.Snmp.RadioState — pure module that maps a frequency in MHz
to the corresponding 802.11 channel across 2.4 GHz, 5 GHz, and 6 GHz
bands; reads the freshest "frequency" sensor (already emitted by
Cambium/MikroTik/AirOS/Mimosa/Exalt/etc. profiles) and persists the
AP-level current_frequency_mhz, current_channel, last_radio_seen_at
fields onto snmp_devices.
- DevicePollerWorker now calls RadioState.update_from_sensors/1 after
each sensor poll so frequency-aware rules read a single canonical
source instead of joining sensors at query time.
- Towerops.Recommendations.Rules.OwnFleetChannelConflict — new rule
that needs no neighbor-scan collection: detects when 2+ of our own
APs at the same site share a channel (self-interference). One
insight per (site, channel) with a clickable list of conflicting
APs and their channel widths. Critical when 3+ APs share a channel.
- Insight.@valid_types extended with own_fleet_channel_conflict.
- UI: amber card showing channel, frequency, AP count, and the linked
list of conflicting devices.
Wired into the existing hourly RecommendationsRunWorker. LLM
enrichment automatically applies via the Phase 1 worker.
Two new multi-source rules using data we already collect, both wired
into the existing hourly RecommendationsRunWorker.
SectorOverload — fires when a Preseem-monitored AP has <25% free
airtime AND active subscribers. Critical urgency when free airtime
drops below 15%, OR when free airtime <25% AND QoE <50. Reuses the
airtime/subscriber/QoE data already pulled by PreseemSyncWorker so
no new collection is needed. UI: orange evidence card with free
airtime, subscriber count, QoE score, and AP model.
CpeRealign — fires when a wireless client has BOTH signal_strength
<=-78 dBm AND SNR <=18 dB seen in the last 2 hours. This is the
classic alignment / obstruction signature, distinct from the
existing single-metric wireless_signal_weak / wireless_snr_low
alerts produced by WirelessInsightWorker. One insight per (AP, CPE)
pair via a metadata.dedup_key. Critical when signal <=-88 OR SNR
<=10. UI: rose evidence card with signal, SNR, TX/RX rate, distance,
hostname.
Insights.insert_insight_if_new/1 dedup logic now prefers a
metadata.dedup_key over device_id when explicitly set, allowing
per-CPE insights without collapsing multiple CPEs on the same AP
into a single insight. Existing rules without dedup_key are
unaffected.
Insight.@valid_types extended with sector_overload and cpe_realign.
LLM enrichment automatically applies to both new types via the
existing Phase 1 worker — no extra wiring.
Also adds k8s/secrets.yaml to .gitignore so operators can drop a
local Secret manifest with real values, kubectl apply manually, and
never accidentally commit it. Documented in k8s/README.md.
When a wireless AP has a strong same-channel neighbor (RSSI >= -75 dBm)
and a measurably cleaner alternative channel exists in the same band,
the system now emits an ap_frequency_change insight with a specific
recommended channel and the top interfering BSSIDs as evidence.
- New schema wireless_neighbor_scans (append-only) storing observed
neighboring APs per device: bssid, ssid, channel, frequency_mhz,
channel_width_mhz, rssi_dbm, is_own_fleet, seen_at
- New columns on snmp_devices: current_channel, current_frequency_mhz,
current_channel_width_mhz, last_radio_seen_at — populated by vendor
SNMP profiles for APs that expose wireless OIDs
- Towerops.Recommendations.Rules.FrequencyChange — pure rule that scores
each candidate channel using sum-of-linear-power over the last 24h of
neighbor scans, recommends the cleanest if it beats the current by at
least 6 dB. Critical urgency when current score >= 100.
- Towerops.Workers.RecommendationsRunWorker — hourly Oban cron, iterates
organizations and runs all rules, uses insert_insight_if_new/1 for
idempotent upsert
- LiveView /insights renders a purple frequency-change card with current
vs recommended channel, dB-cleaner badge, and a top-offender list
(BSSID, SSID, RSSI, own-fleet flag)
- LLM enrichment automatically applies to the new insight type
Also fixes two pre-existing flaky tests that surfaced in CI:
- test/towerops_web/live/agent_live_test.exs:567 — now sets an explicit
Req.Test transport_error stub for ReleaseChecker and uses Req.Test.allow
so the LiveView process inherits it. Previously relied on absence of a
stub, which leaked across tests and produced different behavior under
full-suite parallel runs.
- test/towerops_web/telemetry_test.exs:140 — relaxes the "no log" check
to refute the specific messages the function under test would emit,
rather than asserting an empty capture_log/1 buffer (which catches
stray logs from concurrent tests, e.g. Postgrex sandbox disconnects).
Adds an optional plain-language summary and recommended action to every
active insight. A new Oban cron worker runs every 5 minutes, picks up
unenriched insights, and asks the configured LLM to restate the finding
and suggest one specific action grounded in the structured metadata.
- Migration: adds llm_summary, recommended_action, llm_model,
llm_enriched_at columns to preseem_insights
- Towerops.LLM context with swappable behaviour; real client uses Req
with the standard Req.Test plug for test isolation
- MockClient + InsightPrompt module (builds chat messages, parses JSON
with code-fence stripping and a raw-text fallback)
- Worker logs and skips on rate limits / missing key / parse errors so
insights still render without an AI summary when the LLM is unavailable
- Optional towerops-llm k8s secret with example template; deployment.yaml
references it as optional in both init and main containers
- UI renders summary + recommended action callout on /insights and on
the org-level insights page
ePMP devices were not reporting firmware_version because the YAML's
cambiumCurrentuImageVersion.0 OID returns empty on ePMP3000 firmware.
Add detect_version/1 to the Epmp vendor module with two-OID fallback:
cambiumCurrentSWInfo.0 (verbose, ePMP3000-friendly) → cambiumCurrentuImageVersion.0
(short, ePMP1000/2000), with a regex to extract the version triplet
from whatever the device returns.
ePMP3000 also has a sysObjectID quirk where some firmware reports the
generic NetSNMP-on-Linux ID (.1.3.6.1.4.1.8072.3.2.10) instead of the
Cambium enterprise OID, causing devices to misclassify as the linux
profile. Add a second discovery block to the epmp profile that matches
NetSNMP-OID + PREEMPT_RT armv7l kernel signature + a confirming snmpget
against the Cambium ePMP MIB tree, so detection runs in pass-2
(conditional non-generic) before linux falls through in pass-3.
- Remove Mox.set_mox_global() in DPW extra test; rely on $callers
instead so concurrent vendor SNMP tests stop intermittently failing
with VerificationError.
- Make ToweropsWeb.Plugs.GraphQLIntrospectionTest async: false. It
mutates Application.put_env(:towerops, :env, :prod), which polluted
every async vendor test that reads :env via
Towerops.Snmp.Client.phoenix_snmp_disabled/0.
- Bump test pool queue_target/queue_interval. AgentChannelTest spawns
many `:proc_lib` channel processes that hold sandbox connections;
the default 50ms queue caused intermittent
'could not checkout the connection' errors.
- Tag real-System.cmd("ping") tests as :network so they're excluded
by default. Saves ~20s on every full run.
- Make JobCleanupTask's internal 1s settle sleep configurable via
:job_cleanup_settle_ms; the test overrides it to 0.
Plus the pending CoverageLive.Form edit-save / EIRP recompute and
TraceLive deep-link tests from before the compact.
Add PromEx with Application/BEAM/Phoenix/LiveView/Ecto/Oban plugins plus
a custom plugin that bridges existing Towerops Oban/Redis telemetry events.
The metrics HTTP server runs isolated on port 9568 so scrape traffic never
traverses the public Traefik IngressRoute.
The pod template carries prometheus.io annotations (scrape, port, path, job)
so the external Prometheus on 10.0.15.31 discovers each pod through the
existing kubernetes-pods scrape job (apiserver-proxy).
- Add Codeberg Ansible collection link to API/GraphQL docs and Terraform guide
- Un-skip the previously-disabled DevicePollerWorker tests that still pass
- Expand MibController, CnMaestro.Sync, CheckWorker, ActivityFeed,
MobileController, JobCleanupTask tests
- New UploadMibs Mix task test covering arg validation and upload paths
- Set :mib_dir in test config so MibController upload/delete flows can
exercise the real on-disk paths against a writable tmp directory
The /docs/api and /docs/graphql pages were missing four resource
families that the API has shipped:
- Coverages (RF coverage prediction with async compute, recompute,
KMZ download)
- Maintenance Windows (one-off + recurring with RRULE)
- On-Call Schedules (top-level CRUD + layers, members, overrides,
on_call lookup)
- Escalation Policies (CRUD + rules + targets)
Both REST and GraphQL pages now have sidebar entries and example
requests for each.
Dialyzer:
* Add Lidar.Tile @type, narrow several @specs to match success-typing,
prefix discarded function returns with `_ =`, fix the broken
File.stream! call in the ms-buildings worker (it was causing dialyzer
to mark every private fun unreachable), broaden Antenna.spec key
types from `float()` to `number()` to match catalog int literals.
* Add :geo, :geo_postgis, :db_connection to plt_add_apps so the dep
PLT actually contains the modules we use.
* Result: 0 unsuppressed dialyzer warnings (was 50+ in project code).
Tree canopy:
* Replace the dead landfire.gov URL with the ETH Zurich Global
Canopy Height 10 m product. ETH publishes proper COGs with
overviews on libdrive.ethz.ch — confirmed working with
/vsicurl/ byte-range reads. The module now picks the 3°×3° tile
containing the bbox centroid, with both a `:metres` decoder
(ETH Byte raster, NoData=255) and a `:landfire_evh` decoder for
operators self-hosting LANDFIRE EVH COGs.
* Re-enable canopy by default now that there's a working source.
Tests (+105 tests, 0 failures):
* CoverageLive Index/Show/Form/Map — empty/loaded/auth states,
LiveView event handlers, formatters.
* Coverages.Building changeset + helpers.
* Coverages.TreeCanopy provider tile-URL builder and decoders.
* Workers.MsBuildingsImportWorker (full import flow incl. blank
lines, MultiPolygon, idempotency, hash fallback) — also fixed
the partial-index ON CONFLICT bug that prevented imports from
ever working: the unique index has WHERE ms_footprint_id IS
NOT NULL, so the worker now uses an :unsafe_fragment conflict
target that includes the predicate.
* GraphQLDocsController smoke test (was 0%).
Coverage rose from 73.19% → 74% with the major project modules I
touched all moved from 0% / <50% to >75%.
Rewire the single-coverage show page to mount MultiCoverageMap (the
same hook the org map uses) with one coverage in the array, and lift
the cnHeat-style overlay panels — LOS/NLOS toggle + height slider on
the left, opacity + RSSI filter on the bottom, distance probe on
click. Parameters move into a togglable right-side drawer so the map
dominates the viewport.
Show LiveView gains toggle_params / probe_point / close_probe /
set_probe_units handlers and delegates the formatters to
CoverageLive.Map. Site is now preloaded so coverages_payload/1 can
read site.name without an Ecto.AssociationNotLoaded crash that was
silently nuking the data-coverages JSON encode.
Also fixes a real bug in coverage_hooks.ts: pngForCoverage was
returning before the antenna-marker creation block, leaving the
marker code as unreachable dead code — towers never rendered. Moved
the marker creation into addCoverageLayer where it belongs and added
a no-PNG guard so a missing png_path logs instead of silently
failing the L.imageOverlay constructor.
Updates all docs, UI copy, release checker, proto go_package,
container image refs, and tests to point at
codeberg.org/towerops-agent/towerops-agent. Also deletes stray
empty package.json/package-lock.json (Phoenix uses esbuild — no
npm) and an unused test_encode_decode.exs scratch file.
Coverage rasters were being written to priv/static/coverage on the pod's
ephemeral filesystem — fine on a single-pod dev box, broken under k8s
where pods share NFS at /data and a restart wipes computed predictions.
* New :coverage_storage_dir config (defaults to priv/static/coverage for
dev/test, "/data/coverage" in production).
* Raster.output_dir/absolute_raster_path read this config and resolve
{otp_app, rel} or absolute paths.
* Endpoint now mounts a dedicated Plug.Static at /coverage from that
same config — replicas all serve the same NFS-backed file regardless
of which one ran the compute.
* Drop "coverage" from static_paths/0 since the dedicated plug owns
that prefix now.
`coverage` was missing from static_paths/0, so /coverage/<org>/<id>/rssi*.png
fell through to the router (or downstream CDN) instead of being served as a
static asset. In prod that meant the ingress's hotlink-protection page was
rendered as the Leaflet image overlay — the "Access blocked / Referer is
required by..." graphic the user saw on the map.
CoverageWorker was silent between progress updates, so a stuck
gdalwarp / runaway pixel loop showed up only as 'stuck at 15%' in
the UI with nothing in k8s logs. Now every stage logs with the
coverage id (terrain fetch, building load, per-tier LOS+NLOS, row
band progress every 10%, finalize) and the row stream uses a
10-minute timeout with on_timeout: :kill_task so a hang surfaces
as an explicit row-failed log line + NaN fill instead of a wedged
worker. Unexpected exceptions now log with stacktrace and fail the
coverage cleanly.
gdal_translate doesn't accept -t_srs (only -projwin_srs for input),
so reprojection silently fell over against any non-WGS84 source COG
(3DEP Albers tiles). Switch to gdalwarp which actually reprojects.
Also wrap raw GDAL stderr behind a generic user-facing message and
log the technical detail server-side — operators shouldn't see the
gdal_translate usage banner pasted into a coverage failure card.
cnHeat-2.0-style automation hooks. Coverage prediction is now a
first-class API resource that Terraform / Ansible / cron can drive.
REST API (under /api/v1, Bearer-token auth scoped to one org)
- GET /coverages list
- POST /coverages create + auto-queue compute
- GET /coverages/:id read (includes status / progress_pct
for polling-based providers)
- PATCH /coverages/:id update editable fields
- DELETE /coverages/:id delete + cleanup
- POST /coverages/:id/recompute requeue, returns 202
- GET /coverages/:id/kmz Google Earth bundle (KML + PNG)
Each response includes the full resource — Terraform read-after-write
reconciles cleanly; status polling hits a stable JSON shape.
Email-on-complete (Towerops.Coverages.Notifier)
- Worker fires deliver_compute_complete on both :ready and :failed
paths. Sends a short text email to every member of the owning
organisation with a link to the show page (or the failure
reason). Mirrors cnHeat 2.0's per-project email alerts.
- Failures are non-fatal: a stuck SMTP relay never blocks the
underlying coverage record from transitioning state.
KMZ export
- Builds a flat KMZ in-memory via :zip.create — doc.kml +
overlay.png, with the LatLonBox set from the coverage's bbox.
XML special chars are escaped. 409 if not yet ready.
Tests (11 new, all green)
- Bearer-token auth happy path + 401 missing
- index / show / create / update / delete + 404 cross-org
- create returns status:'queued' confirming the auto-enqueue
- recompute returns 202 with status:'queued'
Plus docs/terraform-coverages.md showing how to drive the new API
from a third-party restapi provider until the first-party
terraform-provider-towerops gets a coverage resource.
Closes the last clutter source. NLOS heatmaps now see trees as
obstacles in addition to buildings:
- New Towerops.Coverages.TreeCanopy module fetches an AAIGrid of
canopy heights for the coverage bbox via the same /vsicurl/
+ Lidar.Reader pipeline used for terrain. Source defaults to
the public LANDFIRE 2022 EVH COG (~30 m, all CONUS) and is
override-able via :tree_canopy_url.
- EVH cells are integer codes; decode_evh maps the canopy range
(101..199 → height-100 m), the herbaceous range (11..29 →
(code-10) * 0.1 m), and zeroes everything else.
- Profile.sample/5 takes a :canopy grid alongside :clutter and
uses max(building_height, canopy_height) so a tree poking above
a one-storey roof still shadows.
- CoverageWorker fetches the canopy grid once per job (best
effort — failures log + continue with nil) and threads it
through the env map. LOS keeps clutter empty + canopy nil
(height-above-clutter view); NLOS gets both.
- Tests: tree_canopy_test for the disabled / override paths;
test_helper disables the LANDFIRE fetch by default so the
rest of the suite doesn't stream remote bytes.
Closes the cnHeat-parity gap on modes and clutter:
Backend
- Towerops.Coverages.Buildings.for_bbox/1 — PostGIS ST_Intersects
query that returns each building polygon as a compact
%{height_m, coords} map ready for in-memory point-in-polygon
testing, so the per-pixel hot loop never round-trips to
Postgres.
- Towerops.Coverages.Profile.sample/5 — accepts an optional
:clutter list and adds the building rooftop height to terrain
whenever a sample point falls inside a polygon. Pure ray-cast
PIP for portability.
- Towerops.Workers.CoverageWorker — fetches buildings once per
job and computes BOTH LOS and NLOS variants per SM-height
tier (12 PNGs total). LOS passes clutter: [] (height-above-
clutter view), NLOS passes the prefetched list (height-above-
ground view with rooftop diffraction). Pixel-loop signature
bundled into a ctx map to keep the function arity within
Credo's max-8 limit.
- Towerops.Workers.MsBuildingsImportWorker — streams Microsoft
GlobalMLBuildingFootprints GeoJSONSeq exports line-by-line,
upserts into coverage_buildings in 500-row batches via
insert_all + ON CONFLICT (source, ms_footprint_id). Handles
Polygon and MultiPolygon geometries; stable-hashes the bbox
when an explicit feature id is missing so re-imports are
idempotent. Public import_file/2 lets ops drive a state at
a time from IEx.
- Schema: nlos_tiers JSONB column on coverages, status_changeset
whitelist, payload exposed via list_ready_for_organization.
Frontend
- LOS / NLOS toggle pills above the height slider (left panel).
Selecting NLOS swaps each L.imageOverlay's URL via setUrl to
the matching nlos_tiers PNG, drops the cached pixel data, and
re-decodes for the RSSI filter; falls back to the other
mode's tiers when one is empty so older coverages still
render. coverage_hooks chunk: 22 KB → 25 KB.
Vertical 'Height above clutter' slider on the left of the map now
matches cnHeat's experience: pick an SM install height, the heatmap
re-renders to show only the coverage achievable from that height.
Backend
- New JSONB height_tiers column on coverages keyed by string metres
→ relative PNG path.
- CoverageWorker runs compute_pixels once per tier in
[1.83, 3.05, 4.57, 6.10, 9.14, 12.19] m (~6/10/15/20/30/40 ft)
by varying receiver_height_m on a copied struct. Each tier
writes its own raster + PNG via Raster.write/4 with a per-height
filename suffix; the default 3.05 m tier also populates the
legacy raster_path / png_path so existing show pages keep
working.
Frontend
- Vertical range slider (writing-mode: vertical-lr) anchored
centre-left. Step picks one of HEIGHT_TIERS_FT; label updates
to '6 ft'..'40 ft'.
- pngForCoverage/2 picks the closest available tier from the
payload's height_tiers map, falling back to the original png
for coverages not yet recomputed.
- On slider change the JS swaps each L.imageOverlay's URL via
setUrl, drops the cached pixel data, and re-decodes so the RSSI
filter still applies. coverage_hooks chunk: 20 KB → 22 KB.
Adds the cnHeat-style horizontal 'Min RSSI' slider beside the opacity
control. Pixels weaker than the threshold turn fully transparent,
client-side, with no server round-trip — same compute output now
powers an interactive 'where can I get -75 dBm or better' exploration.
- Slider range -100..-40 dBm step 1; -100 disables filtering.
- addCoverageLayer fetches each PNG into a hidden canvas and caches
its ImageData. crossOrigin=anonymous keeps reads tainted-free.
- Each pixel is classified by nearest palette colour (-50/-65/-75/
-85/-95 dBm centres matching Raster.write/3) and dropped if its
dBm is below the threshold. Result re-encoded via toDataURL and
swapped onto the L.imageOverlay via setUrl.
- destroyed() revokes blob URLs and clears the filter cache to
avoid leaks during LiveView nav. Bundle: coverage_hooks chunk
16 KB → 20 KB; main bundle unchanged at 1.2 MB.
Earlier commit a9aa8bf9 had the message but only the test tweak made
it through pre-commit stashing. This is the implementation:
- lib/towerops/lidar/sources/usgs_ned.ex
- lib/towerops/lidar.ex (mosaic_from_ned + get_elevation_from_ned
fallback paths)
- lib/towerops/coverages/building.ex + migration scaffolding
- lib/towerops/workers/coverage_worker.ex error message
- test_helper.exs and the lidar test changes
Switch esbuild to ESM with --splitting and load app.js as type=module.
Heavy hooks now ship in their own chunk and only download when their
LiveView mounts.
- assets/js/lib/leaflet.ts: extract ensureLeaflet (vendored Leaflet
loader) into a shared module with a typed window.L declaration.
- assets/js/hooks/coverage_hooks.ts: extract CoverageMap,
MultiCoverageMap, and CoverageLocationPicker (~510 lines) with
proper interface types for hook context and tightened typing on
payloads / event handlers (CoveragePayload is now a real type).
- app.ts: replace those three inline hook bodies with lazyHook
stubs that call import('./hooks/coverage_hooks') on first mount,
forwarding lifecycle hooks once the chunk resolves. Also adds a
scoped `declare const L: any` so the remaining Leaflet hooks
type-check.
- root.html.heex: switch the script tag to type=module so the
emitted ESM bundle can use dynamic import().
- esbuild config: add --splitting --format=esm.
Build output now produces:
app.js 1.2 MB
coverage_hooks-<hash>.js 16 KB (lazy)
chunk-<hash>.js 2.4 KB (shared, lazy)
- /coverage now renders a full-bleed Leaflet map of every ready
coverage in the org as one composite heatmap; the prior table view
moves to /coverage/list.
- Top-right tower toggle list with Show All / Hide All. Disabled
towers grey out and their overlay/marker drop from the map.
- Bottom-center opacity slider controls all overlays in unison; the
cnHeat dBm legend renders next to it.
- Top-left base-layer toggle (OSM streets / Esri satellite) and a
Nominatim address search bar that drops a marker.
- Click anywhere on the map → Distance Tool side panel with each
tower sorted by distance and the predicted RSSI at that point
(read from each coverage's GeoTIFF via gdallocationinfo). ft/m
unit toggle, NA for out-of-bbox, color-coded by signal quality.
- Backend additions: Coverages.list_ready_for_organization/1,
Coverages.query_point/3, Raster.query_rssi/3 (gdallocationinfo
wrapper with bbox guard, NoData sentinel handling).
- New JS hook MultiCoverageMap manages the layer set, toolbar,
search, opacity, click-to-probe, and live updates via PubSub
(coverage:enabled-changed and coverage:list-changed).
create flow only inserted the row and left status as :draft, so the
worker never ran. Chain queue_compute/1 after create_coverage/2 so
the coverage immediately moves to :queued and the show page renders
the compute progress.
