- Change postgresql_16 → postgresql_17 in shell.nix (17.10 available)
- Both postgis and timescaledb confirmed available for PG17
- Remove PG17-only transaction_timeout SET from structure.sql
(dump was already PG17, now compatible with running server)
- Auto-reinit triggers on version file mismatch
Three changes targeting the root cause of production up/down alerting:
1. Skip BruteForceProtection for /health and /health/live paths
(removes unnecessary Repo.get_by(IpBlock) DB query from every
k8s probe, matching the existing /socket/agent exemption).
2. Relax k8s readiness probe: periodSeconds 5→10, timeoutSeconds
2→5, failureThreshold 2→3, successThreshold 3→2 so transient
DB/Redis blips don't cascade into full service unavailability.
3. Use the persistent Towerops.Redix connection for health checks
instead of opening a brand-new TCP connection (handshake→AUTH→
PING→close) on every probe. Towerops.Redix.Fake gains a
set_ping_response/2 toggle for testing failure paths.
Converts the 120KB PNG logo to 17KB WebP (86% smaller). Both logo instances in the marketing layout use <picture> with WebP source and PNG fallback for maximum browser compatibility.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Prod paired the Oban Pro Smart engine + unique workers with core
Oban.Plugins.Lifeline, whose rescue has no unique_violation handling. An
orphaned executing CheckExecutorWorker job rescued back to available collides
with its already-scheduled successor on oban_jobs_unique_index (23505),
crashing the plugin every 60s and leaving ~150 orphans stuck. Switch prod to
Oban.Pro.Plugins.DynamicLifeline, which repairs the conflict via
Smart.clear_uniq_violation. Dev keeps core Lifeline (Basic engine).
Add a shallow /health/live endpoint that does not touch db/redis. k8s liveness
and startup probes now target it so a transient dependency outage can't kill or
block boot of a healthy pod; readiness keeps the deep /health to gate the load
balancer. (deployment.yaml probe/replica change pushed separately, after the
image carrying /health/live is live.)
Also drop unused POSTGRES_* keys from the secrets example.
CheckExecutorWorker had no Oban `unique` constraint, so every insert path
created a fresh job row: self-scheduling, JobHealthCheckWorker, the check
form/API/GraphQL, and — the main amplifier — Monitoring.schedule_check/1
called unconditionally by snmp.ex discovery after create_check/1, which is
an upsert that returns {:ok, check} even when the check already exists.
Every SNMP discovery pass re-enqueued a job for every existing check, so
the check_executors queue grew to ~70k jobs for ~5.7k checks and stopped
draining.
Add `unique: [keys: [:check_id], states: [:available, :scheduled,
:retryable], period: :infinity]`. `:executing` is excluded so perform/1's
self-reschedule does not conflict with its own running job.
Also add the `checks` and `check_executors` queues to config/dev.exs —
they were missing from dev entirely (prod's runtime.exs already had them).
- M3: Replace blacklist-based valid_return_path? with whitelist of known app
route prefixes, plus URI decoding to prevent %2f encoding bypasses
- M6: Replace full-org device/link load in get_node_detail with targeted
join query filtering by discovered node fields
- M7: Add partial unique index on (organization_id, email) for pending
invitations (WHERE accepted_at IS NULL)
- M12: Add on_mount superuser verification to all 7 admin LiveViews for
defense-in-depth
- H1: batch count queries in MobileController (org sites/devices/alerts, site
device counts) — reduces 1+3N+2M queries to 4 + 2
- H2: batch Oban cancel for stop_device_checks — single ANY(?) query instead
of one per check
- H3: resolve_active_alerts_for_device uses update_all + deduped PubSub
broadcasts (was N updates + N broadcasts)
- H4: atomic ON CONFLICT upsert for auto-discovery checks; restore partial
unique index dropped in 20260404000002 (dedup migration included)
- H5: wrap apply_agent_to_all_equipment delete+insert in transaction so a
failed insert_all rolls back the prior delete
- H14: replace String.to_integer/1 on untrusted SNMP OID components with
Integer.parse/1 + validation (neighbor_discovery, discovery storage index,
printer supply index)
- H15: ActivityFeedLive whitelists activity types against @all_types instead
of String.to_existing_atom/1
- H16: defensive page param parsing in user_settings and admin dashboard
LiveViews
- H27: search sanitize/1 delegates to QueryHelpers.sanitize_like/1 which
escapes backslash first
- H30: JobCleanupTask no longer cancels jobs in "executing" state so
in-flight polls complete naturally
Adds a parallel LLM path that reviews the whole org snapshot — Preseem
APs, SNMP signals, backhaul, agents, and the insight types the rules
already covered — and emits novel observations as ai_observation
insights (source=ai). Existing rule-based insights and per-insight
enrichment continue unchanged.
- lib/towerops/llm/network_snapshot.ex builds the bounded snapshot
- lib/towerops/llm/network_insight_prompt.ex builds the prompt and
parses the LLM's JSON observations array
- lib/towerops/workers/ai_network_insight_worker.ex runs nightly at
03:30 UTC (after the rule pass + per-insight enrichment) and on
the superuser regenerate button
- preseem/insight.ex extends @valid_types with "ai_observation" and
@valid_sources with "ai"
- insights_live/index.ex includes the new worker in the regen list
and adds an "AI" source badge style
Bumps the SnmpKit Manager timeout-options test threshold from 500ms
to 3000ms (was flaking on loaded dev machines), and scopes
list_unenriched_insights/1 assertion in insights_test:573 to the
test's org so cross-suite leaks don't fail the count.
Token-usage doesn't belong on the per-insight row — every consumer of
Towerops.LLM (insight enrichment today; billing-anomaly summaries,
monitoring narratives, fleet weekly reports tomorrow) needs to record
into one queryable place. Per-insight columns also made it impossible
to track LLM calls that aren't tied to an insight.
Schema: `llm_usage` keyed (organization_id, day, model, purpose) with a
unique index. organization_id is nullable for system-level analyses.
Each row carries running totals plus a request_count — callers UPSERT
and increment atomically.
API: `Towerops.LLM.Usage.record/1` for write, `summarize/2` for
read-side rollups. The worker now calls `Usage.record(%{... purpose:
"insight_enrichment"})` after each successful enrichment.
Removed: `llm_prompt_tokens` / `llm_completion_tokens` from
preseem_insights — same migration drops the columns and creates
llm_usage in one go. `apply_llm_enrichment` reverts to /3 (no usage
arg).
Some MikroTik models report `mtxrGaugeValue` (mtxrGaugeUnit=1, °C) in
deci-degrees, while others — like the CRS317 in upstream test data —
return whole degrees. The MIB doesn't disclose which, so the YAML
profile can't pick one literal divisor. Result: a sensor that should
read 56.92 °C was being stored and alerted on as 5,692 °C.
- Towerops.Snmp.SensorScale.normalize/2: divide-by-10 a temperature
reading until it falls below 150 °C, capped at 3 iterations. Other
sensor types pass through untouched.
- DevicePollerWorker.poll_simple_sensor wires the normaliser in after
the YAML divisor is applied, so live polling stops storing bogus
values immediately.
- DeviceOverheating rule now bounds the query at 150 °C as defense in
depth so a future scaling glitch never produces a 5,000 °C alert.
- mix towerops.fix_mikrotik_temperature_scaling backfills any rows
already in the DB by re-running normalize/2 over them.
- routeros.yaml profile gets a comment pointing at the runtime
normaliser instead of a wrong unconditional divisor.
Track DeepSeek token spend per enriched insight so cost can be rolled
up by SQL — `SELECT date_trunc('day', llm_enriched_at), organization_id,
SUM(llm_prompt_tokens), SUM(llm_completion_tokens) FROM preseem_insights
GROUP BY 1, 2`.
- Migration adds llm_prompt_tokens + llm_completion_tokens to
preseem_insights
- DeepSeek client surfaces the API's `usage` block (prompt/completion/
total tokens) on the success response
- apply_llm_enrichment/4 takes a usage map and persists the two int
columns
- Worker plumbs the usage from the response through to the context
Fiber operators care a lot about optical Rx power because connectors
get dirty, splices degrade, and patch panels develop bend-radius
problems. The data is already in snmp_transceiver_readings — this
rule turns it into actionable insights.
- Towerops.Recommendations.Rules.OpticalRxLow — single query selects
the latest reading per transceiver (last hour), filters to those
with rx_power_dbm <= -28, joins back to the device. Critical at
<= -32 dBm. One insight per (device, port) via metadata.dedup_key.
- Insight.@valid_types extended with optical_rx_low.
- LLM prompt hint walks the operator through the standard ladder:
clean SC/LC connector with a one-click cleaner, inspect bend
radius, verify splices, then replace the transceiver.
- UI: cyan card with port, Rx power, Tx power, transceiver type, and
wavelength.
- Wired into RecommendationsRunWorker.
8 new tests.
When 50%+ of the access points at one site simultaneously have
Preseem QoE score <60, root cause is almost always a shared upstream
problem (backhaul, transit, DNS, DHCP) — not per-AP RF. This rule
emits one site-level insight that points the operator at the right
investigation layer instead of letting them chase per-AP RF
symptoms across many APs at the same site.
- Towerops.Recommendations.Rules.UpstreamDegradation — single query
pulls all matched Preseem APs grouped by site_id. Critical when
100% of APs at the site are degraded; warning otherwise. Lists
every degraded AP with its QoE score in metadata.
- Insight.@valid_types extended with upstream_degradation.
- LLM prompt hint tells the model to instruct the operator to
investigate upstream FIRST and to NOT chase per-AP RF symptoms.
- UI: indigo card with degraded/total AP counts, an "Investigate
upstream first" badge, and a clickable list of affected APs.
- Wired into RecommendationsRunWorker FIRST in the @rules list so
the upstream signal surfaces before per-AP RF rules.
Also fixes two flakes:
- device_live/show_test.exs is now async: false. It inserts to
InterfaceStat which is a TimescaleDB hypertable; concurrent inserts
deadlock on chunk-table locks under parallel SQL-sandbox tests
(ERROR 40P01 deadlock_detected in setup).
- Cleared an unused-variable warning in device_overheating_test.exs.
8 new tests for UpstreamDegradation.
Operators frequently misdiagnose thermal throttling as RF problems
and waste truck rolls. This rule reads the hottest temperature
sensor per device and, when the device is linked to a Preseem AP
with degraded QoE, makes the thermal-correlation explicit so the
fix is ventilation/shading rather than antenna alignment.
- Towerops.Recommendations.Rules.DeviceOverheating — single query
pulls the hottest temperature sensor per device above 65 °C; one
insight per device. Critical at 75 °C+. Includes the linked
Preseem qoe_score in metadata when present.
- Insight.@valid_types extended with device_overheating.
- LLM prompt hint tells the model to recommend ventilation/fan/
shading inspection BEFORE any RF action when QoE is also low.
- UI: red card with sensor name, temperature, linked QoE score,
and an "Above manufacturer spec" badge for >=75 °C.
Wired into RecommendationsRunWorker. 8 new tests.
Same-channel conflicts at one site are not all equal. A pair of APs
sharing a channel in a clean RF environment is a planning issue;
the same pair in a contaminated environment is an outage waiting to
happen. This commit makes the rule reason across sources before
deciding urgency.
- Towerops.Snmp.RfHealth — cross-source helper that combines:
* Most recent noise-floor / noise sensor on the AP (dBm)
* Preseem rf_score and qoe_score from the linked AccessPoint row
* Average CPE SNR and client count from wireless_clients
Returns a snapshot map and an interference_severity/1 classifier
(:critical | :warning | :ok). Critical when noise floor >= -80 dBm
OR Preseem rf_score < 4.0; warning when avg CPE SNR < 15 with
active clients.
- OwnFleetChannelConflict now enriches each conflicting AP with its
noise floor, rf_score, qoe_score, avg CPE SNR, CPE count, and a
per-AP rf_severity. Urgency promotes to critical when 3+ APs share
a channel OR any conflicting AP shows critical RF health.
- UI: amber card now renders an evidence table with width, noise,
rf_score, CPE SNR, and CPE count per AP, plus a "Critical RF
environment detected" badge when has_critical_rf is true.
Also strengthens the previous flake fix on agent_live_test.exs:567
by switching to Req.Test shared mode (the test is sync), so the
ReleaseChecker stub is visible to the LiveView process regardless of
when it spawns relative to setup. The earlier Req.Test.allow approach
relied on stub timing that didn't always hold under full-suite load.
11 new tests across RfHealth + the enriched rule. LLM enrichment
automatically picks up the richer metadata via the existing Phase 1
worker.
Adds a vendor-agnostic radio-state extractor that lets the frequency-
aware rules actually fire on production data without per-vendor
plumbing for each profile.
- Towerops.Snmp.RadioState — pure module that maps a frequency in MHz
to the corresponding 802.11 channel across 2.4 GHz, 5 GHz, and 6 GHz
bands; reads the freshest "frequency" sensor (already emitted by
Cambium/MikroTik/AirOS/Mimosa/Exalt/etc. profiles) and persists the
AP-level current_frequency_mhz, current_channel, last_radio_seen_at
fields onto snmp_devices.
- DevicePollerWorker now calls RadioState.update_from_sensors/1 after
each sensor poll so frequency-aware rules read a single canonical
source instead of joining sensors at query time.
- Towerops.Recommendations.Rules.OwnFleetChannelConflict — new rule
that needs no neighbor-scan collection: detects when 2+ of our own
APs at the same site share a channel (self-interference). One
insight per (site, channel) with a clickable list of conflicting
APs and their channel widths. Critical when 3+ APs share a channel.
- Insight.@valid_types extended with own_fleet_channel_conflict.
- UI: amber card showing channel, frequency, AP count, and the linked
list of conflicting devices.
Wired into the existing hourly RecommendationsRunWorker. LLM
enrichment automatically applies via the Phase 1 worker.
Two new multi-source rules using data we already collect, both wired
into the existing hourly RecommendationsRunWorker.
SectorOverload — fires when a Preseem-monitored AP has <25% free
airtime AND active subscribers. Critical urgency when free airtime
drops below 15%, OR when free airtime <25% AND QoE <50. Reuses the
airtime/subscriber/QoE data already pulled by PreseemSyncWorker so
no new collection is needed. UI: orange evidence card with free
airtime, subscriber count, QoE score, and AP model.
CpeRealign — fires when a wireless client has BOTH signal_strength
<=-78 dBm AND SNR <=18 dB seen in the last 2 hours. This is the
classic alignment / obstruction signature, distinct from the
existing single-metric wireless_signal_weak / wireless_snr_low
alerts produced by WirelessInsightWorker. One insight per (AP, CPE)
pair via a metadata.dedup_key. Critical when signal <=-88 OR SNR
<=10. UI: rose evidence card with signal, SNR, TX/RX rate, distance,
hostname.
Insights.insert_insight_if_new/1 dedup logic now prefers a
metadata.dedup_key over device_id when explicitly set, allowing
per-CPE insights without collapsing multiple CPEs on the same AP
into a single insight. Existing rules without dedup_key are
unaffected.
Insight.@valid_types extended with sector_overload and cpe_realign.
LLM enrichment automatically applies to both new types via the
existing Phase 1 worker — no extra wiring.
Also adds k8s/secrets.yaml to .gitignore so operators can drop a
local Secret manifest with real values, kubectl apply manually, and
never accidentally commit it. Documented in k8s/README.md.
When a wireless AP has a strong same-channel neighbor (RSSI >= -75 dBm)
and a measurably cleaner alternative channel exists in the same band,
the system now emits an ap_frequency_change insight with a specific
recommended channel and the top interfering BSSIDs as evidence.
- New schema wireless_neighbor_scans (append-only) storing observed
neighboring APs per device: bssid, ssid, channel, frequency_mhz,
channel_width_mhz, rssi_dbm, is_own_fleet, seen_at
- New columns on snmp_devices: current_channel, current_frequency_mhz,
current_channel_width_mhz, last_radio_seen_at — populated by vendor
SNMP profiles for APs that expose wireless OIDs
- Towerops.Recommendations.Rules.FrequencyChange — pure rule that scores
each candidate channel using sum-of-linear-power over the last 24h of
neighbor scans, recommends the cleanest if it beats the current by at
least 6 dB. Critical urgency when current score >= 100.
- Towerops.Workers.RecommendationsRunWorker — hourly Oban cron, iterates
organizations and runs all rules, uses insert_insight_if_new/1 for
idempotent upsert
- LiveView /insights renders a purple frequency-change card with current
vs recommended channel, dB-cleaner badge, and a top-offender list
(BSSID, SSID, RSSI, own-fleet flag)
- LLM enrichment automatically applies to the new insight type
Also fixes two pre-existing flaky tests that surfaced in CI:
- test/towerops_web/live/agent_live_test.exs:567 — now sets an explicit
Req.Test transport_error stub for ReleaseChecker and uses Req.Test.allow
so the LiveView process inherits it. Previously relied on absence of a
stub, which leaked across tests and produced different behavior under
full-suite parallel runs.
- test/towerops_web/telemetry_test.exs:140 — relaxes the "no log" check
to refute the specific messages the function under test would emit,
rather than asserting an empty capture_log/1 buffer (which catches
stray logs from concurrent tests, e.g. Postgrex sandbox disconnects).
Adds an optional plain-language summary and recommended action to every
active insight. A new Oban cron worker runs every 5 minutes, picks up
unenriched insights, and asks the configured LLM to restate the finding
and suggest one specific action grounded in the structured metadata.
- Migration: adds llm_summary, recommended_action, llm_model,
llm_enriched_at columns to preseem_insights
- Towerops.LLM context with swappable behaviour; real client uses Req
with the standard Req.Test plug for test isolation
- MockClient + InsightPrompt module (builds chat messages, parses JSON
with code-fence stripping and a raw-text fallback)
- Worker logs and skips on rate limits / missing key / parse errors so
insights still render without an AI summary when the LLM is unavailable
- Optional towerops-llm k8s secret with example template; deployment.yaml
references it as optional in both init and main containers
- UI renders summary + recommended action callout on /insights and on
the org-level insights page
ePMP devices were not reporting firmware_version because the YAML's
cambiumCurrentuImageVersion.0 OID returns empty on ePMP3000 firmware.
Add detect_version/1 to the Epmp vendor module with two-OID fallback:
cambiumCurrentSWInfo.0 (verbose, ePMP3000-friendly) → cambiumCurrentuImageVersion.0
(short, ePMP1000/2000), with a regex to extract the version triplet
from whatever the device returns.
ePMP3000 also has a sysObjectID quirk where some firmware reports the
generic NetSNMP-on-Linux ID (.1.3.6.1.4.1.8072.3.2.10) instead of the
Cambium enterprise OID, causing devices to misclassify as the linux
profile. Add a second discovery block to the epmp profile that matches
NetSNMP-OID + PREEMPT_RT armv7l kernel signature + a confirming snmpget
against the Cambium ePMP MIB tree, so detection runs in pass-2
(conditional non-generic) before linux falls through in pass-3.
Updates all docs, UI copy, release checker, proto go_package,
container image refs, and tests to point at
codeberg.org/towerops-agent/towerops-agent. Also deletes stray
empty package.json/package-lock.json (Phoenix uses esbuild — no
npm) and an unused test_encode_decode.exs scratch file.
Closes the cnHeat-parity gap on modes and clutter:
Backend
- Towerops.Coverages.Buildings.for_bbox/1 — PostGIS ST_Intersects
query that returns each building polygon as a compact
%{height_m, coords} map ready for in-memory point-in-polygon
testing, so the per-pixel hot loop never round-trips to
Postgres.
- Towerops.Coverages.Profile.sample/5 — accepts an optional
:clutter list and adds the building rooftop height to terrain
whenever a sample point falls inside a polygon. Pure ray-cast
PIP for portability.
- Towerops.Workers.CoverageWorker — fetches buildings once per
job and computes BOTH LOS and NLOS variants per SM-height
tier (12 PNGs total). LOS passes clutter: [] (height-above-
clutter view), NLOS passes the prefetched list (height-above-
ground view with rooftop diffraction). Pixel-loop signature
bundled into a ctx map to keep the function arity within
Credo's max-8 limit.
- Towerops.Workers.MsBuildingsImportWorker — streams Microsoft
GlobalMLBuildingFootprints GeoJSONSeq exports line-by-line,
upserts into coverage_buildings in 500-row batches via
insert_all + ON CONFLICT (source, ms_footprint_id). Handles
Polygon and MultiPolygon geometries; stable-hashes the bbox
when an explicit feature id is missing so re-imports are
idempotent. Public import_file/2 lets ops drive a state at
a time from IEx.
- Schema: nlos_tiers JSONB column on coverages, status_changeset
whitelist, payload exposed via list_ready_for_organization.
Frontend
- LOS / NLOS toggle pills above the height slider (left panel).
Selecting NLOS swaps each L.imageOverlay's URL via setUrl to
the matching nlos_tiers PNG, drops the cached pixel data, and
re-decodes for the RSSI filter; falls back to the other
mode's tiers when one is empty so older coverages still
render. coverage_hooks chunk: 22 KB → 25 KB.
Vertical 'Height above clutter' slider on the left of the map now
matches cnHeat's experience: pick an SM install height, the heatmap
re-renders to show only the coverage achievable from that height.
Backend
- New JSONB height_tiers column on coverages keyed by string metres
→ relative PNG path.
- CoverageWorker runs compute_pixels once per tier in
[1.83, 3.05, 4.57, 6.10, 9.14, 12.19] m (~6/10/15/20/30/40 ft)
by varying receiver_height_m on a copied struct. Each tier
writes its own raster + PNG via Raster.write/4 with a per-height
filename suffix; the default 3.05 m tier also populates the
legacy raster_path / png_path so existing show pages keep
working.
Frontend
- Vertical range slider (writing-mode: vertical-lr) anchored
centre-left. Step picks one of HEIGHT_TIERS_FT; label updates
to '6 ft'..'40 ft'.
- pngForCoverage/2 picks the closest available tier from the
payload's height_tiers map, falling back to the original png
for coverages not yet recomputed.
- On slider change the JS swaps each L.imageOverlay's URL via
setUrl, drops the cached pixel data, and re-decodes so the RSSI
filter still applies. coverage_hooks chunk: 20 KB → 22 KB.
Earlier commit a9aa8bf9 had the message but only the test tweak made
it through pre-commit stashing. This is the implementation:
- lib/towerops/lidar/sources/usgs_ned.ex
- lib/towerops/lidar.ex (mosaic_from_ned + get_elevation_from_ned
fallback paths)
- lib/towerops/coverages/building.ex + migration scaffolding
- lib/towerops/workers/coverage_worker.ex error message
- test_helper.exs and the lidar test changes
Sourced 198 vendor-published MSI Planet pattern files via the
wireless-planning.com aggregator (data1.mlinkplanner.com). Real
patterns now ship for: Cambium (6), Ubiquiti (154), RF Elements
(27), MARS (5), MTI Wireless (1), Tarana (2), Ruckus (1), RFE (2).
Files dropped into priv/antennas/<vendor>-<part>.msi; the existing
boot loader picks them up and they override the synthesized catalog
entries by slug where the slugs match.
Parser improvements to handle real-world MSI file quirks:
- File-extension filter now accepts .ant/.msi (case-insensitive).
- Slug normalisation: spaces, +, and other non-alnum collapse to
hyphens.
- Header parser splits on whitespace runs (tab or space) — different
vendor tools produce different delimiters.
- New parse_frequency_mhz/1 handles "5500", "5 GHz", "5GHz",
"5.6GHz", "5500 MHz", "5150-5850" (centre frequency), and
"5.45 - 5.85 GHz" formats. Replaces the old integer-only parse.
- 2 MARS files with non-360 angular resolution and 1 with
angle-less pattern lines were excluded; the rest parse cleanly.
Total registry now 305 antennas: 198 real vendor patterns + 107
synthesized catalog entries (used as fallback for SKUs without
public .msi files: Cambium ePMP/PMP variants, Tarana G1/G2,
ALPHA, ITELITE, KP Performance, RADWIN, Mimosa, etc.).
Leaflet auto-load fix:
- The location-picker map wasn't appearing because the form page
didn't have the inline <script> tag the existing site map uses
to load Leaflet. AGENTS.md forbids inline <script> in templates,
so I added an ensureLeaflet() helper in app.ts that lazy-loads
/vendor/leaflet/leaflet.{js,css} on first hook mount and caches
the promise. SitesMap, CoverageMap, and CoverageLocationPicker
all use it now — no inline scripts needed.
After picking a site in the coverage form, a new optional dropdown
lets the user attach the coverage to one of the devices at that site.
The dropdown reloads on every site change. If the site has no devices
yet, a hint replaces it.
Schema: nullable device_id FK on coverages (on_delete: nilify_all so
deleting a device leaves the coverage standing). Filtered by both
site_id and organization_id when loading the dropdown options.
Show page lists the attached device by name (or IP) when set.
The compute pipeline now actually runs: per-pixel path loss over LIDAR
terrain, antenna pattern lookup, real GeoTIFF + colored PNG output,
Leaflet imageOverlay rendering with opacity slider and dBm legend.
Propagation:
- Towerops.Coverages.Propagation: Friis FSPL + Bullington single-knife-edge
diffraction (ITU-R P.526). Pure Elixir, closed-form, no NIF dependency.
Clean signature so a future ITM/Longley-Rice backend can drop in.
- Towerops.Coverages.Profile: AAIGrid sampler with elevation_at, sample
along great circle, haversine distance.
- Towerops.Coverages.Raster: Float32 GeoTIFF via gdal_translate (with VRT
wrapper) and colored PNG via gdaldem color-relief. Outputs to
priv/static/coverage/<org>/<id>/, served by existing Plug.Static.
Worker:
- Towerops.Workers.CoverageWorker now does the full pipeline: bbox compute
-> Towerops.Lidar.get_elevation_grid -> per-pixel Task.async_stream
(parallel = schedulers) running antenna pattern lookup + propagation +
RSSI threshold check, write rasters. Configurable terrain source via
:coverage_terrain_module application env so tests can stub. Friendly
failure messages for :no_tile, :grid_too_large, :nodata,
:unknown_antenna, :missing_location.
Bundled antenna catalog (~95 entries):
- AntennaCatalog with compact specs for every antenna in the request list:
Cambium, ALPHA, Antel, ITELITE, KP Performance, L-com, MARS, MikroTik,
Mimosa, MTI, RADWIN, RF Elements, Ruckus, Tarana, Simulate, Ubiquiti.
- Antenna.from_spec/1 synthesizes 360+360 attenuation patterns from
gain+beamwidth (cosine-squared main lobe, smooth transition,
front-to-back floor with mild ripple). Real .ant files in priv/antennas/
override the catalog by slug. Test fixtures moved to test/support.
Schema (migration add_radio_fields_to_coverages):
- tx_power_dbm replaces eirp_dbm; EIRP is now derived as
tx_power + antenna.gain - cable_loss in Coverage.eirp_dbm/2 and shown
live in the form header.
- cable_loss_db, sm_gain_dbi, latitude_override, longitude_override,
height_above_rooftop_m, tx_clearance_m, foliage_tuning (0-100 slider).
- Coverage.location/1 returns {lat, lon} from override or parent site.
UI:
- form.html.heex: cnHeat-style grouped form (Identity / Location /
Mounting / RF / Coverage extent), foliage tuning range slider,
computed EIRP badge in the header.
- show.html.heex: Leaflet map area with L.imageOverlay heatmap,
opacity slider, dBm color legend, antenna marker with directional
wedge for azimuth.
- assets/js/app.ts: CoverageMap hook registered.
Tests: 24 new (propagation reference values, profile sampling,
worker end-to-end with stubbed terrain that writes real GeoTIFF + PNG,
org-mismatch job-cancel guard). Full suite: 10815 tests.
End-to-end CRUD scaffold for per-site, per-antenna RF coverage
prediction. Schema, context, antenna registry, Oban worker stub, and
three LiveViews are wired up; the actual ITM + LIDAR + buildings
compute pipeline lands in a follow-up.
- Migration + Coverage schema with full validation (RF params, pixel-
budget cap, antenna_slug existence). organization_id excluded from
cast (programmatic-only).
- Coverages context: org-scoped CRUD, validate_site_in_organization
rejects cross-tenant site_id refs at insert/update, queue_compute
with PubSub broadcast on per-coverage and per-org topics.
- MSI Planet .ant parser + persistent_term registry loaded at boot
from priv/antennas/. Two synthetic .ant fixtures (omni + 90deg
sector) bundled.
- CoverageWorker on new :coverage Oban queue (concurrency 2). Job
args carry organization_id; worker refuses to run on org mismatch.
Stub flips status to "failed" with "compute not yet implemented"
so the UI flow is exercisable.
- LiveViews: index (org-wide, live status updates), form (antenna
picker grouped by manufacturer), show (status banners, params
panel, map placeholder). Sidebar nav link added.
- Routes /coverage, /coverage/new, /coverage/:id, /coverage/:id/edit
under :require_authenticated_user_with_default_org.
- 41 new tests covering schema validation, .ant parsing, context
CRUD with cross-org guards, Oban job enqueue.
Catalog-only system for LIDAR-derived 1m DEMs covering Texas. We do not
mirror raster data — the DB stores tile metadata (URL + footprint
geometry) and elevation values are streamed on-demand from public USGS
3DEP / TNRIS Cloud-Optimized GeoTIFFs via GDAL /vsicurl/ HTTP byte-range
reads. Used for future tower coverage line-of-sight calculations.
- Enable PostGIS extension; tile footprints indexed via GiST
- Catalog sync from USGS 3DEP (primary) and TNRIS (fallback for gaps)
- Tile dedup by ST_Contains so TNRIS doesn't duplicate 3DEP coverage
- Single-point reads via gdallocationinfo with tile fallthrough on nodata
- Grid retrieval via gdal_translate AAIGrid with multi-tile mosaicing
- Monthly Oban cron worker keeps catalog fresh
- gdal-bin added to runtime image; geo_postgis dep added
unpkg.com fetches were intermittently failing in production, leaving
the /sites-map page without a working map. Vendoring Leaflet (JS, CSS,
marker/layer images) under priv/static/vendor/leaflet/ removes the
runtime CDN dependency, the matching CSP relaxation, and the SRI hash
churn on future updates.
Add 'vendor' to ToweropsWeb.static_paths/0 so Plug.Static serves the
new tree, and revert script-src/style-src to no longer list unpkg.com.
connect-src still allows *.tile.openstreetmap.org for tile fetches.
The 1.7 upgrade migration added the new oban_workflows tracking but left
behind legacy state from earlier versions that the upgrade guide
strongly recommends removing
(https://oban.pro/docs/pro/v1-7.html):
- The uniq_key / partition_key GENERATED ALWAYS columns. The Smart
engine no longer reads them; uniqueness and partition lookups now
use expression indexes on meta. Keeping the columns imposes write
overhead on every oban_jobs insert/update and was contributing to
intermittent transaction-rollback errors.
- The oban_jobs_*_old workflow/chain indexes. v1.7 renamed the
originals and built expression-based replacements; the _old indexes
are dead weight that still slow down writes.
Also extend ErrorTrackerIgnorer to drop transient
DBConnection.ConnectionError noise ("connection is closed",
"transaction rolling back"). These come from the Repo's intentional
queue-target / disconnect-on-error cycling that recovers from stale SSL
connections — Ecto retries automatically, so logging adds noise without
value.
The original equipment->device rename migration's ALTER INDEX RENAME was a
no-op (the index didn't exist by the legacy name at that point in the
migration sequence on existing databases), leaving the unique index named
agent_assignments_equipment_id_index. The schema's unique_constraint now
expects agent_assignments_device_id_index, causing a constraint violation
to surface as Ecto.ConstraintError instead of a changeset error.
Bumps vendored Oban Pro from 1.6.13 to 1.7.0, which transitively bumps
oban core to 2.22.1 (requires migrations v14) and db_connection to 2.10.0.
- mix.exs constraint: ~> 1.6 → ~> 1.7
- vendor/oban_pro/ replaced with v1.7.0 hex package
- migration 20260430142200: Oban core v12 → v14 (must run first)
- migration 20260430142241: Oban Pro 1.7.0 schema additions
Also corrects a pre-existing constraint name in AgentAssignment that was
left stale by the equipment→devices rename migration (20260117190134).
The unique index is named agent_assignments_device_id_index, but the
schema's unique_constraint/3 still pointed at the old equipment_id name,
causing the test suite to surface Ecto.ConstraintError instead of a
changeset error. Required to get the suite green for this upgrade.
Performance:
- schedule_live: preload page once instead of get_schedule!/1 per row
- alert_live + alerts: DB-side status filter; Repo.aggregate counts replace length/Enum.count over 500-row fetches on every event
- dashboard_live: drop duplicate get_device_status_counts; cap active alerts to 20 + use count_active_alerts/1
- maintenance.active_windows_for_device: 3 round-trips collapsed into one query (per-site-id branch keeps Postgres parameter types unambiguous)
- sites.build_site_tree: O(N^2) -> O(N) via group_by(parent_site_id)
- accounts.sole_owner_organizations: single group_by + having instead of per-org Repo.aggregate loop
- agents + agent_live (org + admin): count_assigned_devices_batch/1 + count_agent_polling_targets/1 (no preloads)
- alert_digest_worker: list_alerts_by_ids/1 batches digest fetch
- gaiia: distinct: true at DB; limit 50 on bidirectional ilike
Indexes:
- maintenance_windows(organization_id, starts_at, ends_at) WHERE suppress_alerts = true
- alerts(check_id) WHERE resolved_at IS NULL
Antipatterns:
- agents.delete_agent_token: PubSub.broadcast moved outside Repo.transaction so a rollback no longer leaves subscribers acting on a non-existent deletion
- integrations_controller.to_atom_keys: replaced String.to_existing_atom on user-controlled JSON keys with explicit allowlist
- 9 Task.start callsites converted to Task.Supervisor.start_child(Towerops.TaskSupervisor, ...) so background DB writes survive shutdown (test-mode discovery shims left as-is for sandbox semantics)
Three deferred followups from the chunk-4 plan, all together:
A. Restrictions editor
- Resolver: time_of_week support (weekday set + time-of-day window),
in addition to the existing time_of_day.
- New OnCall.update_layer_restrictions/2 + clear_layer_restrictions/1.
- schedule_live/show: per-layer Restrict on-call shifts form
(Always on call / Time of day / Time of week with Mon-Sun checkboxes),
changes patched live and persisted via the resolver.
B. Drag-and-drop reorder
- New OnCall.reorder_layers/2 + reorder_escalation_rules/2 (atomic
transaction; validates id count + membership).
- assets/js/sortable_list.ts: minimal HTML5 drag-and-drop hook (no
external dep; mirrors the existing DeviceListReorder pattern),
registered as SortableList in app.ts.
- schedule_live/show + escalation_policy_live/show: layer/level cards
now carry data-id + draggable + a visible drag handle. Up/down arrow
buttons remain as a fallback for users on assistive tech.
C. Unified new-schedule UX
- schedule_live/show pre-opens the Add Layer form when the schedule
has zero layers, so the new-schedule -> add-layers flow lands ready
to act on. Two existing tests updated to seed a layer first; two
new tests verify the auto-open behaviour both ways.
Full suite 10,231 / 0 failures. Format/dialyzer clean.
ScheduleLive.Index
- Name search input above the date nav (phx-debounce 300ms).
- Standard <.pagination> footer (10 per page) under the gantt cards.
- Both reflected in URL params (?search= and ?page=) for shareable
links; defaults are stripped from the URL.
- Refactored: load_schedules_with_timeline/3 -> hydrate_schedules/3
so filter/paginate/hydrate are clearly separate steps.
Tests
- 2 new ExUnit integration tests (search filter narrows results +
search input renders).
- e2e/tests/schedules.spec.ts: switched from table-row selectors to
card-link selectors (schedules tab is now gantt cards, not a table),
added smoke tests for date nav controls + search input + REPEAT
footer + Services sidebar, renamed "Add Rule" -> "Add Level".
Full suite 10,216 / 0 failures. Format/dialyzer clean.
OnCall context
- move_layer/2: up/down position swap for schedule layers, mirror of
move_escalation_rule/2 (tested for both directions + boundary no-op).
ScheduleLive.Show
- New move_layer event handler.
- Up/down arrow buttons on each layer card; the top arrow is disabled on
the first layer, the bottom on the last (matches the escalation policy
level reorder UX shipped in chunk 1).
- Replaced two per-hour walks (build_layer_spans + build_final_spans -
~672 resolve calls per render at 14 days) with Resolver.resolve_range/3
+ a small segments_to_spans/3 mapper. Per-layer view wraps each layer
in a temporary single-layer %Schedule{} so the resolver code path is
shared and overrides don't bleed into the per-layer strip.
Tests: 3 new context tests for move_layer, 1 new integration test for
the layer reorder UI. Full suite 10,214 / 0 failures. Format/dialyzer
clean.
Resolver
- Resolver.resolve_range/3: returns the contiguous on-call segments for a
schedule across [start_at, end_at). Walks layer handoffs + override
boundaries, drops gap segments, and merges adjacent same-user segments.
UserColor module
- New Towerops.OnCall.UserColor with deterministic id -> color mapping
(Tailwind class + matching #RRGGBB hex). Refactored ScheduleLive.Show
to use it instead of its own per-page index palette so the same person
gets the same color across the schedules list and detail pages.
ScheduleLive.Index
- Date navigation: Today / prev / next + 1/2/4-week range selector.
- URL-driven: ?start=YYYY-MM-DD&range=N for shareable views; falls back
to defaults on malformed values.
- Per-row card: schedule name link, on-call-now avatar swatch, day-of-week
header strip, gantt strip rendered as a CSS grid with one column per day,
Today marker overlay.
Tests: 5 new resolver tests, 9 new UserColor tests, 5 new ScheduleLive
integration tests. Full suite: 10,210 / 0 failures. Format/dialyzer clean.
Also: cleaned up two stale inline comments in k8s/deployment.yaml that
disagreed with the values they sat next to (replicas/maxSurge).
Chunk 1 of the on-call/escalation redesign plan
(docs/plans/2026-04-28-on-call-pagerduty-style-redesign.md):
Schema
- on_call.ex: list_devices_using_policy/2 (direct + org-default inheritance)
- on_call.ex: move_escalation_rule/2 for up/down level reordering
- list_escalation_policies/1 now preloads :rules
Show page
- Levels rendered as numbered cards with up/down/delete controls
- "Notify the following users immediately and escalate after N minutes."
- REPEAT footer surfaces repeat_count inline (no longer a standalone column)
- Right-side Services sidebar lists devices using the policy
- Header line shows the handoff notifications mode (when in use by a service /
always / never)
Edit form
- Removed standalone Repeat Count input
- Added handoff notifications mode select
- Added inline "If no one acknowledges, repeat this policy N time(s)" block
Index views (policies tab + standalone)
- Replaced Repeat Count column with a Levels (rule count) column
Other
- rate_limit.ex: bind unmatched :ets.new/2 + schedule_clean/1 returns to fix
dialyzer :unmatched_returns warning under the project's strict flags.
All 10,191 tests pass; mix format clean; mix compile --warnings-as-errors clean;
mix dialyzer passes.
Oban 2.18+ introduced the 'suspended' job state for pausing jobs.
Oban.Met.Reporter queries for jobs in this state, causing errors when
the database enum doesn't include it:
ERROR 22P02 (invalid_text_representation) invalid input value
for enum oban_job_state: "suspended"
This migration adds 'suspended' to the oban_job_state enum type.
Required by recent Oban dependency updates (commit 0399b30d).
Note: Uses @disable_ddl_transaction since ALTER TYPE ADD VALUE cannot
run inside a transaction in PostgreSQL.
Reviewed-on: graham/towerops-web#199
The recent dependency update to phoenix_pubsub_redis 3.1.0 introduced
breaking changes in the configuration API. The new version no longer
accepts connection options at the top level - they must be nested in
:redis_opts.
Changes:
- Move Redix connection options into :redis_opts keyword list
- Keep only :node_name at top level per v3.1.0 requirements
- Maintains all resilience settings (timeouts, backoff, keepalive)
Before (invalid for v3.1.0):
[node_name: node(), host: "...", port: 6379, timeout: 5000, ...]
After (valid for v3.1.0):
[node_name: node(), redis_opts: [host: "...", port: 6379, timeout: 5000, ...]]
This fixes the production crash with error:
NimbleOptions.ValidationError: unknown options [:timeout, :backoff_initial,
:backoff_max, :exit_on_disconnection, :sync_connect]
Related: commit 0399b30d (Update Elixir dependencies)
Reviewed-on: graham/towerops-web#197
