Add PromEx with Application/BEAM/Phoenix/LiveView/Ecto/Oban plugins plus
a custom plugin that bridges existing Towerops Oban/Redis telemetry events.
The metrics HTTP server runs isolated on port 9568 so scrape traffic never
traverses the public Traefik IngressRoute.
The pod template carries prometheus.io annotations (scrape, port, path, job)
so the external Prometheus on 10.0.15.31 discovers each pod through the
existing kubernetes-pods scrape job (apiserver-proxy).
- Add Codeberg Ansible collection link to API/GraphQL docs and Terraform guide
- Un-skip the previously-disabled DevicePollerWorker tests that still pass
- Expand MibController, CnMaestro.Sync, CheckWorker, ActivityFeed,
MobileController, JobCleanupTask tests
- New UploadMibs Mix task test covering arg validation and upload paths
- Set :mib_dir in test config so MibController upload/delete flows can
exercise the real on-disk paths against a writable tmp directory
The /docs/api and /docs/graphql pages were missing four resource
families that the API has shipped:
- Coverages (RF coverage prediction with async compute, recompute,
KMZ download)
- Maintenance Windows (one-off + recurring with RRULE)
- On-Call Schedules (top-level CRUD + layers, members, overrides,
on_call lookup)
- Escalation Policies (CRUD + rules + targets)
Both REST and GraphQL pages now have sidebar entries and example
requests for each.
The current ApiDocs/GraphQLDocs templates don't surface the sample_token
back into the rendered HTML, so the previous assertions were checking
for strings that don't exist. Drop them down to a 'page renders' check.
The controller branches that compute sample_token still get exercised.
Add focused tests across many modules to push overall coverage from
75.45% to ~77%.
New test files:
- test/towerops/snmp/topology_test.exs
- test/towerops/vault_test.exs
- test/towerops/workers/check_worker_test.exs
- test/towerops_web/graphql/resolvers/happy_path_test.exs
- test/towerops_web/graphql/schema_test.exs
- test/towerops_web/live/reports_live_test.exs
Expanded existing tests for: Airos vendor, ProfileWatcher, StormDetector,
LLDP, GpsSync, MikrotikBackupWorker, AdminController, MibController,
MobileController, GeoipController, OnboardingLive, UserResetPasswordLive,
SessionManager, Telemetry, CoverageLive.Show.
Also fix a pre-existing dead test in ApplicationSettingTest where the
schema default for value_type made the 'invalid without value_type' test
unreachable.
Apply Testing LiveView course patterns (live/2, has_element?,
render_click, form/3). Add tests for MobileQRLive, WeathermapLive,
StatusPageLive, controllers, plugs, and pure function modules.
Expand proto decode coverage to 85.24%.
Dialyzer:
* Add Lidar.Tile @type, narrow several @specs to match success-typing,
prefix discarded function returns with `_ =`, fix the broken
File.stream! call in the ms-buildings worker (it was causing dialyzer
to mark every private fun unreachable), broaden Antenna.spec key
types from `float()` to `number()` to match catalog int literals.
* Add :geo, :geo_postgis, :db_connection to plt_add_apps so the dep
PLT actually contains the modules we use.
* Result: 0 unsuppressed dialyzer warnings (was 50+ in project code).
Tree canopy:
* Replace the dead landfire.gov URL with the ETH Zurich Global
Canopy Height 10 m product. ETH publishes proper COGs with
overviews on libdrive.ethz.ch — confirmed working with
/vsicurl/ byte-range reads. The module now picks the 3°×3° tile
containing the bbox centroid, with both a `:metres` decoder
(ETH Byte raster, NoData=255) and a `:landfire_evh` decoder for
operators self-hosting LANDFIRE EVH COGs.
* Re-enable canopy by default now that there's a working source.
Tests (+105 tests, 0 failures):
* CoverageLive Index/Show/Form/Map — empty/loaded/auth states,
LiveView event handlers, formatters.
* Coverages.Building changeset + helpers.
* Coverages.TreeCanopy provider tile-URL builder and decoders.
* Workers.MsBuildingsImportWorker (full import flow incl. blank
lines, MultiPolygon, idempotency, hash fallback) — also fixed
the partial-index ON CONFLICT bug that prevented imports from
ever working: the unique index has WHERE ms_footprint_id IS
NOT NULL, so the worker now uses an :unsafe_fragment conflict
target that includes the predicate.
* GraphQLDocsController smoke test (was 0%).
Coverage rose from 73.19% → 74% with the major project modules I
touched all moved from 0% / <50% to >75%.
Rewire the single-coverage show page to mount MultiCoverageMap (the
same hook the org map uses) with one coverage in the array, and lift
the cnHeat-style overlay panels — LOS/NLOS toggle + height slider on
the left, opacity + RSSI filter on the bottom, distance probe on
click. Parameters move into a togglable right-side drawer so the map
dominates the viewport.
Show LiveView gains toggle_params / probe_point / close_probe /
set_probe_units handlers and delegates the formatters to
CoverageLive.Map. Site is now preloaded so coverages_payload/1 can
read site.name without an Ecto.AssociationNotLoaded crash that was
silently nuking the data-coverages JSON encode.
Also fixes a real bug in coverage_hooks.ts: pngForCoverage was
returning before the antenna-marker creation block, leaving the
marker code as unreachable dead code — towers never rendered. Moved
the marker creation into addCoverageLayer where it belongs and added
a no-PNG guard so a missing png_path logs instead of silently
failing the L.imageOverlay constructor.
ConfigTimelineLive's chart panel was rendering a bare 'Loading chart...'
placeholder forever — the phx-hook="ConfigTimelineChart" had no JS
implementation registered anywhere, so the whole point of the page
(correlating QoE regressions with config edits) was lost.
Add a focused lazy hook in hooks/config_timeline_chart.ts that:
* Plots Preseem latency / throughput / loss across three Y-axes.
* Overlays vertical dashed markers at each config-change timestamp
with a small badge showing how many lines changed.
* Drops a red triangle on the time axis for every failed check.
* Reuses the Chart.js vendor chunk (esbuild --splitting factors it
into a shared chunk with sensor_chart, so visiting either page
primes the other).
Bundle delta:
chunk-RSDHLN2L.js (Chart.js, shared) 314.8 KB
config_timeline_chart 7.2 KB
sensor_chart (was 325 KB standalone) 10.6 KB after dedup
DeviceListReorder, SortableList, MikrotikPortSync are each used on a
single page (devices index, escalation/schedule show, device edit
form respectively) — push them out of app.js into their own chunks
via lazyHook. Also extract MikrotikPortSync into its own file under
hooks/ for consistency.
Compress CopyToClipboard's flash feedback by collapsing the duplicated
success/error inline-SVG/setTimeout blocks into one helper, drop the
debug console.logs, and remove the bulk Phoenix-template boilerplate
comment block that was just generated noise.
After-bundle (minified):
app.js 287.6 KB (-2 KB on top of prior split)
device_list_reorder 4.6 KB loaded on /devices only
sortable_list 3.1 KB loaded on escalation+schedule pages
mikrotik_port_sync 1.3 KB loaded on device edit form only
app.js was bundling Chart.js (~200 KB), Cytoscape (~500 KB) and four
heavy hooks (SensorChart, NetworkMap, WeathermapViewer, SitesMap)
into a single 1.25 MB blob that every page paid for.
Extract each into its own module under assets/js/hooks/ and reference
them through the existing lazyHook() wrapper so the implementation is
fetched only when a hook actually mounts. esbuild --splitting then
factors the shared cytoscape vendor blob into a chunk that's only
loaded on topology / weathermap pages.
After-bundle layout (minified):
app.js 297.8 KB (was ~1250 KB; -76%)
chunk-LJGWK7RH (cyto) 555.1 KB loaded on topology pages only
sensor_chart 325.3 KB loaded on sensor pages only
network_map 24.4 KB topology hook glue
weathermap 13.0 KB weathermap hook glue
sites_map 4.1 KB site map hook glue
coverage_hooks 25.0 KB coverage hook glue
LeafletMap also moved to sites_map.ts and now uses ensureLeaflet
instead of polling for window.L with setTimeout.
Updates all docs, UI copy, release checker, proto go_package,
container image refs, and tests to point at
codeberg.org/towerops-agent/towerops-agent. Also deletes stray
empty package.json/package-lock.json (Phoenix uses esbuild — no
npm) and an unused test_encode_decode.exs scratch file.
Coverage rasters were being written to priv/static/coverage on the pod's
ephemeral filesystem — fine on a single-pod dev box, broken under k8s
where pods share NFS at /data and a restart wipes computed predictions.
* New :coverage_storage_dir config (defaults to priv/static/coverage for
dev/test, "/data/coverage" in production).
* Raster.output_dir/absolute_raster_path read this config and resolve
{otp_app, rel} or absolute paths.
* Endpoint now mounts a dedicated Plug.Static at /coverage from that
same config — replicas all serve the same NFS-backed file regardless
of which one ran the compute.
* Drop "coverage" from static_paths/0 since the dedicated plug owns
that prefix now.
`coverage` was missing from static_paths/0, so /coverage/<org>/<id>/rssi*.png
fell through to the router (or downstream CDN) instead of being served as a
static asset. In prod that meant the ingress's hotlink-protection page was
rendered as the Leaflet image overlay — the "Access blocked / Referer is
required by..." graphic the user saw on the map.
CoverageWorker was silent between progress updates, so a stuck
gdalwarp / runaway pixel loop showed up only as 'stuck at 15%' in
the UI with nothing in k8s logs. Now every stage logs with the
coverage id (terrain fetch, building load, per-tier LOS+NLOS, row
band progress every 10%, finalize) and the row stream uses a
10-minute timeout with on_timeout: :kill_task so a hang surfaces
as an explicit row-failed log line + NaN fill instead of a wedged
worker. Unexpected exceptions now log with stacktrace and fail the
coverage cleanly.
gdal_translate doesn't accept -t_srs (only -projwin_srs for input),
so reprojection silently fell over against any non-WGS84 source COG
(3DEP Albers tiles). Switch to gdalwarp which actually reprojects.
Also wrap raw GDAL stderr behind a generic user-facing message and
log the technical detail server-side — operators shouldn't see the
gdal_translate usage banner pasted into a coverage failure card.
cnHeat-2.0-style automation hooks. Coverage prediction is now a
first-class API resource that Terraform / Ansible / cron can drive.
REST API (under /api/v1, Bearer-token auth scoped to one org)
- GET /coverages list
- POST /coverages create + auto-queue compute
- GET /coverages/:id read (includes status / progress_pct
for polling-based providers)
- PATCH /coverages/:id update editable fields
- DELETE /coverages/:id delete + cleanup
- POST /coverages/:id/recompute requeue, returns 202
- GET /coverages/:id/kmz Google Earth bundle (KML + PNG)
Each response includes the full resource — Terraform read-after-write
reconciles cleanly; status polling hits a stable JSON shape.
Email-on-complete (Towerops.Coverages.Notifier)
- Worker fires deliver_compute_complete on both :ready and :failed
paths. Sends a short text email to every member of the owning
organisation with a link to the show page (or the failure
reason). Mirrors cnHeat 2.0's per-project email alerts.
- Failures are non-fatal: a stuck SMTP relay never blocks the
underlying coverage record from transitioning state.
KMZ export
- Builds a flat KMZ in-memory via :zip.create — doc.kml +
overlay.png, with the LatLonBox set from the coverage's bbox.
XML special chars are escaped. 409 if not yet ready.
Tests (11 new, all green)
- Bearer-token auth happy path + 401 missing
- index / show / create / update / delete + 404 cross-org
- create returns status:'queued' confirming the auto-enqueue
- recompute returns 202 with status:'queued'
Plus docs/terraform-coverages.md showing how to drive the new API
from a third-party restapi provider until the first-party
terraform-provider-towerops gets a coverage resource.
Closes the last clutter source. NLOS heatmaps now see trees as
obstacles in addition to buildings:
- New Towerops.Coverages.TreeCanopy module fetches an AAIGrid of
canopy heights for the coverage bbox via the same /vsicurl/
+ Lidar.Reader pipeline used for terrain. Source defaults to
the public LANDFIRE 2022 EVH COG (~30 m, all CONUS) and is
override-able via :tree_canopy_url.
- EVH cells are integer codes; decode_evh maps the canopy range
(101..199 → height-100 m), the herbaceous range (11..29 →
(code-10) * 0.1 m), and zeroes everything else.
- Profile.sample/5 takes a :canopy grid alongside :clutter and
uses max(building_height, canopy_height) so a tree poking above
a one-storey roof still shadows.
- CoverageWorker fetches the canopy grid once per job (best
effort — failures log + continue with nil) and threads it
through the env map. LOS keeps clutter empty + canopy nil
(height-above-clutter view); NLOS gets both.
- Tests: tree_canopy_test for the disabled / override paths;
test_helper disables the LANDFIRE fetch by default so the
rest of the suite doesn't stream remote bytes.
Closes the cnHeat-parity gap on modes and clutter:
Backend
- Towerops.Coverages.Buildings.for_bbox/1 — PostGIS ST_Intersects
query that returns each building polygon as a compact
%{height_m, coords} map ready for in-memory point-in-polygon
testing, so the per-pixel hot loop never round-trips to
Postgres.
- Towerops.Coverages.Profile.sample/5 — accepts an optional
:clutter list and adds the building rooftop height to terrain
whenever a sample point falls inside a polygon. Pure ray-cast
PIP for portability.
- Towerops.Workers.CoverageWorker — fetches buildings once per
job and computes BOTH LOS and NLOS variants per SM-height
tier (12 PNGs total). LOS passes clutter: [] (height-above-
clutter view), NLOS passes the prefetched list (height-above-
ground view with rooftop diffraction). Pixel-loop signature
bundled into a ctx map to keep the function arity within
Credo's max-8 limit.
- Towerops.Workers.MsBuildingsImportWorker — streams Microsoft
GlobalMLBuildingFootprints GeoJSONSeq exports line-by-line,
upserts into coverage_buildings in 500-row batches via
insert_all + ON CONFLICT (source, ms_footprint_id). Handles
Polygon and MultiPolygon geometries; stable-hashes the bbox
when an explicit feature id is missing so re-imports are
idempotent. Public import_file/2 lets ops drive a state at
a time from IEx.
- Schema: nlos_tiers JSONB column on coverages, status_changeset
whitelist, payload exposed via list_ready_for_organization.
Frontend
- LOS / NLOS toggle pills above the height slider (left panel).
Selecting NLOS swaps each L.imageOverlay's URL via setUrl to
the matching nlos_tiers PNG, drops the cached pixel data, and
re-decodes for the RSSI filter; falls back to the other
mode's tiers when one is empty so older coverages still
render. coverage_hooks chunk: 22 KB → 25 KB.
Vertical 'Height above clutter' slider on the left of the map now
matches cnHeat's experience: pick an SM install height, the heatmap
re-renders to show only the coverage achievable from that height.
Backend
- New JSONB height_tiers column on coverages keyed by string metres
→ relative PNG path.
- CoverageWorker runs compute_pixels once per tier in
[1.83, 3.05, 4.57, 6.10, 9.14, 12.19] m (~6/10/15/20/30/40 ft)
by varying receiver_height_m on a copied struct. Each tier
writes its own raster + PNG via Raster.write/4 with a per-height
filename suffix; the default 3.05 m tier also populates the
legacy raster_path / png_path so existing show pages keep
working.
Frontend
- Vertical range slider (writing-mode: vertical-lr) anchored
centre-left. Step picks one of HEIGHT_TIERS_FT; label updates
to '6 ft'..'40 ft'.
- pngForCoverage/2 picks the closest available tier from the
payload's height_tiers map, falling back to the original png
for coverages not yet recomputed.
- On slider change the JS swaps each L.imageOverlay's URL via
setUrl, drops the cached pixel data, and re-decodes so the RSSI
filter still applies. coverage_hooks chunk: 20 KB → 22 KB.
Adds the cnHeat-style horizontal 'Min RSSI' slider beside the opacity
control. Pixels weaker than the threshold turn fully transparent,
client-side, with no server round-trip — same compute output now
powers an interactive 'where can I get -75 dBm or better' exploration.
- Slider range -100..-40 dBm step 1; -100 disables filtering.
- addCoverageLayer fetches each PNG into a hidden canvas and caches
its ImageData. crossOrigin=anonymous keeps reads tainted-free.
- Each pixel is classified by nearest palette colour (-50/-65/-75/
-85/-95 dBm centres matching Raster.write/3) and dropped if its
dBm is below the threshold. Result re-encoded via toDataURL and
swapped onto the L.imageOverlay via setUrl.
- destroyed() revokes blob URLs and clears the filter cache to
avoid leaks during LiveView nav. Bundle: coverage_hooks chunk
16 KB → 20 KB; main bundle unchanged at 1.2 MB.
Earlier commit a9aa8bf9 had the message but only the test tweak made
it through pre-commit stashing. This is the implementation:
- lib/towerops/lidar/sources/usgs_ned.ex
- lib/towerops/lidar.ex (mosaic_from_ned + get_elevation_from_ned
fallback paths)
- lib/towerops/coverages/building.ex + migration scaffolding
- lib/towerops/workers/coverage_worker.ex error message
- test_helper.exs and the lidar test changes
The 3DEP+TNRIS catalog was Texas-only; coverage compute crashed for
any site outside the catalog with 'No LIDAR terrain data available
for this site'. Production hit this for at least one user today.
This commit lays in the rest of the cnHeat-style backend foundation:
- Towerops.Lidar.Sources.UsgsNed: synthesizes Tile structs for the
USGS National Elevation Dataset (1/3 arc-second / ~10 m), keyed
off the predictable n{lat:02d}w{lon:03d} 1° tile naming on the
prd-tnm S3 bucket. Covers all of CONUS, AK, HI, and PR/USVI.
- Towerops.Lidar: when the curated catalog has no tile, falls back
to the synthetic NED tiles via the same Reader / GDAL /vsicurl/
pipeline. New :lidar_ned_fallback config flag (default on);
test_helper.exs disables it so tests can still assert :no_tile
without spinning up GDAL stubs.
- coverage_worker error message reflects the new geographic scope.
- coverage_buildings table + Towerops.Coverages.Building schema:
scaffolding for the next sprint where Microsoft Global ML
Building Footprints get imported and woven into the path
profile as DSM clutter.
- Tests: usgs_ned_test for tile_for_point/2 (CONUS, HI, AK, PR,
out-of-extent); grid_test fallback path verifying the synthetic
URL is hit and an AAIGrid response composes correctly.
Switch esbuild to ESM with --splitting and load app.js as type=module.
Heavy hooks now ship in their own chunk and only download when their
LiveView mounts.
- assets/js/lib/leaflet.ts: extract ensureLeaflet (vendored Leaflet
loader) into a shared module with a typed window.L declaration.
- assets/js/hooks/coverage_hooks.ts: extract CoverageMap,
MultiCoverageMap, and CoverageLocationPicker (~510 lines) with
proper interface types for hook context and tightened typing on
payloads / event handlers (CoveragePayload is now a real type).
- app.ts: replace those three inline hook bodies with lazyHook
stubs that call import('./hooks/coverage_hooks') on first mount,
forwarding lifecycle hooks once the chunk resolves. Also adds a
scoped `declare const L: any` so the remaining Leaflet hooks
type-check.
- root.html.heex: switch the script tag to type=module so the
emitted ESM bundle can use dynamic import().
- esbuild config: add --splitting --format=esm.
Build output now produces:
app.js 1.2 MB
coverage_hooks-<hash>.js 16 KB (lazy)
chunk-<hash>.js 2.4 KB (shared, lazy)
- /coverage now renders a full-bleed Leaflet map of every ready
coverage in the org as one composite heatmap; the prior table view
moves to /coverage/list.
- Top-right tower toggle list with Show All / Hide All. Disabled
towers grey out and their overlay/marker drop from the map.
- Bottom-center opacity slider controls all overlays in unison; the
cnHeat dBm legend renders next to it.
- Top-left base-layer toggle (OSM streets / Esri satellite) and a
Nominatim address search bar that drops a marker.
- Click anywhere on the map → Distance Tool side panel with each
tower sorted by distance and the predicted RSSI at that point
(read from each coverage's GeoTIFF via gdallocationinfo). ft/m
unit toggle, NA for out-of-bbox, color-coded by signal quality.
- Backend additions: Coverages.list_ready_for_organization/1,
Coverages.query_point/3, Raster.query_rssi/3 (gdallocationinfo
wrapper with bbox guard, NoData sentinel handling).
- New JS hook MultiCoverageMap manages the layer set, toolbar,
search, opacity, click-to-probe, and live updates via PubSub
(coverage:enabled-changed and coverage:list-changed).
create flow only inserted the row and left status as :draft, so the
worker never ran. Chain queue_compute/1 after create_coverage/2 so
the coverage immediately moves to :queued and the show page renders
the compute progress.
The status atoms (:computing, :queued, :ready, :failed) may not be
interned in the worker process when it broadcasts. Map status strings
to atoms explicitly instead of relying on prior interning.
- show.html.heex: `@coverage.device_id and ...` raised BadBooleanError
in production because device_id is a UUID string, not a boolean. Use
is_nil/1 checks instead.
- show.ex: handle_info/2 was reloading the coverage without preloading
:device, so the device row vanished from the UI on every status
broadcast. Preload :device alongside the reload.
- coverage_worker: thread the precomputed tx_height through the pixel
loop instead of re-deriving `coverage.height_agl_m + ...` per pixel
with inconsistent nil-safety vs. compute_pixels/6.
