gdal_translate doesn't accept -t_srs (only -projwin_srs for input),
so reprojection silently fell over against any non-WGS84 source COG
(3DEP Albers tiles). Switch to gdalwarp which actually reprojects.
Also wrap raw GDAL stderr behind a generic user-facing message and
log the technical detail server-side — operators shouldn't see the
gdal_translate usage banner pasted into a coverage failure card.
cnHeat-2.0-style automation hooks. Coverage prediction is now a
first-class API resource that Terraform / Ansible / cron can drive.
REST API (under /api/v1, Bearer-token auth scoped to one org)
- GET /coverages list
- POST /coverages create + auto-queue compute
- GET /coverages/:id read (includes status / progress_pct
for polling-based providers)
- PATCH /coverages/:id update editable fields
- DELETE /coverages/:id delete + cleanup
- POST /coverages/:id/recompute requeue, returns 202
- GET /coverages/:id/kmz Google Earth bundle (KML + PNG)
Each response includes the full resource — Terraform read-after-write
reconciles cleanly; status polling hits a stable JSON shape.
Email-on-complete (Towerops.Coverages.Notifier)
- Worker fires deliver_compute_complete on both :ready and :failed
paths. Sends a short text email to every member of the owning
organisation with a link to the show page (or the failure
reason). Mirrors cnHeat 2.0's per-project email alerts.
- Failures are non-fatal: a stuck SMTP relay never blocks the
underlying coverage record from transitioning state.
KMZ export
- Builds a flat KMZ in-memory via :zip.create — doc.kml +
overlay.png, with the LatLonBox set from the coverage's bbox.
XML special chars are escaped. 409 if not yet ready.
Tests (11 new, all green)
- Bearer-token auth happy path + 401 missing
- index / show / create / update / delete + 404 cross-org
- create returns status:'queued' confirming the auto-enqueue
- recompute returns 202 with status:'queued'
Plus docs/terraform-coverages.md showing how to drive the new API
from a third-party restapi provider until the first-party
terraform-provider-towerops gets a coverage resource.
Closes the last clutter source. NLOS heatmaps now see trees as
obstacles in addition to buildings:
- New Towerops.Coverages.TreeCanopy module fetches an AAIGrid of
canopy heights for the coverage bbox via the same /vsicurl/
+ Lidar.Reader pipeline used for terrain. Source defaults to
the public LANDFIRE 2022 EVH COG (~30 m, all CONUS) and is
override-able via :tree_canopy_url.
- EVH cells are integer codes; decode_evh maps the canopy range
(101..199 → height-100 m), the herbaceous range (11..29 →
(code-10) * 0.1 m), and zeroes everything else.
- Profile.sample/5 takes a :canopy grid alongside :clutter and
uses max(building_height, canopy_height) so a tree poking above
a one-storey roof still shadows.
- CoverageWorker fetches the canopy grid once per job (best
effort — failures log + continue with nil) and threads it
through the env map. LOS keeps clutter empty + canopy nil
(height-above-clutter view); NLOS gets both.
- Tests: tree_canopy_test for the disabled / override paths;
test_helper disables the LANDFIRE fetch by default so the
rest of the suite doesn't stream remote bytes.
Closes the cnHeat-parity gap on modes and clutter:
Backend
- Towerops.Coverages.Buildings.for_bbox/1 — PostGIS ST_Intersects
query that returns each building polygon as a compact
%{height_m, coords} map ready for in-memory point-in-polygon
testing, so the per-pixel hot loop never round-trips to
Postgres.
- Towerops.Coverages.Profile.sample/5 — accepts an optional
:clutter list and adds the building rooftop height to terrain
whenever a sample point falls inside a polygon. Pure ray-cast
PIP for portability.
- Towerops.Workers.CoverageWorker — fetches buildings once per
job and computes BOTH LOS and NLOS variants per SM-height
tier (12 PNGs total). LOS passes clutter: [] (height-above-
clutter view), NLOS passes the prefetched list (height-above-
ground view with rooftop diffraction). Pixel-loop signature
bundled into a ctx map to keep the function arity within
Credo's max-8 limit.
- Towerops.Workers.MsBuildingsImportWorker — streams Microsoft
GlobalMLBuildingFootprints GeoJSONSeq exports line-by-line,
upserts into coverage_buildings in 500-row batches via
insert_all + ON CONFLICT (source, ms_footprint_id). Handles
Polygon and MultiPolygon geometries; stable-hashes the bbox
when an explicit feature id is missing so re-imports are
idempotent. Public import_file/2 lets ops drive a state at
a time from IEx.
- Schema: nlos_tiers JSONB column on coverages, status_changeset
whitelist, payload exposed via list_ready_for_organization.
Frontend
- LOS / NLOS toggle pills above the height slider (left panel).
Selecting NLOS swaps each L.imageOverlay's URL via setUrl to
the matching nlos_tiers PNG, drops the cached pixel data, and
re-decodes for the RSSI filter; falls back to the other
mode's tiers when one is empty so older coverages still
render. coverage_hooks chunk: 22 KB → 25 KB.
Vertical 'Height above clutter' slider on the left of the map now
matches cnHeat's experience: pick an SM install height, the heatmap
re-renders to show only the coverage achievable from that height.
Backend
- New JSONB height_tiers column on coverages keyed by string metres
→ relative PNG path.
- CoverageWorker runs compute_pixels once per tier in
[1.83, 3.05, 4.57, 6.10, 9.14, 12.19] m (~6/10/15/20/30/40 ft)
by varying receiver_height_m on a copied struct. Each tier
writes its own raster + PNG via Raster.write/4 with a per-height
filename suffix; the default 3.05 m tier also populates the
legacy raster_path / png_path so existing show pages keep
working.
Frontend
- Vertical range slider (writing-mode: vertical-lr) anchored
centre-left. Step picks one of HEIGHT_TIERS_FT; label updates
to '6 ft'..'40 ft'.
- pngForCoverage/2 picks the closest available tier from the
payload's height_tiers map, falling back to the original png
for coverages not yet recomputed.
- On slider change the JS swaps each L.imageOverlay's URL via
setUrl, drops the cached pixel data, and re-decodes so the RSSI
filter still applies. coverage_hooks chunk: 20 KB → 22 KB.
Adds the cnHeat-style horizontal 'Min RSSI' slider beside the opacity
control. Pixels weaker than the threshold turn fully transparent,
client-side, with no server round-trip — same compute output now
powers an interactive 'where can I get -75 dBm or better' exploration.
- Slider range -100..-40 dBm step 1; -100 disables filtering.
- addCoverageLayer fetches each PNG into a hidden canvas and caches
its ImageData. crossOrigin=anonymous keeps reads tainted-free.
- Each pixel is classified by nearest palette colour (-50/-65/-75/
-85/-95 dBm centres matching Raster.write/3) and dropped if its
dBm is below the threshold. Result re-encoded via toDataURL and
swapped onto the L.imageOverlay via setUrl.
- destroyed() revokes blob URLs and clears the filter cache to
avoid leaks during LiveView nav. Bundle: coverage_hooks chunk
16 KB → 20 KB; main bundle unchanged at 1.2 MB.
Earlier commit a9aa8bf9 had the message but only the test tweak made
it through pre-commit stashing. This is the implementation:
- lib/towerops/lidar/sources/usgs_ned.ex
- lib/towerops/lidar.ex (mosaic_from_ned + get_elevation_from_ned
fallback paths)
- lib/towerops/coverages/building.ex + migration scaffolding
- lib/towerops/workers/coverage_worker.ex error message
- test_helper.exs and the lidar test changes
The 3DEP+TNRIS catalog was Texas-only; coverage compute crashed for
any site outside the catalog with 'No LIDAR terrain data available
for this site'. Production hit this for at least one user today.
This commit lays in the rest of the cnHeat-style backend foundation:
- Towerops.Lidar.Sources.UsgsNed: synthesizes Tile structs for the
USGS National Elevation Dataset (1/3 arc-second / ~10 m), keyed
off the predictable n{lat:02d}w{lon:03d} 1° tile naming on the
prd-tnm S3 bucket. Covers all of CONUS, AK, HI, and PR/USVI.
- Towerops.Lidar: when the curated catalog has no tile, falls back
to the synthetic NED tiles via the same Reader / GDAL /vsicurl/
pipeline. New :lidar_ned_fallback config flag (default on);
test_helper.exs disables it so tests can still assert :no_tile
without spinning up GDAL stubs.
- coverage_worker error message reflects the new geographic scope.
- coverage_buildings table + Towerops.Coverages.Building schema:
scaffolding for the next sprint where Microsoft Global ML
Building Footprints get imported and woven into the path
profile as DSM clutter.
- Tests: usgs_ned_test for tile_for_point/2 (CONUS, HI, AK, PR,
out-of-extent); grid_test fallback path verifying the synthetic
URL is hit and an AAIGrid response composes correctly.
Switch esbuild to ESM with --splitting and load app.js as type=module.
Heavy hooks now ship in their own chunk and only download when their
LiveView mounts.
- assets/js/lib/leaflet.ts: extract ensureLeaflet (vendored Leaflet
loader) into a shared module with a typed window.L declaration.
- assets/js/hooks/coverage_hooks.ts: extract CoverageMap,
MultiCoverageMap, and CoverageLocationPicker (~510 lines) with
proper interface types for hook context and tightened typing on
payloads / event handlers (CoveragePayload is now a real type).
- app.ts: replace those three inline hook bodies with lazyHook
stubs that call import('./hooks/coverage_hooks') on first mount,
forwarding lifecycle hooks once the chunk resolves. Also adds a
scoped `declare const L: any` so the remaining Leaflet hooks
type-check.
- root.html.heex: switch the script tag to type=module so the
emitted ESM bundle can use dynamic import().
- esbuild config: add --splitting --format=esm.
Build output now produces:
app.js 1.2 MB
coverage_hooks-<hash>.js 16 KB (lazy)
chunk-<hash>.js 2.4 KB (shared, lazy)
- /coverage now renders a full-bleed Leaflet map of every ready
coverage in the org as one composite heatmap; the prior table view
moves to /coverage/list.
- Top-right tower toggle list with Show All / Hide All. Disabled
towers grey out and their overlay/marker drop from the map.
- Bottom-center opacity slider controls all overlays in unison; the
cnHeat dBm legend renders next to it.
- Top-left base-layer toggle (OSM streets / Esri satellite) and a
Nominatim address search bar that drops a marker.
- Click anywhere on the map → Distance Tool side panel with each
tower sorted by distance and the predicted RSSI at that point
(read from each coverage's GeoTIFF via gdallocationinfo). ft/m
unit toggle, NA for out-of-bbox, color-coded by signal quality.
- Backend additions: Coverages.list_ready_for_organization/1,
Coverages.query_point/3, Raster.query_rssi/3 (gdallocationinfo
wrapper with bbox guard, NoData sentinel handling).
- New JS hook MultiCoverageMap manages the layer set, toolbar,
search, opacity, click-to-probe, and live updates via PubSub
(coverage:enabled-changed and coverage:list-changed).
create flow only inserted the row and left status as :draft, so the
worker never ran. Chain queue_compute/1 after create_coverage/2 so
the coverage immediately moves to :queued and the show page renders
the compute progress.
The status atoms (:computing, :queued, :ready, :failed) may not be
interned in the worker process when it broadcasts. Map status strings
to atoms explicitly instead of relying on prior interning.
- show.html.heex: `@coverage.device_id and ...` raised BadBooleanError
in production because device_id is a UUID string, not a boolean. Use
is_nil/1 checks instead.
- show.ex: handle_info/2 was reloading the coverage without preloading
:device, so the device row vanished from the UI on every status
broadcast. Preload :device alongside the reload.
- coverage_worker: thread the precomputed tx_height through the pixel
loop instead of re-deriving `coverage.height_agl_m + ...` per pixel
with inconsistent nil-safety vs. compute_pixels/6.
Sourced 198 vendor-published MSI Planet pattern files via the
wireless-planning.com aggregator (data1.mlinkplanner.com). Real
patterns now ship for: Cambium (6), Ubiquiti (154), RF Elements
(27), MARS (5), MTI Wireless (1), Tarana (2), Ruckus (1), RFE (2).
Files dropped into priv/antennas/<vendor>-<part>.msi; the existing
boot loader picks them up and they override the synthesized catalog
entries by slug where the slugs match.
Parser improvements to handle real-world MSI file quirks:
- File-extension filter now accepts .ant/.msi (case-insensitive).
- Slug normalisation: spaces, +, and other non-alnum collapse to
hyphens.
- Header parser splits on whitespace runs (tab or space) — different
vendor tools produce different delimiters.
- New parse_frequency_mhz/1 handles "5500", "5 GHz", "5GHz",
"5.6GHz", "5500 MHz", "5150-5850" (centre frequency), and
"5.45 - 5.85 GHz" formats. Replaces the old integer-only parse.
- 2 MARS files with non-360 angular resolution and 1 with
angle-less pattern lines were excluded; the rest parse cleanly.
Total registry now 305 antennas: 198 real vendor patterns + 107
synthesized catalog entries (used as fallback for SKUs without
public .msi files: Cambium ePMP/PMP variants, Tarana G1/G2,
ALPHA, ITELITE, KP Performance, RADWIN, Mimosa, etc.).
Leaflet auto-load fix:
- The location-picker map wasn't appearing because the form page
didn't have the inline <script> tag the existing site map uses
to load Leaflet. AGENTS.md forbids inline <script> in templates,
so I added an ensureLeaflet() helper in app.ts that lazy-loads
/vendor/leaflet/leaflet.{js,css} on first hook mount and caches
the promise. SitesMap, CoverageMap, and CoverageLocationPicker
all use it now — no inline scripts needed.
Adds a "Radio location" section to the coverage form with an embedded
Leaflet map. The marker starts at the parent site's coordinates and
the user can drag it to the actual radio location. A "Use site
location" button clears the override.
- New CoverageLocationPicker hook in assets/js/app.ts: creates the
map, draggable marker, and pushes "location_picked" {lat, lon} on
dragend. Re-creates the marker when the user picks a site that
initially had no coords.
- form.html.heex: location section with map div (phx-hook), readout
("lat, lon - override" if set), and reset button. Hidden inputs for
latitude_override / longitude_override so the form's params always
carry the chosen coordinates.
- form.ex: handle_event "location_picked" updates the override fields
in the changeset; "use_site_location" clears them. Helper functions
radio_marker_lat/lon walk the form values, falling back to the
selected site.
CI test gate fix:
- Worker test now requires GDAL CLI tools to write GeoTIFF + PNG.
test_helper.exs detects whether gdal_translate / gdaldem are on
PATH; when missing (CI's lean test image), :requires_gdal tagged
tests are excluded. Locally with brew install gdal they still run.
After picking a site in the coverage form, a new optional dropdown
lets the user attach the coverage to one of the devices at that site.
The dropdown reloads on every site change. If the site has no devices
yet, a hint replaces it.
Schema: nullable device_id FK on coverages (on_delete: nilify_all so
deleting a device leaves the coverage standing). Filtered by both
site_id and organization_id when loading the dropdown options.
Show page lists the attached device by name (or IP) when set.
Form simplification per user feedback. Form now shows only the
essentials (Name, Site, Antenna, Frequency, TX power, Range, Height,
Azimuth, Tilt, Foliage tuning) with imperial units throughout (ft, mi,
GHz). Defaults: TX power 18 dBm, height 100 ft, azimuth 0°, frequency
5.8 GHz, range 4 mi.
EIRP is calculated, not entered: shown as a live-updating badge in
the form header as `TX power + antenna.gain`.
Removed from the form (still on the schema with sensible defaults so
existing callers and the worker keep working): cell_size_m (auto-
computed from radius / 200, clamped 5–50 m), cable_loss_db, sm_gain_dbi,
tx_clearance_m, height_above_rooftop_m, receiver_height_m,
rx_threshold_dbm, latitude/longitude_override.
Schema:
- Six virtual imperial fields (height_agl_ft, height_above_rooftop_ft,
receiver_height_ft, tx_clearance_ft, radius_mi, frequency_ghz). The
changeset converts to SI before validation. Tests/workers/API keep
sending SI directly.
- ensure_cell_size_m/1: defaults the cell size when callers omit it.
Show + index pages now display GHz / ft / mi.
Propagation accuracy upgrade:
- Earth-curvature correction (4/3-Earth model, ITU-R P.453) applied
per profile sample. Important for paths over ~5 km.
- Replaced Bullington single-knife-edge with Deygout three-edge
multi-obstacle method (ITU-R P.526 §4.5): catches multi-ridge
terrain shadowing that a single dominant obstacle would miss.
Tests: 67 coverage tests pass (added Earth-curvature and Deygout
multi-obstacle reference checks). Full suite: 10817 tests.
The compute pipeline now actually runs: per-pixel path loss over LIDAR
terrain, antenna pattern lookup, real GeoTIFF + colored PNG output,
Leaflet imageOverlay rendering with opacity slider and dBm legend.
Propagation:
- Towerops.Coverages.Propagation: Friis FSPL + Bullington single-knife-edge
diffraction (ITU-R P.526). Pure Elixir, closed-form, no NIF dependency.
Clean signature so a future ITM/Longley-Rice backend can drop in.
- Towerops.Coverages.Profile: AAIGrid sampler with elevation_at, sample
along great circle, haversine distance.
- Towerops.Coverages.Raster: Float32 GeoTIFF via gdal_translate (with VRT
wrapper) and colored PNG via gdaldem color-relief. Outputs to
priv/static/coverage/<org>/<id>/, served by existing Plug.Static.
Worker:
- Towerops.Workers.CoverageWorker now does the full pipeline: bbox compute
-> Towerops.Lidar.get_elevation_grid -> per-pixel Task.async_stream
(parallel = schedulers) running antenna pattern lookup + propagation +
RSSI threshold check, write rasters. Configurable terrain source via
:coverage_terrain_module application env so tests can stub. Friendly
failure messages for :no_tile, :grid_too_large, :nodata,
:unknown_antenna, :missing_location.
Bundled antenna catalog (~95 entries):
- AntennaCatalog with compact specs for every antenna in the request list:
Cambium, ALPHA, Antel, ITELITE, KP Performance, L-com, MARS, MikroTik,
Mimosa, MTI, RADWIN, RF Elements, Ruckus, Tarana, Simulate, Ubiquiti.
- Antenna.from_spec/1 synthesizes 360+360 attenuation patterns from
gain+beamwidth (cosine-squared main lobe, smooth transition,
front-to-back floor with mild ripple). Real .ant files in priv/antennas/
override the catalog by slug. Test fixtures moved to test/support.
Schema (migration add_radio_fields_to_coverages):
- tx_power_dbm replaces eirp_dbm; EIRP is now derived as
tx_power + antenna.gain - cable_loss in Coverage.eirp_dbm/2 and shown
live in the form header.
- cable_loss_db, sm_gain_dbi, latitude_override, longitude_override,
height_above_rooftop_m, tx_clearance_m, foliage_tuning (0-100 slider).
- Coverage.location/1 returns {lat, lon} from override or parent site.
UI:
- form.html.heex: cnHeat-style grouped form (Identity / Location /
Mounting / RF / Coverage extent), foliage tuning range slider,
computed EIRP badge in the header.
- show.html.heex: Leaflet map area with L.imageOverlay heatmap,
opacity slider, dBm color legend, antenna marker with directional
wedge for azimuth.
- assets/js/app.ts: CoverageMap hook registered.
Tests: 24 new (propagation reference values, profile sampling,
worker end-to-end with stubbed terrain that writes real GeoTIFF + PNG,
org-mismatch job-cancel guard). Full suite: 10815 tests.
End-to-end CRUD scaffold for per-site, per-antenna RF coverage
prediction. Schema, context, antenna registry, Oban worker stub, and
three LiveViews are wired up; the actual ITM + LIDAR + buildings
compute pipeline lands in a follow-up.
- Migration + Coverage schema with full validation (RF params, pixel-
budget cap, antenna_slug existence). organization_id excluded from
cast (programmatic-only).
- Coverages context: org-scoped CRUD, validate_site_in_organization
rejects cross-tenant site_id refs at insert/update, queue_compute
with PubSub broadcast on per-coverage and per-org topics.
- MSI Planet .ant parser + persistent_term registry loaded at boot
from priv/antennas/. Two synthetic .ant fixtures (omni + 90deg
sector) bundled.
- CoverageWorker on new :coverage Oban queue (concurrency 2). Job
args carry organization_id; worker refuses to run on org mismatch.
Stub flips status to "failed" with "compute not yet implemented"
so the UI flow is exercisable.
- LiveViews: index (org-wide, live status updates), form (antenna
picker grouped by manufacturer), show (status banners, params
panel, map placeholder). Sidebar nav link added.
- Routes /coverage, /coverage/new, /coverage/:id, /coverage/:id/edit
under :require_authenticated_user_with_default_org.
- 41 new tests covering schema validation, .ant parsing, context
CRUD with cross-org guards, Oban job enqueue.
Each timeout test was sleeping 55ms vs a 50ms timeout — flaky on slow
CI runners where the sleep occasionally finishes before the timer
fires. Bumps every flaky timing pair to 500ms sleep / 25ms timeout so
the timeout always wins.
Splits the runtime apt installs (gdal-bin, snmp, libsnmp40, locales,
BEAM runtime libs) into k8s/Dockerfile.base, hosted at
codeberg.org/gmcintire/towerops-base:latest. The app Dockerfile now
does FROM that base instead of re-installing gdal (~500 MB) on every
push.
The new build-base workflow rebuilds the base image only when
k8s/Dockerfile.base or the workflow itself changes, weekly via cron
(Sundays 06:00 UTC, with CACHE_BUST=<ISO week> to force apt-get update
on a week boundary), or via workflow_dispatch.
Production workflow now uses buildx + does docker login before the
build so it can pull the private base image.
The repo got renamed towerops-web → towerops on the Codeberg side,
so the published image follows: image is now
codeberg.org/gmcintire/towerops with the same main-<ts>-<sha> tag
pattern. Login URL is the hardcoded env.REGISTRY rather than
secrets.REGISTRY_URL so a stale URL secret can't push to the wrong
registry.
CI: switch postgres service image to timescale/timescaledb-ha:pg17-all
which bundles PostGIS, fixing the lidar migration failure in PR and
production workflows.
Tests: ~30 new test files / extensions, ~600+ new tests, taking total
coverage from 70.42% to 73.4%. Targets included GraphQL resolver
unauthenticated paths, SNMP context queries (mempools, processors,
ip_addresses, neighbors, wireless_clients, arp_entries), worker
routing/lifecycle (alert notification, alert digest, weather sync,
job cleanup, cn_maestro, uisp, mikrotik backup, report worker),
SNMP monitoring executors (storage, interface, processor) end-to-end
via stub adapter, NetBox + Sonar sync integration paths, ConfigChanges
listing and correlator, organizations membership query, and live-view
smoke tests for org-scoped settings, integrations, maintenance, agent,
device deep paths, schedules. Property tests used for query
composition and pure helpers where natural.
Catalog-only system for LIDAR-derived 1m DEMs covering Texas. We do not
mirror raster data — the DB stores tile metadata (URL + footprint
geometry) and elevation values are streamed on-demand from public USGS
3DEP / TNRIS Cloud-Optimized GeoTIFFs via GDAL /vsicurl/ HTTP byte-range
reads. Used for future tower coverage line-of-sight calculations.
- Enable PostGIS extension; tile footprints indexed via GiST
- Catalog sync from USGS 3DEP (primary) and TNRIS (fallback for gaps)
- Tile dedup by ST_Contains so TNRIS doesn't duplicate 3DEP coverage
- Single-point reads via gdallocationinfo with tile fallthrough on nodata
- Grid retrieval via gdal_translate AAIGrid with multi-tile mosaicing
- Monthly Oban cron worker keeps catalog fresh
- gdal-bin added to runtime image; geo_postgis dep added
device_down alerts and active incidents represent the same outage.
Filter device_down alerts from the alerts section for devices already
shown as incidents, so a down device shows as 1 issue not 2.
unpkg.com fetches were intermittently failing in production, leaving
the /sites-map page without a working map. Vendoring Leaflet (JS, CSS,
marker/layer images) under priv/static/vendor/leaflet/ removes the
runtime CDN dependency, the matching CSP relaxation, and the SRI hash
churn on future updates.
Add 'vendor' to ToweropsWeb.static_paths/0 so Plug.Static serves the
new tree, and revert script-src/style-src to no longer list unpkg.com.
connect-src still allows *.tile.openstreetmap.org for tile fetches.
The /sites-map page loads leaflet from unpkg.com (CSS + JS) and fetches
tiles from *.tile.openstreetmap.org. Update Content-Security-Policy to
permit those origins so the map can render.
Adds:
- script-src https://unpkg.com (leaflet.js)
- style-src https://unpkg.com (leaflet.css)
- connect-src https://*.tile.openstreetmap.org (tile fetches)
Mix isn't loaded in mix releases, so Mix.env() raised
UndefinedFunctionError inside YamlProfiles.init/1, crashing the
GenServer at boot. That made Supervisor.start_link return {:error, ...}
in production, which the application-start callback then masked by
exiting on the unrelated Task.Supervisor.start_child noproc — leaving
the pod in CrashLoopBackoff with no clear log of the real failure.
Switch to Application.get_env(:towerops, :env), which is set in all
three environments (dev/test/runtime).
Also harden the post-startup callback so a transient TaskSupervisor
noproc cannot mask a real Supervisor.start_link error in the future.
The 1.7 upgrade migration added the new oban_workflows tracking but left
behind legacy state from earlier versions that the upgrade guide
strongly recommends removing
(https://oban.pro/docs/pro/v1-7.html):
- The uniq_key / partition_key GENERATED ALWAYS columns. The Smart
engine no longer reads them; uniqueness and partition lookups now
use expression indexes on meta. Keeping the columns imposes write
overhead on every oban_jobs insert/update and was contributing to
intermittent transaction-rollback errors.
- The oban_jobs_*_old workflow/chain indexes. v1.7 renamed the
originals and built expression-based replacements; the _old indexes
are dead weight that still slow down writes.
Also extend ErrorTrackerIgnorer to drop transient
DBConnection.ConnectionError noise ("connection is closed",
"transaction rolling back"). These come from the Repo's intentional
queue-target / disconnect-on-error cycling that recovers from stale SSL
connections — Ecto retries automatically, so logging adds noise without
value.
The deferred load via send(self(), :load_profiles) is fine in dev/prod
where SNMP polling starts well after boot, but it races with tests that
exercise discovery immediately after the GenServer starts. With an empty
ETS table, match_profile/2 returns nil and discovery falls back to the
Base profile, breaking tests that rely on YAML-profile-derived
manufacturer strings.
The original equipment->device rename migration's ALTER INDEX RENAME was a
no-op (the index didn't exist by the legacy name at that point in the
migration sequence on existing databases), leaving the unique index named
agent_assignments_equipment_id_index. The schema's unique_constraint now
expects agent_assignments_device_id_index, causing a constraint violation
to surface as Ecto.ConstraintError instead of a changeset error.
Bookworm runner images use /etc/apt/sources.list.d/*.sources (deb822),
which the legacy sed didn't touch — non-free stayed off and
snmp-mibs-downloader was unavailable. Patch the Components: line in
deb822 sources too.
Wire up Towerops.Sites.SiteQuery and Towerops.Monitoring.CheckQuery,
which were created in the recent context split but had no callers.
Sites: 5 callsites now compose via SiteQuery.for_organization/roots/
order_by_display.
Monitoring: 6 callsites compose via CheckQuery.for_organization/
for_device/of_type/with_enabled/order_by_name.
Switches lib/towerops/snmp.ex over to defdelegate calls into the new
submodules introduced in the prior commit. Drops the inline definitions
and unused aliases. Final size: 450 lines (was 2742).
Splits the rest of lib/towerops/snmp.ex into dedicated submodules:
- Towerops.Snmp.WirelessClients
- Towerops.Snmp.EntityPhysicals
- Towerops.Snmp.Transceivers
- Towerops.Snmp.PrinterSupplies
- Towerops.Snmp.ArpEntries
- Towerops.Snmp.MacAddresses
- Towerops.Snmp.DiscoveredDevices (cross-source aggregation + OUI lookup)
- Towerops.Snmp.Topology (graph nodes/edges/subnets)
DiscoveredDevices and Topology use Towerops.Devices.DeviceQuery.for_organization/1
to remove inline 'site.organization_id' joins. Public API preserved via
defdelegate in Towerops.Snmp.
snmp.ex shrinks from 2159 to 450 lines (originally 2742).
Splits cohesive sub-domains out of lib/towerops/snmp.ex into dedicated
submodules under lib/towerops/snmp/. Public API preserved via defdelegate
in Towerops.Snmp.
- Towerops.Snmp.Vlans, IpAddresses, Processors, StorageQueries
- Towerops.Snmp.Mempools, Sensors, Interfaces, Neighbors
Neighbors uses Towerops.Devices.DeviceQuery.for_organization/1 to remove
inline org_id filters. snmp.ex shrinks from 2742 to 2159 lines.
Replaces the last inline 'where: w.organization_id == ^...' clause in
maintenance.ex with composable filters from the existing MaintenanceWindowQuery
module.
Adds 7 new composable Query modules following the existing
DeviceQuery/AlertQuery pattern. Each exposes base/0 and for_organization/1,2
plus filters used by 2+ callers.
New modules:
- Towerops.OnCall.ScheduleQuery
- Towerops.OnCall.EscalationPolicyQuery
- Towerops.Maintenance.MaintenanceWindowQuery
- Towerops.Organizations.MembershipQuery
- Towerops.Organizations.InvitationQuery
- Towerops.Agents.AgentTokenQuery
- Towerops.Gaiia.DeviceSubscriberLinkQuery
Refactors callsites in alerts, agents, devices, gaiia, maintenance, on_call,
organizations, search, and subscription_limits to use existing or new Query
modules instead of inline 'where: x.organization_id == ^...' clauses.
Continues the extraction begun with Accounts.TOTP and Accounts.Sessions.
The context module shrinks from ~1198 to 577 lines by moving cohesive
sub-domains into dedicated submodules under lib/towerops/accounts/:
* Consents - grant/revoke/list user consents (+ private
grant_consents_sequential helper used by
register_user/1)
* PolicyVersions - policy version CRUD + reconsent detection
* LoginHistory - record/list/count login attempts, GDPR anonymize
* MagicLinks - magic-link login flow
* Passwords - change/update/reset + reset-instruction email
* Emails - change/update + update-instruction email
The Accounts module retains a defdelegate facade so existing callers
across lib/towerops_web and other contexts continue to work unchanged.
Two shared transaction helpers (update_user_and_delete_all_tokens/1 and
unwrap_transaction_result/1) are used by multiple extracted modules; they
remain in Accounts but are now public with @doc false rather than
duplicated.
