Graham McIntire
b5e3c1f977
Three changes targeting the root cause of production up/down alerting: 1. Skip BruteForceProtection for /health and /health/live paths (removes unnecessary Repo.get_by(IpBlock) DB query from every k8s probe, matching the existing /socket/agent exemption). 2. Relax k8s readiness probe: periodSeconds 5→10, timeoutSeconds 2→5, failureThreshold 2→3, successThreshold 3→2 so transient DB/Redis blips don't cascade into full service unavailability. 3. Use the persistent Towerops.Redix connection for health checks instead of opening a brand-new TCP connection (handshake→AUTH→ PING→close) on every probe. Towerops.Redix.Fake gains a set_ping_response/2 toggle for testing failure paths.
1312 lines
77 KiB
Text
1312 lines
77 KiB
Text
2026-06-08
|
||
fix: reduce health-check probe pressure to stop readiness-flapping downtime
|
||
Three changes to eliminate the root cause of production up/down alerting:
|
||
1. BruteForceProtection plug now skips /health and /health/live paths (like
|
||
/socket/agent already does), removing an unnecessary Repo.get_by(IpBlock)
|
||
query from every k8s readiness probe.
|
||
2. k8s readiness probe relaxed: periodSeconds 5→10, timeoutSeconds 2→5,
|
||
failureThreshold 2→3 to tolerate brief DB/Redis blips without marking the
|
||
pod unready; successThreshold 3→2 to recover faster after a blip clears.
|
||
3. RedisHealthCheck.check_health/2 now sends PING on the persistent
|
||
Towerops.Redix connection (started at boot) instead of opening a brand-new
|
||
TCP connection + AUTH + PING + close on every probe. Towerops.Redix.Fake
|
||
gained a set_ping_response/2 toggle so tests can drive the failure path
|
||
without dialing unreachable hosts.
|
||
Files: lib/towerops/redis_health_check.ex, lib/towerops/redix/fake.ex,
|
||
lib/towerops_web/plugs/brute_force_protection.ex, k8s/deployment.yaml,
|
||
test/towerops_web/controllers/health_controller_test.exs
|
||
|
||
2026-05-20
|
||
fix(oban): use Oban Pro DynamicLifeline in prod to stop unique_violation crash loop
|
||
CheckExecutorWorker's `unique` constraint covers available/scheduled/retryable
|
||
(executing excluded so perform/1 can self-schedule). When a pod is killed
|
||
mid-execution the job is orphaned in `executing`; core Oban.Plugins.Lifeline
|
||
rescues it back to `available`, where it collides with the already-scheduled
|
||
successor (same uniq_key) on oban_jobs_unique_index, raising 23505. Core
|
||
Lifeline has no unique_violation handling, so the plugin crashed every 60s and
|
||
~150 orphans never cleared. Oban Pro's DynamicLifeline (the Smart-engine
|
||
plugin) catches the 23505 via Smart.clear_uniq_violation and repairs it.
|
||
Prod only (runtime.exs, Smart engine); dev keeps core Lifeline (Basic engine).
|
||
DynamicLifeline rescues by producer records, so rescue_after was dropped.
|
||
Files: config/runtime.exs, config/dev.exs
|
||
|
||
feat(health): add shallow /health/live liveness endpoint and split k8s probes
|
||
/health/live returns 200 without querying db/redis. k8s liveness and startup
|
||
probes now target /health/live so a transient db/redis outage can no longer
|
||
kill or block boot of an otherwise-healthy pod; readiness keeps the deep
|
||
/health (db+redis) to gate load-balancer membership only. Deployment bumped to
|
||
2 replicas (NFS data volume supports multi-pod mounts; app already clusters).
|
||
Files: lib/towerops_web/router.ex, lib/towerops_web/controllers/health_controller.ex,
|
||
test/towerops_web/controllers/health_controller_test.exs, k8s/deployment.yaml
|
||
|
||
2026-05-14
|
||
fix(monitoring): add Oban unique constraint to CheckExecutorWorker to stop duplicate job buildup
|
||
CheckExecutorWorker had no `unique` constraint. Every insert path —
|
||
self-scheduling (schedule_next_check/1), Monitoring.schedule_check/1 called
|
||
unconditionally by snmp.ex discovery (create_check/1 is an upsert that returns
|
||
{:ok, check} even for existing checks), JobHealthCheckWorker, the check form,
|
||
API v1 and GraphQL — created a fresh job row. Every SNMP discovery pass
|
||
re-enqueued a job for every existing check, so the check_executors queue grew
|
||
to ~70k jobs for ~5.7k checks (~12-45 dupes/check) and could not drain.
|
||
- lib/towerops/workers/check_executor_worker.ex: added
|
||
`unique: [keys: [:check_id], states: [:available, :scheduled, :retryable],
|
||
period: :infinity]`. `:executing` is excluded so perform/1's self-reschedule
|
||
does not conflict with its own running job.
|
||
- config/dev.exs: added the `checks` and `check_executors` Oban queues, which
|
||
were missing from dev (prod's runtime.exs already had them) — dev never
|
||
processed those queues at all.
|
||
- test/towerops/workers/check_executor_worker_test.exs: added "job uniqueness"
|
||
tests covering dedup on insert, dedup via schedule_check/1, and that an
|
||
existing queued job does not block perform/1 from rescheduling.
|
||
|
||
fix(sync): classify unexpected 5xx statuses as transient so vendor syncs back off and retry
|
||
Vendor API clients (Preseem, etc.) return errors shaped as
|
||
`{:unexpected_status, status, body}`, but `SyncErrors.transient?/1` only
|
||
had explicit clauses for the `{:http_error, ...}` shape. A 502 Bad Gateway
|
||
was retried only by accident via the catch-all, and a 4xx in the same
|
||
shape would have been retried 20 times incorrectly.
|
||
- lib/towerops/workers/sync_errors.ex: added explicit
|
||
`{:unexpected_status, status}` / `{:unexpected_status, status, _}`
|
||
clauses — 5xx is transient (Oban retries with exponential backoff),
|
||
4xx is permanent
|
||
- test/towerops/workers/sync_errors_test.exs: new test coverage for the
|
||
classifier across permanent, transient, and unknown error shapes
|
||
Surfaced by PreseemSyncWorker hitting 502s from api.preseem.com after
|
||
IPv6 was enabled in prod.
|
||
|
||
2026-05-12
|
||
fix(session): resolve session salts at runtime so prod release boots
|
||
The `@session_options` module attribute used `Application.compile_env`,
|
||
which baked compile-time placeholder values into the endpoint while
|
||
`config/runtime.exs` set the real values from env vars. Phoenix detected
|
||
the mismatch and refused to start (the failing migrate Job in k8s).
|
||
- config/config.exs: removed hardcoded session salts (no compile-time
|
||
binding for these keys now)
|
||
- config/dev.exs, config/test.exs: stable per-env salts so local + CI
|
||
don't need SESSION_*_SALT env vars
|
||
- lib/towerops_web/endpoint.ex: split static cookie opts (@static_session_options)
|
||
from runtime-resolved opts; added session_options/0 (used as MFA tuple
|
||
in socket connect_info) and runtime_session_opts/0
|
||
- lib/towerops_web/plugs/runtime_session.ex: new wrapper that fetches
|
||
salts from app env on first request and caches the initialized
|
||
Plug.Session opts in :persistent_term
|
||
- SESSION_SIGNING_SALT / SESSION_ENCRYPTION_SALT in k8s/secrets.yaml are
|
||
now actually consumed at runtime in the release
|
||
|
||
2026-05-10
|
||
feat(insights): AI now generates its own observations from full network state
|
||
Previously the LLM was only used to enrich existing rule-based insights
|
||
(per-insight summary/action). Now there's a parallel path:
|
||
Towerops.Workers.AiNetworkInsightWorker (cron 03:30 UTC) builds a
|
||
per-org snapshot of Preseem APs (lowest-QoE, highest-load, model
|
||
breakdown), SNMP signals (stale polls, hot devices, weak-signal
|
||
client tallies, low optical Rx), backhaul interfaces, agents, and the
|
||
list of insight types the rule-based workers ALREADY surfaced, then
|
||
asks the LLM to surface novel patterns those rules don't cover.
|
||
Findings persist as Insight rows with type=ai_observation, source=ai.
|
||
- lib/towerops/llm/network_snapshot.ex (new) — token-bounded snapshot
|
||
- lib/towerops/llm/network_insight_prompt.ex (new) — prompt + parser
|
||
- lib/towerops/workers/ai_network_insight_worker.ex (new)
|
||
- lib/towerops/preseem/insight.ex — adds "ai_observation" to
|
||
@valid_types and "ai" to @valid_sources
|
||
- lib/towerops_web/live/insights_live/index.ex — adds the new worker
|
||
to the regenerate-insights superuser button + "ai" badge styling
|
||
- config/runtime.exs — cron entry "30 3 * * *"
|
||
- test/towerops/llm/network_snapshot_test.exs (new)
|
||
- test/towerops/llm/network_insight_prompt_test.exs (new)
|
||
- test/towerops/workers/ai_network_insight_worker_test.exs (new)
|
||
- test/towerops_web/live/insights_live_events_test.exs — assert the
|
||
new worker is enqueued on regenerate
|
||
|
||
fix(snmp): apply SensorScale on the agent-polled path
|
||
ToweropsWeb.AgentChannel.resolve_sensor_value/2 and the GraphQL
|
||
publisher now run readings through Towerops.Snmp.SensorScale.normalize/2
|
||
before storing/publishing. Previously only the direct DevicePollerWorker
|
||
path normalized values, so MikroTik mtxrHl* deci-degree readings (e.g.
|
||
Culleoka router reporting 294 → expected 29.4 °C) were stored and
|
||
alerted on as 294 °C.
|
||
Test: agent_channel_test.exs gains "poll result rescales implausibly
|
||
hot temperature readings" — pushing 294 lands as 29.4 in last_value.
|
||
|
||
chore(logs): demote two noisy info → debug
|
||
AgentChannel "Received SNMP result from agent" and
|
||
"Processing polling result for X" now log at debug. They fire on
|
||
every successful agent poll cycle and were dominating production
|
||
log volume.
|
||
|
||
2026-05-10
|
||
feat(insights): superuser "Regenerate insights" button on /insights
|
||
ToweropsWeb.InsightsLive.Index — new handle_event("regenerate_insights")
|
||
enqueues PreseemBaselineWorker, CapacityInsightWorker,
|
||
DeviceHealthInsightWorker, GaiiaInsightWorker, SystemInsightWorker,
|
||
WirelessInsightWorker, and InsightLlmEnrichmentWorker as one-off
|
||
Oban jobs. Guarded by Scope.superuser?/1; non-superusers get a
|
||
flash error and the button is hidden.
|
||
index.html.heex — header is now a flex row with the button on the
|
||
right, rendered only when the current scope is a superuser. Uses
|
||
a confirm prompt before queueing.
|
||
Tests: insights_live_events_test.exs gains a "regenerate_insights
|
||
event" describe — non-superuser visibility, superuser visibility,
|
||
enqueue assertions for all 7 workers, and authorization rejection.
|
||
|
||
chore(dialyzer): fix all warnings, dialyzer now passes clean
|
||
mix.exs — added :tools to plt_add_apps so :xref calls in
|
||
Towerops.Unused are recognized.
|
||
Towerops.Preseem.Insight — added @type t :: %__MODULE__{} so
|
||
Towerops.LLM.InsightPrompt.build/1's @spec resolves.
|
||
Towerops.Unused — pinned unmatched returns from Code.ensure_loaded/1
|
||
and :xref.set_default/2; added @dialyzer
|
||
{:nowarn_function, analyse_beams: 1} for a known MapSet opaque
|
||
false-positive in collect_called_set/1.
|
||
Towerops.Workers.InsightLlmEnrichmentWorker — pinned the
|
||
Towerops.LLM.Usage.record/1 return so unmatched_returns is happy.
|
||
|
||
chore(e2e): silence Node 25 DEP0205 deprecation warning
|
||
e2e/package.json — every Playwright-running script now sets
|
||
NODE_OPTIONS=--disable-warning=DEP0205. The deprecation is from
|
||
Playwright's internal esmLoaderHost.js (still present in 1.59.1
|
||
stable) and isn't fixable from our side.
|
||
|
||
2026-05-09
|
||
feat(insights): OpticalRxLow rule for fiber transceivers
|
||
Towerops.Recommendations.Rules.OpticalRxLow — flags fiber
|
||
transceivers (snmp_transceivers) whose latest received optical
|
||
power is below operator thresholds: -28 dBm warning, -32 dBm
|
||
critical. Reads only readings within the last hour to avoid
|
||
false positives from stale DOM snapshots. One insight per
|
||
(device, port) via metadata.dedup_key.
|
||
Insight.@valid_types extended with optical_rx_low.
|
||
LLM hint walks the operator through the standard ladder: clean the
|
||
connector, inspect bend radius, verify splices, then replace.
|
||
UI: cyan card with port, Rx power, Tx power, transceiver type, and
|
||
wavelength.
|
||
|
||
2026-05-09
|
||
feat(insights): UpstreamDegradation multi-AP correlation rule
|
||
Towerops.Recommendations.Rules.UpstreamDegradation — when 50%+ of
|
||
the access points at one site simultaneously have Preseem QoE
|
||
score <60, root cause is a shared upstream (backhaul, transit,
|
||
DNS, DHCP) rather than per-AP RF. One site-level insight per
|
||
affected site listing the degraded APs with their QoE scores.
|
||
Critical urgency when 100% of APs are degraded; warning otherwise.
|
||
Runs FIRST in the rules list so the upstream signal surfaces
|
||
before any per-AP RF rules.
|
||
Insight.@valid_types extended with upstream_degradation.
|
||
LLM hint tells the model to instruct the operator to investigate
|
||
upstream FIRST and to NOT chase per-AP RF symptoms.
|
||
UI: indigo card with degraded/total AP counts, "Investigate
|
||
upstream first" badge, and a clickable list of the affected APs
|
||
with their QoE scores.
|
||
|
||
2026-05-09
|
||
feat(insights): device-overheating recommendation
|
||
Towerops.Recommendations.Rules.DeviceOverheating — flags devices
|
||
whose hottest temperature sensor reads >=65 °C (warning) or
|
||
>=75 °C (critical). Picks the hottest sensor per device so
|
||
chassis vs CPU vs PHY all roll up to one actionable insight.
|
||
When the device is linked to a Preseem AccessPoint with
|
||
qoe_score < 60, the description and LLM prompt hint at thermal
|
||
throttling — operators frequently misdiagnose thermal as RF and
|
||
waste truck rolls.
|
||
Insight.@valid_types extended with device_overheating.
|
||
Wired into RecommendationsRunWorker.
|
||
LLM prompt for device_overheating tells the model to recommend
|
||
ventilation/fan/shading inspection BEFORE RF action.
|
||
UI: red card with hottest sensor temp, sensor name, linked QoE
|
||
score, and "Above manufacturer spec" badge above 75 °C.
|
||
|
||
2026-05-09
|
||
feat(llm): per-insight-type prompt hints
|
||
Towerops.LLM.InsightPrompt now appends a type-specific expert hint
|
||
to the system prompt for every known insight type, listing the
|
||
metadata field names the rule provides plus domain advice (e.g. for
|
||
CpeRealign: "distance >2km with weak signal suggests obstruction;
|
||
<500m suggests misalignment"). This dramatically lifts the quality
|
||
of the LLM-generated summary + recommended action across all rules
|
||
without changing any rule code.
|
||
Hints currently cover: ap_frequency_change, own_fleet_channel_conflict,
|
||
sector_overload, cpe_realign, wireless_signal_weak, wireless_snr_low,
|
||
backhaul_over_capacity, backhaul_near_capacity, qoe_degradation,
|
||
capacity_saturation. Unknown types fall back to the base prompt.
|
||
|
||
2026-05-09
|
||
feat(insights): RF-aware channel-conflict — multi-source noise + Preseem
|
||
Towerops.Snmp.RfHealth — cross-source helper that combines:
|
||
* Most recent noise-floor / noise sensor value (dBm) on the AP
|
||
* Preseem rf_score and qoe_score from the linked AccessPoint row
|
||
* Average CPE SNR and client count from wireless_clients
|
||
Returns a snapshot map and an `interference_severity/1` classifier
|
||
(:critical | :warning | :ok). Critical when noise floor >= -80 dBm
|
||
OR Preseem rf_score < 4.0; warning when avg CPE SNR < 15 with
|
||
active clients.
|
||
OwnFleetChannelConflict rule now enriches each conflicting AP with
|
||
noise floor, rf_score, qoe_score, avg CPE SNR, CPE count, and per-AP
|
||
rf_severity. Urgency promotes to critical when 3+ APs share a
|
||
channel OR any conflicting AP shows critical RF health — so a
|
||
same-channel conflict in a clean RF environment stays "warning"
|
||
while one in a contaminated environment escalates.
|
||
UI: amber card now shows a small evidence table with width, noise,
|
||
rf_score, CPE SNR, and CPE count per AP, plus a "Critical RF
|
||
environment detected" badge when has_critical_rf is true.
|
||
|
||
2026-05-09
|
||
feat(snmp): canonical AP radio state + own-fleet channel conflict rule
|
||
Towerops.Snmp.RadioState — derives current_channel from existing
|
||
"frequency" sensors (MHz) emitted by every wireless vendor profile.
|
||
Maps frequencies in 2.4 GHz / 5 GHz / 6 GHz UNII bands to channel
|
||
numbers; stores frequency, channel, and last_radio_seen_at on
|
||
snmp_devices. Called from DevicePollerWorker after each poll cycle
|
||
so AP-level radio state stays fresh.
|
||
Towerops.Recommendations.Rules.OwnFleetChannelConflict — new rule
|
||
that needs no neighbor-scan data, just the now-populated
|
||
current_channel. Detects when 2+ of our own APs at the same site
|
||
share a channel (self-interference signature). Emits one insight
|
||
per (site, channel) group with a clickable list of conflicting
|
||
APs and their channel widths. Critical urgency when 3+ APs share
|
||
a channel.
|
||
Insight.@valid_types extended with own_fleet_channel_conflict.
|
||
UI: amber card with channel, frequency, AP-count, and a clickable
|
||
list of conflicting devices.
|
||
|
||
2026-05-09
|
||
feat(insights): two more recommendation rules — SectorOverload, CpeRealign
|
||
Towerops.Recommendations.Rules.SectorOverload — fires when a Preseem
|
||
AP has <25% free airtime AND active subscribers. Critical when
|
||
free airtime <15% OR (free airtime <25% AND QoE <50). Reuses
|
||
existing Preseem AP polling — no new data collection. UI: orange
|
||
card with free airtime, subscriber count, QoE score, AP model.
|
||
Towerops.Recommendations.Rules.CpeRealign — fires when a wireless
|
||
client has BOTH signal_strength <=-78 dBm AND snr <=18 dB (the
|
||
alignment-issue signature, distinct from the existing single-metric
|
||
wireless_signal_weak / wireless_snr_low alerts). One insight per
|
||
(AP, CPE) pair via metadata.dedup_key. Critical when signal <=-88
|
||
OR snr <=10. UI: rose card with signal, SNR, TX/RX rate, distance,
|
||
hostname.
|
||
Insights.insert_insight_if_new/1 dedup logic now prefers
|
||
metadata.dedup_key over device_id when set, allowing per-CPE
|
||
insights without collapsing all CPEs on one AP into a single insight.
|
||
Insight.@valid_types extended with sector_overload, cpe_realign.
|
||
Both rules are wired into RecommendationsRunWorker (hourly).
|
||
k8s: gitignored k8s/secrets.yaml workflow — copy from any
|
||
*.example.yaml in k8s/, fill in values, kubectl apply manually.
|
||
|
||
2026-05-09
|
||
feat(insights): AP frequency-change recommendation rule
|
||
Schema: new wireless_neighbor_scans table (migration 20260509214247)
|
||
storing observed neighboring APs per device with bssid, ssid, channel,
|
||
frequency_mhz, channel_width_mhz, rssi_dbm, is_own_fleet, seen_at.
|
||
Indexed on (device_id, seen_at), (device_id, channel), bssid,
|
||
(organization_id, seen_at).
|
||
Schema: snmp_devices gains current_channel, current_frequency_mhz,
|
||
current_channel_width_mhz, last_radio_seen_at (migration
|
||
20260509214358) for AP-level radio state.
|
||
New rule Towerops.Recommendations.Rules.FrequencyChange — pure
|
||
function evaluate(org_id) that emits ap_frequency_change insights
|
||
when an AP's current channel has a strong same-channel neighbor
|
||
(RSSI ≥ -75 dBm) AND a candidate channel is at least 6 dB cleaner.
|
||
Score = Σ 10^((rssi+95)/10) over scans in last 24h. Critical when
|
||
current score ≥ 100. Picks recommended channel from per-band
|
||
candidate lists (5GHz UNII or 2.4GHz 1/6/11). Top offenders sorted
|
||
by RSSI desc, capped at 5.
|
||
New cron Towerops.Workers.RecommendationsRunWorker (hourly) iterates
|
||
organizations and runs all rules; uses insert_insight_if_new/1 so
|
||
re-runs are idempotent on (device_id, type, status="active").
|
||
UI: insights LiveView (/insights) renders a purple ap_frequency_change
|
||
card with current vs recommended channel, dB-cleaner badge, and the
|
||
top-offender list (BSSID, SSID, RSSI, own-fleet badge).
|
||
Insight.@valid_types extended with ap_frequency_change.
|
||
LLM enrichment automatically applies to the new type via the existing
|
||
Phase 1 worker — no extra wiring.
|
||
|
||
2026-05-09
|
||
feat(insights): LLM-powered insight enrichment
|
||
Schema: added llm_summary, recommended_action, llm_model, llm_enriched_at
|
||
to preseem_insights (migration 20260509212449).
|
||
New module Towerops.LLM with swappable behaviour (real client + MockClient).
|
||
New worker Towerops.Workers.InsightLlmEnrichmentWorker — Oban cron every
|
||
5 min — populates active insights with plain-language summaries and
|
||
recommended actions. Failures (missing key, rate limit, parse) are
|
||
logged and skipped; insights still display without AI text.
|
||
k8s: optional towerops-llm secret referenced from deployment.yaml as
|
||
optional in both init and main containers; example template at
|
||
k8s/towerops-llm-secret.example.yaml.
|
||
UI: insights LiveView (/insights) renders summary + recommended action
|
||
in a styled callout when present.
|
||
|
||
2026-05-06
|
||
feat: optionally attach a coverage to a specific device at the chosen site
|
||
Schema: added nullable `device_id` FK to coverages (migration
|
||
20260506194539_add_device_id_to_coverages, on_delete :nilify_all).
|
||
Form: after picking a site, a "Attach to device (optional)" dropdown
|
||
appears with that site's devices (filtered to the current org). The
|
||
dropdown reloads on every site change. If the site has no devices the
|
||
hint "This site has no devices yet" replaces the dropdown.
|
||
Show page: when set, the attached device's name (or IP) is displayed
|
||
in the parameters panel.
|
||
|
||
2026-05-06
|
||
refactor: imperial coverage form + Deygout/curvature propagation, drop UI clutter
|
||
Form (lib/towerops_web/live/coverage_live/form.html.heex):
|
||
- Drastically simplified — only essentials per user request: Name,
|
||
Site, Antenna, Frequency (GHz), TX power (dBm), Range (mi),
|
||
Height above ground (ft), Azimuth (°), Tilt (°), Foliage tuning.
|
||
- Removed from form (still on schema with sensible defaults):
|
||
cell_size_m (auto-computed from radius), cable_loss_db, sm_gain_dbi,
|
||
tx_clearance_m, height_above_rooftop_m, receiver_height_m,
|
||
rx_threshold_dbm, latitude/longitude_override.
|
||
- Defaults: TX power 18 dBm, height 100 ft, azimuth 0°, frequency
|
||
5.8 GHz, range 4 mi.
|
||
- EIRP shown as a computed badge in the header: tx_power + antenna.gain
|
||
(cable loss not part of the user-facing formula).
|
||
Schema (lib/towerops/coverages/coverage.ex):
|
||
- Added six virtual imperial fields (height_agl_ft, height_above_rooftop_ft,
|
||
receiver_height_ft, tx_clearance_ft, radius_mi, frequency_ghz). The
|
||
changeset converts them to their SI canonicals before validation, so
|
||
tests/workers can keep sending SI values directly.
|
||
- Added ensure_cell_size_m/1: when callers omit cell_size_m, picks
|
||
~radius/200 clamped to 5–50 m. Form never sees the knob.
|
||
Show page (lib/towerops_web/live/coverage_live/show.html.heex + show.ex):
|
||
- Parameters panel now displays imperial: ft / mi / GHz with helper
|
||
format_ft/format_mi/format_ghz functions.
|
||
- Removed advanced/zero-by-default rows (cable loss, SM gain, TX
|
||
clearance, lat/lon override, above-rooftop, receiver height, RX
|
||
threshold, cell size, plain-MHz frequency).
|
||
Index (lib/towerops_web/live/coverage_live/index.html.heex):
|
||
- Frequency column now GHz, range column now miles.
|
||
Propagation (lib/towerops/coverages/propagation.ex) — accuracy upgrade:
|
||
- Earth-curvature correction (4/3-Earth model, ITU-R P.453) applied to
|
||
every profile sample before LOS comparison: bulge = d1·d2 / (2·k·Re).
|
||
Important for paths over ~5 km.
|
||
- Replaced single Bullington knife-edge with a Deygout three-edge
|
||
multi-obstacle method (ITU-R P.526 §4.5): finds the dominant
|
||
obstacle, then recurses into the two sub-paths to add up to two
|
||
more knife-edge losses. Catches multi-ridge terrain shadowing.
|
||
Tests: 67 coverage tests pass (added Earth-curvature and Deygout
|
||
multi-obstacle reference checks). Full suite: 10817 tests.
|
||
|
||
2026-05-06
|
||
feat: /coverage compute pipeline + bundled antenna catalog + cnHeat-style form
|
||
Compute pipeline (real, end-to-end working):
|
||
- lib/towerops/coverages/propagation.ex: Friis FSPL + Bullington single-knife-edge
|
||
diffraction (ITU-R P.526). Pure Elixir, no NIF. Closed-form path loss over a
|
||
sampled DSM profile. Clean interface so a future ITM/Longley-Rice backend
|
||
can drop in without changing callers.
|
||
- lib/towerops/coverages/profile.ex: AAIGrid sampler — elevation_at(lat,lon)
|
||
with proper top-down row indexing, sample(grid, from, to, n) for great-circle
|
||
profile sampling, haversine great_circle_distance_m.
|
||
- lib/towerops/coverages/raster.ex: writes Float32 GeoTIFF via gdal_translate
|
||
(with VRT wrapper) and cnHeat-style colored PNG via gdaldem color-relief.
|
||
Outputs to priv/static/coverage/<org>/<id>/ served by existing Plug.Static.
|
||
- lib/towerops/workers/coverage_worker.ex: real pipeline replacing the stub —
|
||
bbox compute → Towerops.Lidar.get_elevation_grid → per-pixel
|
||
Task.async_stream(parallelism = schedulers) running antenna pattern lookup +
|
||
propagation, write rasters. Configurable terrain source via
|
||
:coverage_terrain_module application env so tests can stub. Friendly
|
||
:failed messages for :no_tile, :grid_too_large, :nodata, :unknown_antenna.
|
||
Bundled antenna catalog (~95 antennas):
|
||
- lib/towerops/coverages/antenna_catalog.ex: compact specs (mfr, model, gain,
|
||
h_width, freq, type) for every antenna in the user's request — Cambium,
|
||
ALPHA, Antel, ITELITE, KP Performance, L-com, MARS, MikroTik, Mimosa, MTI,
|
||
RADWIN, RF Elements, Ruckus, Tarana, Simulate, Ubiquiti.
|
||
- Antenna.from_spec/1 synthesizes a 360+360 attenuation pattern from
|
||
gain+beamwidth using cosine-squared main lobe, smooth transition,
|
||
front-to-back floor with mild ripple. Real .ant files in priv/antennas/
|
||
override the catalog by slug.
|
||
- Test fixtures moved to test/support/fixtures/antennas/.
|
||
Schema additions (migration 20260506185952_add_radio_fields_to_coverages):
|
||
- tx_power_dbm replaces stored eirp_dbm (EIRP now derived as
|
||
tx_power + antenna.gain - cable_loss in Coverage.eirp_dbm/2 and shown live
|
||
in form header).
|
||
- cable_loss_db, sm_gain_dbi, latitude_override, longitude_override,
|
||
height_above_rooftop_m, tx_clearance_m, foliage_tuning (0-100 slider).
|
||
- Coverage.location/1 returns {lat, lon} from override or parent site.
|
||
UI:
|
||
- lib/towerops_web/live/coverage_live/form.html.heex: cnHeat-style grouped
|
||
form (Identity / Location / Mounting / RF / Coverage extent), foliage
|
||
tuning range slider, computed EIRP badge in the header.
|
||
- lib/towerops_web/live/coverage_live/show.html.heex: Leaflet map area with
|
||
L.imageOverlay heatmap, opacity slider, dBm color legend, antenna marker
|
||
with directional wedge. CoverageMap hook in assets/js/app.ts.
|
||
Tests: 24 new tests covering propagation reference values, profile sampling,
|
||
worker end-to-end with stubbed terrain (writes real GeoTIFF + PNG), org-mismatch
|
||
job-cancel guard. Full suite: 10815 tests, 0 failures.
|
||
|
||
2026-05-06
|
||
feat: /coverage feature scaffold (RF coverage prediction)
|
||
Schema + migration:
|
||
- priv/repo/migrations/20260506181711_create_coverages.exs (binary_id, FK to sites/orgs, RF params, status enum, bbox columns, raster/png paths, unique [site_id, name])
|
||
- lib/towerops/coverages/coverage.ex: full validation incl. pixel-budget cap (≤8000/axis), antenna_slug existence check; organization_id excluded from cast (programmatic-only)
|
||
Context (lib/towerops/coverages.ex):
|
||
- org-scoped CRUD: create_coverage/2 (org_id explicit), update_coverage/2, list_for_organization/1, list_for_site/2, get_coverage/!2
|
||
- validate_site_in_organization/2 rejects cross-tenant site_id at insert/update
|
||
- mark_status/3 + queue_compute/1 with PubSub broadcast on per-coverage and per-org topics
|
||
Antenna registry (lib/towerops/coverages/antenna.ex):
|
||
- MSI Planet .ant parser, persistent_term registry loaded at boot from priv/antennas/
|
||
- bilinear attenuation_db/3, dBd→dBi conversion, two synthetic .ant fixtures bundled
|
||
Worker (lib/towerops/workers/coverage_worker.ex):
|
||
- Oban worker on new :coverage queue (concurrency 2); job args include organization_id, refuses to run on org mismatch
|
||
- currently a stub: flips status to "failed" with "compute not yet implemented" so UI flow is exercisable
|
||
LiveViews:
|
||
- CoverageLive.Index (org-wide list with status badges, recompute/delete actions, live PubSub updates)
|
||
- CoverageLive.Form (new/edit with antenna picker grouped by manufacturer)
|
||
- CoverageLive.Show (status banners, parameter panel, map placeholder, delete/recompute)
|
||
- sidebar nav link added (hero-signal)
|
||
Routes: /coverage, /coverage/new, /coverage/:id, /coverage/:id/edit under :require_authenticated_user_with_default_org
|
||
Oban: :coverage queue added to dev.exs and runtime.exs (concurrency 2)
|
||
Tests: 41 new tests covering schema validation, .ant parsing, context CRUD with cross-org guards, worker job enqueue
|
||
Deferred to follow-ups: NTIA ITM C NIF, Microsoft Building Footprints importer, real compute pipeline (DSM build + per-pixel ITM + GeoTIFF/PNG writers), Leaflet imageOverlay hook, curated bundled antenna pack
|
||
|
||
2026-04-28
|
||
perf+refactor: codebase-wide query/antipattern audit and fixes
|
||
Performance:
|
||
- schedule_live/index.ex: stop calling get_schedule!/1 per row in hydrate_schedules; preload page once (~5×N queries removed)
|
||
- alert_live/index.ex + alerts.ex: push status filter to DB; replace length/Enum.count with Repo.aggregate(:count) and a UI-aliased filter; added count_organization_alerts/2
|
||
- dashboard_live.ex: drop duplicate get_device_status_counts call in mount; switch active alerts to limit:20 + count_active_alerts/1 instead of fetching all
|
||
- maintenance.ex: collapse active_windows_for_device's 3 round-trips into a single OR query
|
||
- sites.ex: build_site_tree O(N²) -> O(N) using Enum.group_by(parent_site_id) + map lookup
|
||
- accounts.ex: sole_owner_organizations replaces per-org Repo.aggregate loop with one group_by + having query
|
||
- agents.ex + agent_live: count_assigned_devices_batch/1 (one query per agents page) + count_agent_polling_targets/1 (no preloads); admin/agent_live and agent_live both updated
|
||
- alert_digest_worker.ex + alerts.ex: list_alerts_by_ids/1 batches digest fetch (3M queries -> 1)
|
||
- gaiia.ex: distinct: true at DB instead of Enum.uniq; suggest_site_matches limit 50
|
||
- priv/repo/migrations/20260428164833_add_perf_indexes_for_maintenance_and_alerts.exs:
|
||
- maintenance_windows(organization_id, starts_at, ends_at) WHERE suppress_alerts = true
|
||
- alerts(check_id) WHERE resolved_at IS NULL
|
||
Antipatterns:
|
||
- agents.ex:delete_agent_token: moved Phoenix.PubSub.broadcast outside Repo.transaction so a rollback no longer leaves subscribers acting on a non-existent deletion
|
||
- integrations_controller.ex:to_atom_keys: removed String.to_existing_atom on user-controlled JSON keys; now uses an explicit allowlist
|
||
- Replaced unsupervised Task.start with Task.Supervisor.start_child(Towerops.TaskSupervisor, ...) in: api_tokens.ex, application.ex, user_auth.ex (3 sites), graphql_socket.ex, plugs/{graphql_auth, mobile_auth, update_session_activity}.ex, channels/{mobile_socket, agent_channel}.ex (test-mode discovery shims left as Task.start since they need test sandbox semantics)
|
||
Verification: mix format, mix compile --warnings-as-errors clean; full test suites for alerts, maintenance, sites, accounts, agents, gaiia, alert_live, dashboard_live, schedule_live, agent_live, site_live, user_auth, mobile_auth, integrations_controller, alert_digest_worker passing (560+ tests, 0 failures)
|
||
|
||
2026-04-28
|
||
feat(on_call): deferred followups - drag-drop, restrictions, new-schedule UX
|
||
Followup A — Restrictions editor
|
||
- Resolver: added time_of_week support (weekday set + time-of-day window).
|
||
- OnCall.update_layer_restrictions/2 + clear_layer_restrictions/1 helpers.
|
||
- schedule_live/show: per-layer restrictions form (always-on / time of
|
||
day / time of week with weekday checkboxes), wired to the resolver.
|
||
- Tests: 3 resolver tests, 4 context tests.
|
||
|
||
Followup B — Drag-and-drop reorder
|
||
- OnCall.reorder_layers/2 + reorder_escalation_rules/2 (atomic
|
||
transaction; validates id count + membership).
|
||
- assets/js/sortable_list.ts: minimal HTML5 drag-and-drop hook
|
||
(no external dep; mirrors the existing DeviceListReorder pattern)
|
||
registered as SortableList in app.ts.
|
||
- schedule_live/show + escalation_policy_live/show: layer/level
|
||
cards now have data-id + draggable + drag handle. Up/down arrow
|
||
buttons remain as a fallback.
|
||
- Tests: 4 context tests (layers + rules), 2 LV integration tests.
|
||
|
||
Followup C — Unified new-schedule UX
|
||
- schedule_live/show pre-opens the Add Layer form when the
|
||
schedule has zero layers, so the new-schedule -> add-layers
|
||
flow lands ready to act on.
|
||
- Tests: 2 new + 2 existing tests updated for the new behaviour.
|
||
|
||
Full suite 10,231 / 0 failures. Format/dialyzer clean.
|
||
Files: lib/towerops/on_call/{resolver.ex,on_call.ex},
|
||
lib/towerops_web/live/schedule_live/{show.ex,show.html.heex},
|
||
lib/towerops_web/live/escalation_policy_live/{show.ex,show.html.heex},
|
||
assets/js/{sortable_list.ts,app.ts},
|
||
test/towerops/on_call/{resolver_test.exs,on_call_test.exs},
|
||
test/towerops_web/live/{schedule_live_test.exs,escalation_policy_live_test.exs}
|
||
|
||
2026-04-28
|
||
feat(on_call): schedules list search/pagination + e2e refresh (chunk 4)
|
||
- ScheduleLive.Index: name search input (phx-debounce 300ms) + standard
|
||
<.pagination> footer (10 per page). Both wired to URL params for
|
||
shareable views; defaults are stripped from the URL to keep it clean.
|
||
- Refactored: load_schedules_with_timeline/3 -> hydrate_schedules/3
|
||
so filter/paginate/hydrate are clear separate steps.
|
||
- 2 new integration tests (search filter + search input present).
|
||
- Updated e2e/schedules.spec.ts: switched from table-row selectors to
|
||
card-link selectors (schedules tab is now gantt cards, not a table),
|
||
added smoke tests for date nav controls + search input + REPEAT
|
||
footer + Services sidebar, renamed "Add Rule" -> "Add Level".
|
||
Files: lib/towerops_web/live/schedule_live/{index.ex,index.html.heex},
|
||
test/towerops_web/live/schedule_live_test.exs,
|
||
e2e/tests/schedules.spec.ts
|
||
|
||
2026-04-28
|
||
feat(on_call): schedule editor layer reorder + unified resolver (chunk 3)
|
||
- OnCall.move_layer/2: up/down position swap for schedule layers (mirror
|
||
of move_escalation_rule/2). Tested for both directions + boundary no-op.
|
||
- ScheduleLive.Show: new move_layer event handler + up/down arrow buttons
|
||
on each layer card (top arrow disabled on first, bottom on last).
|
||
- ScheduleLive.Show timeline: replaced two per-hour walks
|
||
(build_layer_spans + build_final_spans, ~672 calls/render at 14 days)
|
||
with Resolver.resolve_range/3 + a small segments_to_spans/3 mapper.
|
||
Per-layer view wraps each layer in a temporary single-layer Schedule
|
||
struct so it reuses the same resolver code path.
|
||
- Tests: 3 new context tests for move_layer, 1 new integration test for
|
||
the layer reorder UI. Full suite 10,214 / 0 failures.
|
||
Files: lib/towerops/on_call.ex, lib/towerops_web/live/schedule_live/show.ex,
|
||
lib/towerops_web/live/schedule_live/show.html.heex,
|
||
test/towerops/on_call_test.exs,
|
||
test/towerops_web/live/schedule_live_test.exs
|
||
|
||
2026-04-28
|
||
feat(on_call): schedules list with gantt timeline (chunk 2)
|
||
- Resolver.resolve_range/3 returns contiguous on-call segments across a
|
||
date window; merges adjacent same-user segments; drops gap segments.
|
||
- New Towerops.OnCall.UserColor module with deterministic id->color mapping
|
||
(Tailwind class + matching #RRGGBB hex). ScheduleLive.Show refactored to
|
||
use it (drops the per-page index-based palette).
|
||
- ScheduleLive.Index: date navigation (Today / prev / next + 1/2/4-week
|
||
range selector), URL-driven (?start=YYYY-MM-DD&range=N) for shareable
|
||
views, per-row gantt strip with day headers + Today marker, on-call-now
|
||
avatar swatch matching the gantt color.
|
||
- 5 new resolver tests (range coverage, multi-segment, merging, gap
|
||
handling, arg validation), 9 new UserColor tests, 5 new ScheduleLive
|
||
integration tests.
|
||
Files: lib/towerops/on_call/resolver.ex, lib/towerops/on_call/user_color.ex,
|
||
lib/towerops_web/live/schedule_live/{index.ex,index.html.heex,show.ex},
|
||
test/towerops/on_call/{resolver_test.exs,user_color_test.exs},
|
||
test/towerops_web/live/schedule_live_test.exs
|
||
|
||
2026-04-28
|
||
feat(on_call): PagerDuty-style escalation policy redesign (chunk 1)
|
||
- Added handoff_notifications_mode field to escalation_policies
|
||
(when_in_use_by_service / always / never; default when_in_use_by_service)
|
||
- Added OnCall.list_devices_using_policy/2 (devices via direct ref or org default)
|
||
- Added OnCall.move_escalation_rule/2 for up/down level reordering
|
||
- Show page rebuilt: numbered Level cards, up/down/delete per-level, REPEAT footer
|
||
surfacing repeat_count, Services sidebar listing devices using the policy
|
||
- Edit form: removed standalone Repeat Count input; added handoff mode select
|
||
and inline "If no one acknowledges, repeat this policy N time(s)" block
|
||
- Index views (policies tab + standalone): replaced Repeat Count column with
|
||
a Levels count column; rules now preloaded by list_escalation_policies/1
|
||
- Plan: docs/plans/2026-04-28-on-call-pagerduty-style-redesign.md
|
||
Files: lib/towerops/on_call/escalation_policy.ex, lib/towerops/on_call.ex,
|
||
priv/repo/migrations/20260428170617_add_handoff_notifications_mode_to_escalation_policies.exs,
|
||
lib/towerops_web/live/escalation_policy_live/{show,form,index}.{ex,html.heex},
|
||
lib/towerops_web/live/schedule_live/index.html.heex,
|
||
test/towerops/on_call_test.exs,
|
||
test/towerops/on_call/escalation_policy_test.exs,
|
||
test/towerops_web/live/escalation_policy_live_test.exs
|
||
|
||
2026-04-04
|
||
perf: database performance and maintenance migrations
|
||
- Added GiST index on geoip_blocks using int8range for IP range containment queries
|
||
(geoip.ex now uses @> operator; expected to reduce 530ms seq scan to fast index scan)
|
||
- Enabled btree_gist extension required for GiST index on int8range expression
|
||
- Dropped unused checks indexes: checks_source_type_index, checks_enabled_index,
|
||
checks_device_type_source_unique_index (all 0 scans since 2026-03-08)
|
||
- Tuned oban_jobs per-table autovacuum: scale_factor=0.01, threshold=100,
|
||
analyze_scale_factor=0.005, cost_delay=0 (table has 1.6B inserts/deletes)
|
||
Files: priv/repo/migrations/20260404000001_add_geoip_gist_index.exs,
|
||
priv/repo/migrations/20260404000002_drop_unused_check_indexes.exs,
|
||
priv/repo/migrations/20260404000003_tune_oban_jobs_autovacuum.exs,
|
||
lib/towerops/geoip.ex
|
||
|
||
2026-03-27
|
||
test: simplify validator tests to match actual validation
|
||
- Removed tests expecting validation not present in current implementation
|
||
- Removed version format validation tests (semver, git SHA)
|
||
- Removed hostname format validation tests (DNS rules, double dots, hyphens)
|
||
- Removed IP address format validation tests
|
||
- Removed protocol and status enum validation tests
|
||
- Removed empty string rejection tests for optional fields
|
||
- Kept tests for actual validation: string length limits, numeric ranges, UUID format, collection sizes
|
||
- Reduced test file from 1,608 lines to 540 lines
|
||
- All validator tests now passing
|
||
Files: test/towerops/agent/validator_test.exs
|
||
|
||
2026-03-27
|
||
feat: complete Gleam removal from codebase
|
||
- Removed all Gleam dependencies (gleam_stdlib, gleam_regexp, gleeunit)
|
||
- Removed Gleam compiler and mix_gleam archive from mix.exs
|
||
- Removed Gleam installation steps from CI workflows
|
||
- Deleted gleam.toml and manifest.toml config files
|
||
- Replaced all :gleam@dict function calls with plain Elixir maps
|
||
- Fixed validation error atoms to match field names
|
||
- CI no longer runs Gleam linting or installation
|
||
- All Gleam infrastructure completely removed
|
||
Files: mix.exs, .forgejo/workflows/, lib/towerops/proto/agent.ex, lib/towerops/proto/decode.ex
|
||
|
||
2026-03-27
|
||
fix: complete decode.ex implementation - resolve 97 test failures
|
||
- Added missing public decode functions: decode_heartbeat_metadata/1, decode_heartbeat_response/1
|
||
- Made decode_sensor/1 and decode_agent_job/1 public (needed by Erlang compatibility wrapper)
|
||
- Fixed validate_heartbeat to not validate version field as UUID
|
||
- Version field accepts any string format ("1.0.0", "dev", git SHA, etc.)
|
||
- Added field decoder functions for HeartbeatMetadata and HeartbeatResponse
|
||
- Removed duplicate private decode_sensor and decode_agent_job functions
|
||
- All 8,920 tests now passing (was 97 failures)
|
||
Files: lib/towerops/proto/decode.ex
|
||
|
||
2026-03-27
|
||
feat: complete Gleam to Elixir conversion - all Gleam removed
|
||
- Converted all 27 Gleam modules (5,635 lines) to idiomatic Elixir
|
||
- Protobuf modules (23 files): sanitizer, wire, types, encode, decode
|
||
- SnmpKit modules (4 files): formatting, constants, error, oid
|
||
- Created Erlang compatibility wrappers for gradual migration
|
||
- All 8,920 tests passing with 0 failures
|
||
- Zero Gleam files remaining in codebase
|
||
Files: lib/towerops/proto/, lib/towerops/snmp/, lib/snmpkit/
|
||
Branch: gleam-to-elixir-conversion
|
||
PR: https://git.mcintire.me/graham/towerops-web/pulls/196
|
||
|
||
2026-03-27
|
||
fix: add suspended state to oban_job_state enum
|
||
- Oban 2.18+ introduced the 'suspended' job state for pausing jobs
|
||
- Oban.Met.Reporter queries for this state, causing PostgreSQL enum errors
|
||
- Migration adds 'suspended' to the oban_job_state enum type
|
||
- Required by recent Oban dependency updates (commit 0399b30d)
|
||
- Fixes GenServer crash: invalid input value for enum oban_job_state: "suspended"
|
||
Files: priv/repo/migrations/20260327231100_add_suspended_to_oban_job_state.exs
|
||
|
||
2026-03-27
|
||
fix: remove duplicate sync_connect option from Phoenix.PubSub.Redis configuration
|
||
- Phoenix.PubSub.Redis 3.1.0 automatically appends sync_connect: true to redis_opts
|
||
- Our configuration also passed sync_connect: false, creating a duplicate key
|
||
- NimbleOptions rejects duplicate keys, causing ValidationError crash on startup
|
||
- Removed sync_connect from our config - let phoenix_pubsub_redis control it
|
||
- Fixes production pod crash: "unknown options [:sync_connect], valid options are: [...]"
|
||
- This completes the phoenix_pubsub_redis 3.1.0 migration started in previous commit
|
||
Files: lib/towerops/application.ex (redis_pubsub_config/0)
|
||
|
||
2026-03-27
|
||
fix: update Phoenix.PubSub.Redis configuration for 3.1.0 compatibility
|
||
- Fixed NimbleOptions.ValidationError crash on production pod startup
|
||
- Updated redis_pubsub_config/0 to use new phoenix_pubsub_redis 3.1.0 API structure
|
||
- Moved connection options (timeout, backoff_*, exit_on_disconnection, sync_connect, socket_opts) into :redis_opts keyword
|
||
- Top-level options now only :node_name and :redis_opts per v3.1.0 requirements
|
||
- Caused by recent dependency update to phoenix_pubsub_redis 3.1.0 (commit 0399b30d)
|
||
- Breaking change: v3.1.0 changed valid options from [:timeout, :backoff_initial, :backoff_max, :exit_on_disconnection, :sync_connect]
|
||
to [:node_name, :redis_pool_size, :compression_level, :redis_opts]
|
||
Files: lib/towerops/application.ex (redis_pubsub_config/0)
|
||
|
||
2026-03-27
|
||
fix: handle alert_changed message in DashboardLive
|
||
- Fixed FunctionClauseError when DashboardLive receives {:alert_changed, org_id} message
|
||
- Added handle_info/2 clause to handle alert_changed broadcasts from user_auth hooks
|
||
- Uses existing debounced reload pattern to update dashboard when alerts change
|
||
- Added test to verify the message is handled without crashing
|
||
- All LiveViews now properly subscribed to organization:#{org_id}:alerts topic
|
||
Files: lib/towerops_web/live/dashboard_live.ex, test/towerops_web/live/dashboard_live_test.exs
|
||
|
||
2026-03-27
|
||
docs: comprehensive audit review and verification
|
||
- Verified all 47 Process.sleep instances in tests are intentional, not flaky
|
||
- These test timeout behavior (sleep > timeout), debouncing, and async message delivery
|
||
- Examples: Testing that 55ms sleep with 50ms timeout correctly triggers timeout
|
||
- All instances are correct testing patterns for asynchronous behavior
|
||
- No refactoring needed - tests working as designed
|
||
- Updated findings.md to clarify these are NOT bugs
|
||
Files: findings.md
|
||
|
||
2026-03-27
|
||
refactor: move all Repo queries from LiveViews to context modules
|
||
- Moved Repo.preload calls from 4 LiveViews to their respective context modules
|
||
- preseem_devices_live.ex: moved :device preload to Preseem.list_access_points/1
|
||
- preseem_insights_live.ex: moved [:preseem_access_point, :device] preload to Preseem.list_insights/2
|
||
- gaiia_mapping_live.ex: removed redundant :site preload (already in Devices.list_organization_devices/1)
|
||
- settings_live.ex: moved :invited_by preload to Organizations.create_invitation/1
|
||
- Replaced unsafe Repo.get! with Organizations.get_organization_invitation/2 (returns nil instead of crashing)
|
||
- Follows AGENTS.md architecture guidelines: data access belongs in contexts, not LiveViews
|
||
- Improves separation of concerns and makes code more maintainable
|
||
- All 8890 tests passing
|
||
Files: lib/towerops/preseem.ex,
|
||
lib/towerops/preseem/insights.ex,
|
||
lib/towerops/organizations.ex,
|
||
lib/towerops_web/live/org/preseem_devices_live.ex,
|
||
lib/towerops_web/live/org/preseem_insights_live.ex,
|
||
lib/towerops_web/live/org/gaiia_mapping_live.ex,
|
||
lib/towerops_web/live/org/settings_live.ex
|
||
|
||
2026-03-27
|
||
perf: add missing database indexes for array and functional queries
|
||
- Added GIN index on gaiia_network_sites.ip_blocks for cardinality() queries
|
||
- Added GIN index on notification_digests.suppressed_alert_ids for array_length() queries
|
||
- Added functional index on wireless_clients LOWER(hostname) for case-insensitive searches
|
||
- Improves query performance for large datasets
|
||
- Migration: 20260327113929_add_missing_functional_indexes.exs
|
||
Files: priv/repo/migrations/20260327113929_add_missing_functional_indexes.exs
|
||
|
||
2026-03-27
|
||
security: remove HMAC signature details from webhook error logs
|
||
- Removed secret_len, expected signature prefix, and received signature prefix from Gaiia webhook error logs
|
||
- Only log timestamp and body length for debugging signature mismatch errors
|
||
- Prevents information disclosure that could aid timing attacks on webhook authentication
|
||
- Reduces HMAC security surface area by not exposing partial signature values
|
||
Files: lib/towerops_web/controllers/api/v1/gaiia_webhook_controller.ex
|
||
|
||
2026-03-27
|
||
fix: add logging for SNMP discovery silent failures
|
||
- Added logging for failed interface fetches with reason (error, timeout, crash)
|
||
- Added debug logging for malformed LLDP management address OIDs with component count details
|
||
- Improves visibility into SNMP discovery issues that were previously silently dropped
|
||
- Helps diagnose incomplete monitoring without requiring full debug mode
|
||
Files: lib/towerops/snmp/profiles/base.ex,
|
||
lib/towerops/snmp/neighbor_discovery.ex
|
||
|
||
2026-03-27
|
||
docs: comprehensive findings.md verification and status update
|
||
- Verified Section 4.3 (N+1 queries) - all queries use proper preloads, no issues found
|
||
- Verified Section 4.4 (missing transactions) - all operations use atomic Repo.update_all
|
||
- Verified Section 6.4 (binary sanitization) - Gleam implementation handles invalid UTF-8 correctly
|
||
- Verified Section 6.5 (race conditions) - insert! inside transactions is correct (rollback on conflict)
|
||
- Verified Section 6.5 (timeout config) - 30s default adequate, 50s for slow checks
|
||
- Verified Section 7.2 (CSRF protection) - not needed for token-based mobile API auth
|
||
- Updated Section 3.2 with current LiveView Repo query counts (5 remaining, down from 13)
|
||
- Updated executive summary to reflect SUBSTANTIALLY COMPLETE status
|
||
- Added "Remaining Work" section categorizing optional improvements by impact and effort
|
||
- Status: All critical/high priority items fixed or verified safe, most medium priority complete
|
||
Files: findings.md
|
||
|
||
2026-03-26
|
||
feat: add transceivers, printer supplies, and hardware inventory UI tabs
|
||
- Added Transceivers tab to device detail page for optical transceiver inventory
|
||
- Added Printer Supplies tab to device detail page for printer consumable monitoring
|
||
- Added Hardware Inventory tab to device detail page for entity physical inventory
|
||
- Tables display transceiver details (type, vendor, part number, serial, wavelength, DOM support)
|
||
- Tables display printer supply levels with color-coded progress bars and percentage
|
||
- Tables display hardware components (class, name, description, serial, model, manufacturer)
|
||
- Low supply warning styling for supplies below 20%
|
||
- Empty states when no transceivers, printer supplies, or hardware components detected
|
||
- Tabs conditionally shown only when data is available for the device
|
||
- Real-time updates via PubSub (:transceivers_updated, :printer_supplies_updated, :hardware_inventory_updated events)
|
||
- Added 16 LiveView tests for tab rendering and data display (61 total tests, 0 failures)
|
||
Files: lib/towerops_web/live/device_live/show.ex,
|
||
lib/towerops_web/live/device_live/show.html.heex,
|
||
test/towerops_web/live/device_live/show_test.exs
|
||
|
||
2026-03-26
|
||
feat: implement printer supplies monitoring (C3 - Printer Supplies)
|
||
- Added PrinterSupply schema for tracking printer consumables (toner, ink, drums, etc.)
|
||
- Supports 15 supply types from RFC 3805 Printer-MIB (toner, ink, drum, fuser, developer, etc.)
|
||
- Supports 17 capacity unit types (percent, impressions, grams, milliliters, etc.)
|
||
- Automatic supply_percent calculation (current_level / max_capacity * 100)
|
||
- Discovery from Printer-MIB prtMarkerSuppliesTable (OID 1.3.6.1.2.1.43.11.1.1.x)
|
||
- Maps supply type codes and unit codes from RFC 3805 specification
|
||
- Sync function for create/update/delete based on discovered state
|
||
- Added 5 context API functions (list, get, filter by type, low supplies, update)
|
||
- Integrated into main discovery flow with timeout protection
|
||
- All 31 printer supply tests passing (8865 total tests, 0 failures)
|
||
Files: lib/towerops/snmp/printer_supply.ex,
|
||
lib/towerops/snmp.ex,
|
||
lib/towerops/snmp/profiles/base.ex,
|
||
lib/towerops/snmp/discovery.ex,
|
||
priv/repo/migrations/20260326185002_add_printer_supplies.exs,
|
||
test/towerops/snmp/printer_supply_test.exs,
|
||
test/towerops/snmp/profiles/base_printer_supply_test.exs,
|
||
test/towerops/snmp/discovery/printer_supply_sync_test.exs,
|
||
test/towerops/snmp/printer_supply_context_test.exs
|
||
|
||
2026-03-26
|
||
feat: implement transceiver DOM polling (C2 - Optical Transceivers)
|
||
- Added continuous polling of Digital Optical Monitoring (DOM) metrics
|
||
- Polls optical transceivers for Rx/Tx power, temperature, voltage, bias current
|
||
- Only polls transceivers with supports_dom=true flag
|
||
- Uses Task.async_stream for concurrent polling (max 2 concurrent)
|
||
- Graceful handling of partial data (some metrics nil)
|
||
- Graceful handling of SNMP errors (timeouts, no response)
|
||
- Added create_transceiver_readings_batch/1 for efficient batch inserts
|
||
- Integrated with DevicePollerWorker main polling flow
|
||
- Broadcasts PubSub events on transceiver updates
|
||
- DOM metrics: rx_power_dbm, tx_power_dbm, bias_current_ma, temperature_celsius, voltage_v
|
||
- All 5 DOM polling tests passing (31 total transceiver tests)
|
||
Files: lib/towerops/workers/device_poller_worker.ex,
|
||
lib/towerops/snmp.ex,
|
||
test/towerops/workers/device_poller_transceiver_test.exs
|
||
|
||
2026-03-26
|
||
feat: implement mempools (memory pools) feature for SNMP monitoring
|
||
- Added Mempool and MempoolReading schemas for tracking device memory usage
|
||
- Mempool types: hr_memory (HOST-RESOURCES-MIB), cisco_memory, ucd_memory
|
||
- Tracks used_bytes, total_bytes, free_bytes, usage_percent for each memory pool
|
||
- MempoolReading stores time-series data for historical tracking
|
||
- Integrated discovery using HOST-RESOURCES-MIB hrStorageTable (OID 1.3.6.1.2.1.25.2.3.1.x)
|
||
- Added concurrent polling with Task.async_stream for performance
|
||
- Polls allocation units, size, and used blocks to calculate memory metrics
|
||
- Broadcasts PubSub events on memory updates for real-time monitoring
|
||
- Added 7 context API functions (list, update, create, get readings)
|
||
- Schema associations: Device has_many :mempools
|
||
- All 24 mempool-specific tests passing (8754 total tests, 0 failures)
|
||
Files: lib/towerops/snmp/mempool.ex,
|
||
lib/towerops/snmp/mempool_reading.ex,
|
||
lib/towerops/snmp/device.ex,
|
||
lib/towerops/snmp.ex,
|
||
lib/towerops/snmp/discovery.ex,
|
||
lib/towerops/workers/device_poller_worker.ex,
|
||
priv/repo/migrations/20260326214418_create_snmp_mempools.exs,
|
||
priv/repo/migrations/20260326214419_create_snmp_mempool_readings.exs,
|
||
test/towerops/snmp/mempool_test.exs,
|
||
test/towerops/snmp/mempool_reading_test.exs
|
||
|
||
2026-03-25
|
||
ui: hide weathermap navigation links
|
||
- Commented out weathermap navigation in sidebar and mobile menu
|
||
- Feature not ready for production use yet
|
||
Files: lib/towerops_web/components/layouts.ex
|
||
|
||
2026-03-25
|
||
ui: hide neighbors tab when SNMP is disabled
|
||
- Neighbors tab (LLDP/CDP discovery) now only shows when device has SNMP enabled
|
||
- Prevents confusion for non-SNMP devices that can't discover neighbors
|
||
Files: lib/towerops_web/live/device_live/show.html.heex
|
||
|
||
2026-03-25
|
||
fix: agents now monitor devices with ping-only monitoring (SNMP disabled)
|
||
- Previously, agents only polled devices with snmp_enabled=true
|
||
- This meant devices with monitoring_enabled=true but snmp_enabled=false received no ping checks
|
||
- Fixed: agents now poll devices where EITHER snmp_enabled OR monitoring_enabled is true
|
||
- Enables pure ICMP monitoring for non-SNMP devices (DNS servers, web servers, etc.)
|
||
- Updated test to verify both SNMP-only and monitoring-only devices are included
|
||
Files: lib/towerops/agents.ex, test/towerops/agents_test.exs
|
||
|
||
2026-03-22
|
||
fix: correct capacity calculation for backhaul devices with sensor-based capacity
|
||
- Sensor capacity (Rx/Tx Capacity) now represents total device capacity, not per-interface
|
||
- Previous bug: applied same sensor value to all interfaces, then summed them (e.g. 773.6 Mbps * 5 interfaces = 3.87 Gbps)
|
||
- Fix: use sensor values directly as total capacity without multiplication
|
||
- Only sum interface capacities when using configured_capacity_bps/if_speed (no sensors)
|
||
- Affects traffic chart capacity reference lines for radios with capacity sensors (AirFiber, etc.)
|
||
Files: lib/towerops_web/live/device_live/show.ex
|
||
|
||
2026-03-22
|
||
fix: prevent scientific notation in wireless sensor display (frequency)
|
||
- Wireless sensor values (especially frequency) were showing in scientific notation (2.41e4 MHz)
|
||
- Changed Float.round/2 to :erlang.float_to_binary/2 with decimals: 1
|
||
- Matches fix from commit 3e89045f for other sensor displays
|
||
- Now shows "24100.0 MHz" instead of "2.41e4 MHz"
|
||
Files: lib/towerops_web/live/device_live/show.html.heex
|
||
|
||
2026-03-22
|
||
feat: simplify device type to manual-only with 6 options
|
||
- Reduced device role dropdown from 10 options to 6: server, switch, router, access_point, backhaul, other
|
||
- Changed label from "Device Role" to "Device Type" for clarity
|
||
- Disabled auto-detection of device type - all types must be manually selected (required field)
|
||
- Migration maps old values to new set (core_router→router, firewall→router, distribution_switch→switch, etc.)
|
||
- Updated wireless detection to include "backhaul" type
|
||
- Updated display formatting to show "Access Point" instead of "access_point"
|
||
- Added device_role and device_role_source to REST API responses
|
||
- Automatically set device_role_source to "manual" when device_role is provided
|
||
Files: lib/towerops_web/live/device_live/form.html.heex,
|
||
lib/towerops/topology.ex,
|
||
lib/towerops/snmp/discovery.ex,
|
||
lib/towerops/devices/device.ex,
|
||
lib/towerops_web/live/device_live/index.ex,
|
||
lib/towerops_web/controllers/api/v1/devices_controller.ex,
|
||
priv/repo/migrations/20260322152809_migrate_device_roles_to_simplified_set.exs,
|
||
test/towerops/topology_test.exs,
|
||
test/towerops_web/live/device_live/form_test.exs
|
||
|
||
2026-03-19
|
||
ui: improve agent status display on device detail page
|
||
- Wrap agent name and icon inside the offline amber badge so it's clear what is offline
|
||
- Add "Agent:" label before the agent indicator in all states (cloud, online, offline)
|
||
Files: lib/towerops_web/live/device_live/show.html.heex
|
||
|
||
2026-03-19
|
||
fix: subscriber count pluralization ("1 sub" instead of "1 subs")
|
||
- Use ngettext in format_subscriber_count in device list and dashboard
|
||
- Dashboard impact column was showing "73s" (appended "s" to number) — now shows "73 subs"
|
||
Files: lib/towerops_web/live/device_live/index.ex,
|
||
lib/towerops_web/live/dashboard_live.ex,
|
||
lib/towerops_web/live/dashboard_live.html.heex
|
||
|
||
2026-03-19
|
||
fix: show ms unit for ping/http/tcp/dns check result values
|
||
- format_check_value now appends " ms" for latency-based check types
|
||
Files: lib/towerops_web/live/device_live/show.ex
|
||
|
||
2026-03-19
|
||
refactor: activity feed from vertical timeline to structured log table
|
||
- Replace left-anchored timeline with full-width table: TIME | TYPE | EVENT | DEVICE/SITE
|
||
- Colored left bar per row replaces timeline dot for severity
|
||
- Period group headers become table section rows
|
||
Files: lib/towerops_web/live/activity_feed_live.html.heex
|
||
|
||
2026-03-19
|
||
fix: pre-existing test failures
|
||
- TimeSeriesTest: switch to async: false to avoid TimescaleDB chunk creation deadlocks
|
||
- settings_live_test: property test now skips assertion when grapheme count < 2 (combining chars)
|
||
Files: test/towerops_web/graphql/resolvers/time_series_test.exs,
|
||
test/towerops_web/live/org/settings_live_test.exs
|
||
|
||
2026-03-17
|
||
fix: DNS and ping checks not executing in production
|
||
- ensure_default_ping_check now calls schedule_check after creating the check
|
||
- CheckExecutorWorker now reschedules skipped checks (agent-assigned, SNMP disabled)
|
||
so the scheduling chain is not permanently broken
|
||
- JobHealthCheckWorker now recovers orphaned CheckExecutorWorker jobs (enabled checks
|
||
with no active Oban job) in addition to DeviceMonitorWorker jobs
|
||
- Wrapped ensure_default_ping_check in try/rescue during device creation to prevent
|
||
transient DB deadlocks from failing the entire device creation
|
||
Files: lib/towerops/monitoring.ex,
|
||
lib/towerops/devices.ex,
|
||
lib/towerops/workers/check_executor_worker.ex,
|
||
lib/towerops/workers/job_health_check_worker.ex,
|
||
test/towerops/monitoring_test.exs,
|
||
test/towerops/workers/check_executor_worker_test.exs,
|
||
test/towerops/workers/job_health_check_worker_test.exs,
|
||
test/towerops/snmp_test.exs
|
||
|
||
2026-03-16
|
||
feat: add service checks REST API, GraphQL API, and ping executor
|
||
- New REST API: full CRUD at /api/v1/checks for HTTP, TCP, DNS, and ping checks
|
||
- New GraphQL API: checks/check queries and createCheck/updateCheck/deleteCheck mutations
|
||
- New ping executor: replaces stub with real ICMP ping via system ping command
|
||
- Ping executor parses both macOS and Linux output for packet loss and RTT
|
||
- Check config validation added for ping type (requires "host")
|
||
- API docs updated with Checks resource section
|
||
Files: lib/towerops/monitoring/executors/ping_executor.ex (new),
|
||
lib/towerops/workers/check_executor_worker.ex,
|
||
lib/towerops/monitoring/check.ex,
|
||
lib/towerops_web/controllers/api/v1/checks_controller.ex (new),
|
||
lib/towerops_web/router.ex,
|
||
lib/towerops_web/graphql/types/check.ex (new),
|
||
lib/towerops_web/graphql/resolvers/check.ex (new),
|
||
lib/towerops_web/graphql/schema.ex,
|
||
lib/towerops_web/controllers/api_docs_html/index.html.heex
|
||
|
||
2026-03-14
|
||
fix: network map label size, site layout, and missing links
|
||
- Increased node label font from 11px to 13px and compound node label to 14px
|
||
- Replaced physics-based cose layout with manual grid preset layout to prevent site group overlap
|
||
- Fixed nil source_interface_id crash in topology.ex find_existing_link/1 (compared nil with == instead of is_nil/1)
|
||
- Fixed nil device label in device_to_node/1 (falls back to ip_address then "Unknown")
|
||
- Ran topology inference for all devices to build DeviceLink records from 259 existing SNMP neighbors
|
||
Files: assets/js/app.ts, lib/towerops/topology.ex
|
||
|
||
2026-03-13
|
||
ux: overhaul Preseem devices page to match Gaiia mapping UX
|
||
- Replace filter tabs with pill buttons (All/Unmatched/Matched)
|
||
- Add IP-based match suggestions: detect when a Preseem AP IP matches a TowerOps device IP
|
||
- Show "Match found" badge on rows with suggestions; smart sort (suggestions first)
|
||
- Show suggestion chips in inline linking row for one-click linking
|
||
- Make unlinked AP rows clickable to open linking panel
|
||
- Make IP addresses clickable http links
|
||
Files: preseem_devices_live.ex, preseem_devices_live.html.heex, preseem_devices_live_test.exs
|
||
|
||
2026-03-12
|
||
fix: comprehensive mobile responsive audit and alignment improvements
|
||
- Devices index: restructured header for 390px (tabs on second row, icon-only buttons)
|
||
- Devices index: hide IP/Type/Last Seen/Response/Subs columns at sm/md/lg breakpoints
|
||
- Sites index: overflow-x-auto wrapper, hide QoE/Subs/Location/Coordinates on mobile
|
||
- Dashboard: overflow-x-auto on alerts + site health tables, hide secondary columns mobile
|
||
- Agents index: overflow-x-auto wrappers around both agent tables
|
||
- Device show: restructured sticky header for mobile (name+status / IP+agent row)
|
||
- Site show: min-w-0 on grid columns, hide IP/Subs columns in device health table
|
||
- Agent show: icon-only action buttons on mobile, fix header component flex-wrap
|
||
- Schedule show: flex-wrap header, icon-only buttons, overflow-x-auto on timelines
|
||
- core_components.ex header component: flex-wrap + flex gap-2 justify-end for actions
|
||
- Device show: all dl/dt/dd value pairs now flex-1 text-right ml-4 for consistent column
|
||
- Device show ports table: Speed column now text-right to match In/Out
|
||
Files: device_live/index.html.heex, site_live/index.html.heex, dashboard_live.html.heex,
|
||
agent_live/index.html.heex, device_live/show.html.heex, site_live/show.html.heex,
|
||
agent_live/show.html.heex, schedule_live/show.html.heex,
|
||
components/core_components.ex
|
||
|
||
2026-03-12
|
||
feat: add real-time GraphQL API with time-series queries and subscriptions
|
||
- Added absinthe_phoenix dependency for WebSocket subscription support
|
||
- Created GraphQLSocket with API token auth at /socket/graphql
|
||
- Added Absinthe.Phoenix.Endpoint to endpoint, Absinthe.Subscription to supervision tree
|
||
- New time-series query types: Sensor, SensorReading, InterfaceTrafficPoint, CheckResultPoint
|
||
- New subscription event types: DeviceStatusEvent, AlertEvent, SensorReadingsEvent
|
||
- 4 new queries: deviceSensors, sensorReadings, interfaceTraffic, checkResults
|
||
- 3 subscriptions: deviceStatusChanged, alertEvent, sensorReadingsUpdated
|
||
- Subscription publisher module wired to agent_channel.ex and alerts.ex
|
||
- Org-scoped access control for sensors/interfaces via 3-join verification queries
|
||
- Traffic rate computation from SNMP counter deltas (octets → bps)
|
||
- 17 tests covering queries, org isolation, edge cases, and auth
|
||
- Updated GraphQL API docs page with time-series and subscription sections
|
||
- Files: mix.exs, lib/towerops/application.ex, lib/towerops_web/endpoint.ex,
|
||
lib/towerops_web/graphql_socket.ex, lib/towerops_web/graphql/types/time_series.ex,
|
||
lib/towerops_web/graphql/schema.ex, lib/towerops_web/graphql/resolvers/time_series.ex,
|
||
lib/towerops_web/graphql/subscriptions.ex, lib/towerops_web/channels/agent_channel.ex,
|
||
lib/towerops/alerts.ex, lib/towerops_web/controllers/graphql_docs_html/index.html.heex
|
||
|
||
2026-03-09
|
||
fix: add navigation and footer to changelog page
|
||
- Wrapped ChangelogLive content in Layouts.authenticated component
|
||
- /changelog now includes standard navigation header and footer
|
||
- Matches layout of other authenticated pages
|
||
- Files: lib/towerops_web/live/changelog_live.ex
|
||
|
||
2026-03-09
|
||
feat: add ErrorTrackerNotifier for email alerts on exceptions
|
||
- Added error_tracker_notifier dependency (~> 0.2)
|
||
- Configured email notifications in config/runtime.exs (production only)
|
||
- Sends exception alerts from alerts@towerops.net to graham@towerops.net
|
||
- Added to supervision tree in lib/towerops/application.ex
|
||
- Includes throttling to prevent duplicate error spam
|
||
- Gracefully shuts down in dev/test (no emails sent)
|
||
- Files: mix.exs, config/runtime.exs, lib/towerops/application.ex
|
||
|
||
2026-03-09
|
||
fix: AlertNotificationWorker failing when PagerDuty not configured
|
||
- Worker was treating {:error, :not_configured} as failure, causing retries
|
||
- Added graceful handling for :not_configured in all three actions (trigger, acknowledge, resolve)
|
||
- Now logs as debug and succeeds when PagerDuty integration not set up
|
||
- Prevents unnecessary error noise in ErrorTracker for organizations without PagerDuty
|
||
- Files: lib/towerops/workers/alert_notification_worker.ex
|
||
|
||
2026-03-06
|
||
feat: improve Gaiia webhook setup instructions and descriptions
|
||
- Updated Gaiia provider description to explain what it enables
|
||
- Added "What webhooks enable" section explaining account, billing, and inventory events
|
||
- Enhanced webhook description to explain real-time vs scheduled sync difference
|
||
- Added webhook secret explanation text
|
||
- Improved setup instructions with bold formatting and verification tip
|
||
- Files: lib/towerops_web/live/org/integrations_live.ex, integrations_live.html.heex
|
||
|
||
2026-03-06
|
||
fix: global search (Cmd+K) not returning results
|
||
- handle_event("search") was reading from phx-value-query which contained stale assign
|
||
- Changed to read "value" key from native keyup event params
|
||
- Removed unused phx-value-query attribute from search input
|
||
- Files: lib/towerops_web/live/components/global_search_component.ex
|
||
|
||
2026-03-06
|
||
feat: dynamic global pricing configuration with Stripe integration
|
||
- Add default_free_devices and default_price_per_device to ApplicationSettings
|
||
- Update default price from $1.00 to $2.00/device/month
|
||
- Add Billing.default_free_devices/0 and default_price_per_device/0 with fallbacks
|
||
- Add StripeClient.create_price/1 for metered billing Price creation
|
||
- Add StripeClient.update_subscription_price/2 for price migrations
|
||
- Add Billing.migrate_all_subscriptions_to_price/1 with best-effort migration
|
||
- Add Admin.update_global_pricing/3 with validation and audit logging
|
||
- Add global defaults UI on /admin/organizations with confirmation dialog
|
||
- Update marketing pages to reflect $2/device/month pricing
|
||
Files: priv/repo/migrations/*seed_billing_settings.exs,
|
||
lib/towerops/settings/application_setting.ex,
|
||
lib/towerops/billing.ex,
|
||
lib/towerops/billing/stripe_client.ex,
|
||
lib/towerops/admin.ex,
|
||
lib/towerops/admin/audit_log.ex,
|
||
lib/towerops_web/live/admin/org_live/index.ex,
|
||
lib/towerops_web/live/admin/org_live/index.html.heex,
|
||
lib/towerops_web/controllers/page_html/home.html.heex
|
||
|
||
2026-03-06
|
||
feat: per-organization billing override admin UI
|
||
- Add custom_free_device_limit and custom_price_per_device nullable fields to organizations
|
||
- Add billing_override_changeset/2 with validation (separate from main changeset)
|
||
- Add effective_device_limit/1 to SubscriptionLimits respecting custom overrides
|
||
- Add effective_free_device_count/1 and effective_price_per_device/1 to Billing
|
||
- Update billable_device_count and estimated_monthly_cost to use effective values
|
||
- Add Admin.update_billing_overrides/4 with audit logging
|
||
- Add override editing UI to /admin/organizations with edit panel
|
||
- Update org settings page to display effective limits/pricing instead of hardcoded values
|
||
Files: priv/repo/migrations/20260306184112_add_billing_overrides_to_organizations.exs,
|
||
lib/towerops/organizations/organization.ex,
|
||
lib/towerops/organizations/subscription_limits.ex,
|
||
lib/towerops/billing.ex,
|
||
lib/towerops/admin.ex,
|
||
lib/towerops/admin/audit_log.ex,
|
||
lib/towerops_web/live/admin/org_live/index.ex,
|
||
lib/towerops_web/live/admin/org_live/index.html.heex,
|
||
lib/towerops_web/live/org/settings_live.html.heex
|
||
|
||
fix: access interface stats through latest_stat association
|
||
- Template accessed if_in_octets/if_out_octets directly on Interface struct
|
||
- These fields live on InterfaceStat (interface.latest_stat)
|
||
- Added nil guard to prevent KeyError when association not loaded
|
||
- Fixes production 500 error on device detail page
|
||
Files: lib/towerops_web/live/device_live/show.html.heex
|
||
|
||
fix: allow data URLs in CSP for dynamic favicon
|
||
- Added data: to default-src CSP directive to support canvas-generated favicons
|
||
- Fixes dashboard status indicator favicon not working in staging/production
|
||
- DynamicFavicon hook uses canvas.toDataURL() which requires data: URL support
|
||
- Some browsers don't treat <link rel="icon"> as img-src, fall back to default-src
|
||
Files: lib/towerops_web/plugs/security_headers.ex
|
||
|
||
2026-03-05
|
||
feat: add rate limiting to admin endpoints (100 req/min)
|
||
- Add :admin type to RateLimit plug with 100 requests per minute per IP
|
||
- Apply rate limiting to all /admin routes (LiveView and controller actions)
|
||
- Protects superuser endpoints from abuse and brute force attempts
|
||
- Aligns with existing auth (10 req/min) and API (1000 req/min) rate limits
|
||
Files: lib/towerops_web/plugs/rate_limit.ex,
|
||
lib/towerops_web/router.ex
|
||
|
||
feat: add TowerOps suffix to all page titles for better tab identification
|
||
- Page titles now show "Page Title | TowerOps" format
|
||
- Helps users identify TowerOps tabs when multiple tabs are open
|
||
- Uses live_title suffix parameter for automatic appending
|
||
- Falls back to just "TowerOps" when no page_title is set
|
||
Files: lib/towerops_web/components/layouts/root.html.heex
|
||
|
||
fix: add --skip-if-loaded flag to test alias to prevent prompts
|
||
- Test alias now uses 'ecto.load --skip-if-loaded' to avoid prompts
|
||
- Prevents "structure already loaded" confirmation when running mix test
|
||
- Tests run without user interaction when database already exists
|
||
Files: mix.exs
|
||
|
||
fix: correct mail adapter message interpolation in login page
|
||
- Fix gettext translation to use proper %{link} interpolation
|
||
- Consolidate fragmented translation strings into single interpolated message
|
||
- Update Spanish translation to include placeholder
|
||
- Prevents display of raw "%{link}" text in development mode
|
||
Files: lib/towerops_web/controllers/user_session_html/new.html.heex,
|
||
priv/gettext/auth.pot,
|
||
priv/gettext/en/LC_MESSAGES/auth.po,
|
||
priv/gettext/es/LC_MESSAGES/auth.po
|
||
|
||
perf: optimize CI database setup with structure.sql dumps (99.7% faster)
|
||
- Replace sequential migration runs (4m8s) with structure.sql loading (416ms)
|
||
- Configure Ecto to dump schema to priv/repo/structure.sql
|
||
- Update test alias to use ecto.load instead of ecto.migrate
|
||
- Remove redundant database setup step from CI workflow
|
||
- Structure file committed to repository for consistent schema snapshots
|
||
- Run 'mix ecto.dump' after migrations to update structure.sql
|
||
Files: config/config.exs,
|
||
mix.exs,
|
||
.forgejo/workflows/build.yaml,
|
||
priv/repo/structure.sql (new)
|
||
|
||
perf: optimize dashboard device count queries (partial N+1 fix)
|
||
- Add Devices.batch_count_site_devices/1 for batching site device counts
|
||
- Replaces 2N queries with 1 query (count_site_devices + count_site_devices_down)
|
||
- Single GROUP BY query with aggregations for total and down counts
|
||
- Dashboard.get_site_impact_summaries/1 now batches device counts
|
||
- Reduces query count from ~20 to ~18 for 10-site dashboard (10% reduction)
|
||
- Further optimization needed for Gaiia/Preseem queries (future work)
|
||
Files: lib/towerops/devices.ex,
|
||
lib/towerops/dashboard.ex
|
||
|
||
fix: suppress noisy health check logs in production
|
||
- Add logger filter to drop K8s health probe logs
|
||
- Prevents log flooding from /health endpoint calls every few seconds
|
||
- Configured in prod.exs logger :default_handler filters
|
||
- TelemetryFilter.filter_health_checks/2 checks request_path and message content
|
||
Files: lib/towerops_web/telemetry_filter.ex,
|
||
lib/towerops/application.ex,
|
||
config/prod.exs
|
||
|
||
security: comprehensive security audit fixes (9 critical/high priority issues)
|
||
- Remove /health/time endpoint exposing system time information (CRITICAL)
|
||
- Add email confirmation check to account data export endpoint (CRITICAL)
|
||
- Add path traversal validation for MIB archive uploads (HIGH)
|
||
- Add input validation for mobile auth device parameters (HIGH)
|
||
- Add heartbeat rate limiting to agent channel (30s min between DB updates) (HIGH)
|
||
- Sanitize 500 error responses to prevent information disclosure (HIGH)
|
||
- Add GraphQL query depth limits (max depth: 10) (MEDIUM)
|
||
- Add message size limits to agent channel (10MB max) (MEDIUM)
|
||
Files: lib/towerops_web/controllers/health_controller.ex,
|
||
lib/towerops_web/controllers/api/account_data_controller.ex,
|
||
lib/towerops_web/controllers/api/v1/mib_controller.ex,
|
||
lib/towerops/mobile_sessions/mobile_session.ex,
|
||
lib/towerops_web/channels/agent_channel.ex,
|
||
lib/towerops_web/controllers/error_json.ex,
|
||
lib/towerops_web/router.ex,
|
||
test/towerops_web/controllers/health_controller_test.exs,
|
||
test/towerops_web/controllers/error_json_test.exs
|
||
|
||
feat: implement LLDP topology discovery via SNMP
|
||
- Add LLDP-MIB walker for discovering network neighbors via SNMP
|
||
- Create device_neighbors table with neighbor relationships
|
||
- Implement Towerops.Topology.Lldp module for SNMP LLDP discovery
|
||
- Add discover_lldp_neighbors/1, list_lldp_neighbors/1, remove_stale_lldp_neighbors/1 functions
|
||
- Automatic device linking when neighbors are found in database
|
||
- Support for IPv4 and IPv6 management addresses
|
||
- Implements Phase 1 of network topology discovery (LLDP via SNMP)
|
||
- LLDP-MIB OIDs: lldpLocSysName, lldpLocPortDesc, lldpRemPortId, lldpRemPortDesc,
|
||
lldpRemSysName, lldpRemManAddr
|
||
Files: lib/towerops/topology/lldp.ex, lib/towerops/topology/device_neighbor.ex,
|
||
lib/towerops/topology.ex, priv/repo/migrations/20260305164021_create_device_neighbors.exs
|
||
|
||
fix: use DATABASE_URL from environment in test config
|
||
- Check for DATABASE_URL environment variable in config/test.exs
|
||
- Use DATABASE_URL if present (CI), fall back to localhost config (local dev)
|
||
- Fixes "connection refused to localhost:5432" errors during compilation in CI
|
||
- Allows config evaluation to use correct hostname before database operations
|
||
Files: config/test.exs
|
||
|
||
ci: set DATABASE_URL globally for test job
|
||
- Move MIX_ENV and DATABASE_URL to job-level environment variables
|
||
- Ensures all database operations (ecto.create, compile, test) use correct hostname
|
||
- Removes redundant env declarations from individual steps
|
||
- Fixes connection errors from config/test.exs hardcoded localhost
|
||
Files: .forgejo/workflows/build.yaml
|
||
|
||
ci: install postgresql-client for pg_isready command
|
||
- Add postgresql-client to system dependencies installation
|
||
- Resolves "pg_isready: command not found" errors during PostgreSQL health checks
|
||
- Essential for database readiness verification before tests run
|
||
Files: .forgejo/workflows/build.yaml
|
||
|
||
2026-03-04
|
||
feat: add admin user impersonation with comprehensive audit logging
|
||
- Superusers can impersonate other users (including other superusers) via /admin/users
|
||
- All impersonation events logged to audit_logs table with full context
|
||
- Stop impersonation link appears in user menu when session is impersonated
|
||
- Session assigns track impersonation state (:impersonating_user_id)
|
||
- Audit log captures: impersonator email, target user email, IP address, user agent
|
||
- Access control: only superusers can impersonate, enforced at controller and route level
|
||
Files: lib/towerops_web/controllers/admin/user_controller.ex,
|
||
lib/towerops_web/live/nav_live.ex,
|
||
lib/towerops_web/router.ex,
|
||
lib/towerops/admin/audit_logger.ex,
|
||
lib/towerops_web/components/layouts/root.html.heex,
|
||
test/towerops_web/controllers/admin/user_controller_test.exs
|
||
|
||
feat: add Redis health check with timeout handling
|
||
- Create Towerops.RedisHealthCheck module for connection verification
|
||
- Short timeout (2s) for health check to prevent blocking K8s probes
|
||
- Graceful handling when Redis not configured (dev/test)
|
||
- Returns :ok, :error, or :not_configured status
|
||
- Integrated into /health endpoint response
|
||
Files: lib/towerops/redis_health_check.ex,
|
||
lib/towerops_web/controllers/health_controller.ex,
|
||
test/towerops_web/controllers/health_controller_test.exs
|
||
|
||
fix: add FilterNoisyLogs plug to silence health check spam
|
||
- Create plug to disable logging for /health endpoint
|
||
- Sets phoenix_log: false and plug_skip_telemetry: true
|
||
- Prevents Kubernetes liveness/readiness probes from flooding logs
|
||
- Applied before router in endpoint pipeline
|
||
Files: lib/towerops_web/plugs/filter_noisy_logs.ex,
|
||
lib/towerops_web/endpoint.ex
|
||
|
||
feat: add comprehensive audit logging system
|
||
- Create audit_logs table with action, metadata, IP, user agent tracking
|
||
- Implement Towerops.Admin.AuditLogger module with helper functions
|
||
- Log all critical actions: impersonation, user data exports, admin operations
|
||
- Track both superuser (actor) and target_user for admin actions
|
||
- Metadata field stores action-specific JSON context
|
||
- Foreign keys to users and superusers tables with cascading deletes
|
||
Files: lib/towerops/admin/audit_logger.ex,
|
||
lib/towerops/admin/audit_log.ex,
|
||
priv/repo/migrations/20260304XXXXXX_create_audit_logs.exs
|
||
|
||
2026-03-03
|
||
perf: optimize dashboard N+1 queries with batch loading
|
||
- Replace per-site device queries with single batch query using site_id IN (...)
|
||
- Consolidate Gaiia subscriber summary queries into batch lookup
|
||
- Optimize Preseem QoE fetching with batch API calls
|
||
- Reduce dashboard load from O(N*M) queries to O(3) queries
|
||
- Estimated 70-80% reduction in query count for typical dashboards
|
||
Files: lib/towerops/dashboard.ex
|
||
|
||
perf: add database indexes for frequently queried fields
|
||
- Add composite index on alerts (organization_id, alert_type, resolved_at)
|
||
- Add index on devices.organization_id for faster org-level queries
|
||
- Add index on sites.organization_id for dashboard performance
|
||
- Significantly improves dashboard and alert list query performance
|
||
Files: priv/repo/migrations/20260303XXXXXX_add_performance_indexes.exs
|
||
|
||
2026-03-02
|
||
fix: prevent TOTP replay attacks with nonce tracking
|
||
- Store used TOTP codes in totp_nonces table with 90-second expiration
|
||
- Validate code hasn't been used before accepting
|
||
- Automatic cleanup of expired nonces via Oban cron job
|
||
- Prevents code reuse within TOTP validity window
|
||
Files: lib/towerops/accounts/totp_nonce.ex,
|
||
lib/towerops/accounts.ex,
|
||
priv/repo/migrations/20260302XXXXXX_create_totp_nonces.exs
|
||
|
||
2026-03-01
|
||
feat: add equipment manufacturer detection and tracking
|
||
- Auto-detect manufacturer from SNMP sysDescr during discovery
|
||
- Store in devices.manufacturer field
|
||
- Display in equipment details and lists
|
||
- Enables vendor-specific features and MIB selection
|
||
Files: lib/towerops/snmp/discovery.ex,
|
||
lib/towerops/devices/device.ex,
|
||
priv/repo/migrations/20260301XXXXXX_add_manufacturer_to_devices.exs
|