- Override mint to 1.9.3 to fix HTTP/1 chunk-size parser CVE-2026-59249
- Remove plug_cowboy dependency by switching PromEx standalone metrics server
to PromEx.Plug in the main endpoint (Bandit), eliminating cowlib CVEs
- Update castore transitive dep 1.0.19→1.0.20
L2: Remove 2-second Process.sleep from post_startup — the try/catch
already handles transient noproc errors from the TaskSupervisor.
L3: Remove Process.sleep from delete_device — in-flight jobs already
handle the race condition via verify_polling_assignment_unchanged.
The blocking sleep was unnecessary and delayed web requests by 500ms.
Prod paired the Oban Pro Smart engine + unique workers with core
Oban.Plugins.Lifeline, whose rescue has no unique_violation handling. An
orphaned executing CheckExecutorWorker job rescued back to available collides
with its already-scheduled successor on oban_jobs_unique_index (23505),
crashing the plugin every 60s and leaving ~150 orphans stuck. Switch prod to
Oban.Pro.Plugins.DynamicLifeline, which repairs the conflict via
Smart.clear_uniq_violation. Dev keeps core Lifeline (Basic engine).
Add a shallow /health/live endpoint that does not touch db/redis. k8s liveness
and startup probes now target it so a transient dependency outage can't kill or
block boot of a healthy pod; readiness keeps the deep /health to gate the load
balancer. (deployment.yaml probe/replica change pushed separately, after the
image carrying /health/live is live.)
Also drop unused POSTGRES_* keys from the secrets example.
CheckExecutorWorker had no Oban `unique` constraint, so every insert path
created a fresh job row: self-scheduling, JobHealthCheckWorker, the check
form/API/GraphQL, and — the main amplifier — Monitoring.schedule_check/1
called unconditionally by snmp.ex discovery after create_check/1, which is
an upsert that returns {:ok, check} even when the check already exists.
Every SNMP discovery pass re-enqueued a job for every existing check, so
the check_executors queue grew to ~70k jobs for ~5.7k checks and stopped
draining.
Add `unique: [keys: [:check_id], states: [:available, :scheduled,
:retryable], period: :infinity]`. `:executing` is excluded so perform/1's
self-reschedule does not conflict with its own running job.
Also add the `checks` and `check_executors` queues to config/dev.exs —
they were missing from dev entirely (prod's runtime.exs already had them).
The @session_options module attribute used Application.compile_env, which
baked compile-time placeholders into the endpoint while runtime.exs set
the real values from SESSION_SIGNING_SALT / SESSION_ENCRYPTION_SALT env
vars. Phoenix detected the mismatch and refused to start (failing migrate
Job in k8s).
- Remove hardcoded salts from config/config.exs (no compile-time binding)
- Add stable per-env salts in dev.exs / test.exs so local + CI don't need
the env vars
- Split static cookie opts (@static_session_options) from runtime-resolved
opts in endpoint.ex; expose session_options/0 as an MFA tuple in socket
connect_info so LiveView decodes sessions with the same runtime salts
- New ToweropsWeb.Plugs.RuntimeSession wraps Plug.Session, fetches salts
from app env on first request, and caches the initialized opts in
:persistent_term (zero per-request overhead after warm-up)
SESSION_SIGNING_SALT, SESSION_ENCRYPTION_SALT, and LIVE_VIEW_SIGNING_SALT
are now loaded from environment variables in production. Dev/test keep the
previous defaults via config.exs. k8s/secrets.yaml has placeholder entries;
the user fills in real values before applying.
- H12: session and remember-me cookies get http_only + secure (prod-only).
Cookie can no longer be read via document.cookie (XSS exfil defense)
and the Secure flag is set in production via config/prod.exs.
- L2: 404 tracker uses EXPIRE … NX so a sustained probe can't keep
refreshing the 60s window and dodge the threshold ban.
- L5: vault only reads CLOAK_KEY when :env == :prod — a developer with
a prod env var set in their shell won't accidentally encrypt local
data with the production key.
- L6: health endpoint no longer leaks the app version.
- L8: Preseem.dismiss_insight/2 + InsightsLive uses it — dismissing now
requires the insight to belong to the user's org (closes IDOR).
- L10: SidebarCollapse JS hook stores its click handler and removes it
in destroyed(); listeners no longer accumulate across LV navigation.
- L11: WebMCP navigate tool rejects anything that isn't a same-origin
absolute path (blocks javascript:, data:, off-site URLs).
Adds a parallel LLM path that reviews the whole org snapshot — Preseem
APs, SNMP signals, backhaul, agents, and the insight types the rules
already covered — and emits novel observations as ai_observation
insights (source=ai). Existing rule-based insights and per-insight
enrichment continue unchanged.
- lib/towerops/llm/network_snapshot.ex builds the bounded snapshot
- lib/towerops/llm/network_insight_prompt.ex builds the prompt and
parses the LLM's JSON observations array
- lib/towerops/workers/ai_network_insight_worker.ex runs nightly at
03:30 UTC (after the rule pass + per-insight enrichment) and on
the superuser regenerate button
- preseem/insight.ex extends @valid_types with "ai_observation" and
@valid_sources with "ai"
- insights_live/index.ex includes the new worker in the regen list
and adds an "AI" source badge style
Bumps the SnmpKit Manager timeout-options test threshold from 500ms
to 3000ms (was flaking on loaded dev machines), and scopes
list_unenriched_insights/1 assertion in insights_test:573 to the
test's org so cross-suite leaks don't fail the count.
If a rule keeps re-firing the underlying condition, the dedup query
suppresses duplicate inserts but `inserted_at` of the live row stays
old — so a real 6-month-old issue and a one-off blip both linger
forever. Auto-resolving any active insight older than 7 days clears
the dashboard. Still-real issues will be re-created by the next rule
pass (dedup only checks `status: "active"`), so the operator sees
fresh evidence rather than stale text.
- Towerops.Workers.InsightExpiryWorker (queue: :maintenance) flips
`status` to "resolved" on rows where `inserted_at < now - 7d`.
- Window is configurable via `:max_age_days` for ops who want a
different cadence per env.
- Oban cron: 04:00 UTC nightly, after the 02:00 rule pass and 03:00
LLM enrichment.
UI:
- Type-specific evidence cards (red/amber/orange/rose/purple/cyan/
indigo/blue) had `dark:text-{color}-300` labels and
`dark:bg-{color}-800 dark:text-{color}-100` badges that washed out on
the dark background. Bumped 31 label shades to -200 and switched the
three problem badges (Above manufacturer spec, Investigate upstream
first, own fleet, crit) to solid -600 with white text.
- upstream_degradation card got brighter device-link text and a
hover state in dark mode.
DeepSeek:
- Empty-string DEEPSEEK_MODEL / DEEPSEEK_BASE_URL env vars were not
treated as missing — `||` only short-circuits on nil/false — so a
blank secret would override the @default_model and the API would
reject `model: ""`. Both runtime.exs and the client itself now treat
blank values as missing.
Replaces the previous :requires_redis skip with a real test double so
the suite stays self-contained — no external Redis service needed in
CI, but the call paths through RedisHealthCheck and FourOhFourTracker
still get exercised end-to-end.
- Towerops.Redix.Fake: GenServer that speaks PING / SADD / SCARD /
SISMEMBER / SMEMBERS / EXPIRE / DEL / FLUSHDB. start_link refuses
anything other than localhost:6379 so failure-path tests still fail
the way they did against real Redix.
- Application supervisor starts the fake under name Towerops.Redix in
test env (instead of the dummy Agent stub).
- RedisHealthCheck and FourOhFourTracker now route Redix calls through
Application.get_env(:towerops, :redix_module, Redix). Test config
points it at the fake.
- Drops the test_helper :requires_redis exclusion and the @moduletag
on the two test files; FourOhFourTrackerTest setup just resets the
fake between runs.
DeepSeek API spend isn't justified by minute-by-minute freshness when
ops review insights in the morning. Run rules at 02:00 UTC and the
enrichment pass at 03:00 UTC so summaries are ready by 9 AM local.
- RecommendationsRunWorker: hourly → 02:00 UTC daily
- InsightLlmEnrichmentWorker: every 5 min → 03:00 UTC daily
Performance:
- HealthController: honor `:timeout` from `:redis` config (was hardcoded
2s) so the 503-when-unreachable test exits immediately
- DevicePollerBranches: drop `collect_events` window 1500ms → 100ms
(PubSub broadcasts are synchronous from `run_perform`)
- Towerops.Unused: cache xref analysis in `:persistent_term` keyed by
BEAM mtimes so repeated `mix unused` invocations skip re-analysis
- DiscoveryWorker: make wait_loop interval configurable via
`:discovery_wait_interval_ms` (25ms in tests vs 500ms in prod)
- ProfileWatcher: configurable `:profile_reloader` so tests can stub
the YAML reload instead of paying the full disk-load cost
- PingExecutor: tag the loopback test `:network` (it shells out to
`ping`); coverage already provided by 4 other `:network`-tagged tests
Flake fix:
- FrequencyChange tests used hardcoded `~U[2026-05-09 12:00:00Z]`,
which now falls outside the rule's 24h lookback window. Switched to
`DateTime.utc_now()` so scans stay inside the window regardless of
clock progression.
When a wireless AP has a strong same-channel neighbor (RSSI >= -75 dBm)
and a measurably cleaner alternative channel exists in the same band,
the system now emits an ap_frequency_change insight with a specific
recommended channel and the top interfering BSSIDs as evidence.
- New schema wireless_neighbor_scans (append-only) storing observed
neighboring APs per device: bssid, ssid, channel, frequency_mhz,
channel_width_mhz, rssi_dbm, is_own_fleet, seen_at
- New columns on snmp_devices: current_channel, current_frequency_mhz,
current_channel_width_mhz, last_radio_seen_at — populated by vendor
SNMP profiles for APs that expose wireless OIDs
- Towerops.Recommendations.Rules.FrequencyChange — pure rule that scores
each candidate channel using sum-of-linear-power over the last 24h of
neighbor scans, recommends the cleanest if it beats the current by at
least 6 dB. Critical urgency when current score >= 100.
- Towerops.Workers.RecommendationsRunWorker — hourly Oban cron, iterates
organizations and runs all rules, uses insert_insight_if_new/1 for
idempotent upsert
- LiveView /insights renders a purple frequency-change card with current
vs recommended channel, dB-cleaner badge, and a top-offender list
(BSSID, SSID, RSSI, own-fleet flag)
- LLM enrichment automatically applies to the new insight type
Also fixes two pre-existing flaky tests that surfaced in CI:
- test/towerops_web/live/agent_live_test.exs:567 — now sets an explicit
Req.Test transport_error stub for ReleaseChecker and uses Req.Test.allow
so the LiveView process inherits it. Previously relied on absence of a
stub, which leaked across tests and produced different behavior under
full-suite parallel runs.
- test/towerops_web/telemetry_test.exs:140 — relaxes the "no log" check
to refute the specific messages the function under test would emit,
rather than asserting an empty capture_log/1 buffer (which catches
stray logs from concurrent tests, e.g. Postgrex sandbox disconnects).
Adds an optional plain-language summary and recommended action to every
active insight. A new Oban cron worker runs every 5 minutes, picks up
unenriched insights, and asks the configured LLM to restate the finding
and suggest one specific action grounded in the structured metadata.
- Migration: adds llm_summary, recommended_action, llm_model,
llm_enriched_at columns to preseem_insights
- Towerops.LLM context with swappable behaviour; real client uses Req
with the standard Req.Test plug for test isolation
- MockClient + InsightPrompt module (builds chat messages, parses JSON
with code-fence stripping and a raw-text fallback)
- Worker logs and skips on rate limits / missing key / parse errors so
insights still render without an AI summary when the LLM is unavailable
- Optional towerops-llm k8s secret with example template; deployment.yaml
references it as optional in both init and main containers
- UI renders summary + recommended action callout on /insights and on
the org-level insights page
- Remove Mox.set_mox_global() in DPW extra test; rely on $callers
instead so concurrent vendor SNMP tests stop intermittently failing
with VerificationError.
- Make ToweropsWeb.Plugs.GraphQLIntrospectionTest async: false. It
mutates Application.put_env(:towerops, :env, :prod), which polluted
every async vendor test that reads :env via
Towerops.Snmp.Client.phoenix_snmp_disabled/0.
- Bump test pool queue_target/queue_interval. AgentChannelTest spawns
many `:proc_lib` channel processes that hold sandbox connections;
the default 50ms queue caused intermittent
'could not checkout the connection' errors.
- Tag real-System.cmd("ping") tests as :network so they're excluded
by default. Saves ~20s on every full run.
- Make JobCleanupTask's internal 1s settle sleep configurable via
:job_cleanup_settle_ms; the test overrides it to 0.
Plus the pending CoverageLive.Form edit-save / EIRP recompute and
TraceLive deep-link tests from before the compact.
Add PromEx with Application/BEAM/Phoenix/LiveView/Ecto/Oban plugins plus
a custom plugin that bridges existing Towerops Oban/Redis telemetry events.
The metrics HTTP server runs isolated on port 9568 so scrape traffic never
traverses the public Traefik IngressRoute.
The pod template carries prometheus.io annotations (scrape, port, path, job)
so the external Prometheus on 10.0.15.31 discovers each pod through the
existing kubernetes-pods scrape job (apiserver-proxy).
- Add Codeberg Ansible collection link to API/GraphQL docs and Terraform guide
- Un-skip the previously-disabled DevicePollerWorker tests that still pass
- Expand MibController, CnMaestro.Sync, CheckWorker, ActivityFeed,
MobileController, JobCleanupTask tests
- New UploadMibs Mix task test covering arg validation and upload paths
- Set :mib_dir in test config so MibController upload/delete flows can
exercise the real on-disk paths against a writable tmp directory
Coverage rasters were being written to priv/static/coverage on the pod's
ephemeral filesystem — fine on a single-pod dev box, broken under k8s
where pods share NFS at /data and a restart wipes computed predictions.
* New :coverage_storage_dir config (defaults to priv/static/coverage for
dev/test, "/data/coverage" in production).
* Raster.output_dir/absolute_raster_path read this config and resolve
{otp_app, rel} or absolute paths.
* Endpoint now mounts a dedicated Plug.Static at /coverage from that
same config — replicas all serve the same NFS-backed file regardless
of which one ran the compute.
* Drop "coverage" from static_paths/0 since the dedicated plug owns
that prefix now.
Switch esbuild to ESM with --splitting and load app.js as type=module.
Heavy hooks now ship in their own chunk and only download when their
LiveView mounts.
- assets/js/lib/leaflet.ts: extract ensureLeaflet (vendored Leaflet
loader) into a shared module with a typed window.L declaration.
- assets/js/hooks/coverage_hooks.ts: extract CoverageMap,
MultiCoverageMap, and CoverageLocationPicker (~510 lines) with
proper interface types for hook context and tightened typing on
payloads / event handlers (CoveragePayload is now a real type).
- app.ts: replace those three inline hook bodies with lazyHook
stubs that call import('./hooks/coverage_hooks') on first mount,
forwarding lifecycle hooks once the chunk resolves. Also adds a
scoped `declare const L: any` so the remaining Leaflet hooks
type-check.
- root.html.heex: switch the script tag to type=module so the
emitted ESM bundle can use dynamic import().
- esbuild config: add --splitting --format=esm.
Build output now produces:
app.js 1.2 MB
coverage_hooks-<hash>.js 16 KB (lazy)
chunk-<hash>.js 2.4 KB (shared, lazy)
End-to-end CRUD scaffold for per-site, per-antenna RF coverage
prediction. Schema, context, antenna registry, Oban worker stub, and
three LiveViews are wired up; the actual ITM + LIDAR + buildings
compute pipeline lands in a follow-up.
- Migration + Coverage schema with full validation (RF params, pixel-
budget cap, antenna_slug existence). organization_id excluded from
cast (programmatic-only).
- Coverages context: org-scoped CRUD, validate_site_in_organization
rejects cross-tenant site_id refs at insert/update, queue_compute
with PubSub broadcast on per-coverage and per-org topics.
- MSI Planet .ant parser + persistent_term registry loaded at boot
from priv/antennas/. Two synthetic .ant fixtures (omni + 90deg
sector) bundled.
- CoverageWorker on new :coverage Oban queue (concurrency 2). Job
args carry organization_id; worker refuses to run on org mismatch.
Stub flips status to "failed" with "compute not yet implemented"
so the UI flow is exercisable.
- LiveViews: index (org-wide, live status updates), form (antenna
picker grouped by manufacturer), show (status banners, params
panel, map placeholder). Sidebar nav link added.
- Routes /coverage, /coverage/new, /coverage/:id, /coverage/:id/edit
under :require_authenticated_user_with_default_org.
- 41 new tests covering schema validation, .ant parsing, context
CRUD with cross-org guards, Oban job enqueue.
Catalog-only system for LIDAR-derived 1m DEMs covering Texas. We do not
mirror raster data — the DB stores tile metadata (URL + footprint
geometry) and elevation values are streamed on-demand from public USGS
3DEP / TNRIS Cloud-Optimized GeoTIFFs via GDAL /vsicurl/ HTTP byte-range
reads. Used for future tower coverage line-of-sight calculations.
- Enable PostGIS extension; tile footprints indexed via GiST
- Catalog sync from USGS 3DEP (primary) and TNRIS (fallback for gaps)
- Tile dedup by ST_Contains so TNRIS doesn't duplicate 3DEP coverage
- Single-point reads via gdallocationinfo with tile fallthrough on nodata
- Grid retrieval via gdal_translate AAIGrid with multi-tile mosaicing
- Monthly Oban cron worker keeps catalog fresh
- gdal-bin added to runtime image; geo_postgis dep added
Severe index bloat discovered on oban_jobs:
- oban_jobs_pkey: 82.2 bloat ratio, 3.2GB waste
- oban_jobs_state_queue_priority_scheduled_at_id_index: 216.8 bloat, 7.9GB waste
Root cause: High DELETE churn from aggressive Pruner config (10k jobs/sec)
causes PostgreSQL VACUUM to leave dead index entries. VACUUM only marks space
as reusable but cannot shrink indexes.
The Oban Reindexer plugin was only reindexing args_index and meta_index by
default, missing the primary key and main scheduling index.
Solution: Explicitly configure Reindexer to include all high-churn indexes:
- oban_jobs_pkey (primary key, used by all queries)
- oban_jobs_state_queue_priority_scheduled_at_id_index (job scheduling)
- oban_jobs_args_index (default, GIN index)
- oban_jobs_meta_index (default, GIN index)
Daily REINDEX CONCURRENTLY at 2 AM will prevent future bloat accumulation.
Manual immediate reindex required to reclaim 11GB:
REINDEX INDEX CONCURRENTLY oban_jobs_pkey;
REINDEX INDEX CONCURRENTLY oban_jobs_state_queue_priority_scheduled_at_id_index;
Reviewed-on: graham/towerops-web#165
Add data_retention_days field to organizations (default 365, range 30-730).
DataRetentionWorker runs nightly at 1 AM to purge expired time-series data
across 8 tables, respecting each org's retention setting.
TimescaleDB global retention bumped from 90 to 730 days as safety net.
Resolved alerts are cleaned up; unresolved alerts kept regardless of age.
Reviewed-on: graham/towerops-web#144
Default Pruner settings (10k rows/30s = 333/s) can't keep up with job
completion rate (~600+/s from SNMP polling + sync workers), causing
oban_jobs to accumulate millions of completed rows. The bloated table
(47M rows, 29GB) makes Oban.Met.Reporter COUNT queries slow, blocking
on IO and holding DB connections until they timeout at 15s — the actual
root cause of the "ssl recv: closed" connection pool exhaustion.
Increase to 50k rows/5s (10,000/s) to keep the table small.
Reviewed-on: graham/towerops-web#95
- TCP keepalive more aggressive (14s detection vs 30s) so dead
connections are found before the 15s checkout timeout fires
- Reduce pool size 30→15 per pod (2 pods = 30 total, avoids
exhaustion during rolling deploys when 4 pods run briefly)
- Add idle_interval: 1s to proactively ping idle connections and
detect dead SSL sockets before they're handed to real queries
- Simplify SSL config: remove redundant verify_fun and SNI disable
when verify_none already skips certificate validation
Reviewed-on: graham/towerops-web#93
staging was hitting postgres max_connections due to 252 concurrent oban
workers plus pool connections. OBAN_QUEUE_SCALE divides all queue sizes
(e.g. "5" = 1/5th capacity). defaults to 1, no change for production.
Reviewed-on: graham/towerops-web#41
- StormDetector GenServer: sliding-window per org, >10 alerts/min triggers
suppression mode with single summary notification
- Site correlation: >3 devices at same site down within 120s creates
single 'Site Outage' alert instead of N individual alerts
- Notification rate limiter: max 5 per user per 15min window, excess
batched into digest via AlertDigestWorker
- PagerDuty client: retry with exponential backoff on 429, respects
Retry-After header
- Batch maintenance checks: new Maintenance.devices_in_maintenance/1
takes list of device_ids, returns MapSet (single query vs N queries)
- Migration: alert_storms, site_outages, notification_digests tables
plus site_outage_id/storm_suppressed on alerts
- DeviceMonitorWorker integrates storm detection and site correlation
before creating alerts
- Tests for StormDetector, SiteCorrelation, NotificationRateLimiter,
and batch maintenance checks
- Add 1.5s debounce to dashboard LiveView for alert events, preventing
excessive re-renders during mass outages where hundreds of alerts fire
simultaneously
- Replace global 'device:events' PubSub topic with org-scoped
'device_events:org:{id}' topics across all broadcasters (device_poller_worker,
sensor_change_detector, discovery) so LiveView processes only receive
events for their tenant
- Update EventLogger GenServer to subscribe to per-org topics dynamically,
with runtime subscription support for newly created orgs
- Add Organizations.list_organization_ids/0 for EventLogger bootstrap
- Increase default DB pool_size from 20 to 25 for production concurrent users
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The server only sent polling/ping jobs to agents once on WebSocket join
and never re-dispatched them. After the first batch completed (~5-10s),
devices were never polled again until the agent reconnected.
:send_jobs now schedules a follow-up dispatch every 60s (configurable).
Unifies the timer management with :assignments_changed to prevent
accumulation.
Start a named Finch pool with CAStore certificate path and configure
Req to use it by default. Fixes integration sync failures on Dokku
where the OS CA trust store is unavailable.
- Add error_tracker_notifier for email alerts on exceptions
* Sends alerts to graham@towerops.net
* Configured in production only (gracefully shuts down in dev/test)
* Includes throttling to prevent duplicate error spam
- Fix AlertNotificationWorker when PagerDuty not configured
* Worker now handles :not_configured gracefully instead of failing
* Prevents unnecessary retries and error noise
* Logs as debug when PagerDuty integration isn't set up
Files changed:
- mix.exs, mix.lock
- config/runtime.exs
- lib/towerops/application.ex
- lib/towerops/workers/alert_notification_worker.ex
- CHANGELOG.txt, priv/static/changelog.txt
Redis was configured to connect to localhost:6379 in dev.exs, but no Redis
server exists in CI environments. This caused /health endpoint to return 503
during e2e test startup, blocking test execution.
Solution: Remove Redis config from dev - application will fall back to stub
Agent (as designed in application.ex). If developers need Redis locally, they
can set REDIS_HOST environment variable.
Add CapacityInsightWorker (every 15 min) that generates critical/warning
insights when backhaul utilization exceeds 90%/75% and auto-resolves
when it drops below 70%.
Add capacity and utilization columns to the device ports tab with
set/clear capacity controls. Add organization-level /capacity page
with summary cards and per-site capacity table. Add capacity summary
card to site show page. Add Capacity link to nav menu.
- Add stripe_meter_id to dev, runtime, and test configs
- Add STRIPE_METER_ID to k8s deployment secret references
- Use real Stripe meter and price IDs from Stripe setup
- Meter: mtr_test_61UHPU067veEZ7bhl41S77kvnTfgyODY
- Price: price_1T81XBS77kvnTfgyPlw1jF8N ($1/device/month)
- Include setup script for Stripe billing configuration
- Add Stripe test API keys to dev.exs for local development
- Add placeholder Stripe env vars to k8s deployment to prevent startup crashes
- Includes secret_key, webhook_secret, and price_id configuration
