Adds a parallel LLM path that reviews the whole org snapshot — Preseem
APs, SNMP signals, backhaul, agents, and the insight types the rules
already covered — and emits novel observations as ai_observation
insights (source=ai). Existing rule-based insights and per-insight
enrichment continue unchanged.
- lib/towerops/llm/network_snapshot.ex builds the bounded snapshot
- lib/towerops/llm/network_insight_prompt.ex builds the prompt and
parses the LLM's JSON observations array
- lib/towerops/workers/ai_network_insight_worker.ex runs nightly at
03:30 UTC (after the rule pass + per-insight enrichment) and on
the superuser regenerate button
- preseem/insight.ex extends @valid_types with "ai_observation" and
@valid_sources with "ai"
- insights_live/index.ex includes the new worker in the regen list
and adds an "AI" source badge style
Bumps the SnmpKit Manager timeout-options test threshold from 500ms
to 3000ms (was flaking on loaded dev machines), and scopes
list_unenriched_insights/1 assertion in insights_test:573 to the
test's org so cross-suite leaks don't fail the count.
If a rule keeps re-firing the underlying condition, the dedup query
suppresses duplicate inserts but `inserted_at` of the live row stays
old — so a real 6-month-old issue and a one-off blip both linger
forever. Auto-resolving any active insight older than 7 days clears
the dashboard. Still-real issues will be re-created by the next rule
pass (dedup only checks `status: "active"`), so the operator sees
fresh evidence rather than stale text.
- Towerops.Workers.InsightExpiryWorker (queue: :maintenance) flips
`status` to "resolved" on rows where `inserted_at < now - 7d`.
- Window is configurable via `:max_age_days` for ops who want a
different cadence per env.
- Oban cron: 04:00 UTC nightly, after the 02:00 rule pass and 03:00
LLM enrichment.
UI:
- Type-specific evidence cards (red/amber/orange/rose/purple/cyan/
indigo/blue) had `dark:text-{color}-300` labels and
`dark:bg-{color}-800 dark:text-{color}-100` badges that washed out on
the dark background. Bumped 31 label shades to -200 and switched the
three problem badges (Above manufacturer spec, Investigate upstream
first, own fleet, crit) to solid -600 with white text.
- upstream_degradation card got brighter device-link text and a
hover state in dark mode.
DeepSeek:
- Empty-string DEEPSEEK_MODEL / DEEPSEEK_BASE_URL env vars were not
treated as missing — `||` only short-circuits on nil/false — so a
blank secret would override the @default_model and the API would
reject `model: ""`. Both runtime.exs and the client itself now treat
blank values as missing.
DeepSeek API spend isn't justified by minute-by-minute freshness when
ops review insights in the morning. Run rules at 02:00 UTC and the
enrichment pass at 03:00 UTC so summaries are ready by 9 AM local.
- RecommendationsRunWorker: hourly → 02:00 UTC daily
- InsightLlmEnrichmentWorker: every 5 min → 03:00 UTC daily
When a wireless AP has a strong same-channel neighbor (RSSI >= -75 dBm)
and a measurably cleaner alternative channel exists in the same band,
the system now emits an ap_frequency_change insight with a specific
recommended channel and the top interfering BSSIDs as evidence.
- New schema wireless_neighbor_scans (append-only) storing observed
neighboring APs per device: bssid, ssid, channel, frequency_mhz,
channel_width_mhz, rssi_dbm, is_own_fleet, seen_at
- New columns on snmp_devices: current_channel, current_frequency_mhz,
current_channel_width_mhz, last_radio_seen_at — populated by vendor
SNMP profiles for APs that expose wireless OIDs
- Towerops.Recommendations.Rules.FrequencyChange — pure rule that scores
each candidate channel using sum-of-linear-power over the last 24h of
neighbor scans, recommends the cleanest if it beats the current by at
least 6 dB. Critical urgency when current score >= 100.
- Towerops.Workers.RecommendationsRunWorker — hourly Oban cron, iterates
organizations and runs all rules, uses insert_insight_if_new/1 for
idempotent upsert
- LiveView /insights renders a purple frequency-change card with current
vs recommended channel, dB-cleaner badge, and a top-offender list
(BSSID, SSID, RSSI, own-fleet flag)
- LLM enrichment automatically applies to the new insight type
Also fixes two pre-existing flaky tests that surfaced in CI:
- test/towerops_web/live/agent_live_test.exs:567 — now sets an explicit
Req.Test transport_error stub for ReleaseChecker and uses Req.Test.allow
so the LiveView process inherits it. Previously relied on absence of a
stub, which leaked across tests and produced different behavior under
full-suite parallel runs.
- test/towerops_web/telemetry_test.exs:140 — relaxes the "no log" check
to refute the specific messages the function under test would emit,
rather than asserting an empty capture_log/1 buffer (which catches
stray logs from concurrent tests, e.g. Postgrex sandbox disconnects).
Adds an optional plain-language summary and recommended action to every
active insight. A new Oban cron worker runs every 5 minutes, picks up
unenriched insights, and asks the configured LLM to restate the finding
and suggest one specific action grounded in the structured metadata.
- Migration: adds llm_summary, recommended_action, llm_model,
llm_enriched_at columns to preseem_insights
- Towerops.LLM context with swappable behaviour; real client uses Req
with the standard Req.Test plug for test isolation
- MockClient + InsightPrompt module (builds chat messages, parses JSON
with code-fence stripping and a raw-text fallback)
- Worker logs and skips on rate limits / missing key / parse errors so
insights still render without an AI summary when the LLM is unavailable
- Optional towerops-llm k8s secret with example template; deployment.yaml
references it as optional in both init and main containers
- UI renders summary + recommended action callout on /insights and on
the org-level insights page
End-to-end CRUD scaffold for per-site, per-antenna RF coverage
prediction. Schema, context, antenna registry, Oban worker stub, and
three LiveViews are wired up; the actual ITM + LIDAR + buildings
compute pipeline lands in a follow-up.
- Migration + Coverage schema with full validation (RF params, pixel-
budget cap, antenna_slug existence). organization_id excluded from
cast (programmatic-only).
- Coverages context: org-scoped CRUD, validate_site_in_organization
rejects cross-tenant site_id refs at insert/update, queue_compute
with PubSub broadcast on per-coverage and per-org topics.
- MSI Planet .ant parser + persistent_term registry loaded at boot
from priv/antennas/. Two synthetic .ant fixtures (omni + 90deg
sector) bundled.
- CoverageWorker on new :coverage Oban queue (concurrency 2). Job
args carry organization_id; worker refuses to run on org mismatch.
Stub flips status to "failed" with "compute not yet implemented"
so the UI flow is exercisable.
- LiveViews: index (org-wide, live status updates), form (antenna
picker grouped by manufacturer), show (status banners, params
panel, map placeholder). Sidebar nav link added.
- Routes /coverage, /coverage/new, /coverage/:id, /coverage/:id/edit
under :require_authenticated_user_with_default_org.
- 41 new tests covering schema validation, .ant parsing, context
CRUD with cross-org guards, Oban job enqueue.
Catalog-only system for LIDAR-derived 1m DEMs covering Texas. We do not
mirror raster data — the DB stores tile metadata (URL + footprint
geometry) and elevation values are streamed on-demand from public USGS
3DEP / TNRIS Cloud-Optimized GeoTIFFs via GDAL /vsicurl/ HTTP byte-range
reads. Used for future tower coverage line-of-sight calculations.
- Enable PostGIS extension; tile footprints indexed via GiST
- Catalog sync from USGS 3DEP (primary) and TNRIS (fallback for gaps)
- Tile dedup by ST_Contains so TNRIS doesn't duplicate 3DEP coverage
- Single-point reads via gdallocationinfo with tile fallthrough on nodata
- Grid retrieval via gdal_translate AAIGrid with multi-tile mosaicing
- Monthly Oban cron worker keeps catalog fresh
- gdal-bin added to runtime image; geo_postgis dep added
Severe index bloat discovered on oban_jobs:
- oban_jobs_pkey: 82.2 bloat ratio, 3.2GB waste
- oban_jobs_state_queue_priority_scheduled_at_id_index: 216.8 bloat, 7.9GB waste
Root cause: High DELETE churn from aggressive Pruner config (10k jobs/sec)
causes PostgreSQL VACUUM to leave dead index entries. VACUUM only marks space
as reusable but cannot shrink indexes.
The Oban Reindexer plugin was only reindexing args_index and meta_index by
default, missing the primary key and main scheduling index.
Solution: Explicitly configure Reindexer to include all high-churn indexes:
- oban_jobs_pkey (primary key, used by all queries)
- oban_jobs_state_queue_priority_scheduled_at_id_index (job scheduling)
- oban_jobs_args_index (default, GIN index)
- oban_jobs_meta_index (default, GIN index)
Daily REINDEX CONCURRENTLY at 2 AM will prevent future bloat accumulation.
Manual immediate reindex required to reclaim 11GB:
REINDEX INDEX CONCURRENTLY oban_jobs_pkey;
REINDEX INDEX CONCURRENTLY oban_jobs_state_queue_priority_scheduled_at_id_index;
Reviewed-on: graham/towerops-web#165
Add data_retention_days field to organizations (default 365, range 30-730).
DataRetentionWorker runs nightly at 1 AM to purge expired time-series data
across 8 tables, respecting each org's retention setting.
TimescaleDB global retention bumped from 90 to 730 days as safety net.
Resolved alerts are cleaned up; unresolved alerts kept regardless of age.
Reviewed-on: graham/towerops-web#144
Default Pruner settings (10k rows/30s = 333/s) can't keep up with job
completion rate (~600+/s from SNMP polling + sync workers), causing
oban_jobs to accumulate millions of completed rows. The bloated table
(47M rows, 29GB) makes Oban.Met.Reporter COUNT queries slow, blocking
on IO and holding DB connections until they timeout at 15s — the actual
root cause of the "ssl recv: closed" connection pool exhaustion.
Increase to 50k rows/5s (10,000/s) to keep the table small.
Reviewed-on: graham/towerops-web#95
- TCP keepalive more aggressive (14s detection vs 30s) so dead
connections are found before the 15s checkout timeout fires
- Reduce pool size 30→15 per pod (2 pods = 30 total, avoids
exhaustion during rolling deploys when 4 pods run briefly)
- Add idle_interval: 1s to proactively ping idle connections and
detect dead SSL sockets before they're handed to real queries
- Simplify SSL config: remove redundant verify_fun and SNI disable
when verify_none already skips certificate validation
Reviewed-on: graham/towerops-web#93
staging was hitting postgres max_connections due to 252 concurrent oban
workers plus pool connections. OBAN_QUEUE_SCALE divides all queue sizes
(e.g. "5" = 1/5th capacity). defaults to 1, no change for production.
Reviewed-on: graham/towerops-web#41
- StormDetector GenServer: sliding-window per org, >10 alerts/min triggers
suppression mode with single summary notification
- Site correlation: >3 devices at same site down within 120s creates
single 'Site Outage' alert instead of N individual alerts
- Notification rate limiter: max 5 per user per 15min window, excess
batched into digest via AlertDigestWorker
- PagerDuty client: retry with exponential backoff on 429, respects
Retry-After header
- Batch maintenance checks: new Maintenance.devices_in_maintenance/1
takes list of device_ids, returns MapSet (single query vs N queries)
- Migration: alert_storms, site_outages, notification_digests tables
plus site_outage_id/storm_suppressed on alerts
- DeviceMonitorWorker integrates storm detection and site correlation
before creating alerts
- Tests for StormDetector, SiteCorrelation, NotificationRateLimiter,
and batch maintenance checks
- Add 1.5s debounce to dashboard LiveView for alert events, preventing
excessive re-renders during mass outages where hundreds of alerts fire
simultaneously
- Replace global 'device:events' PubSub topic with org-scoped
'device_events:org:{id}' topics across all broadcasters (device_poller_worker,
sensor_change_detector, discovery) so LiveView processes only receive
events for their tenant
- Update EventLogger GenServer to subscribe to per-org topics dynamically,
with runtime subscription support for newly created orgs
- Add Organizations.list_organization_ids/0 for EventLogger bootstrap
- Increase default DB pool_size from 20 to 25 for production concurrent users
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Add error_tracker_notifier for email alerts on exceptions
* Sends alerts to graham@towerops.net
* Configured in production only (gracefully shuts down in dev/test)
* Includes throttling to prevent duplicate error spam
- Fix AlertNotificationWorker when PagerDuty not configured
* Worker now handles :not_configured gracefully instead of failing
* Prevents unnecessary retries and error noise
* Logs as debug when PagerDuty integration isn't set up
Files changed:
- mix.exs, mix.lock
- config/runtime.exs
- lib/towerops/application.ex
- lib/towerops/workers/alert_notification_worker.ex
- CHANGELOG.txt, priv/static/changelog.txt
Add CapacityInsightWorker (every 15 min) that generates critical/warning
insights when backhaul utilization exceeds 90%/75% and auto-resolves
when it drops below 70%.
Add capacity and utilization columns to the device ports tab with
set/clear capacity controls. Add organization-level /capacity page
with summary cards and per-site capacity table. Add capacity summary
card to site show page. Add Capacity link to nav menu.
- Add stripe_meter_id to dev, runtime, and test configs
- Add STRIPE_METER_ID to k8s deployment secret references
- Use real Stripe meter and price IDs from Stripe setup
- Meter: mtr_test_61UHPU067veEZ7bhl41S77kvnTfgyODY
- Price: price_1T81XBS77kvnTfgyPlw1jF8N ($1/device/month)
- Include setup script for Stripe billing configuration
All sync workers (Preseem, Gaiia, NetBox) now fire every 5 minutes
via Oban cron. Each worker's should_sync? check compares elapsed
time against the user's sync_interval_minutes, so the actual sync
frequency matches what the user configured.
- NetBox.Sync: pull devices/sites from NetBox, match by name/IP, upsert
- NetBoxSyncWorker: Oban cron every 30min for enabled NetBox integrations
- Gaiia sync: show actual error type instead of generic 'unexpected error'
Transform the dashboard into a single-pane-of-glass command center that
blends operational metrics with business context from Gaiia subscriber data.
- Extend insight schema with multi-source support (snmp, gaiia, system)
- Add Oban workers for automated insight generation (device health, system, gaiia)
- New Dashboard context with health score computation and site summaries
- Rewrite dashboard LiveView with health overview, alert/insight feeds, site grid
- Add source filter pills for insights with URL state management
- Add metrics bar to site detail pages (device count, alerts, subscribers, MRR)
- Add subscriber/MRR context to alert list site links
- Add subscriber badges to device list site headers
- Real-time PubSub updates for alerts on dashboard
- create MonitoringCheck schema for the monitoring_checks table, fixing
silent data loss where ping results were dropped by the wrong schema
- update agent channel and device monitor worker to use new schema
- fix Stats queries to use MonitoringCheck instead of Check schema
- add list_online_cloud_pollers and list_cloud_polled_devices queries
- add CloudLatencyProbeWorker to dispatch cross-agent latency probes
every 8 hours via PubSub to all online cloud pollers
- scope AgentLatencyEvaluator to cloud pollers only with cloud_only
filter, lower min_checks (5), wider time window (48h), and
device-level assignments to avoid changing site/org defaults
- update cron schedule: probes at 0/8/16h, evaluation at 0:30/8:30/16:30
- refactor monitoring.ex and http_executor.ex to fix credo complexity
Implements Icinga2-style generic check system for comprehensive monitoring.
Phoenix Backend:
- Generic checks table with JSONB config for all check types
- TimescaleDB check_results hypertable for time-series data
- Check executors (HTTP, TCP, DNS)
- CheckWorker with self-scheduling and state transitions
- Extended alerts system for check-based alerting
Check Types: HTTP/HTTPS, TCP, DNS
Architecture: Icinga2 Checkable pattern, soft/hard states, flapping detection
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Filter port_died and write_failed/epipe errors from Honeybadger reports,
email notifications, and logs. Reorder supervision tree so Oban drains
before its dependencies shut down, and add 40s shutdown grace period.
POST /api/v1/webhooks/agent-release triggers all connected agents
to self-update via their existing WebSocket channels. Authenticated
with a shared secret (AGENT_WEBHOOK_SECRET env var).
- Add ReleaseChecker.invalidate_cache/0 for fresh GitHub fetch
- Add Agents.list_updatable_agents/0 (enabled + seen < 10min)
- Add Agents.broadcast_mass_update/0 orchestration function
- Add WebhookAuth plug with timing-safe secret comparison
- Add AgentReleaseWebhookController with :webhook pipeline
- Configure AGENT_WEBHOOK_SECRET in dev/test/runtime configs
The plug was incorrectly placed in the endpoint, causing it to run on
all requests including APIs. Moved to :browser pipeline where it belongs,
and removed redundant fetch_session call since the pipeline handles it.
