Default Pruner settings (10k rows/30s = 333/s) can't keep up with job
completion rate (~600+/s from SNMP polling + sync workers), causing
oban_jobs to accumulate millions of completed rows. The bloated table
(47M rows, 29GB) makes Oban.Met.Reporter COUNT queries slow, blocking
on IO and holding DB connections until they timeout at 15s — the actual
root cause of the "ssl recv: closed" connection pool exhaustion.
Increase to 50k rows/5s (10,000/s) to keep the table small.
Reviewed-on: graham/towerops-web#95
- TCP keepalive more aggressive (14s detection vs 30s) so dead
connections are found before the 15s checkout timeout fires
- Reduce pool size 30→15 per pod (2 pods = 30 total, avoids
exhaustion during rolling deploys when 4 pods run briefly)
- Add idle_interval: 1s to proactively ping idle connections and
detect dead SSL sockets before they're handed to real queries
- Simplify SSL config: remove redundant verify_fun and SNI disable
when verify_none already skips certificate validation
Reviewed-on: graham/towerops-web#93
staging was hitting postgres max_connections due to 252 concurrent oban
workers plus pool connections. OBAN_QUEUE_SCALE divides all queue sizes
(e.g. "5" = 1/5th capacity). defaults to 1, no change for production.
Reviewed-on: graham/towerops-web#41
- StormDetector GenServer: sliding-window per org, >10 alerts/min triggers
suppression mode with single summary notification
- Site correlation: >3 devices at same site down within 120s creates
single 'Site Outage' alert instead of N individual alerts
- Notification rate limiter: max 5 per user per 15min window, excess
batched into digest via AlertDigestWorker
- PagerDuty client: retry with exponential backoff on 429, respects
Retry-After header
- Batch maintenance checks: new Maintenance.devices_in_maintenance/1
takes list of device_ids, returns MapSet (single query vs N queries)
- Migration: alert_storms, site_outages, notification_digests tables
plus site_outage_id/storm_suppressed on alerts
- DeviceMonitorWorker integrates storm detection and site correlation
before creating alerts
- Tests for StormDetector, SiteCorrelation, NotificationRateLimiter,
and batch maintenance checks
- Add 1.5s debounce to dashboard LiveView for alert events, preventing
excessive re-renders during mass outages where hundreds of alerts fire
simultaneously
- Replace global 'device:events' PubSub topic with org-scoped
'device_events:org:{id}' topics across all broadcasters (device_poller_worker,
sensor_change_detector, discovery) so LiveView processes only receive
events for their tenant
- Update EventLogger GenServer to subscribe to per-org topics dynamically,
with runtime subscription support for newly created orgs
- Add Organizations.list_organization_ids/0 for EventLogger bootstrap
- Increase default DB pool_size from 20 to 25 for production concurrent users
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The server only sent polling/ping jobs to agents once on WebSocket join
and never re-dispatched them. After the first batch completed (~5-10s),
devices were never polled again until the agent reconnected.
:send_jobs now schedules a follow-up dispatch every 60s (configurable).
Unifies the timer management with :assignments_changed to prevent
accumulation.
Start a named Finch pool with CAStore certificate path and configure
Req to use it by default. Fixes integration sync failures on Dokku
where the OS CA trust store is unavailable.
- Add error_tracker_notifier for email alerts on exceptions
* Sends alerts to graham@towerops.net
* Configured in production only (gracefully shuts down in dev/test)
* Includes throttling to prevent duplicate error spam
- Fix AlertNotificationWorker when PagerDuty not configured
* Worker now handles :not_configured gracefully instead of failing
* Prevents unnecessary retries and error noise
* Logs as debug when PagerDuty integration isn't set up
Files changed:
- mix.exs, mix.lock
- config/runtime.exs
- lib/towerops/application.ex
- lib/towerops/workers/alert_notification_worker.ex
- CHANGELOG.txt, priv/static/changelog.txt
Redis was configured to connect to localhost:6379 in dev.exs, but no Redis
server exists in CI environments. This caused /health endpoint to return 503
during e2e test startup, blocking test execution.
Solution: Remove Redis config from dev - application will fall back to stub
Agent (as designed in application.ex). If developers need Redis locally, they
can set REDIS_HOST environment variable.
Add CapacityInsightWorker (every 15 min) that generates critical/warning
insights when backhaul utilization exceeds 90%/75% and auto-resolves
when it drops below 70%.
Add capacity and utilization columns to the device ports tab with
set/clear capacity controls. Add organization-level /capacity page
with summary cards and per-site capacity table. Add capacity summary
card to site show page. Add Capacity link to nav menu.
- Add stripe_meter_id to dev, runtime, and test configs
- Add STRIPE_METER_ID to k8s deployment secret references
- Use real Stripe meter and price IDs from Stripe setup
- Meter: mtr_test_61UHPU067veEZ7bhl41S77kvnTfgyODY
- Price: price_1T81XBS77kvnTfgyPlw1jF8N ($1/device/month)
- Include setup script for Stripe billing configuration
- Add Stripe test API keys to dev.exs for local development
- Add placeholder Stripe env vars to k8s deployment to prevent startup crashes
- Includes secret_key, webhook_secret, and price_id configuration
Make the 500ms device deletion delay configurable via application config.
Set to 10ms in test environment (vs 500ms in production).
This sleep waits for in-flight polling jobs to complete before deleting
a device. In tests with mocked jobs, this delay is unnecessary.
Affected tests:
- All device deletion tests (were 500-650ms, now ~100-150ms)
- Equipment context tests
- API controller tests
- LiveView tests with device deletion
Performance: 28.3s → 25.2s (11% faster)
Total improvement from baseline: 52s → 25.2s (51.5% faster)
Replace sequential migration execution (4m8s for 172 migrations) with
structure.sql loading (416ms), reducing CI database setup time by 99.7%.
Changes:
- Configure Ecto to dump/load schema from priv/repo/structure.sql
- Update test alias: ecto.migrate → ecto.load
- Remove redundant database setup step from CI workflow
- Commit structure.sql to repository for version control
Benefits:
- CI database setup: 4m8s → 416ms (248s → 0.4s)
- Consistent schema snapshots across environments
- Faster local test runs
Maintenance:
- Run 'mix ecto.dump' after adding new migrations
- Structure file auto-updates with schema changes
Add logger filter to drop Kubernetes health probe logs that were flooding
production logs every few seconds.
Implementation:
- Create ToweropsWeb.TelemetryFilter module with filter_health_checks/2
- Configure as logger :default_handler filter in prod.exs
- Filters based on request_path metadata and message content
- Drops logs for GET /health and HEAD / requests
Impact:
- Eliminates ~120 log entries per minute per pod from K8s probes
- Keeps application logs focused on actual user activity and errors
- No impact on health check functionality - only suppresses logging
Files:
- lib/towerops_web/telemetry_filter.ex (new)
- lib/towerops/application.ex (attach filter on startup)
- config/prod.exs (add filter to logger config)
- CHANGELOG.txt
Implements Phase 1 of network topology discovery using LLDP (Link Layer
Discovery Protocol) via SNMP, inspired by concepts from lldp2map.
New Features:
- LLDP-MIB walker for discovering network neighbors via SNMP
- Database schema for storing neighbor relationships
- Functions to discover, store, and query device neighbors
- Automatic device linking when neighbors are found in database
Components Added:
- Migration: device_neighbors table with unique constraint on
(device_id, local_port, neighbor_name)
- Schema: Towerops.Topology.DeviceNeighbor for neighbor relationships
- Module: Towerops.Topology.Lldp for SNMP LLDP-MIB walking
- Functions: discover_lldp_neighbors/1, list_lldp_neighbors/1,
remove_stale_lldp_neighbors/1 in Topology context
LLDP-MIB OIDs:
- 1.0.8802.1.1.2.1.3.3.0 (lldpLocSysName)
- 1.0.8802.1.1.2.1.3.7.1.4 (lldpLocPortDesc)
- 1.0.8802.1.1.2.1.4.1.1.7 (lldpRemPortId)
- 1.0.8802.1.1.2.1.4.1.1.8 (lldpRemPortDesc)
- 1.0.8802.1.1.2.1.4.1.1.9 (lldpRemSysName)
- 1.0.8802.1.1.2.1.4.2.1.3 (lldpRemManAddr)
CI Fix:
- Updated config/test.exs to use DATABASE_URL from environment in CI
- Fixes "connection refused to localhost:5432" errors in CI builds
- Falls back to localhost config for local development
Code Quality:
- Refactored LLDP module to reduce cyclomatic complexity
- Extracted helper functions to improve readability
- Used 'with' statement to reduce nesting in parse_management_address
- All Credo checks passing
Next Steps (Phase 2):
- Background worker for automated topology discovery
- Topology visualization in LiveView
- Recursive discovery from seed devices
Add ErrorTrackerIgnorer to drop port_died, write_failed, epipe, and
broken pipe errors that occur during K8s pod shutdown. Same patterns
already filtered by HoneybadgerFilter and LoggerFilters.
Replace the HoneybadgerNoticeFilter that emailed raw stacktraces on
every exception with ErrorTracker for self-hosted error tracking.
Honeybadger itself is retained. ErrorTracker dashboard is mounted at
/admin/errors behind the superuser auth wall.
All sync workers (Preseem, Gaiia, NetBox) now fire every 5 minutes
via Oban cron. Each worker's should_sync? check compares elapsed
time against the user's sync_interval_minutes, so the actual sync
frequency matches what the user configured.
- NetBox.Sync: pull devices/sites from NetBox, match by name/IP, upsert
- NetBoxSyncWorker: Oban cron every 30min for enabled NetBox integrations
- Gaiia sync: show actual error type instead of generic 'unexpected error'
Transform the dashboard into a single-pane-of-glass command center that
blends operational metrics with business context from Gaiia subscriber data.
- Extend insight schema with multi-source support (snmp, gaiia, system)
- Add Oban workers for automated insight generation (device health, system, gaiia)
- New Dashboard context with health score computation and site summaries
- Rewrite dashboard LiveView with health overview, alert/insight feeds, site grid
- Add source filter pills for insights with URL state management
- Add metrics bar to site detail pages (device count, alerts, subscribers, MRR)
- Add subscriber/MRR context to alert list site links
- Add subscriber badges to device list site headers
- Real-time PubSub updates for alerts on dashboard
- create MonitoringCheck schema for the monitoring_checks table, fixing
silent data loss where ping results were dropped by the wrong schema
- update agent channel and device monitor worker to use new schema
- fix Stats queries to use MonitoringCheck instead of Check schema
- add list_online_cloud_pollers and list_cloud_polled_devices queries
- add CloudLatencyProbeWorker to dispatch cross-agent latency probes
every 8 hours via PubSub to all online cloud pollers
- scope AgentLatencyEvaluator to cloud pollers only with cloud_only
filter, lower min_checks (5), wider time window (48h), and
device-level assignments to avoid changing site/org defaults
- update cron schedule: probes at 0/8/16h, evaluation at 0:30/8:30/16:30
- refactor monitoring.ex and http_executor.ex to fix credo complexity
Implements Icinga2-style generic check system for comprehensive monitoring.
Phoenix Backend:
- Generic checks table with JSONB config for all check types
- TimescaleDB check_results hypertable for time-series data
- Check executors (HTTP, TCP, DNS)
- CheckWorker with self-scheduling and state transitions
- Extended alerts system for check-based alerting
Check Types: HTTP/HTTPS, TCP, DNS
Architecture: Icinga2 Checkable pattern, soft/hard states, flapping detection
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Filter port_died and write_failed/epipe errors from Honeybadger reports,
email notifications, and logs. Reorder supervision tree so Oban drains
before its dependencies shut down, and add 40s shutdown grace period.
Add a Honeybadger NoticeFilter that sends an email with the full
error class, message, stacktrace, and server info to graham@mcintire.me
whenever an exception is reported. The notice passes through unchanged
so Honeybadger still receives it normally.
POST /api/v1/webhooks/agent-release triggers all connected agents
to self-update via their existing WebSocket channels. Authenticated
with a shared secret (AGENT_WEBHOOK_SECRET env var).
- Add ReleaseChecker.invalidate_cache/0 for fresh GitHub fetch
- Add Agents.list_updatable_agents/0 (enabled + seen < 10min)
- Add Agents.broadcast_mass_update/0 orchestration function
- Add WebhookAuth plug with timing-safe secret comparison
- Add AgentReleaseWebhookController with :webhook pipeline
- Configure AGENT_WEBHOOK_SECRET in dev/test/runtime configs
