Commit graph

510 commits

Author SHA1 Message Date
daf66038c6
fix: auto-resolve Gaiia reconciliation insights when issues are fixed 2026-03-04 16:31:08 -06:00
11d976531b
fix: auto-dismiss agent offline insights when agent reconnects 2026-03-04 16:25:08 -06:00
8109bd71c8 fix: auto-dismiss poll gap insights when polling resumes
When a device resumes polling (last_snmp_poll_at updated), automatically
dismiss any active 'device_poll_gap' insights for that device.

This prevents stale insights from lingering after connectivity issues
are resolved (e.g., agent reconnection after being blocked).

Changes:
- Auto-dismiss device_poll_gap insights in update_snmp_poll_time()
- Log when insights are auto-dismissed for visibility
- Fix brute force protection compilation error
2026-03-04 16:19:28 -06:00
89949fdfd6 fix: deduplicate Preseem sync entries in activity feed
Group sync entries by minute to show only one entry per minute instead
of multiple duplicate entries with the same timestamp. This aggregates:
- Total records synced in that minute
- Average sync duration
- Most recent status

This keeps the activity feed informative while reducing noise from
rapid syncs happening within the same minute.
2026-03-04 15:18:03 -06:00
74ad58bada fix: hide successful Preseem syncs from activity feed
Only show failed Preseem syncs in the activity feed to reduce noise.
Successful syncs happen every 10-20 minutes and clutter the dashboard
without providing useful information to users.

Users can still see all sync history (including successful syncs) in
the Preseem integration settings page if needed.
2026-03-04 15:14:06 -06:00
f0fea0dd3e
fix: retry firmware fetch with exponential backoff on network errors
Increase max_attempts from 3 to 10 and add custom backoff/1 that uses
quadratic growth (attempt² × 60s) capped at 2 hours, so transient
network failures retry over ~9 hours rather than being discarded after
3 quick attempts.
2026-02-18 10:34:51 -06:00
7081e5daed
fix: correct subscriber counts by filtering CGNAT IPs and excluding routers
Router devices were inflating per-site subscriber counts because:
1. CGNAT IPs (100.64.0.0/10) in router ARP tables caused false matches
2. Routers see all traffic and shouldn't have subscribers attributed
3. Site totals were computed from raw links without filtering

Changes:
- Filter CGNAT (RFC 6598) IPs from device subscriber links
- Deduplicate accounts across devices by match method priority
- Detect routers via SNMP sys_descr and zero out per-device counts
- Derive site totals from filtered per-device impacts
- Add per-device Subs/MRR columns to site device tables
- Add device link fallback for sites without IP blocks
- Fix Gaiia sync case mismatch and execution order
- Improve device list toolbar button icons and tooltips
2026-02-18 09:15:10 -06:00
0f32086933
Ignore pod shutdown errors in ErrorTracker
Add ErrorTrackerIgnorer to drop port_died, write_failed, epipe, and
broken pipe errors that occur during K8s pod shutdown. Same patterns
already filtered by HoneybadgerFilter and LoggerFilters.
2026-02-17 10:59:51 -06:00
2929848f51
fix: validate NetBox URL scheme before making HTTP requests
The NetBox client crashed with "scheme is required for url" when
integration credentials had a nil or blank URL. normalize_url/1 now
validates the scheme and returns a clear error instead of letting
Finch blow up.
2026-02-17 10:50:00 -06:00
c92dfff6c9
Add ErrorTracker, remove Honeybadger email notice filter
Replace the HoneybadgerNoticeFilter that emailed raw stacktraces on
every exception with ErrorTracker for self-hosted error tracking.
Honeybadger itself is retained. ErrorTracker dashboard is mounted at
/admin/errors behind the superuser auth wall.
2026-02-17 08:13:34 -06:00
1712e7ac9d
Skip SNMP check execution in CheckExecutorWorker for agent-polled devices
CheckExecutorWorker was missing the phoenix_snmp_disabled and
should_phoenix_poll_device guards that DevicePollerWorker and
DeviceMonitorWorker already have. This caused SNMP checks to execute
through Phoenix even when devices are assigned to agents, creating
an infinite loop of failing jobs that fill the Oban queue.
2026-02-17 07:47:15 -06:00
78aa415073 Skip Phoenix poller/monitor scheduling when SNMP disabled
JobCleanupTask was scheduling poller and monitor jobs for all SNMP-enabled
devices on every startup without checking phoenix_snmp_disabled. These jobs
would immediately bail out but still consumed queue capacity.

JobHealthCheckWorker had the same issue - recovering 'missing' monitor jobs
that would just no-op.

Both now check Client.phoenix_snmp_disabled() before scheduling.
2026-02-17 07:12:15 -06:00
de986bddf6
Prevent Oban polling/monitoring job stacking per device
When the Oban queue backs up, the 60-second uniqueness window expires
and duplicate jobs stack up per device. Switch to period: :infinity so
only one poll/monitor job exists per device at any time. Add replace
option to supersede stale scheduled jobs with updated scheduled_at.
Remove :executing from unique states so self-scheduling works while
the current job runs. Set max_attempts: 1 since retrying stale polls
is pointless.

Also fix all credo --strict issues across the codebase.
2026-02-16 16:37:48 -06:00
b12d46c716 Add wireless client table discovery and improve subscriber matching
Poll connected client/SM tables from APs via vendor-specific SNMP:
- Cambium ePMP: cambiumAPConnectedSTATable
- Ubiquiti airOS: ubntStaTable
- MikroTik RouterOS: mtxrWlRtabTable

Wireless client MAC matches now take highest priority for
subscriber-to-AP mapping, giving accurate per-radio subscriber counts.
2026-02-16 11:16:52 -06:00
520f7b0c54 Add subscriber-to-AP matching via ARP + Gaiia integration
Matches Gaiia subscriber accounts to TowerOps devices using:
- ARP table IP matching (high confidence)
- ARP table MAC matching (high confidence)
- Site CIDR fallback (medium confidence)

Enables subscriber impact display on alerts and device pages.
2026-02-16 11:07:20 -06:00
5b786aac08 Add maintenance windows and weather overlay (backend)
Maintenance Windows:
- maintenance_windows table with org/site/device scoping
- Full CRUD context (Towerops.Maintenance)
- device_in_maintenance?/1 checks device → site → org hierarchy
- Alert suppression in DeviceMonitorWorker (skips alerts during maintenance)
- maintenance_badge/1 helper for UI display

Weather Overlay:
- weather_observations + weather_alerts tables
- OpenWeatherMap free API client (lat/lon based)
- WeatherSyncWorker (Oban, 15min interval, respects rate limits)
- Observation parsing from OWM response format
- Correlation helpers: severe_weather_near?/2, weather_at/2
- Severity classification tuned for WISP ops (wind, rain, ice thresholds)
- Per-org latest observations query for dashboard
- 30-day automatic cleanup

Both features are backend-only — no UI yet.
2026-02-16 10:01:19 -06:00
47b364a112
Fix Gaiia inventory matching, mapping UX, and logo link
- Fetch MAC address from Gaiia custom fields during inventory sync
- Remove name-based device matching (model names caused false matches)
- Keep IP-only matching for device suggestions
- Make unmapped rows clickable in the Gaiia mapping table
- Point authenticated logo link to /dashboard instead of /orgs
- Fix flaky monitor test teardown race condition
2026-02-15 17:41:43 -06:00
b4c0407ee0 Add granular org roles: executive, technician
New role system:
- owner: full access + financials
- admin: full access + financials (except org deletion)
- executive: read-only + full financial visibility (MRR, revenue)
- technician: field ops (devices, sites, alerts) without financials
- member: legacy alias for technician (migrated)
- viewer: read-only, no financials

Financial data (MRR, revenue, billing) gated behind can_view_financials
assign in all LiveView templates: dashboard, alerts, trace, site show.

Includes migration to rename existing member roles to technician.
2026-02-15 17:40:53 -06:00
8b51b02d45
Compute per-site subscriber counts from IP block matching
After Gaiia sync, match inventory item IPs against network site CIDR
blocks to populate account_count and total_mrr on each site. This
enables the Site Health table to show subscriber and MRR data per site.
2026-02-15 17:11:23 -06:00
de075ea8d9
Store billing subscriber/MRR data on integrations for dashboard
Each billing sync (Gaiia, Splynx, VISP, Sonar) already computes
subscriber counts and MRR but only stored them in sync messages.
Now persists them on the integrations table so the dashboard can
aggregate across all billing providers instead of only querying
Gaiia network sites (which were never populated).

Also fixes SnmpKit Config environment detection to use Mix.env()
as fallback when MIX_ENV env var is not set.
2026-02-15 16:51:33 -06:00
3cb18a47fc Fix Credo nesting in Splynx MRR calculation
Extract parse_mrr/1 helper to reduce nesting depth in calculate_total_mrr.
Credo --strict: 0 issues. Dialyzer: pass (no new warnings).
2026-02-15 15:18:36 -06:00
6be3910fc4 Fix Splynx integration: token-based auth, per-provider credential fields
- Splynx client: replace broken Basic Auth with proper token flow
  (POST /api/2.0/admin/auth/tokens → Splynx-EA bearer header)
- Support both api_key and admin login auth types with fallback
- Add MRR calculation to Splynx sync using mrr_total field
- Integrations UI: per-provider credential fields (instance_url,
  api_key, api_secret for Splynx; instance_url, api_token for Sonar)
- Wire up test_connection for Splynx, Sonar, and VISP providers
- Verified against live Splynx demo (demo.splynx.com):
  auth, customers, routers, services all working
- All 7350 tests passing
2026-02-15 15:02:17 -06:00
74332e2203
fix: Gaiia and Preseem sync bugs from API response mismatches
- Billing subscription and inventory item status fields are scalars, not objects
- IP blocks use "block" key, not "subnet"
- Strip nil variables from GraphQL pagination (Gaiia returns null for after:null)
- Handle nil root keys in paginated responses gracefully
- Handle empty string response from Preseem scores endpoint
2026-02-15 12:23:47 -06:00
c790794191 Fix tests, credo issues, and Gaiia sync bugs
- Fix CLOAK_KEY placeholder in nix shell (invalid base64)
- Fix error HTML test assertion to match actual template
- Fix Gaiia sync: read 'subnet' field instead of 'block' for IP blocks
- Fix Gaiia sync: extract nested status name from subscription objects
- Fix security headers tests for environment detection
- Fix mobile auth tests to use raw_token
- Fix LiveView test assertions to match actual template content
- Fix SNMP config test expectations for test environment
- Refactor: resolve all Credo complexity/nesting issues
  - Extract helper functions in NetBox sync, VISP sync, Sonar sync
  - Flatten nested conditionals in settings, integrations, alerts
  - Use with/validate_present pattern for connection testing
2026-02-15 11:55:49 -06:00
34fe5d7e49 Security fixes: mask credentials in logs/API, fix cookie/CSP/LIKE injection/webhooks
CRITICAL fixes:
- Mask SNMP community string in agent channel logs (CRITICAL-1)
- Remove snmpv3 passwords from REST API responses, return _set booleans (CRITICAL-2)
- Replace snmp_community with snmp_community_set in GraphQL type (CRITICAL-3)

HIGH fixes:
- Fix cookie same_site from invalid 'Towerops' to 'Lax' (HIGH-4)
- Remove unsafe-eval from CSP script-src (HIGH-6)
- Block GraphQL introspection queries in production (HIGH-7)
- Sanitize LIKE wildcards in SNMP device name search (HIGH-8)
- Reject webhooks when no secret configured instead of accepting (HIGH-9)

MEDIUM fixes:
- Hash mobile session tokens (SHA-256) before DB storage (MEDIUM-10)
- Apply security headers in all environments, not just prod (MEDIUM-14)
- Add GraphQL query complexity limit (500) in production (MEDIUM-16)
- Fix X-Frame-Options to DENY to match frame-ancestors 'none' (MEDIUM-13)
2026-02-15 09:09:04 -06:00
669dec5d89 proto: sync agent.proto with towerops-agent
- Add transport field to SnmpDevice and SnmpConfig
- Add arch field to AgentHeartbeat (already in .pb.ex)
- Add Check, CheckResult, CheckList messages for service checks (HTTP/TCP/DNS)
- Add checks field to AgentConfig
- Add CheckResult to Metric oneof
- Sync proto source with compiled .pb.ex
2026-02-15 08:56:48 -06:00
02474c529f ui: major redesign - NOC dashboard, consolidated nav, dense tables, flat alerts 2026-02-14 20:15:07 -06:00
8752dfec49 fix: netbox url field type, gaiia ipRange→block, remove unknown webhook log 2026-02-14 17:44:01 -06:00
a3730db579 i18n: add Spanish locale with full translations, language selector, and locale hook 2026-02-14 17:44:01 -06:00
c8e398ea05 Add NetBox sync worker + improve Gaiia error messages
- NetBox.Sync: pull devices/sites from NetBox, match by name/IP, upsert
- NetBoxSyncWorker: Oban cron every 30min for enabled NetBox integrations
- Gaiia sync: show actual error type instead of generic 'unexpected error'
2026-02-14 17:11:38 -06:00
b8a737ddf4 Fix Gaiia: subnet -> ipRange on NetworkSiteIpBlock 2026-02-14 16:16:44 -06:00
e3c730a8e9 Fix Gaiia readable_id: coerce to string (can be integer) 2026-02-14 16:15:59 -06:00
ebcba70925 Fix Gaiia GraphQL: SubscriptionStatus is scalar, not object 2026-02-14 15:09:05 -06:00
7b72ed3164 PagerDuty two-way sync: webhook receiver for resolve/acknowledge
- POST /api/v1/webhooks/pagerduty/:organization_id endpoint
- HMAC-SHA256 signature verification (optional, via webhook_secret)
- incident.resolved → resolve_alert_silent (no re-notify to PagerDuty)
- incident.acknowledged → acknowledge_alert_silent (no re-notify)
- Matches alerts via dedup_key format: towerops-alert-<uuid>
- Webhook secret configurable in org settings PagerDuty panel
- Setup instructions in UI for PagerDuty Generic Webhooks (v3)
2026-02-14 14:58:58 -06:00
29c2039832 Auto-resolve device_up recovery alerts on creation
device_up alerts are informational recovery events, not ongoing issues.
Previously they were created without resolved_at, inflating the
unresolved alert count. Now they're born resolved.

Includes a data migration to fix existing stale device_up alerts.
2026-02-14 14:53:17 -06:00
61f803afe5 Add language user setting (default: English) 2026-02-14 14:53:17 -06:00
1cfe306373
format 2026-02-14 14:52:08 -06:00
d1cd608ae9 feat: actionable insights with fix-it buttons, proper pluralization via gettext
- Gaiia 'not tracked' insight: Map to Gaiia inventory + View reconciliation buttons
- Gaiia 'ghost' insight: Clean up stale mappings button
- Gaiia 'mismatch' insight: Review mismatches button
- Fixed device(s) → proper ngettext pluralization
- Fixed Towerops → TowerOps capitalization
2026-02-14 14:46:19 -06:00
db40a3e5b7 feat: show human-readable sync status messages for integrations
- Added last_sync_message field to integrations schema
- Gaiia sync: shows item counts on success, friendly error on failure
- Preseem sync: shows AP/device counts on success, friendly error on failure
- Error messages translated from technical errors to user-friendly text
- Message displayed on integration card next to status badge
- Exposed in REST API response
2026-02-14 14:11:02 -06:00
5917190efe feat: add NetBox integration with sync direction, entity selection, and filtering
- Three sync modes: NetBox→TowerOps (pull), TowerOps→NetBox (push), Bidirectional
- Configurable entity sync: devices, sites, IP addresses, interfaces
- Optional filters by device role, site, and tag (comma-separated)
- NetBox client with test connection, CRUD operations
- Radio card UI for sync direction, checkbox cards for entity selection
- Credentials stored encrypted (URL, API token, sync config)
2026-02-14 14:09:01 -06:00
a028695d3b
format 2026-02-14 13:57:25 -06:00
392b1e135c fix: trace page crash - detected_at → changed_at to match ConfigChangeEvent schema 2026-02-14 13:44:07 -06:00
950f4d9ae0 fix: implement Gaiia webhook signature verification per their spec
- Header: X-Gaiia-Webhook-Signature with t=timestamp,v1=signature format
- Signed payload: timestamp.body (not just body)
- 5-minute timestamp tolerance to prevent replay attacks
- Updated tests to match new 4-arity verify_signature/4
2026-02-14 13:37:26 -06:00
5df5c97c45 feat: add geographic location to sites with geocoding, address fields, and Leaflet.js network map 2026-02-14 13:07:14 -06:00
74f112642d feat: make network insights actionable with affected device links and metadata 2026-02-14 13:07:14 -06:00
146f5745cf
fix: resolve compilation errors, test failures, and credo issues
- Escape HEEx template braces in GraphQL/API docs with raw(~S[...])
- Fix test assertions for updated marketing copy and UI text
- Extract helper functions in GraphQL resolvers to reduce nesting depth
- Create shared ErrorHelpers module for API controllers
- Fix ETS race condition in brute force whitelist cache for async tests
- Fix property test generators to use ASCII instead of printable unicode
- Add alert_severity helper to site_live/show
- Update accounts fixtures for explicit user confirmation
2026-02-14 12:23:10 -06:00
bb4ff232e3 feat: add PagerDuty Events API v2 integration with 2-way alert sync 2026-02-14 11:28:57 -06:00
f326eb5dd4 feat: require email verification before first login
- Remove auto-confirmation on registration (register_user and register_user_with_organization)
- Add deliver_user_confirmation_instructions/2 and confirm_user/1 to Accounts context
- Add verify_email_token_query/2 and by_user_and_contexts_query/2 to UserToken
- Make deliver_confirmation_email public in UserNotifier
- Send confirmation email after registration in both flows
- Block unconfirmed users at password login with helpful message
- Add UserConfirmationController with confirm/resend routes
- Add resend confirmation link to login page
- Update test fixtures to confirm users after registration
2026-02-14 11:28:57 -06:00
8d00a04187
feat: add FluxCD image automation, fix Preseem duplicate AP crash
Switch deployment pipeline from GitLab Agent kubectl to FluxCD image
automation. CI now only builds and pushes tagged images to Forgejo
registry; FluxCD detects new tags and updates the deployment manifest.

Fix Ecto.MultipleResultsError on /dashboard when a device has multiple
Preseem access point records by using limit(1) instead of Repo.get_by.
2026-02-14 10:30:12 -06:00
12b1a64f8d fix: use atom for Ecto.Enum alert_type in activity feed query 2026-02-13 19:52:29 -06:00