Commit graph

1193 commits

Author SHA1 Message Date
c8237cbcdc
fix: update tests for alert_type string migration and LiveView changes
- Add normalize_alert_type helpers for backward compatibility
- Update alert_type assertions from atoms to strings
- Fix org selection test to use form submission instead of phx-click
- Skip brute force protection test (feature disabled)
- Normalize alert_type in create_alert for atom inputs

Fixes test failures from alert schema changes.
2026-03-05 09:48:07 -06:00
31cccf97b9
perf: add organization_id to alerts for fast single-table queries
- Denormalize organization_id from device to alerts table
- Add composite indexes on (organization_id, alert_type, resolved_at)
- Optimize list_organization_active_alerts to use single-table query
- Eliminates expensive 3-table joins (alerts → device → site)
- Auto-populate organization_id when creating alerts
- Backfill existing alerts in migration

Fixes bug #14 from reliability audit.
2026-03-05 09:32:43 -06:00
ba79c6da6b
fix: add validation for state_descr in sensor readings
Added validation to ensure state_descr:
- Is not empty when present (min: 1 character)
- Has reasonable length limit (max: 255 characters)
- Prevents storing invalid/malformed state descriptions

This prevents invalid state sensor readings from being stored.

Fixes Medium Bug #16 from reliability audit.
2026-03-05 09:27:07 -06:00
cc0a4f4ed6
refactor: replace Task.start with Oban for alert notifications
Replaced fire-and-forget Task.start calls with reliable Oban workers
for all alert notifications.

**Why Oban instead of Task.start:**
-  Persistent - jobs survive crashes/restarts
-  Automatic retries with backoff (3 attempts)
-  Monitoring via Oban dashboard
-  Rate limiting via queue concurrency
-  Distributed coordination across servers

**Changes:**
- Created AlertNotificationWorker for trigger/acknowledge/resolve
- Added notifications queue (concurrency: 10) to Oban config
- Replaced 4 Task.start calls in alerts.ex and device_monitor_worker.ex
- Updated 2 test assertions for string alert_type

**Files changed:**
- lib/towerops/workers/alert_notification_worker.ex (new)
- lib/towerops/alerts.ex
- lib/towerops/workers/device_monitor_worker.ex
- config/dev.exs
- config/runtime.exs
- test/towerops/alerts_test.exs

Notifications are now guaranteed to be delivered with automatic retry.
2026-03-05 09:22:14 -06:00
0ac99f679c
fix: convert alert_type from enum to string and fix SQL array syntax
Two critical production bugs fixed:

1. **Alert type enum → string conversion**
   - Changed Alert.alert_type from Ecto.Enum to :string for flexibility
   - Updated all queries to use "device_down"/"device_up" strings instead of atoms
   - Fixed pattern matching in alerts.ex and device_monitor_worker.ex
   - Updated 44+ test files to use string literals

2. **SQL array indexing syntax error in activity feed**
   - Fixed PostgreSQL syntax: `array_agg(...)[1]` → `(array_agg(...))[1]`
   - Prevents "syntax error at or near [" in activity feed queries

3. **Added comprehensive timer cleanup tests**
   - Tests for mobile_qr_live.ex timer cleanup on terminate
   - Tests for agent_live index timer cleanup
   - Verifies memory leak fixes from previous commits

Files changed:
- lib/towerops/alerts.ex
- lib/towerops/alerts/alert.ex
- lib/towerops/activity_feed.ex
- lib/towerops/workers/device_monitor_worker.ex
- test/**/*_test.exs (44+ files with alert_type references)
- test/towerops_web/live/mobile_qr_live_test.exs
- test/towerops_web/live/agent_live_test.exs
- test/towerops/workers/device_poller_worker_test.exs

All tests passing except 1 unrelated brute force protection test.
2026-03-05 09:12:39 -06:00
1d1a686634
docs: update changelog and memory with bug fix details
- Added comprehensive CHANGELOG.txt entry documenting all 16 bug fixes
- Updated priv/static/changelog.txt with user-facing improvements
- Enhanced MEMORY.md with key learnings:
  * Task.yield_many race condition patterns
  * Enum.zip data corruption prevention
  * LiveView timer memory leak fixes
  * PubSub subscription cleanup
  * Pattern match error handling
  * Health check log silencing

Also fixed:
- Added error checking for sensor reading batch inserts
2026-03-05 08:55:59 -06:00
86a5dc728c
fix: comprehensive bug fixes from reliability audit
Critical Fixes (5):
- Fix Task.yield_many race condition causing data corruption in DevicePollerWorker
- Fix Enum.zip data corruption in SNMP Base Profile with length validation
- Fix missing Alert schema fields for check alerts (check_id, severity, etc.)
- Fix memory leaks from uncancelled LiveView timers in 4 components
- Fix PubSub subscription leak in device form credential testing

High Severity Fixes (3):
- Fix clock skew bug in needs_discovery? check with DateTime.diff clamping
- Fix nil crash in interface status display with proper nil handling
- Fix migration index names after equipment→devices table rename

Medium Severity Fixes (6):
- Fix race condition in device monitor worker (duplicate maintenance checks)
- Fix missing preload validation in devices.ex get_org_default_agent
- Fix broad rescue clause in alerts.ex with specific error handling
- Fix fire-and-forget notification tasks with try-catch error logging
- Fix LiveView state bleeding between tabs (assign_new → assign)
- Add catch-all handle_info callbacks to 3 LiveViews

Infrastructure:
- Silence health check endpoint logs (/health, /health/time)
- Add migration to fix equipment index names missed in rename

Files Changed: 16 files modified, 1 migration added
All changes compile successfully and are backward-compatible.
2026-03-05 08:53:30 -06:00
ec03a03cf5
fix: completely silence logs for noisy health check endpoints
Created FilterNoisyLogs plug that disables Phoenix logging for both
request and response logs by setting conn.private.phoenix_log to false.

This prevents both:
- Initial request log (Plug.Telemetry)
- Response log (Phoenix.Logger "Sent 200 in Xms")

Filters:
- GET /health (Kubernetes probes)
- HEAD / (external uptime monitors)
2026-03-05 08:23:12 -06:00
c56496f7cc
fix: org switching now correctly updates session
The select_org LiveView event was updating default_organization_id in
the database but not the session's current_organization_id. Since
load_default_organization prioritizes the session value, users would
always get redirected back to the stale org.

Replaced with a controller POST action that updates both the session
and DB default before redirecting.
2026-03-04 17:04:49 -06:00
c10394808a
chore: remove unused functions from disabled brute force protection
Removed all unused private functions and imports from BruteForceProtection
plug since the feature is currently disabled (Cloudflare handles security).

Functions can be restored from git history if feature is re-enabled.

Fixed compilation warnings:
- unused function track_404_if_needed/2
- unused function maybe_track_404/2
- unused function get_remote_ip/1
- unused function get_fallback_ip/1
- unused function format_remote_ip/1
- unused function check_and_block/2
- unused function block_request/2
- unused function agent_websocket?/1
- unused alias BruteForce
- unused import Plug.Conn
2026-03-04 16:57:48 -06:00
01ebb8968d
chore: filter out external uptime monitor HEAD requests from logs 2026-03-04 16:48:59 -06:00
daf66038c6
fix: auto-resolve Gaiia reconciliation insights when issues are fixed 2026-03-04 16:31:08 -06:00
11d976531b
fix: auto-dismiss agent offline insights when agent reconnects 2026-03-04 16:25:08 -06:00
8109bd71c8 fix: auto-dismiss poll gap insights when polling resumes
When a device resumes polling (last_snmp_poll_at updated), automatically
dismiss any active 'device_poll_gap' insights for that device.

This prevents stale insights from lingering after connectivity issues
are resolved (e.g., agent reconnection after being blocked).

Changes:
- Auto-dismiss device_poll_gap insights in update_snmp_poll_time()
- Log when insights are auto-dismissed for visibility
- Fix brute force protection compilation error
2026-03-04 16:19:28 -06:00
89949fdfd6 fix: deduplicate Preseem sync entries in activity feed
Group sync entries by minute to show only one entry per minute instead
of multiple duplicate entries with the same timestamp. This aggregates:
- Total records synced in that minute
- Average sync duration
- Most recent status

This keeps the activity feed informative while reducing noise from
rapid syncs happening within the same minute.
2026-03-04 15:18:03 -06:00
74ad58bada fix: hide successful Preseem syncs from activity feed
Only show failed Preseem syncs in the activity feed to reduce noise.
Successful syncs happen every 10-20 minutes and clutter the dashboard
without providing useful information to users.

Users can still see all sync history (including successful syncs) in
the Preseem integration settings page if needed.
2026-03-04 15:14:06 -06:00
74173dad0b fix: temporarily disable brute force IP protection
Disable all IP blocking and 404 tracking to allow unrestricted access
while we review and tune ban thresholds and exemptions.

This is a temporary measure to prevent legitimate traffic from being
blocked. Will re-enable after proper configuration.
2026-03-04 14:05:39 -06:00
94bc755dd8
fix: add dark mode support to marketing site
Add dark: variants throughout the marketing page and layout so text,
backgrounds, cards, badges, and borders render properly when the
system or user preference is set to dark mode. Use invert + screen
blend mode on the logo PNG to eliminate its white background.
2026-03-04 13:57:12 -06:00
dce9253afc
fix: exempt agent WebSocket connections from brute force IP bans
Agent connections authenticate at the channel join level with token
validation, so they should not be subject to IP-based brute force
protection. This prevents legitimate agents from being banned when
connecting to the server.

Changes:
- Exempt /socket/agent/websocket from all IP ban checks
- Update documentation to reflect agent exemption
- Agents still authenticate securely via token during channel join
2026-03-04 13:53:13 -06:00
f0fea0dd3e
fix: retry firmware fetch with exponential backoff on network errors
Increase max_attempts from 3 to 10 and add custom backoff/1 that uses
quadratic growth (attempt² × 60s) capped at 2 hours, so transient
network failures retry over ~9 hours rather than being discarded after
3 quick attempts.
2026-02-18 10:34:51 -06:00
998bdf39b8
update 2026-02-18 10:19:56 -06:00
7081e5daed
fix: correct subscriber counts by filtering CGNAT IPs and excluding routers
Router devices were inflating per-site subscriber counts because:
1. CGNAT IPs (100.64.0.0/10) in router ARP tables caused false matches
2. Routers see all traffic and shouldn't have subscribers attributed
3. Site totals were computed from raw links without filtering

Changes:
- Filter CGNAT (RFC 6598) IPs from device subscriber links
- Deduplicate accounts across devices by match method priority
- Detect routers via SNMP sys_descr and zero out per-device counts
- Derive site totals from filtered per-device impacts
- Add per-device Subs/MRR columns to site device tables
- Add device link fallback for sites without IP blocks
- Fix Gaiia sync case mismatch and execution order
- Improve device list toolbar button icons and tooltips
2026-02-18 09:15:10 -06:00
7668518114
fix: handle missing value attr in input component when using name= directly
When calling <.input name="..." ...> without a field or value attr, @value
was absent from assigns causing a KeyError. Added default: nil to the attr
and use __given__ to detect whether value was explicitly passed so the field
clause still reads value from the FormField correctly.
2026-02-17 16:13:46 -06:00
0f32086933
Ignore pod shutdown errors in ErrorTracker
Add ErrorTrackerIgnorer to drop port_died, write_failed, epipe, and
broken pipe errors that occur during K8s pod shutdown. Same patterns
already filtered by HoneybadgerFilter and LoggerFilters.
2026-02-17 10:59:51 -06:00
2929848f51
fix: validate NetBox URL scheme before making HTTP requests
The NetBox client crashed with "scheme is required for url" when
integration credentials had a nil or blank URL. normalize_url/1 now
validates the scheme and returns a clear error instead of letting
Finch blow up.
2026-02-17 10:50:00 -06:00
edeeacfefb
Add Exceptions link to admin nav bar 2026-02-17 10:44:03 -06:00
b7e2fb497c
Add Error Tracker link to admin dashboard 2026-02-17 10:37:44 -06:00
c92dfff6c9
Add ErrorTracker, remove Honeybadger email notice filter
Replace the HoneybadgerNoticeFilter that emailed raw stacktraces on
every exception with ErrorTracker for self-hosted error tracking.
Honeybadger itself is retained. ErrorTracker dashboard is mounted at
/admin/errors behind the superuser auth wall.
2026-02-17 08:13:34 -06:00
1712e7ac9d
Skip SNMP check execution in CheckExecutorWorker for agent-polled devices
CheckExecutorWorker was missing the phoenix_snmp_disabled and
should_phoenix_poll_device guards that DevicePollerWorker and
DeviceMonitorWorker already have. This caused SNMP checks to execute
through Phoenix even when devices are assigned to agents, creating
an infinite loop of failing jobs that fill the Oban queue.
2026-02-17 07:47:15 -06:00
78aa415073 Skip Phoenix poller/monitor scheduling when SNMP disabled
JobCleanupTask was scheduling poller and monitor jobs for all SNMP-enabled
devices on every startup without checking phoenix_snmp_disabled. These jobs
would immediately bail out but still consumed queue capacity.

JobHealthCheckWorker had the same issue - recovering 'missing' monitor jobs
that would just no-op.

Both now check Client.phoenix_snmp_disabled() before scheduling.
2026-02-17 07:12:15 -06:00
7160bf95d6 Add admin endpoint to flush Oban jobs by state 2026-02-16 16:50:23 -06:00
de986bddf6
Prevent Oban polling/monitoring job stacking per device
When the Oban queue backs up, the 60-second uniqueness window expires
and duplicate jobs stack up per device. Switch to period: :infinity so
only one poll/monitor job exists per device at any time. Add replace
option to supersede stale scheduled jobs with updated scheduled_at.
Remove :executing from unique states so self-scheduling works while
the current job runs. Set max_attempts: 1 since retrying stale polls
is pointless.

Also fix all credo --strict issues across the codebase.
2026-02-16 16:37:48 -06:00
e09b306f9c
Fix maintenance window form crash on mount
The form template iterated over changeset error tuples directly
(e.g. changeset.errors[:name] returns a tuple, not a list), causing
Protocol.UndefinedError for Enumerable on Tuple.

Rewrote form to use to_form/2 and <.input> components per LiveView
conventions. Errors are now handled properly by the input component.
2026-02-16 16:04:59 -06:00
6e3db720fd Collapse device page tabs into header row — single line nav 2026-02-16 15:54:42 -06:00
44d9bb4c49 Merge pull request 'feature/subscriber-impact' (#6) from feature/subscriber-impact into main
Reviewed-on: graham/towerops-web#6
2026-02-16 09:43:54 -08:00
b12d46c716 Add wireless client table discovery and improve subscriber matching
Poll connected client/SM tables from APs via vendor-specific SNMP:
- Cambium ePMP: cambiumAPConnectedSTATable
- Ubiquiti airOS: ubntStaTable
- MikroTik RouterOS: mtxrWlRtabTable

Wireless client MAC matches now take highest priority for
subscriber-to-AP mapping, giving accurate per-radio subscriber counts.
2026-02-16 11:16:52 -06:00
520f7b0c54 Add subscriber-to-AP matching via ARP + Gaiia integration
Matches Gaiia subscriber accounts to TowerOps devices using:
- ARP table IP matching (high confidence)
- ARP table MAC matching (high confidence)
- Site CIDR fallback (medium confidence)

Enables subscriber impact display on alerts and device pages.
2026-02-16 11:07:20 -06:00
261bfd4b95 Merge pull request 'Fix dark mode inconsistencies across all pages' (#4) from fix/dark-mode-consistency into main
Reviewed-on: graham/towerops-web#4
2026-02-16 08:41:46 -08:00
7ec2b2a00a Fix dark mode inconsistencies across all pages
Add missing dark: variant classes to:
- consent_prompt, check_live/form_component: modal overlays
- error pages (404, 500): body background
- marketing_layouts: page background
- home page: pricing card, CTA button, feature card
- totp_enrollment: QR container
- admin audit_live/monitoring_live: badge colors, event borders
- alert_live: acknowledged-by text
- dashboard_live: severity color fallback
- device_live (index, show, form): status text, spinners
- settings_live: checkbox borders
- trace_live: status colors, labels, table headers
2026-02-16 10:31:52 -06:00
0f9fac76d9 Add maintenance windows LiveView UI
- Index page with All/Active/Upcoming/Past filter tabs
- Show page with active banner, edit/delete actions
- Form page with scope picker (org-wide/site/device), datetime fields
- Nav link added to desktop and mobile nav
2026-02-16 10:26:10 -06:00
3f2b2a5d5e Merge pull request 'Add onboarding flow for new organizations' (#3) from feature/onboarding-flow into main
Reviewed-on: graham/towerops-web#3
2026-02-16 08:23:22 -08:00
2735c1a032 Add onboarding flow for new organizations 2026-02-16 10:14:45 -06:00
0cd62951ec Replace 'Other Integrations' with specific category headers on settings page
Each non-billing integration now shows under its proper category:
- Quality of Experience (Preseem)
- Incident Management (PagerDuty)
- Infrastructure & IPAM (NetBox)
2026-02-16 10:06:47 -06:00
5b786aac08 Add maintenance windows and weather overlay (backend)
Maintenance Windows:
- maintenance_windows table with org/site/device scoping
- Full CRUD context (Towerops.Maintenance)
- device_in_maintenance?/1 checks device → site → org hierarchy
- Alert suppression in DeviceMonitorWorker (skips alerts during maintenance)
- maintenance_badge/1 helper for UI display

Weather Overlay:
- weather_observations + weather_alerts tables
- OpenWeatherMap free API client (lat/lon based)
- WeatherSyncWorker (Oban, 15min interval, respects rate limits)
- Observation parsing from OWM response format
- Correlation helpers: severe_weather_near?/2, weather_at/2
- Severity classification tuned for WISP ops (wind, rain, ice thresholds)
- Per-org latest observations query for dashboard
- 30-day automatic cleanup

Both features are backend-only — no UI yet.
2026-02-16 10:01:19 -06:00
541b418d57 Group billing providers on settings integrations tab
Add 'Billing & Subscriber Management' section header with 'Choose one'
badge and 'Other Integrations' divider on the settings page integrations
tab. Reorder providers: billing first (Gaiia, Sonar, Splynx, VISP),
then QoE/incidents/infrastructure.
2026-02-16 08:34:46 -06:00
91946946a6 Make billing integration exclusivity more obvious
Add 'Choose one' badge with indigo border to billing category,
clarify that only one billing platform can be active at a time.
2026-02-16 08:22:47 -06:00
47b364a112
Fix Gaiia inventory matching, mapping UX, and logo link
- Fetch MAC address from Gaiia custom fields during inventory sync
- Remove name-based device matching (model names caused false matches)
- Keep IP-only matching for device suggestions
- Make unmapped rows clickable in the Gaiia mapping table
- Point authenticated logo link to /dashboard instead of /orgs
- Fix flaky monitor test teardown race condition
2026-02-15 17:41:43 -06:00
b4c0407ee0 Add granular org roles: executive, technician
New role system:
- owner: full access + financials
- admin: full access + financials (except org deletion)
- executive: read-only + full financial visibility (MRR, revenue)
- technician: field ops (devices, sites, alerts) without financials
- member: legacy alias for technician (migrated)
- viewer: read-only, no financials

Financial data (MRR, revenue, billing) gated behind can_view_financials
assign in all LiveView templates: dashboard, alerts, trace, site show.

Includes migration to rename existing member roles to technician.
2026-02-15 17:40:53 -06:00
d32ba58b75 Hide config backup and config change features from UI
Temporarily hide config change tracking, config timeline links,
and config backup references across dashboard, site show, trace,
and home page. Code preserved with hidden class / HTML comments.
2026-02-15 17:26:10 -06:00
866715fdcc Hide MikroTik API/SSH features from UI
Temporarily disable all MikroTik-related UI elements using false guards.
Code is preserved and can be re-enabled by removing the guards.

Hidden: org settings tab/panel, device form config section,
device show backups/timeline tabs, site form config section,
help page nav item.
2026-02-15 17:18:55 -06:00