Commit graph

1833 commits

Author SHA1 Message Date
0ac99f679c
fix: convert alert_type from enum to string and fix SQL array syntax
Two critical production bugs fixed:

1. **Alert type enum → string conversion**
   - Changed Alert.alert_type from Ecto.Enum to :string for flexibility
   - Updated all queries to use "device_down"/"device_up" strings instead of atoms
   - Fixed pattern matching in alerts.ex and device_monitor_worker.ex
   - Updated 44+ test files to use string literals

2. **SQL array indexing syntax error in activity feed**
   - Fixed PostgreSQL syntax: `array_agg(...)[1]` → `(array_agg(...))[1]`
   - Prevents "syntax error at or near [" in activity feed queries

3. **Added comprehensive timer cleanup tests**
   - Tests for mobile_qr_live.ex timer cleanup on terminate
   - Tests for agent_live index timer cleanup
   - Verifies memory leak fixes from previous commits

Files changed:
- lib/towerops/alerts.ex
- lib/towerops/alerts/alert.ex
- lib/towerops/activity_feed.ex
- lib/towerops/workers/device_monitor_worker.ex
- test/**/*_test.exs (44+ files with alert_type references)
- test/towerops_web/live/mobile_qr_live_test.exs
- test/towerops_web/live/agent_live_test.exs
- test/towerops/workers/device_poller_worker_test.exs

All tests passing except 1 unrelated brute force protection test.
2026-03-05 09:12:39 -06:00
CI
a87b729abd chore: update towerops-web image to git.mcintire.me/graham/towerops-web:main-1772722775-cf412e2 [skip ci] 2026-03-05 15:03:24 +00:00
cf412e2261
test: add reliability test for Task.yield_many race condition fix 2026-03-05 08:58:23 -06:00
1d1a686634
docs: update changelog and memory with bug fix details
- Added comprehensive CHANGELOG.txt entry documenting all 16 bug fixes
- Updated priv/static/changelog.txt with user-facing improvements
- Enhanced MEMORY.md with key learnings:
  * Task.yield_many race condition patterns
  * Enum.zip data corruption prevention
  * LiveView timer memory leak fixes
  * PubSub subscription cleanup
  * Pattern match error handling
  * Health check log silencing

Also fixed:
- Added error checking for sensor reading batch inserts
2026-03-05 08:55:59 -06:00
86a5dc728c
fix: comprehensive bug fixes from reliability audit
Critical Fixes (5):
- Fix Task.yield_many race condition causing data corruption in DevicePollerWorker
- Fix Enum.zip data corruption in SNMP Base Profile with length validation
- Fix missing Alert schema fields for check alerts (check_id, severity, etc.)
- Fix memory leaks from uncancelled LiveView timers in 4 components
- Fix PubSub subscription leak in device form credential testing

High Severity Fixes (3):
- Fix clock skew bug in needs_discovery? check with DateTime.diff clamping
- Fix nil crash in interface status display with proper nil handling
- Fix migration index names after equipment→devices table rename

Medium Severity Fixes (6):
- Fix race condition in device monitor worker (duplicate maintenance checks)
- Fix missing preload validation in devices.ex get_org_default_agent
- Fix broad rescue clause in alerts.ex with specific error handling
- Fix fire-and-forget notification tasks with try-catch error logging
- Fix LiveView state bleeding between tabs (assign_new → assign)
- Add catch-all handle_info callbacks to 3 LiveViews

Infrastructure:
- Silence health check endpoint logs (/health, /health/time)
- Add migration to fix equipment index names missed in rename

Files Changed: 16 files modified, 1 migration added
All changes compile successfully and are backward-compatible.
2026-03-05 08:53:30 -06:00
CI
8ead55d2bf chore: update towerops-web image to git.mcintire.me/graham/towerops-web:main-1772720897-c231cbd [skip ci] 2026-03-05 14:32:07 +00:00
c231cbdf4c
docs: update changelog for log filtering improvements 2026-03-05 08:23:52 -06:00
ec03a03cf5
fix: completely silence logs for noisy health check endpoints
Created FilterNoisyLogs plug that disables Phoenix logging for both
request and response logs by setting conn.private.phoenix_log to false.

This prevents both:
- Initial request log (Plug.Telemetry)
- Response log (Phoenix.Logger "Sent 200 in Xms")

Filters:
- GET /health (Kubernetes probes)
- HEAD / (external uptime monitors)
2026-03-05 08:23:12 -06:00
CI
61cad41f5c chore: update towerops-web image to git.mcintire.me/graham/towerops-web:main-1772719169-53ab811 [skip ci] 2026-03-05 14:01:21 +00:00
53ab811564
docs: update changelog for migration fix 2026-03-05 07:54:58 -06:00
212f9089e0
fix: use current_database() in TimescaleDB migration
Previous version used hardcoded 'towerops_dev' which doesn't exist in production.
Now uses current_database() to get the actual database name at runtime.

This fixes the crash: ERROR 3D000 (invalid_catalog_name) database "towerops_dev" does not exist
2026-03-05 07:54:20 -06:00
CI
be1add2e1a chore: update towerops-web image to git.mcintire.me/graham/towerops-web:main-1772668279-5e1f8d3 [skip ci] 2026-03-04 23:55:04 +00:00
5e1f8d3ebe
docs: update changelog for TimescaleDB limit fix 2026-03-04 17:44:44 -06:00
a7ab2831b3
fix: increase TimescaleDB tuple decompression limit to unlimited
Fixes ERROR 53400 (configuration_limit_exceeded) during large interface
sync operations in device discovery.

The sync_interfaces operation was decompressing 146,603 tuples but the
default limit was 100,000. Setting to 0 (unlimited) removes this constraint.

Error occurred in: Towerops.Snmp.Discovery.sync_interfaces/2
2026-03-04 17:43:06 -06:00
CI
1b11bb25fc chore: update towerops-web image to git.mcintire.me/graham/towerops-web:main-1772667234-f1b6151 [skip ci] 2026-03-04 23:34:19 +00:00
f1b61513d6
fix: handle concurrent pushes in build workflow
Add git pull --rebase before pushing deployment.yaml update
to handle race conditions when multiple builds run.
2026-03-04 17:33:26 -06:00
FluxCD
65b41f82fe chore: update towerops image to git.mcintire.me/graham/towerops-web:main-1772667132-044e21a [skip ci] 2026-03-04 23:33:03 +00:00
FluxCD
33cbd99cb7 chore: update towerops image to git.mcintire.me/graham/towerops-web:main-1772666722-10df6d8 [skip ci] 2026-03-04 23:32:06 +00:00
044e21a823
fix: downgrade checkout to v4 and add Docker mirror to build.yaml
- Use actions/checkout@v4 to avoid punycode deprecation warning
- Add docker-mirror.mcintire.me configuration for faster builds
2026-03-04 17:30:48 -06:00
afafe3fa09
chore: remove redundant build-deploy.yml workflow
Keep build.yaml which updates k8s/deployment.yaml for FluxCD GitOps.
2026-03-04 17:30:34 -06:00
12af350937
chore: remove GitLab CI in favor of Forgejo Actions
FluxCD handles deployment automatically via ImagePolicy,
so only build workflow is needed. GitLab CI is redundant.
2026-03-04 17:29:43 -06:00
e29981b307
Revert "feat: migrate deployment to Forgejo Actions from GitLab CI"
This reverts commit 27261f107b.
2026-03-04 17:29:36 -06:00
27261f107b
feat: migrate deployment to Forgejo Actions from GitLab CI
Consolidated build and deployment into Forgejo Actions workflow:
- Added deployment job that runs after successful build
- Uses kubectl to deploy to Kubernetes cluster
- Sets image and deployment timestamp
- Removed GitLab CI configuration

Required setup:
- Add KUBECONFIG secret to Forgejo (base64 encoded kubeconfig file)
- Secret should contain context: towerops/towerops:home-cluster-agent
2026-03-04 17:27:38 -06:00
FluxCD
9e28d37256 chore: update towerops image to git.mcintire.me/graham/towerops-web:main-1772666343-10df6d8 [skip ci] 2026-03-04 23:26:03 +00:00
10df6d8145
test: trigger CI build with fixed runner config 2026-03-04 17:18:28 -06:00
49d11b601b
fix: downgrade checkout action to v4 to avoid punycode deprecation
The punycode module deprecation warning in Node.js is coming from
actions/checkout@v6. Downgrading to v4 which is more stable and
doesn't use the deprecated punycode module.
2026-03-04 17:16:05 -06:00
0a07d6e09e
Revert "fix: resolve Forgejo CI Docker daemon access issues"
This reverts commit 54280fd733.
2026-03-04 17:11:52 -06:00
54280fd733
fix: resolve Forgejo CI Docker daemon access issues
Fixed two CI/CD issues:

1. Docker daemon not accessible
   - Added explicit dockerd startup step
   - Configured buildx to use docker-container driver
   - Driver creates its own buildkitd instance

2. Node.js punycode deprecation warning
   - Downgraded actions/checkout from v6 to v4
   - v4 is more stable and doesn't use deprecated punycode module

Changes ensure the workflow runs successfully in Forgejo Actions runners
that may not have Docker socket access by default.
2026-03-04 17:09:28 -06:00
c56496f7cc
fix: org switching now correctly updates session
The select_org LiveView event was updating default_organization_id in
the database but not the session's current_organization_id. Since
load_default_organization prioritizes the session value, users would
always get redirected back to the stale org.

Replaced with a controller POST action that updates both the session
and DB default before redirecting.
2026-03-04 17:04:49 -06:00
c10394808a
chore: remove unused functions from disabled brute force protection
Removed all unused private functions and imports from BruteForceProtection
plug since the feature is currently disabled (Cloudflare handles security).

Functions can be restored from git history if feature is re-enabled.

Fixed compilation warnings:
- unused function track_404_if_needed/2
- unused function maybe_track_404/2
- unused function get_remote_ip/1
- unused function get_fallback_ip/1
- unused function format_remote_ip/1
- unused function check_and_block/2
- unused function block_request/2
- unused function agent_websocket?/1
- unused alias BruteForce
- unused import Plug.Conn
2026-03-04 16:57:48 -06:00
70838af14d
chore: update vendored Oban packages
- oban_pro: 1.6.11 → 1.6.13
  - Enhanced Smart engine functionality
  - Refactored Dynamic Cron plugin
  - Added diagnostics and executing stage modules

- oban_met: already at 1.0.6 (verified)
- oban_web: already at 2.11.8 (latest)
2026-03-04 16:51:55 -06:00
01ebb8968d
chore: filter out external uptime monitor HEAD requests from logs 2026-03-04 16:48:59 -06:00
daf66038c6
fix: auto-resolve Gaiia reconciliation insights when issues are fixed 2026-03-04 16:31:08 -06:00
11d976531b
fix: auto-dismiss agent offline insights when agent reconnects 2026-03-04 16:25:08 -06:00
8109bd71c8 fix: auto-dismiss poll gap insights when polling resumes
When a device resumes polling (last_snmp_poll_at updated), automatically
dismiss any active 'device_poll_gap' insights for that device.

This prevents stale insights from lingering after connectivity issues
are resolved (e.g., agent reconnection after being blocked).

Changes:
- Auto-dismiss device_poll_gap insights in update_snmp_poll_time()
- Log when insights are auto-dismissed for visibility
- Fix brute force protection compilation error
2026-03-04 16:19:28 -06:00
89949fdfd6 fix: deduplicate Preseem sync entries in activity feed
Group sync entries by minute to show only one entry per minute instead
of multiple duplicate entries with the same timestamp. This aggregates:
- Total records synced in that minute
- Average sync duration
- Most recent status

This keeps the activity feed informative while reducing noise from
rapid syncs happening within the same minute.
2026-03-04 15:18:03 -06:00
74ad58bada fix: hide successful Preseem syncs from activity feed
Only show failed Preseem syncs in the activity feed to reduce noise.
Successful syncs happen every 10-20 minutes and clutter the dashboard
without providing useful information to users.

Users can still see all sync history (including successful syncs) in
the Preseem integration settings page if needed.
2026-03-04 15:14:06 -06:00
74173dad0b fix: temporarily disable brute force IP protection
Disable all IP blocking and 404 tracking to allow unrestricted access
while we review and tune ban thresholds and exemptions.

This is a temporary measure to prevent legitimate traffic from being
blocked. Will re-enable after proper configuration.
2026-03-04 14:05:39 -06:00
94bc755dd8
fix: add dark mode support to marketing site
Add dark: variants throughout the marketing page and layout so text,
backgrounds, cards, badges, and borders render properly when the
system or user preference is set to dark mode. Use invert + screen
blend mode on the logo PNG to eliminate its white background.
2026-03-04 13:57:12 -06:00
dce9253afc
fix: exempt agent WebSocket connections from brute force IP bans
Agent connections authenticate at the channel join level with token
validation, so they should not be subject to IP-based brute force
protection. This prevents legitimate agents from being banned when
connecting to the server.

Changes:
- Exempt /socket/agent/websocket from all IP ban checks
- Update documentation to reflect agent exemption
- Agents still authenticate securely via token during channel join
2026-03-04 13:53:13 -06:00
61586d14e2
chore: update all hex dependencies
Upgrades: phoenix 1.8.4, phoenix_live_view 1.1.26, ecto_sql 3.13.5,
bandit 1.10.3, credo 1.7.17, swoosh 1.22.1, yaml_elixir 2.12.1,
styler 1.11.0, error_tracker 0.8.0, finch 0.21.0, honeybadger 0.25.0.

Fix site_aggregation_test assertion to match actual DB default (NULL,
not 0) for account_count on skipped sites.
2026-03-04 13:30:32 -06:00
FluxCD
18c610ca50 chore: update towerops image to git.mcintire.me/graham/towerops-web:main-1772578710-f761290 [skip ci] 2026-03-03 23:05:29 +00:00
FluxCD
19dd54114a chore: update towerops image to git.mcintire.me/graham/towerops-web:main-1772578563-f761290 [skip ci] 2026-03-03 22:58:30 +00:00
f761290ed7
chore(config): add Renovate and build workflows 2026-03-03 16:54:19 -06:00
FluxCD
8882012a65 chore: update towerops image to git.mcintire.me/graham/towerops-web:main-1771432526-f0fea0d [skip ci] 2026-02-18 16:39:38 +00:00
f0fea0dd3e
fix: retry firmware fetch with exponential backoff on network errors
Increase max_attempts from 3 to 10 and add custom backoff/1 that uses
quadratic growth (attempt² × 60s) capped at 2 hours, so transient
network failures retry over ~9 hours rather than being discarded after
3 quick attempts.
2026-02-18 10:34:51 -06:00
FluxCD
cc26b05a9b chore: update towerops image to git.mcintire.me/graham/towerops-web:main-1771431639-998bdf3 [skip ci] 2026-02-18 16:26:36 +00:00
998bdf39b8
update 2026-02-18 10:19:56 -06:00
7081e5daed
fix: correct subscriber counts by filtering CGNAT IPs and excluding routers
Router devices were inflating per-site subscriber counts because:
1. CGNAT IPs (100.64.0.0/10) in router ARP tables caused false matches
2. Routers see all traffic and shouldn't have subscribers attributed
3. Site totals were computed from raw links without filtering

Changes:
- Filter CGNAT (RFC 6598) IPs from device subscriber links
- Deduplicate accounts across devices by match method priority
- Detect routers via SNMP sys_descr and zero out per-device counts
- Derive site totals from filtered per-device impacts
- Add per-device Subs/MRR columns to site device tables
- Add device link fallback for sites without IP blocks
- Fix Gaiia sync case mismatch and execution order
- Improve device list toolbar button icons and tooltips
2026-02-18 09:15:10 -06:00
076b8454de
docs: merge upstream Phoenix AGENTS.md with project-specific patterns
- Add module design/complexity guidelines, quality checks (credo, sobelow,
  coverage), security guidelines (deps.audit), updated auth section with
  live_session details, OTP guidelines, and expanded Elixir core rules
  from cheezy/kanban upstream AGENTS.md
- Move project-specific patterns into AGENTS.md: browser navigation URL
  state, JS hook memory leak prevention, LiveView form data access,
  SNMP test mocking patterns
- Trim CLAUDE.md to project architecture docs only, removing sections
  now covered in AGENTS.md
2026-02-18 09:08:33 -06:00