graham/towerops
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
towerops/lib/towerops_web
History
Graham McIntire cedbc5abd1
fix: resolve critical data loss and protobuf schema bugs (Phase 1) 
This commit addresses Phase 1 (CRITICAL - Data Loss Prevention) issues
from the comprehensive bugs and inconsistencies analysis.

CRITICAL FIXES:

1. Protobuf Schema Inconsistencies
   - Added missing job_id field (field 5) to SnmpResult message
   - Fixed AgentError message structure (job_id, message, timestamp)
   - Removed obsolete transport field from SnmpDevice struct
   - Regenerated protobuf code with protoc-gen-elixir
   - Prevents crashes when agent sends SNMP results or errors

2. Interface/Sensor Discovery Timeout Data Loss
   - Changed timeout behavior to return error instead of empty list
   - Prevents deletion of ALL existing interfaces/sensors on slow networks
   - Discovery fails gracefully instead of destroying historical data
   - Addresses commit 7a57f7c sensor discovery pipeline data loss issue

3. Polling Offset Schedule Mismatch
   - Fixed schedule_next_poll() to use offset only (not interval + offset)
   - Maintains consistent polling intervals across all executions
   - Prevents load bunching that offset was designed to prevent
   - Example: 300s interval with offset=94s now consistently polls every 5m

4. Always Reschedule Race Condition
   - Added should_continue_polling?() and should_continue_monitoring?()
   - Only reschedule if device is still enabled and should be polled by Phoenix
   - Prevents zombie jobs from continuing to poll disabled/deleted/reassigned devices
   - Stops race condition where stop_polling() is bypassed by in-flight jobs

Files Changed:
- priv/proto/agent.proto
- lib/towerops/proto/agent.pb.ex (regenerated)
- lib/towerops_web/channels/agent_channel.ex
- lib/towerops/snmp/discovery.ex
- lib/towerops/workers/device_poller_worker.ex
- lib/towerops/workers/device_monitor_worker.ex
- test/towerops/workers/device_poller_worker_test.exs

Test Results: All passing
- device_poller_worker_test.exs: 8 tests, 0 failures
- device_monitor_worker_test.exs: 3 tests, 0 failures
- discovery_test.exs: 13 tests, 0 failures

Remaining Phases:
- Phase 2 (HIGH): 8 data integrity issues
- Phase 3 (MEDIUM): 10 reliability issues
- Phase 4 (LOW): 3 cleanup issues

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-09 09:16:03 -06:00
..
channels fix: resolve critical data loss and protobuf schema bugs (Phase 1) 2026-02-09 09:16:03 -06:00
components UI improvements 2026-02-05 14:57:11 -06:00
controllers fix: prevent site assignment when sites are disabled for org 2026-02-08 13:24:48 -06:00
helpers more tests and fixes 2026-02-07 11:50:18 -06:00
live update 2026-02-09 08:47:53 -06:00
plugs security: add session inactivity timeout and comprehensive security documentation 2026-02-08 10:47:44 -06:00
endpoint.ex fix: improve agent polling and SNMP testing 2026-02-05 12:34:28 -06:00
gettext.ex init 2025-12-21 11:10:43 -06:00
gettext_helpers.ex feat: migrate email templates to gettext (Phase 2 - Emails) 2026-02-02 09:48:30 -06:00
permissions.ex handle mikrotik ssh 2026-02-02 16:42:18 -06:00
plug_exceptions.ex handle exceptions gracefully on api endpoints 2026-02-04 12:18:14 -06:00
router.ex feat: create MonitoringLive base structure with PubSub subscription 2026-02-06 18:27:40 -06:00
telemetry.ex fix: add Oban config to dev.exs and replace Exq telemetry with Oban 2026-01-24 16:39:45 -06:00
user_auth.ex fix: always redirect to /devices when stopping impersonation 2026-02-08 14:01:15 -06:00