graham/towerops
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
towerops/lib
History
Graham McIntire cedbc5abd1
fix: resolve critical data loss and protobuf schema bugs (Phase 1) 
This commit addresses Phase 1 (CRITICAL - Data Loss Prevention) issues
from the comprehensive bugs and inconsistencies analysis.

CRITICAL FIXES:

1. Protobuf Schema Inconsistencies
   - Added missing job_id field (field 5) to SnmpResult message
   - Fixed AgentError message structure (job_id, message, timestamp)
   - Removed obsolete transport field from SnmpDevice struct
   - Regenerated protobuf code with protoc-gen-elixir
   - Prevents crashes when agent sends SNMP results or errors

2. Interface/Sensor Discovery Timeout Data Loss
   - Changed timeout behavior to return error instead of empty list
   - Prevents deletion of ALL existing interfaces/sensors on slow networks
   - Discovery fails gracefully instead of destroying historical data
   - Addresses commit 7a57f7c sensor discovery pipeline data loss issue

3. Polling Offset Schedule Mismatch
   - Fixed schedule_next_poll() to use offset only (not interval + offset)
   - Maintains consistent polling intervals across all executions
   - Prevents load bunching that offset was designed to prevent
   - Example: 300s interval with offset=94s now consistently polls every 5m

4. Always Reschedule Race Condition
   - Added should_continue_polling?() and should_continue_monitoring?()
   - Only reschedule if device is still enabled and should be polled by Phoenix
   - Prevents zombie jobs from continuing to poll disabled/deleted/reassigned devices
   - Stops race condition where stop_polling() is bypassed by in-flight jobs

Files Changed:
- priv/proto/agent.proto
- lib/towerops/proto/agent.pb.ex (regenerated)
- lib/towerops_web/channels/agent_channel.ex
- lib/towerops/snmp/discovery.ex
- lib/towerops/workers/device_poller_worker.ex
- lib/towerops/workers/device_monitor_worker.ex
- test/towerops/workers/device_poller_worker_test.exs

Test Results: All passing
- device_poller_worker_test.exs: 8 tests, 0 failures
- device_monitor_worker_test.exs: 3 tests, 0 failures
- discovery_test.exs: 13 tests, 0 failures

Remaining Phases:
- Phase 2 (HIGH): 8 data integrity issues
- Phase 3 (MEDIUM): 10 reliability issues
- Phase 4 (LOW): 3 cleanup issues

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-09 09:16:03 -06:00
..
mix/tasks Remove debugging logs, fix Mix.env call 2026-02-04 12:10:48 -06:00
snmpkit security: fix critical vulnerabilities and atom exhaustion risks 2026-02-08 10:30:30 -06:00
towerops fix: resolve critical data loss and protobuf schema bugs (Phase 1) 2026-02-09 09:16:03 -06:00
towerops_web fix: resolve critical data loss and protobuf schema bugs (Phase 1) 2026-02-09 09:16:03 -06:00
snmp_lib.ex bring in snmpkit 2026-01-23 12:52:17 -06:00
snmp_mgr.ex credo fixes 2026-01-23 14:01:52 -06:00
snmpkit.ex complete overhaul of snmp engine 2026-01-30 10:41:07 -06:00
towerops.ex init 2025-12-21 11:10:43 -06:00
towerops_native.ex complete overhaul of snmp engine to C nif 2026-01-30 12:27:12 -06:00
towerops_web.ex feat: implement gettext infrastructure (Phase 1) 2026-02-02 09:40:32 -06:00