Final fixes to pass all tests after audit improvements:
- Fixed reader goroutine context handling: added dedicated goroutine that
closes WebSocket connection on context cancellation, enabling fast shutdown
- Fixed HTTP client timeout: shared transports but per-check client with
custom timeout and redirect policy
- Fixed HTTP redirect handling: CheckRedirect function properly respects
FollowRedirects config
- Updated TestWSDialWriteHandshakeError: removed duplicate assertion, now
accepts both 'write handshake' and 'set deadline' errors
- Updated job executor tests: changed to expect error results on nil device
(executeSnmpJob, executeCredentialTest, executePingJob, executeMikrotikJob)
- Added missing 'strings' import to snmp_test.go
All 249 tests now pass with 97.6% coverage. go vet clean.
When the WebSocket breaks (broken pipe), the main loop can get stuck
in handleMessage -> dispatchJob -> submit if the worker pool is full.
submit blocks on ctx.Done() but ctx is the parent context which only
cancels on SIGINT/SIGTERM, not on connection errors.
Create a session-scoped context that cancels when read/write errors
occur. Pass it to handleMessage so blocked submit calls unblock
immediately, allowing the main loop to receive the write error and
reconnect.
Previously the agent called os.Exit(0) when receiving a server restart
event, requiring Docker or manual intervention to bring it back. Now
it reconnects immediately with a fresh session, which is better for
both dev (no lost process) and production (faster recovery).
When the WebSocket connection drops, session cleanup waits for all
worker pools to drain. SNMP/MikroTik operations against unresponsive
devices can block for 30+ seconds per worker, preventing the agent
from reconnecting during that time.
Add stopWithTimeout(5s) so pool shutdown is bounded. If workers don't
finish within the timeout, they're abandoned and reconnection proceeds
immediately.
CRITICAL: The agent never reset its exponential backoff delay after
successful connections, causing permanent 60s reconnect delays after
multiple deploys.
Root cause:
- retryDelay started at 1s, doubled to 60s cap on each disconnect
- Never reset when connection succeeded
- After ~6 deploys, agent permanently waited 60s between reconnects
Fixes:
1. Reset backoff to 1s after session runs for 30s+ (indicates stable connection)
2. Reduce max retry from 60s → 10s (60s is too long for production)
Impact:
- After Phoenix deploy, agent reconnects within seconds instead of minutes
- Exponential backoff still prevents connection storms on real failures
- Typical reconnect sequence: 1s → 2s → 4s → 8s → 10s (cap)
- Fix blocking MikroTik channel sends that could leak goroutines
- Default to 10s timeout when check TimeoutMs is 0
- Add payload size limit to check_jobs event
- Add 30s write deadline to WebSocket writes
- Use random ICMP sequence numbers to prevent ping ID collisions
- Atomic host key file writes via temp+rename
- Sanitize update URLs in log output
- Zero correct buffer in sendBinaryResult after MarshalAppend
Adds LLDP (Link Layer Discovery Protocol) topology discovery capability
to the agent for local network neighbor detection.
New Features:
- LLDP-MIB walker for discovering network neighbors via SNMP
- New LLDP_TOPOLOGY job type for topology discovery jobs
- Automatic IPv4 and IPv6 management address parsing
- Bulk neighbor discovery with full port information
Protocol Buffers:
- Added LldpTopologyResult message for topology data
- Added LldpNeighbor message for individual neighbor info
- Extended JobType enum with LLDP_TOPOLOGY (value 5)
Implementation:
- lldp.go: LLDP discovery logic with SNMP walking
- agent.go: Job dispatch and result channel handling
- Updated handleMessage signature to support LLDP results
- Result sent to Phoenix via "lldp_topology_result" event
LLDP-MIB OIDs:
- 1.0.8802.1.1.2.1.3.3.0 (lldpLocSysName)
- 1.0.8802.1.1.2.1.3.7.1.4 (lldpLocPortDesc)
- 1.0.8802.1.1.2.1.4.1.1.7 (lldpRemPortId)
- 1.0.8802.1.1.2.1.4.1.1.8 (lldpRemPortDesc)
- 1.0.8802.1.1.2.1.4.1.1.9 (lldpRemSysName)
- 1.0.8802.1.1.2.1.4.2.1.3 (lldpRemManAddr)
This enables the Phoenix web application to dispatch LLDP discovery
jobs to agents for local neighbor detection, reducing SNMP load on the
Phoenix server and improving discovery performance.
Extends agent to execute HTTP, TCP, and DNS service checks.
Agent Changes:
- Extended protobuf schema with Check, CheckResult, CheckList messages
- Added checks.go with HTTP/TCP/DNS executors (completely stateless)
- Added checks worker pool (50 workers)
- Handles check_jobs WebSocket event from Phoenix
- Reports check_result back to Phoenix
Check Executors:
- HTTP: Full request support with regex, SSL, headers
- TCP: Port connectivity with send/expect patterns
- DNS: Resolution for A, AAAA, CNAME, MX, TXT records
Architecture: Agent is stateless - receives jobs, executes, reports back.
Phoenix handles all scheduling and state tracking.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Previously the agent proceeded to the main loop without confirming
the server accepted the channel join. A rejected join (e.g. invalid
token) left the agent running but silently non-functional. Now waits
for the join reply with a 10s timeout, returns error on rejection.
Reader goroutine starts before join send so the reply can be received.
Writer goroutine starts after join confirmation.
Previously, resetting slice length to 0 left previous protobuf data
(device IDs, OID values, credentials) in the backing array. Now zeros
the full capacity before returning to the pool.
handleMessage returns bool to signal session end instead of calling
osExit directly. Defers (pool stop, writeCh close, writerWg wait)
now run before exit, preventing in-flight result loss and panics
from writes to closed channels.
Fix 1: Limit inbound job payload size to 4MB to prevent OOM from malicious server
Fix 2: Worker pool submit accepts context and returns bool, prevents blocking event loop
Fix 3: Panic recovery in worker pool goroutines prevents permanent worker death
Fix 4: WebSocket handshake timeout (30s) prevents indefinite blocking on slow server
Fix 5: Writer goroutine propagates errors to main loop via writeErrCh
Use sync.Pool to reuse 4KB byte slices for protobuf marshaling via
MarshalAppend instead of allocating fresh buffers per result message.
Reduces GC pressure under high-throughput polling.
Accumulate SNMP results and flush in batches of 50 or every 100ms,
whichever comes first. Reduces per-message overhead when thousands
of devices produce results near-simultaneously. Also flushes
remaining results on disconnect/shutdown.
Raise write channel from 500 to 10000, SNMP/monitoring result
channels from 1000 to 10000, MikroTik/credential channels to 5000.
Prevents silent data loss when polling thousands of devices.
Replace unbounded goroutine spawning with fixed-size worker pools:
100 workers for SNMP, 20 for MikroTik, 50 for ping. Prevents file
descriptor exhaustion and device overload when handling thousands
of concurrent jobs. Pool provides backpressure when saturated.
Use strconv.FormatInt/FormatUint instead of fmt.Sprintf for integer
conversions in snmpValueToString. Use encoding/hex + strings.Builder
instead of fmt.Sprintf loop in formatHex. Use strconv.FormatUint for
WebSocket ref counter. Reduces allocations per OID significantly.
