Accumulate SNMP results and flush in batches of 50 or every 100ms,
whichever comes first. Reduces per-message overhead when thousands
of devices produce results near-simultaneously. Also flushes
remaining results on disconnect/shutdown.
Build the entire frame (header + mask key + masked payload) in one
buffer and write it with a single syscall instead of two separate
writes. Removes the failOnNthWrite test that tested the old
two-write implementation detail.
Wrap the WebSocket connection in an 8KB bufio.Reader so that small
reads (2-byte frame headers, length extensions) are served from
buffer instead of making individual syscalls.
Raise write channel from 500 to 10000, SNMP/monitoring result
channels from 1000 to 10000, MikroTik/credential channels to 5000.
Prevents silent data loss when polling thousands of devices.
Replace unbounded goroutine spawning with fixed-size worker pools:
100 workers for SNMP, 20 for MikroTik, 50 for ping. Prevents file
descriptor exhaustion and device overload when handling thousands
of concurrent jobs. Pool provides backpressure when saturated.
Use strconv.FormatInt/FormatUint instead of fmt.Sprintf for integer
conversions in snmpValueToString. Use encoding/hex + strings.Builder
instead of fmt.Sprintf loop in formatHex. Use strconv.FormatUint for
WebSocket ref counter. Reduces allocations per OID significantly.
Previously each OID was fetched individually, meaning 1000 OIDs
required 1000 round trips. Now OIDs are batched into groups of 60
(the SNMP max PDU size), reducing round trips by ~50x. Also
pre-sizes the OID value map to avoid repeated rehashing.
Docker tags cannot contain colons. Use a separate tag-safe format
(colons replaced with hyphens) for Docker tags while keeping the
RFC 3339 timestamp as the version baked into the binary.
add libcap and setcap cap_net_raw+p on /bin/ping so ICMP works
without container-level capabilities. add NET_RAW cap_add to
docker-compose.example.yml as fallback.
- Dockerfile: add BuildKit cache mounts for Go module and build caches,
pin both stages to Alpine 3.23
- CI: replace BUILDKIT_INLINE_CACHE with registry-based layer caching
(mode=max) for branch and release builds
- .dockerignore: exclude test files, Rust legacy, nix, and other
non-build artifacts from Docker context
- SNMP: lower MaxRepetitions from default 50 to 10 to prevent Mikrotik
routers from timing out during bulk walks
Job receipt, start, and completion messages were all at debug level,
making it impossible to tell if the agent was processing jobs without
setting LOG_LEVEL=debug.
musl libc has weak fork-safety guarantees in multi-threaded processes.
When Tokio's runtime is running and we fork() for SNMP operations, musl's
internal state can be inconsistent in the child, causing SIGSEGV on every
SNMP walk to Ubiquiti devices. glibc handles this via pthread_atfork.
- Dockerfile: Alpine → rust:1.93-bookworm build + debian:12-slim runtime
- CI: Remove ensure-netsnmp-base/manifest jobs, simplify build deps
- Delete Dockerfile.netsnmp and netsnmp-base.yml workflow
- Add NULL pointer guards in snmp_helper.c for fork-safety
- Add SNMP exception type handling (NoSuchObject/Instance/EndOfMibView)
- Add crash reproduction tests
The ensure-netsnmp-base jobs check if the pre-built image exists in
GHCR before any Docker builds start. First run compiles from source
on native runners (~5 min), subsequent runs skip with a ~5s check.
build-branch and build-release now depend on ensure-netsnmp-manifest.
Alpine 3.23 ships net-snmp 5.9.4 which has segfault bugs in varbind
construction and buffer overflows in octet string handling. Version
5.9.5.2 fixes these issues.
Adds a pre-built base image (Dockerfile.netsnmp) compiled from source
so the net-snmp build doesn't run on every agent deploy. The base
image is built separately via the netsnmp-base workflow and referenced
in the main Dockerfile.
In Fork mode the parent never calls snmp_init_library(), so
pthread_once remains untriggered. The child called init_snmp()
directly, then snmp_open_session() triggered pthread_once calling
init_snmp() a second time. This double-initialization of libnetsnmp
caused signal 11 in every child process.
Let snmp_open_session() handle initialization via pthread_once,
which correctly initializes exactly once and sets numeric OID format.
Net-SNMP was crashing (signal 11) on every operation despite fork isolation.
The issue was that init_snmp() tries to load MIB files by default, which can
cause crashes if MIB files are missing, corrupted, or incompatible.
Since we only use numeric OIDs (e.g., "1.3.6.1.2.1.1.1.0"), we don't need
MIB name resolution at all. This change explicitly disables MIB loading in
the child process by:
1. Setting MIBS and MIBDIRS environment variables to empty
2. Calling netsnmp_set_mib_directory("") to disable MIB directory search
3. These must happen BEFORE calling init_snmp()
This should fix the persistent SIGSEGV crashes in SNMP operations.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Each SNMP GET/WALK now runs in a forked child process. If libnetsnmp
triggers a SIGSEGV or other fatal signal, only the child dies - the
parent agent logs the crash and continues operating for all other devices.
Key changes:
- C helper: snmp_get_isolated() and snmp_walk_isolated() using
fork+pipe pattern with 60s alarm watchdog and mutex-serialized forks
- Rust: IsolationMode enum (Fork/Direct) controlled by
TOWEROPS_SNMP_ISOLATION env var, defaults to Fork
- New CrashRecovered error variant with signal info and logging
- Device poller logs crash recovery events at error level
- Startup logs active isolation mode
Set TOWEROPS_SNMP_ISOLATION=direct to disable isolation for debugging.
