Replace unbounded goroutine spawning with fixed-size worker pools:
100 workers for SNMP, 20 for MikroTik, 50 for ping. Prevents file
descriptor exhaustion and device overload when handling thousands
of concurrent jobs. Pool provides backpressure when saturated.
Use strconv.FormatInt/FormatUint instead of fmt.Sprintf for integer
conversions in snmpValueToString. Use encoding/hex + strings.Builder
instead of fmt.Sprintf loop in formatHex. Use strconv.FormatUint for
WebSocket ref counter. Reduces allocations per OID significantly.
Previously each OID was fetched individually, meaning 1000 OIDs
required 1000 round trips. Now OIDs are batched into groups of 60
(the SNMP max PDU size), reducing round trips by ~50x. Also
pre-sizes the OID value map to avoid repeated rehashing.
Docker tags cannot contain colons. Use a separate tag-safe format
(colons replaced with hyphens) for Docker tags while keeping the
RFC 3339 timestamp as the version baked into the binary.
add libcap and setcap cap_net_raw+p on /bin/ping so ICMP works
without container-level capabilities. add NET_RAW cap_add to
docker-compose.example.yml as fallback.
- Dockerfile: add BuildKit cache mounts for Go module and build caches,
pin both stages to Alpine 3.23
- CI: replace BUILDKIT_INLINE_CACHE with registry-based layer caching
(mode=max) for branch and release builds
- .dockerignore: exclude test files, Rust legacy, nix, and other
non-build artifacts from Docker context
- SNMP: lower MaxRepetitions from default 50 to 10 to prevent Mikrotik
routers from timing out during bulk walks
Job receipt, start, and completion messages were all at debug level,
making it impossible to tell if the agent was processing jobs without
setting LOG_LEVEL=debug.
musl libc has weak fork-safety guarantees in multi-threaded processes.
When Tokio's runtime is running and we fork() for SNMP operations, musl's
internal state can be inconsistent in the child, causing SIGSEGV on every
SNMP walk to Ubiquiti devices. glibc handles this via pthread_atfork.
- Dockerfile: Alpine → rust:1.93-bookworm build + debian:12-slim runtime
- CI: Remove ensure-netsnmp-base/manifest jobs, simplify build deps
- Delete Dockerfile.netsnmp and netsnmp-base.yml workflow
- Add NULL pointer guards in snmp_helper.c for fork-safety
- Add SNMP exception type handling (NoSuchObject/Instance/EndOfMibView)
- Add crash reproduction tests
The ensure-netsnmp-base jobs check if the pre-built image exists in
GHCR before any Docker builds start. First run compiles from source
on native runners (~5 min), subsequent runs skip with a ~5s check.
build-branch and build-release now depend on ensure-netsnmp-manifest.
Alpine 3.23 ships net-snmp 5.9.4 which has segfault bugs in varbind
construction and buffer overflows in octet string handling. Version
5.9.5.2 fixes these issues.
Adds a pre-built base image (Dockerfile.netsnmp) compiled from source
so the net-snmp build doesn't run on every agent deploy. The base
image is built separately via the netsnmp-base workflow and referenced
in the main Dockerfile.
In Fork mode the parent never calls snmp_init_library(), so
pthread_once remains untriggered. The child called init_snmp()
directly, then snmp_open_session() triggered pthread_once calling
init_snmp() a second time. This double-initialization of libnetsnmp
caused signal 11 in every child process.
Let snmp_open_session() handle initialization via pthread_once,
which correctly initializes exactly once and sets numeric OID format.
Net-SNMP was crashing (signal 11) on every operation despite fork isolation.
The issue was that init_snmp() tries to load MIB files by default, which can
cause crashes if MIB files are missing, corrupted, or incompatible.
Since we only use numeric OIDs (e.g., "1.3.6.1.2.1.1.1.0"), we don't need
MIB name resolution at all. This change explicitly disables MIB loading in
the child process by:
1. Setting MIBS and MIBDIRS environment variables to empty
2. Calling netsnmp_set_mib_directory("") to disable MIB directory search
3. These must happen BEFORE calling init_snmp()
This should fix the persistent SIGSEGV crashes in SNMP operations.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Each SNMP GET/WALK now runs in a forked child process. If libnetsnmp
triggers a SIGSEGV or other fatal signal, only the child dies - the
parent agent logs the crash and continues operating for all other devices.
Key changes:
- C helper: snmp_get_isolated() and snmp_walk_isolated() using
fork+pipe pattern with 60s alarm watchdog and mutex-serialized forks
- Rust: IsolationMode enum (Fork/Direct) controlled by
TOWEROPS_SNMP_ISOLATION env var, defaults to Fork
- New CrashRecovered error variant with signal info and logging
- Device poller logs crash recovery events at error level
- Startup logs active isolation mode
Set TOWEROPS_SNMP_ISOLATION=direct to disable isolation for debugging.
The AMD64 builds were crashing due to cross-compilation. Blacksmith runners
are ARM64, so when building linux/amd64 on them, Docker uses QEMU emulation
which breaks the C FFI code in native/snmp_helper.c.
Solution:
- AMD64 builds now use GitHub's ubuntu-latest runners (native x86_64)
- ARM64 builds continue using Blacksmith ARM64 runners (native)
- Both architectures now build natively, eliminating FFI segfaults
The builder (rust:1.93-alpine) uses Alpine 3.23 but the runtime used
Alpine 3.19. The net-snmp-dev headers from 3.23 don't match
net-snmp-libs from 3.19, causing a segfault on the first SNMP call.
Also adds a SIGSEGV/SIGBUS/SIGABRT signal handler that prints a
diagnostic message instead of silently exiting with code 139.
Tests against unreachable hosts (192.0.2.1) were waiting 10s * 3 attempts
per request. With cfg(test) overrides (1s timeout, 0 retries), the full
test suite runs in ~3s instead of ~90s.
Verifies that ring is properly installed as the default crypto provider
and that all TLS-dependent operations work: rustls ClientConfig, reqwest
client, tokio-rustls connector, and WebSocket TLS with webpki roots.
These tests catch the CryptoProvider panic before it reaches production.
