add libcap and setcap cap_net_raw+p on /bin/ping so ICMP works
without container-level capabilities. add NET_RAW cap_add to
docker-compose.example.yml as fallback.
- Dockerfile: add BuildKit cache mounts for Go module and build caches,
pin both stages to Alpine 3.23
- CI: replace BUILDKIT_INLINE_CACHE with registry-based layer caching
(mode=max) for branch and release builds
- .dockerignore: exclude test files, Rust legacy, nix, and other
non-build artifacts from Docker context
- SNMP: lower MaxRepetitions from default 50 to 10 to prevent Mikrotik
routers from timing out during bulk walks
Job receipt, start, and completion messages were all at debug level,
making it impossible to tell if the agent was processing jobs without
setting LOG_LEVEL=debug.
musl libc has weak fork-safety guarantees in multi-threaded processes.
When Tokio's runtime is running and we fork() for SNMP operations, musl's
internal state can be inconsistent in the child, causing SIGSEGV on every
SNMP walk to Ubiquiti devices. glibc handles this via pthread_atfork.
- Dockerfile: Alpine → rust:1.93-bookworm build + debian:12-slim runtime
- CI: Remove ensure-netsnmp-base/manifest jobs, simplify build deps
- Delete Dockerfile.netsnmp and netsnmp-base.yml workflow
- Add NULL pointer guards in snmp_helper.c for fork-safety
- Add SNMP exception type handling (NoSuchObject/Instance/EndOfMibView)
- Add crash reproduction tests
The ensure-netsnmp-base jobs check if the pre-built image exists in
GHCR before any Docker builds start. First run compiles from source
on native runners (~5 min), subsequent runs skip with a ~5s check.
build-branch and build-release now depend on ensure-netsnmp-manifest.
Alpine 3.23 ships net-snmp 5.9.4 which has segfault bugs in varbind
construction and buffer overflows in octet string handling. Version
5.9.5.2 fixes these issues.
Adds a pre-built base image (Dockerfile.netsnmp) compiled from source
so the net-snmp build doesn't run on every agent deploy. The base
image is built separately via the netsnmp-base workflow and referenced
in the main Dockerfile.
In Fork mode the parent never calls snmp_init_library(), so
pthread_once remains untriggered. The child called init_snmp()
directly, then snmp_open_session() triggered pthread_once calling
init_snmp() a second time. This double-initialization of libnetsnmp
caused signal 11 in every child process.
Let snmp_open_session() handle initialization via pthread_once,
which correctly initializes exactly once and sets numeric OID format.
Net-SNMP was crashing (signal 11) on every operation despite fork isolation.
The issue was that init_snmp() tries to load MIB files by default, which can
cause crashes if MIB files are missing, corrupted, or incompatible.
Since we only use numeric OIDs (e.g., "1.3.6.1.2.1.1.1.0"), we don't need
MIB name resolution at all. This change explicitly disables MIB loading in
the child process by:
1. Setting MIBS and MIBDIRS environment variables to empty
2. Calling netsnmp_set_mib_directory("") to disable MIB directory search
3. These must happen BEFORE calling init_snmp()
This should fix the persistent SIGSEGV crashes in SNMP operations.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Each SNMP GET/WALK now runs in a forked child process. If libnetsnmp
triggers a SIGSEGV or other fatal signal, only the child dies - the
parent agent logs the crash and continues operating for all other devices.
Key changes:
- C helper: snmp_get_isolated() and snmp_walk_isolated() using
fork+pipe pattern with 60s alarm watchdog and mutex-serialized forks
- Rust: IsolationMode enum (Fork/Direct) controlled by
TOWEROPS_SNMP_ISOLATION env var, defaults to Fork
- New CrashRecovered error variant with signal info and logging
- Device poller logs crash recovery events at error level
- Startup logs active isolation mode
Set TOWEROPS_SNMP_ISOLATION=direct to disable isolation for debugging.
The AMD64 builds were crashing due to cross-compilation. Blacksmith runners
are ARM64, so when building linux/amd64 on them, Docker uses QEMU emulation
which breaks the C FFI code in native/snmp_helper.c.
Solution:
- AMD64 builds now use GitHub's ubuntu-latest runners (native x86_64)
- ARM64 builds continue using Blacksmith ARM64 runners (native)
- Both architectures now build natively, eliminating FFI segfaults
The builder (rust:1.93-alpine) uses Alpine 3.23 but the runtime used
Alpine 3.19. The net-snmp-dev headers from 3.23 don't match
net-snmp-libs from 3.19, causing a segfault on the first SNMP call.
Also adds a SIGSEGV/SIGBUS/SIGABRT signal handler that prints a
diagnostic message instead of silently exiting with code 139.
Tests against unreachable hosts (192.0.2.1) were waiting 10s * 3 attempts
per request. With cfg(test) overrides (1s timeout, 0 retries), the full
test suite runs in ~3s instead of ~90s.
Verifies that ring is properly installed as the default crypto provider
and that all TLS-dependent operations work: rustls ClientConfig, reqwest
client, tokio-rustls connector, and WebSocket TLS with webpki roots.
These tests catch the CryptoProvider panic before it reaches production.
Both ring and aws-lc-rs were enabled transitively (webpki-roots enables
rustls default features), causing rustls to panic at runtime because it
couldn't auto-detect which crypto provider to use.
- Add rustls as direct dep with ring only, disable default features
- Switch russh from aws-lc-rs (default) to ring crypto backend
- Call ring::default_provider().install_default() at startup
- Add macOS build fix: detect Homebrew OpenSSL path for netsnmp's libcrypto
Since we now use command-line ping (setuid root) instead of surge-ping
library, the agent no longer requires CAP_NET_RAW capability.
Reverted documentation changes from previous commit that added the
capability requirement.
Switched from surge-ping Rust library to system ping command (iputils)
to avoid requiring CAP_NET_RAW capability for ICMP ping.
Changes:
- Use tokio::process::Command to execute /bin/ping or /bin/ping6
- Parse response time from ping output (time=X.XX ms)
- Removed surge-ping and rand dependencies from Cargo.toml
- Added tokio "process" feature
Benefits:
- No special capabilities required (ping binary has setuid root)
- Works in restricted container environments
- Already available in Docker image (iputils package)
- More portable across different container runtimes
The system ping command is already installed in the Docker image
and has setuid root permissions, allowing it to create raw sockets
without requiring the container to have CAP_NET_RAW.
Agent was crashing with segmentation fault (exit code 139) when
attempting ICMP ping health checks. The surge-ping library requires
CAP_NET_RAW capability to create raw sockets for ICMP.
Changes:
- Added cap_add: NET_RAW to docker-compose.example.yml
- Updated README docker-compose example
- Added troubleshooting section for exit code 139 crashes
Without this capability, the agent crashes immediately after
successful ping execution when the surge-ping library attempts
to clean up raw socket resources.
Handle server-initiated restart (exit for Docker restart) and
self-update (download binary, verify SHA256, replace via exec)
commands received over the WebSocket channel. Report architecture
in heartbeat so the server can select the correct binary.
- Add restart and update handlers in websocket_client.rs
- Add self_update() with download, checksum verification, and exec
- Add arch field to AgentHeartbeat protobuf
- Add reqwest and sha2 dependencies
- chown binary in Dockerfile for non-root self-update
- Extract standalone binaries from Docker images in CI
- Create GitHub Releases with binary assets on version tags
