Net-SNMP was crashing (signal 11) on every operation despite fork isolation.
The issue was that init_snmp() tries to load MIB files by default, which can
cause crashes if MIB files are missing, corrupted, or incompatible.
Since we only use numeric OIDs (e.g., "1.3.6.1.2.1.1.1.0"), we don't need
MIB name resolution at all. This change explicitly disables MIB loading in
the child process by:
1. Setting MIBS and MIBDIRS environment variables to empty
2. Calling netsnmp_set_mib_directory("") to disable MIB directory search
3. These must happen BEFORE calling init_snmp()
This should fix the persistent SIGSEGV crashes in SNMP operations.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Each SNMP GET/WALK now runs in a forked child process. If libnetsnmp
triggers a SIGSEGV or other fatal signal, only the child dies - the
parent agent logs the crash and continues operating for all other devices.
Key changes:
- C helper: snmp_get_isolated() and snmp_walk_isolated() using
fork+pipe pattern with 60s alarm watchdog and mutex-serialized forks
- Rust: IsolationMode enum (Fork/Direct) controlled by
TOWEROPS_SNMP_ISOLATION env var, defaults to Fork
- New CrashRecovered error variant with signal info and logging
- Device poller logs crash recovery events at error level
- Startup logs active isolation mode
Set TOWEROPS_SNMP_ISOLATION=direct to disable isolation for debugging.
The AMD64 builds were crashing due to cross-compilation. Blacksmith runners
are ARM64, so when building linux/amd64 on them, Docker uses QEMU emulation
which breaks the C FFI code in native/snmp_helper.c.
Solution:
- AMD64 builds now use GitHub's ubuntu-latest runners (native x86_64)
- ARM64 builds continue using Blacksmith ARM64 runners (native)
- Both architectures now build natively, eliminating FFI segfaults
The builder (rust:1.93-alpine) uses Alpine 3.23 but the runtime used
Alpine 3.19. The net-snmp-dev headers from 3.23 don't match
net-snmp-libs from 3.19, causing a segfault on the first SNMP call.
Also adds a SIGSEGV/SIGBUS/SIGABRT signal handler that prints a
diagnostic message instead of silently exiting with code 139.
Tests against unreachable hosts (192.0.2.1) were waiting 10s * 3 attempts
per request. With cfg(test) overrides (1s timeout, 0 retries), the full
test suite runs in ~3s instead of ~90s.
Verifies that ring is properly installed as the default crypto provider
and that all TLS-dependent operations work: rustls ClientConfig, reqwest
client, tokio-rustls connector, and WebSocket TLS with webpki roots.
These tests catch the CryptoProvider panic before it reaches production.
Both ring and aws-lc-rs were enabled transitively (webpki-roots enables
rustls default features), causing rustls to panic at runtime because it
couldn't auto-detect which crypto provider to use.
- Add rustls as direct dep with ring only, disable default features
- Switch russh from aws-lc-rs (default) to ring crypto backend
- Call ring::default_provider().install_default() at startup
- Add macOS build fix: detect Homebrew OpenSSL path for netsnmp's libcrypto
Since we now use command-line ping (setuid root) instead of surge-ping
library, the agent no longer requires CAP_NET_RAW capability.
Reverted documentation changes from previous commit that added the
capability requirement.
Switched from surge-ping Rust library to system ping command (iputils)
to avoid requiring CAP_NET_RAW capability for ICMP ping.
Changes:
- Use tokio::process::Command to execute /bin/ping or /bin/ping6
- Parse response time from ping output (time=X.XX ms)
- Removed surge-ping and rand dependencies from Cargo.toml
- Added tokio "process" feature
Benefits:
- No special capabilities required (ping binary has setuid root)
- Works in restricted container environments
- Already available in Docker image (iputils package)
- More portable across different container runtimes
The system ping command is already installed in the Docker image
and has setuid root permissions, allowing it to create raw sockets
without requiring the container to have CAP_NET_RAW.
Agent was crashing with segmentation fault (exit code 139) when
attempting ICMP ping health checks. The surge-ping library requires
CAP_NET_RAW capability to create raw sockets for ICMP.
Changes:
- Added cap_add: NET_RAW to docker-compose.example.yml
- Updated README docker-compose example
- Added troubleshooting section for exit code 139 crashes
Without this capability, the agent crashes immediately after
successful ping execution when the surge-ping library attempts
to clean up raw socket resources.
Handle server-initiated restart (exit for Docker restart) and
self-update (download binary, verify SHA256, replace via exec)
commands received over the WebSocket channel. Report architecture
in heartbeat so the server can select the correct binary.
- Add restart and update handlers in websocket_client.rs
- Add self_update() with download, checksum verification, and exec
- Add arch field to AgentHeartbeat protobuf
- Add reqwest and sha2 dependencies
- chown binary in Dockerfile for non-root self-update
- Extract standalone binaries from Docker images in CI
- Create GitHub Releases with binary assets on version tags
Agents deployed with the old Docker Hub image (gmcintire/towerops-agent)
auto-update via Watchtower. This pushes release builds to both GHCR and
Docker Hub so those agents continue receiving updates.
Drop ratatui, crossterm, hostname dependencies and the tui feature flag.
Delete src/tui/ directory, remove event bus plumbing from websocket client,
and simplify connect/init_logger to single implementations.
-1,979 lines of conditional compilation removed.
- Cache ~/.cargo/git for git-sourced dependencies
- Cache ~/.rustup to avoid re-downloading toolchain each run
- Move format check before cargo check (fail fast on formatting)
- Map "Failed to parse OID" C library errors to InvalidOid variant
- Walk to unreachable host may return empty results instead of error
on some Linux configurations
Implements health monitoring in the agent using ICMP ping, allowing
devices assigned to local (non-cloud) agents to be monitored.
Changes:
- Add surge-ping library for async ICMP ping operations
- Create ping.rs module with ping_device() function (5s timeout)
- Rename JobType::MONITOR to JobType::PING in protobuf
- Add monitoring_check channels for result communication
- Implement execute_ping_job() to handle PING jobs
- Update TUI to display ping result events
The agent now receives PING jobs from Phoenix, executes ICMP pings,
and sends MonitoringCheck results back for storage in the database.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
