Commit graph

64 commits

Author SHA1 Message Date
98d1bbbb21
structured logging 2026-01-31 15:40:42 -06:00
b8aecf0a55
more tests 2026-01-31 15:26:36 -06:00
4cd357777f
Add TRAP_ENABLED env var to conditionally enable trap listener 2026-01-31 14:39:24 -06:00
89adc3a6b3
Add SNMP trap listener for v1 and v2c traps
- Add trap.rs with BER/ASN.1 parser for SNMP trap PDUs
- Support both SNMPv1 and SNMPv2c trap formats
- Listen on configurable UDP port (default 162)
- Log received traps at INFO level
- Add LOG_LEVEL env var for log verbosity control
- Add TRAP_PORT env var and CLI flag
- Update docker-compose with trap port mapping
2026-01-31 14:14:56 -06:00
8ed2208271
poll each device simultaneously 2026-01-31 14:00:00 -06:00
bf1c6314cb
agent version tracking 2026-01-30 17:25:08 -06:00
c3e6e8c5b0
fix clippy error 2026-01-30 16:00:33 -06:00
380beaca1a
Agent overhaul with zero persistence 2026-01-30 15:50:20 -06:00
3793477e2f
relax ping failures 2026-01-29 12:12:01 -06:00
b8932f16b0
fix clippy issues 2026-01-29 12:02:20 -06:00
cee8d693da
make websocket client more robust 2026-01-29 11:55:01 -06:00
dbf66af5c0
support snmp v1 2026-01-29 11:28:45 -06:00
0548ca313d
include device id and ip in snmp error logs 2026-01-25 12:20:45 -06:00
cf08b0589b
cleanup 2026-01-24 12:43:24 -06:00
8ee809d21e
websocket improvements 2026-01-23 12:43:11 -06:00
9cb924fd77
connectivity improvements 2026-01-23 10:33:13 -06:00
ee7ce7fe00
feat: use system ping instead of raw ICMP sockets
- Replace raw ICMP socket implementation with system ping command
- Add regex-lite dependency for parsing ping output
- Add tokio process feature for async command execution
- Support macOS and Linux ping output formats
- Add iputils to Dockerfile for setuid-root ping
- Remove socket2 dependency (no longer needed)

This eliminates the need for CAP_NET_RAW capability in containers.
2026-01-21 13:11:24 -06:00
84b7519ad0
cargo clippy fix 2026-01-20 11:33:52 -06:00
6cea690e6b
format 2026-01-20 11:33:37 -06:00
71267870b0
refactor: remove log dependency, replace with custom macros
- Added custom logging macros (log_error!, log_warn!, log_info!, log_debug!)
- Implemented LogLevel enum and LOG_LEVEL static for runtime level control
- Exported macros with #[macro_export] for use across all modules
- Replaced all log:: calls in websocket_client.rs, health.rs, and version.rs
- Removed obsolete test code referencing old log crate
- Removed log dependency from Cargo.toml
- Fixed useless comparison warning in test
- Binary size: still 1.5M (no change - log was lightweight)
- Tests: 65 passing, 0 warnings

This completes the removal of 9 dependencies (thiserror, hostname, chrono,
base64, rand, ureq, tiny_http, anyhow, log). Dependency count: 19 → 10.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-19 17:31:12 -06:00
0daaf9ed68
refactor: start replacing log crate with custom macros (WIP)
Added custom logging macros to main.rs to prepare for removing the
log crate dependency:
- Added LogLevel enum and LOG_LEVEL static
- Created log_error!, log_warn!, log_info!, log_debug! macros
- Replaced log calls in main.rs with custom macros
- Made format_timestamp() public for use in macros

Still TODO:
- Update websocket_client.rs, health.rs, version.rs
- Remove log from Cargo.toml

This is work in progress toward removing more dependencies.
2026-01-19 17:23:59 -06:00
55fb691440
refactor: remove 4 more dependencies (rand, ureq, tiny_http, anyhow)
Removed dependencies:
- rand: Replaced with SystemTime-based random for ICMP ID/sequence
- ureq: Removed Docker Hub version checking (Watchtower handles updates)
- tiny_http: Replaced with raw TCP listener for /health endpoint
- anyhow: Replaced with std::error::Error trait

Benefits:
- Binary size: 2.3M → 1.5M (800KB reduction, 35% smaller!)
- Dependencies: 19 → 11 direct dependencies (-8 total)
- Tests: 80 → 67 (removed Docker Hub API tests)
- All tests passing

Custom implementations:
- ICMP random: Use nanosecond timestamp for ID/sequence
- Health endpoint: Simple HTTP/1.1 parser on raw TCP
- Error handling: type Result<T> = std::result::Result<T, Box<dyn Error>>

Total progress from initial state:
- Binary: 2.4M → 1.5M (37.5% reduction!)
- Dependencies removed: 8 (thiserror, hostname, chrono, base64, rand, ureq, tiny_http, anyhow)
2026-01-19 17:20:49 -06:00
d02193921d
Increase SNMP timeout from 5s to 30s to reduce timeout errors
Problem:
- Rust agent was experiencing frequent SNMP timeouts
- 5-second timeout is too aggressive for slow devices or congested networks
- Phoenix app uses 30-second timeout with better reliability

Solution:
- Increased SNMP_TIMEOUT_SECS constant from 5 to 30 seconds
- Applied to both get() and walk() operations
- Now matches Phoenix application timeout configuration

Benefits:
- Reduces false positive timeout errors
- Better handling of slow network devices
- Consistent timeout behavior across Elixir and Rust pollers
- Maintains reliability during network congestion

Note: SNMP timeout is per-request. Total polling time may still be
limited by Task::spawn_blocking timeout in poller executor (40s).

🤖 Generated with Claude Code
2026-01-19 17:17:45 -06:00
d132f03d8b
refactor: replace 4 dependencies with custom implementations
Removed dependencies and replaced with custom code:
- thiserror: Not used (only pulled by tokio-tungstenite)
- hostname: Replaced with /proc/sys/kernel/hostname reader
- chrono: Replaced with std::time::SystemTime + custom formatting
- base64: Replaced with custom base64 encoder/decoder

Benefits:
- Binary size: 2.4M → 2.3M (100KB reduction)
- Reduced dependency count: 19 → 15 direct dependencies
- Added 7 new tests (73 → 80 total tests)
- All tests pass

Custom implementations:
- format_timestamp(): YYYY-MM-DD HH:MM:SS.mmm format
- get_hostname(): Read from /proc/sys/kernel/hostname
- get_uptime_seconds(): Read from /proc/uptime
- base64_encode/decode(): Standard base64 encoding

No functionality changes, all existing features work identically.
2026-01-19 17:15:07 -06:00
da3088f87f
fix: relax ICMP identifier validation for DGRAM sockets
When using SOCK_DGRAM for ICMP on Linux, the kernel manages the
identifier field and may overwrite the value we set. This causes
validation failures when we receive replies with kernel-assigned
identifiers.

The fix is to only validate:
- Type (0 = ECHO REPLY)
- Code (0)
- Sequence number (matches our random value)

The sequence number is sufficient for uniqueness given our polling
intervals (300+ seconds per device). This is the standard approach
for DGRAM ICMP sockets.
2026-01-19 15:27:51 -06:00
2b5a67d26c
fix: correct ICMP diagnostic message when IP header present
The diagnostic error message was reading ICMP fields from the wrong
offset when an IP header was present. It was showing bytes from the
IP header instead of the actual ICMP identifier and sequence fields.

Now properly detects and skips IP header before extracting ICMP
fields for diagnostic output. Also added total packet length to help
distinguish between raw ICMP and IP-wrapped packets.

Example before:
  Invalid ICMP reply packet (expected id=24079, seq=12918):
  type=0 code=0 id=29 seq=12918 len=21
  (id=29 was reading from IP header, not ICMP)

Example after:
  Invalid ICMP reply packet (expected id=24079, seq=12918):
  type=0 code=0 id=24079 seq=12918 len=21 (total=41)
  (now correctly shows ICMP identifier)
2026-01-19 15:20:10 -06:00
0b3cc9121e
Add comprehensive unit test coverage (29.19%)
Add 73 unit tests covering all testable business logic:

- src/snmp/types.rs: 100% coverage (17/17 lines)
  * SnmpError display formatting
  * SnmpValue conversions (as_i64, as_f64)

- src/snmp/client.rs: 34.4% coverage (32/93 lines)
  * OID parsing/formatting/validation
  * SNMP value conversion for all types
  * Error mapping from snmp crate
  * Helper functions (starts_with, format_oid)

- src/ping.rs: 65.2% coverage (58/89 lines)
  * ICMP checksum calculation and verification
  * Echo request packet building
  * Reply packet parsing (raw and IP-wrapped)
  * IP header length extraction (IHL field)
  * Error handling for invalid packets

- src/version.rs: 54.0% coverage (27/50 lines)
  * Version parsing with optional 'v' prefix
  * Version comparison and sorting
  * Docker Hub response deserialization
  * Latest version extraction from tags

- src/websocket_client.rs: 7.7% coverage (17/221 lines)
  * SnmpValue to string conversion
  * Agent ID generation
  * Phoenix message serialization/deserialization
  * Helper functions (get_uptime_seconds, get_local_ip)

- src/main.rs: 20.0% coverage (11/55 lines)
  * SimpleLogger enabled() logic
  * HTTP/HTTPS to WebSocket URL conversion

- .gitlab-ci.yml: Add 'cargo test' to CI pipeline

Uncovered code requires integration testing:
- Network I/O (WebSocket, HTTP, Docker Hub API)
- System privileges (raw ICMP sockets)
- External services (SNMP devices, WebSocket servers)
- Runtime initialization (tokio main, logger setup)

All 73 tests pass. No test failures.
2026-01-19 15:07:00 -06:00
aa32be93be
fix: remove unused code to pass clippy CI checks 2026-01-19 13:42:00 -06:00
4269591803
Add ICMP monitoring support via WebSocket
- Add MonitoringCheck message to protobuf definitions
- Add monitoring_enabled and check_interval_seconds to Device and SnmpDevice
- Implement continuous ICMP ping monitoring for devices
- Send monitoring check results to Phoenix via WebSocket
- Integrate existing ping module with agent client
- Spawn background tasks for devices with monitoring enabled
2026-01-19 13:38:37 -06:00
f8fdcacd39
add raw ICMP ping module with socket2 and tests 2026-01-18 10:58:53 -06:00
a514cef8d0
update to new schema 2026-01-17 15:26:55 -06:00
02ea81cae3
Fix double slash in WebSocket URL by stripping trailing slash from base URL 2026-01-16 20:08:13 -06:00
451b530641
Add exponential backoff retry logic for WebSocket reconnection
- Starts with 1 second delay, doubles each retry
- Caps at 60 seconds between attempts
- Resets delay counter on successful connection
- Prevents constant reconnection hammering
2026-01-16 18:28:42 -06:00
6223638acf
Move token from URL to channel join payload 2026-01-16 18:23:37 -06:00
5f39f748a9
Fix WebSocket URL: add /websocket suffix for Phoenix Channels 2026-01-16 18:16:15 -06:00
cc0e250a88
Add Phoenix channel join handshake after WebSocket connection 2026-01-16 18:15:31 -06:00
13e45febce
Add detailed WebSocket connection error logging 2026-01-16 18:14:43 -06:00
b57c6ee3ec
format 2026-01-16 18:06:01 -06:00
812ee08ac5
Fix clippy warnings to pass CI build
- Remove unused SnmpError import
- Replace deprecated from_i32 with TryFrom
- Add #[allow(dead_code)] to unused SnmpValue methods
- Remove unused perform_self_update function
- Remove unused token field from AgentClient
- Remove unused send_error method
- Add #[allow(dead_code)] to protobuf generated module
2026-01-16 18:02:33 -06:00
967d317b69
Complete WebSocket migration for agent communication
Major architectural change from REST API polling to WebSocket-based
bidirectional communication:

**What Changed:**
- Agent now uses persistent WebSocket connection instead of REST API
- Server pushes SNMP query jobs to agent via Phoenix Channels
- Agent executes raw SNMP queries and returns results
- Removed complex polling/scheduling/buffering architecture

**New Files:**
- src/websocket_client.rs - WebSocket client with SNMP job execution
- Extended proto/agent.proto with WebSocket message types

**Modified Files:**
- src/main.rs - Simplified to connect and run WebSocket client
- src/health.rs - Simplified health endpoint (no storage needed)
- src/snmp/mod.rs - Export SnmpValue for WebSocket client

**Removed Files:**
- src/api_client.rs - Old REST API client
- src/config.rs - Old config types
- src/buffer/ - SQLite buffering (no longer needed)
- src/metrics/ - Old metric types
- src/poller/ - Polling/scheduling logic
- src/snmp/neighbor.rs - High-level neighbor discovery

**Dependencies:**
- Switched from native-tls to rustls for WebSocket TLS
- Uses tokio-tungstenite for WebSocket communication
- Protobuf for efficient binary message encoding

**Benefits:**
- Simpler agent architecture (~500 lines vs 5000+)
- Real-time job execution (<1s vs 60s polling)
- No duplicate SNMP profile logic
- No local storage/buffering complexity
- 68% smaller message payloads (protobuf vs JSON)
2026-01-16 17:57:41 -06:00
efa8404ea6
rewrite with much simpler runtime 2026-01-16 17:27:10 -06:00
097c4bd581
Add automatic semver versioning for agent
Features:
- Parse and compare semantic versions from Docker Hub
- Check if current version is outdated on startup
- Only pull updates when newer version is available
- GitLab CI now tags images with Cargo.toml version
- Created bump-version.sh script for easy version bumping

How it works:
1. Cargo.toml contains source of truth version (0.1.0)
2. GitLab CI extracts version and tags Docker images with it
3. Agent queries Docker Hub for all semver tags
4. Compares current version against latest available
5. Only pulls and restarts if newer version exists

Version bumping workflow:
  ./scripts/bump-version.sh patch  # 0.1.0 -> 0.1.1
  ./scripts/bump-version.sh minor  # 0.1.0 -> 0.2.0
  ./scripts/bump-version.sh major  # 0.1.0 -> 1.0.0

This creates git commit and tag, ready to push.
2026-01-15 12:52:36 -06:00
38c3451266
Simplify auto-update logic to always pull latest tag
The previous version checking didn't work because:
- GitLab CI only creates semver tags when pushing Git tags
- Most builds use SHA hash tags, not version tags
- Comparing versions was unreliable

New approach:
- Simply pulls latest tag every hour
- Checks docker pull output to see if image changed
- Only restarts if a new image was actually pulled
- Logs the last_updated timestamp from Docker Hub for visibility
2026-01-15 08:39:22 -06:00
d00e3782c3
Remove unused health server methods
- Removed update_config_fetch_time() and record_error()
- These methods were not integrated with the scheduler
- Fixes cargo clippy dead_code warnings
2026-01-15 08:06:29 -06:00
e01a71be39
Add timestamps with dates to agent logs
Added comprehensive timestamp formatting to both agent logs and UI:

Rust Agent Logger:
- Added chrono dependency to Cargo.toml
- Updated SimpleLogger to include timestamps in log output
- Format: [2026-01-15 19:45:23.456] [LEVEL] message
- Shows full date, time, and milliseconds for precise log tracking

Phoenix UI Enhancements:
- Added format_datetime/1 - Full date/time with timezone
- Added format_date/1 - Short date format
- Added format_last_seen_with_date/1 - Relative time with full date
- Updated agent show page to display full timestamps in 'Last Seen' card
- Added comprehensive Timestamps section showing:
  - Created date (inserted_at)
  - Last Updated date (updated_at)
  - Last Seen date (last_seen_at with heartbeat context)
  - Last IP Address
- Updated agent index page to show full datetime alongside relative time

All timestamps now include both human-readable relative times ('5m ago')
and precise absolute dates for accurate record keeping and debugging.
2026-01-15 08:01:45 -06:00
ec50b7faa5
Fix semaphore error handling and parallelize interface polling
1. Fixed semaphore acquire to handle errors gracefully instead of panicking
   when permit acquisition fails during concurrent polling

2. Parallelized interface counter polling using tokio::join! to fetch all
   6 SNMP counters (in/out octets/errors/discards) concurrently instead of
   sequentially

Performance improvement: Reduces per-interface polling latency from ~30ms
(6 × 5ms timeout) to ~5ms (1 parallel batch).
2026-01-14 19:04:50 -06:00
6462bdbf81
Add scalability improvements for 10,000+ equipment
Implemented two critical optimizations for handling large equipment counts:

1. **Concurrent polling limiter**: Added semaphore to limit concurrent
   SNMP polling tasks to 100 at a time, preventing system overload when
   polling 10,000+ devices simultaneously.

2. **Batched metrics flushing**: Increased batch size from 100 to 500
   metrics and added loop to process up to 10,000 metrics per flush cycle
   (20 batches × 500). Prevents metric backlog with high-volume polling.

Performance characteristics:
- 10,000 equipment with 5 sensors each = 50,000 metrics per poll cycle
- Flush cycle handles 10,000 metrics every 30 seconds
- Concurrent polling processes 100 devices at a time instead of unlimited

System resource usage remains bounded regardless of equipment count.
2026-01-14 19:01:04 -06:00
b9db4133be
Fix Mutex unwrap calls to handle poisoned mutex errors
Replaced all .unwrap() calls on Mutex::lock() with proper error handling
using map_err to prevent panics when the mutex is poisoned.

This prevents silent crashes in production when any thread panics while
holding the storage mutex lock.
2026-01-14 18:58:27 -06:00
656992221a
Implement parallel SNMP polling for better performance
- Modified scheduler to poll equipment items concurrently using tokio::spawn
- Each equipment item now polls in its own async task
- Added Clone derives to Executor and SnmpClient to support parallel execution
- Sensors and interfaces within each equipment still poll in parallel via tokio::join!
- All tasks are awaited to ensure completion before returning

Performance improvement: Multiple devices can now be polled simultaneously
instead of sequentially, significantly reducing total polling time for
agents monitoring many devices.
2026-01-14 18:51:47 -06:00
f7ac5f48e8
Add health endpoint for agent monitoring
Added /health HTTP endpoint on port 8080 that returns:
- Agent status and version
- Uptime in seconds
- Pending metrics count
- Last error (if any)

Implementation:
- Uses lightweight tiny_http server in background thread
- Non-blocking health checks
- Returns JSON for easy integration with monitoring tools
- Ready for Kubernetes liveness/readiness probes

Example response:
{
  "status": "healthy",
  "version": "0.1.0",
  "uptime_seconds": 3600,
  "config_last_fetch": "2026-01-14T23:00:00Z",
  "metrics_pending": 0,
  "last_error": null
}
2026-01-14 18:46:25 -06:00