The agent version was always showing 0.1.0 because build.rs tried to
run 'git describe' inside the Docker build, but .git directory wasn't
copied into the image.
Changes:
- Add VERSION build arg to Dockerfile
- Pass VERSION as env var to cargo build commands
- Update GitLab CI to pass --build-arg VERSION
- Modify build.rs to prefer BUILD_VERSION env var over git commands
This ensures the version displayed by the agent matches the Docker
image tag it was built with.
When using SOCK_DGRAM for ICMP on Linux, the kernel manages the
identifier field and may overwrite the value we set. This causes
validation failures when we receive replies with kernel-assigned
identifiers.
The fix is to only validate:
- Type (0 = ECHO REPLY)
- Code (0)
- Sequence number (matches our random value)
The sequence number is sufficient for uniqueness given our polling
intervals (300+ seconds per device). This is the standard approach
for DGRAM ICMP sockets.
The agent uses WebSocket streaming instead of SQLite buffering,
so rusqlite is not needed. Removing it reduces compile time and
simplifies the dependency tree.
The diagnostic error message was reading ICMP fields from the wrong
offset when an IP header was present. It was showing bytes from the
IP header instead of the actual ICMP identifier and sequence fields.
Now properly detects and skips IP header before extracting ICMP
fields for diagnostic output. Also added total packet length to help
distinguish between raw ICMP and IP-wrapped packets.
Example before:
Invalid ICMP reply packet (expected id=24079, seq=12918):
type=0 code=0 id=29 seq=12918 len=21
(id=29 was reading from IP header, not ICMP)
Example after:
Invalid ICMP reply packet (expected id=24079, seq=12918):
type=0 code=0 id=24079 seq=12918 len=21 (total=41)
(now correctly shows ICMP identifier)
- Add MonitoringCheck message to protobuf definitions
- Add monitoring_enabled and check_interval_seconds to Device and SnmpDevice
- Implement continuous ICMP ping monitoring for devices
- Send monitoring check results to Phoenix via WebSocket
- Integrate existing ping module with agent client
- Spawn background tasks for devices with monitoring enabled
- Starts with 1 second delay, doubles each retry
- Caps at 60 seconds between attempts
- Resets delay counter on successful connection
- Prevents constant reconnection hammering
Documents:
- Semantic versioning workflow
- How to bump versions with script
- CI/CD pipeline behavior
- Version checking at startup and hourly
- Docker Hub tagging strategy
- Best practices for when to bump
- Troubleshooting guide
- Development tips
Features:
- Parse and compare semantic versions from Docker Hub
- Check if current version is outdated on startup
- Only pull updates when newer version is available
- GitLab CI now tags images with Cargo.toml version
- Created bump-version.sh script for easy version bumping
How it works:
1. Cargo.toml contains source of truth version (0.1.0)
2. GitLab CI extracts version and tags Docker images with it
3. Agent queries Docker Hub for all semver tags
4. Compares current version against latest available
5. Only pulls and restarts if newer version exists
Version bumping workflow:
./scripts/bump-version.sh patch # 0.1.0 -> 0.1.1
./scripts/bump-version.sh minor # 0.1.0 -> 0.2.0
./scripts/bump-version.sh major # 0.1.0 -> 1.0.0
This creates git commit and tag, ready to push.
- Detect host Docker socket GID at runtime
- Create docker group with matching GID
- Add towerops user to docker group
- Allows non-root container to access Docker socket for self-updates
The previous version checking didn't work because:
- GitLab CI only creates semver tags when pushing Git tags
- Most builds use SHA hash tags, not version tags
- Comparing versions was unreliable
New approach:
- Simply pulls latest tag every hour
- Checks docker pull output to see if image changed
- Only restarts if a new image was actually pulled
- Logs the last_updated timestamp from Docker Hub for visibility
Added comprehensive timestamp formatting to both agent logs and UI:
Rust Agent Logger:
- Added chrono dependency to Cargo.toml
- Updated SimpleLogger to include timestamps in log output
- Format: [2026-01-15 19:45:23.456] [LEVEL] message
- Shows full date, time, and milliseconds for precise log tracking
Phoenix UI Enhancements:
- Added format_datetime/1 - Full date/time with timezone
- Added format_date/1 - Short date format
- Added format_last_seen_with_date/1 - Relative time with full date
- Updated agent show page to display full timestamps in 'Last Seen' card
- Added comprehensive Timestamps section showing:
- Created date (inserted_at)
- Last Updated date (updated_at)
- Last Seen date (last_seen_at with heartbeat context)
- Last IP Address
- Updated agent index page to show full datetime alongside relative time
All timestamps now include both human-readable relative times ('5m ago')
and precise absolute dates for accurate record keeping and debugging.
Implemented two critical optimizations for handling large equipment counts:
1. **Concurrent polling limiter**: Added semaphore to limit concurrent
SNMP polling tasks to 100 at a time, preventing system overload when
polling 10,000+ devices simultaneously.
2. **Batched metrics flushing**: Increased batch size from 100 to 500
metrics and added loop to process up to 10,000 metrics per flush cycle
(20 batches × 500). Prevents metric backlog with high-volume polling.
Performance characteristics:
- 10,000 equipment with 5 sensors each = 50,000 metrics per poll cycle
- Flush cycle handles 10,000 metrics every 30 seconds
- Concurrent polling processes 100 devices at a time instead of unlimited
System resource usage remains bounded regardless of equipment count.
Replaced all .unwrap() calls on Mutex::lock() with proper error handling
using map_err to prevent panics when the mutex is poisoned.
This prevents silent crashes in production when any thread panics while
holding the storage mutex lock.
- Modified scheduler to poll equipment items concurrently using tokio::spawn
- Each equipment item now polls in its own async task
- Added Clone derives to Executor and SnmpClient to support parallel execution
- Sensors and interfaces within each equipment still poll in parallel via tokio::join!
- All tasks are awaited to ensure completion before returning
Performance improvement: Multiple devices can now be polled simultaneously
instead of sequentially, significantly reducing total polling time for
agents monitoring many devices.
Added /health HTTP endpoint on port 8080 that returns:
- Agent status and version
- Uptime in seconds
- Pending metrics count
- Last error (if any)
Implementation:
- Uses lightweight tiny_http server in background thread
- Non-blocking health checks
- Returns JSON for easy integration with monitoring tools
- Ready for Kubernetes liveness/readiness probes
Example response:
{
"status": "healthy",
"version": "0.1.0",
"uptime_seconds": 3600,
"config_last_fetch": "2026-01-14T23:00:00Z",
"metrics_pending": 0,
"last_error": null
}
The agent now checks for updates every hour and automatically updates
itself when a new version is available on Docker Hub.
Features:
- Periodic update checks (hourly via scheduler)
- Automatic Docker image pull when update available
- Graceful exit and restart with new version
- Non-blocking, runs in background task
- Requires Docker socket mount for self-update
Changes:
- Add UpdateInfo struct and get_update_info() function
- Add perform_self_update() to pull new image and restart
- Add update check ticker to scheduler (hourly)
- Include docker-cli in Dockerfile runtime stage
- Update docker-compose.example.yml with socket mount
- Update README and CLAUDE.md with auto-update docs
The agent will log update status:
- "Already running latest version" - no action
- "Performing self-update: X -> Y" - pulling new image
- "Exiting to allow restart with new version" - restarting
Requires:
- Docker socket mounted: /var/run/docker.sock:/var/run/docker.sock
- restart: unless-stopped in docker-compose (to restart after exit)
The agent now checks Docker Hub on startup to see if a newer version
is available and logs a warning if an update is detected.
Changes:
- Add version.rs module to query Docker Hub API
- Check for newer versions on startup (non-blocking)
- Log warning with docker pull command if update available
- Gracefully handle Docker Hub API failures
- Update CLAUDE.md with version checking documentation
The agent was failing with 'no TLS backend is configured' because
ureq was configured with 'default-features = false' which disables
TLS entirely. Removing this restriction allows ureq to use its default
TLS implementation (rustls on supported platforms).
This fixes HTTPS connections to the Towerops API.
The multi-architecture builds (amd64 + arm64) were taking 15-20+ minutes
due to QEMU emulation for ARM64 cross-compilation. This is too slow for
regular development iteration.
Changes:
- Main branch builds: amd64 only (much faster, ~3-5 minutes)
- Tagged releases: Still build multi-arch (amd64 + arm64)
- Added Docker layer caching to speed up subsequent builds
- Cache stored in registry for persistence across CI runs
The 'latest' tag will be amd64-only from main branch. Users needing
ARM64 should use a specific version tag (e.g., v0.1.0).
Fixed docker buildx errors in CI/CD pipeline:
1. Set DOCKER_TLS_CERTDIR to empty string to disable TLS
(buildx with docker-in-docker doesn't work well with TLS certs)
2. Added fallback to reuse existing builder if creation fails
(docker buildx create ... || docker buildx use ...)
This fixes the error:
'could not create a builder instance with TLS data loaded from environment'
The Docker container now handles data directory permissions automatically
without requiring manual user setup.
Changes:
- Added entrypoint.sh script that runs as root, fixes /data permissions,
then drops to non-root user (towerops) using su-exec
- Updated Dockerfile to install su-exec and use the entrypoint script
- Container starts as root but immediately drops privileges after fixing
permissions
The agent will now start successfully with just 'docker-compose up -d'
without users needing to run chown commands manually.
