Documents:
- Semantic versioning workflow
- How to bump versions with script
- CI/CD pipeline behavior
- Version checking at startup and hourly
- Docker Hub tagging strategy
- Best practices for when to bump
- Troubleshooting guide
- Development tips
Features:
- Parse and compare semantic versions from Docker Hub
- Check if current version is outdated on startup
- Only pull updates when newer version is available
- GitLab CI now tags images with Cargo.toml version
- Created bump-version.sh script for easy version bumping
How it works:
1. Cargo.toml contains source of truth version (0.1.0)
2. GitLab CI extracts version and tags Docker images with it
3. Agent queries Docker Hub for all semver tags
4. Compares current version against latest available
5. Only pulls and restarts if newer version exists
Version bumping workflow:
./scripts/bump-version.sh patch # 0.1.0 -> 0.1.1
./scripts/bump-version.sh minor # 0.1.0 -> 0.2.0
./scripts/bump-version.sh major # 0.1.0 -> 1.0.0
This creates git commit and tag, ready to push.
- Detect host Docker socket GID at runtime
- Create docker group with matching GID
- Add towerops user to docker group
- Allows non-root container to access Docker socket for self-updates
The previous version checking didn't work because:
- GitLab CI only creates semver tags when pushing Git tags
- Most builds use SHA hash tags, not version tags
- Comparing versions was unreliable
New approach:
- Simply pulls latest tag every hour
- Checks docker pull output to see if image changed
- Only restarts if a new image was actually pulled
- Logs the last_updated timestamp from Docker Hub for visibility
Added comprehensive timestamp formatting to both agent logs and UI:
Rust Agent Logger:
- Added chrono dependency to Cargo.toml
- Updated SimpleLogger to include timestamps in log output
- Format: [2026-01-15 19:45:23.456] [LEVEL] message
- Shows full date, time, and milliseconds for precise log tracking
Phoenix UI Enhancements:
- Added format_datetime/1 - Full date/time with timezone
- Added format_date/1 - Short date format
- Added format_last_seen_with_date/1 - Relative time with full date
- Updated agent show page to display full timestamps in 'Last Seen' card
- Added comprehensive Timestamps section showing:
- Created date (inserted_at)
- Last Updated date (updated_at)
- Last Seen date (last_seen_at with heartbeat context)
- Last IP Address
- Updated agent index page to show full datetime alongside relative time
All timestamps now include both human-readable relative times ('5m ago')
and precise absolute dates for accurate record keeping and debugging.
Implemented two critical optimizations for handling large equipment counts:
1. **Concurrent polling limiter**: Added semaphore to limit concurrent
SNMP polling tasks to 100 at a time, preventing system overload when
polling 10,000+ devices simultaneously.
2. **Batched metrics flushing**: Increased batch size from 100 to 500
metrics and added loop to process up to 10,000 metrics per flush cycle
(20 batches × 500). Prevents metric backlog with high-volume polling.
Performance characteristics:
- 10,000 equipment with 5 sensors each = 50,000 metrics per poll cycle
- Flush cycle handles 10,000 metrics every 30 seconds
- Concurrent polling processes 100 devices at a time instead of unlimited
System resource usage remains bounded regardless of equipment count.
Replaced all .unwrap() calls on Mutex::lock() with proper error handling
using map_err to prevent panics when the mutex is poisoned.
This prevents silent crashes in production when any thread panics while
holding the storage mutex lock.
- Modified scheduler to poll equipment items concurrently using tokio::spawn
- Each equipment item now polls in its own async task
- Added Clone derives to Executor and SnmpClient to support parallel execution
- Sensors and interfaces within each equipment still poll in parallel via tokio::join!
- All tasks are awaited to ensure completion before returning
Performance improvement: Multiple devices can now be polled simultaneously
instead of sequentially, significantly reducing total polling time for
agents monitoring many devices.
Added /health HTTP endpoint on port 8080 that returns:
- Agent status and version
- Uptime in seconds
- Pending metrics count
- Last error (if any)
Implementation:
- Uses lightweight tiny_http server in background thread
- Non-blocking health checks
- Returns JSON for easy integration with monitoring tools
- Ready for Kubernetes liveness/readiness probes
Example response:
{
"status": "healthy",
"version": "0.1.0",
"uptime_seconds": 3600,
"config_last_fetch": "2026-01-14T23:00:00Z",
"metrics_pending": 0,
"last_error": null
}
The agent now checks for updates every hour and automatically updates
itself when a new version is available on Docker Hub.
Features:
- Periodic update checks (hourly via scheduler)
- Automatic Docker image pull when update available
- Graceful exit and restart with new version
- Non-blocking, runs in background task
- Requires Docker socket mount for self-update
Changes:
- Add UpdateInfo struct and get_update_info() function
- Add perform_self_update() to pull new image and restart
- Add update check ticker to scheduler (hourly)
- Include docker-cli in Dockerfile runtime stage
- Update docker-compose.example.yml with socket mount
- Update README and CLAUDE.md with auto-update docs
The agent will log update status:
- "Already running latest version" - no action
- "Performing self-update: X -> Y" - pulling new image
- "Exiting to allow restart with new version" - restarting
Requires:
- Docker socket mounted: /var/run/docker.sock:/var/run/docker.sock
- restart: unless-stopped in docker-compose (to restart after exit)
The agent now checks Docker Hub on startup to see if a newer version
is available and logs a warning if an update is detected.
Changes:
- Add version.rs module to query Docker Hub API
- Check for newer versions on startup (non-blocking)
- Log warning with docker pull command if update available
- Gracefully handle Docker Hub API failures
- Update CLAUDE.md with version checking documentation
The agent was failing with 'no TLS backend is configured' because
ureq was configured with 'default-features = false' which disables
TLS entirely. Removing this restriction allows ureq to use its default
TLS implementation (rustls on supported platforms).
This fixes HTTPS connections to the Towerops API.
The multi-architecture builds (amd64 + arm64) were taking 15-20+ minutes
due to QEMU emulation for ARM64 cross-compilation. This is too slow for
regular development iteration.
Changes:
- Main branch builds: amd64 only (much faster, ~3-5 minutes)
- Tagged releases: Still build multi-arch (amd64 + arm64)
- Added Docker layer caching to speed up subsequent builds
- Cache stored in registry for persistence across CI runs
The 'latest' tag will be amd64-only from main branch. Users needing
ARM64 should use a specific version tag (e.g., v0.1.0).
Fixed docker buildx errors in CI/CD pipeline:
1. Set DOCKER_TLS_CERTDIR to empty string to disable TLS
(buildx with docker-in-docker doesn't work well with TLS certs)
2. Added fallback to reuse existing builder if creation fails
(docker buildx create ... || docker buildx use ...)
This fixes the error:
'could not create a builder instance with TLS data loaded from environment'
The Docker container now handles data directory permissions automatically
without requiring manual user setup.
Changes:
- Added entrypoint.sh script that runs as root, fixes /data permissions,
then drops to non-root user (towerops) using su-exec
- Updated Dockerfile to install su-exec and use the entrypoint script
- Container starts as root but immediately drops privileges after fixing
permissions
The agent will now start successfully with just 'docker-compose up -d'
without users needing to run chown commands manually.
- Document Protocol Buffers integration in its own section
- Update API client description to mention protobuf usage
- Update build status to show only 1 warning (down from 7)
- Add note about GitLab CI/CD Docker Hub deployment
- Update fetch_config to use protobuf instead of JSON
- Add conversion functions from protobuf types to internal config types
- Update heartbeat to use protobuf encoding
- All API endpoints now use Protocol Buffers (config, metrics, heartbeat)
- Reduces 7 unused struct warnings to 1 (HeartbeatResponse)
- Change registry from GitLab to Docker Hub (gmcintire/towerops-agent)
- Update authentication to use DOCKERHUB_USERNAME and DOCKERHUB_TOKEN
- Remove vntx runner tags to use GitLab shared runners
