towerops-agent/AUTO_UPDATE_SETUP.md

Automatic Updates Setup

Overview

The Towerops agent now supports automatic updates using Watchtower, eliminating the need for manual updates or self-update code in the agent.

How It Works

┌─────────────────┐
│  Push to main   │
└────────┬────────┘
         │
         ↓
┌─────────────────┐
│  GitLab CI/CD   │ ← Builds Docker image
│  builds image   │   Tags: main, latest, timestamp
└────────┬────────┘
         │
         ↓
┌─────────────────┐
│  Docker Hub     │ ← New image published
│  gmcintire/     │   Tag: main (updated)
│  towerops-agent │
└────────┬────────┘
         │
         ↓
┌─────────────────┐
│  Watchtower     │ ← Polls every 5 minutes
│  (on customer   │   Detects new 'main' tag
│   network)      │   Pulls new image
└────────┬────────┘   Restarts container
         │
         ↓
┌─────────────────┐
│  Agent updated  │ ← Zero downtime
│  automatically  │   New version running
└─────────────────┘

Versioning Strategy

Git-Based Versioning

The agent version is automatically generated from git:

Scenario	Version Format	Example
Exact tag	X.Y.Z	0.2.0
After tag	X.Y.Z.N.hash	0.2.0.5.831588e
No tags	X.Y.Z.hash	0.1.0.831588e
Dirty tree	X.Y.Z-modified	0.2.0-modified

Generated by: git describe --tags --always --dirty=-modified

Docker Image Tags

Every push to main creates multiple tags:

Tag	Purpose	Updated On
main	Auto-update tracking	Every push
latest	Stable/production	Every push
0.1.0.5.831588e	Specific version	Every push
main-831588e	Commit reference	Every push
main-20260116-143022	Timestamp	Every push

Release Tags

Git tags trigger multi-arch releases:

git tag v0.2.0
git push --tags
# CI builds: linux/amd64, linux/arm64
# Tags: 0.2.0, v0.2.0, latest

Setup Instructions

1. Docker Compose (Recommended)

Copy docker-compose.example.yml to docker-compose.yml:

services:
  towerops-agent:
    image: gmcintire/towerops-agent:main  # ← Track main for auto-updates
    # ... environment vars ...
    labels:
      - "com.centurylinklabs.watchtower.enable=true"

  watchtower:
    image: containrrr/watchtower:latest
    environment:
      - WATCHTOWER_POLL_INTERVAL=300  # Check every 5 minutes
      - WATCHTOWER_LABEL_ENABLE=true
      - WATCHTOWER_CLEANUP=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Start services:

docker-compose up -d

2. Docker Run (Manual)

Agent:

docker run -d \
  --name towerops-agent \
  --restart unless-stopped \
  --label com.centurylinklabs.watchtower.enable=true \
  -e TOWEROPS_API_URL=https://app.towerops.com \
  -e TOWEROPS_AGENT_TOKEN=your-token \
  -v $(pwd)/data:/data \
  gmcintire/towerops-agent:main

Watchtower:

docker run -d \
  --name watchtower \
  --restart unless-stopped \
  -e WATCHTOWER_POLL_INTERVAL=300 \
  -e WATCHTOWER_LABEL_ENABLE=true \
  -e WATCHTOWER_CLEANUP=true \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower

3. Kubernetes

Use a Kubernetes CronJob with a Docker image updater like:

• Keel (https://keel.sh/)
• Flux (https://fluxcd.io/)
• ArgoCD Image Updater (https://argocd-image-updater.readthedocs.io/)

Example with Keel:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: towerops-agent
  labels:
    keel.sh/policy: force
    keel.sh/trigger: poll
spec:
  template:
    spec:
      containers:
      - name: agent
        image: gmcintire/towerops-agent:main

Watchtower Configuration

Update Frequency

Default: Every 5 minutes (WATCHTOWER_POLL_INTERVAL=300)

Recommended settings:

• Aggressive: 300s (5 min) - Get updates quickly
• Balanced: 1800s (30 min) - Reduce Docker Hub API calls
• Conservative: 3600s (1 hour) - Minimize load

Notifications

Get notified when updates happen:

Slack

environment:
  - WATCHTOWER_NOTIFICATIONS=slack
  - WATCHTOWER_NOTIFICATION_SLACK_HOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
  - WATCHTOWER_NOTIFICATION_SLACK_IDENTIFIER=Towerops Agent

Email

environment:
  - WATCHTOWER_NOTIFICATIONS=email
  - WATCHTOWER_NOTIFICATION_EMAIL_FROM=watchtower@yourdomain.com
  - WATCHTOWER_NOTIFICATION_EMAIL_TO=alerts@yourdomain.com
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER=smtp.gmail.com
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT=587
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER=your-email@gmail.com
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=your-app-password

Discord, Telegram, etc.

See: https://containrrr.dev/watchtower/notifications/

Update Scheduling

Run updates at specific times (cron format):

environment:
  # Daily at 4am
  - WATCHTOWER_SCHEDULE=0 0 4 * * *

  # Every Sunday at 2am
  - WATCHTOWER_SCHEDULE=0 0 2 * * 0

Rolling Updates

Update only specific containers:

environment:
  - WATCHTOWER_SCOPE=towerops  # Only update containers with this scope label

# On agent:
labels:
  - "com.centurylinklabs.watchtower.scope=towerops"

Tag Selection Strategy

Production (Stable)

Use latest tag for stable releases only:

image: gmcintire/towerops-agent:latest

• Updated only on git tags (v0.2.0, v0.3.0, etc.)
• Tested releases
• Manual version control

Auto-Updates (Continuous)

Use main tag for automatic updates:

image: gmcintire/towerops-agent:main

• Updated on every push to main branch
• Latest features and bug fixes
• Automated deployment
• Recommended for most users

Pinned Version

Use specific version to prevent updates:

image: gmcintire/towerops-agent:0.2.0.5.831588e

• Never updates automatically
• Full control over upgrades
• Use when stability is critical

Monitoring Updates

Check Watchtower Logs

# Docker Compose
docker-compose logs -f watchtower

# Docker
docker logs -f watchtower

Check Agent Version

# View agent logs for version on startup
docker logs towerops-agent | grep "Current version"

# Output: Current version: 0.2.0.5.831588e

Manual Update Check

Force Watchtower to check immediately:

# Send SIGUSR1 to Watchtower
docker kill --signal=SIGUSR1 watchtower

Rollback

To Previous Version

Find previous image:

# List available tags
docker image ls gmcintire/towerops-agent

# Use timestamp tag for specific build
docker pull gmcintire/towerops-agent:main-20260116-143022

Update docker-compose.yml:

image: gmcintire/towerops-agent:main-20260116-143022

Restart:

docker-compose up -d --force-recreate towerops-agent

Disable Auto-Updates

Remove or comment out Watchtower:

# watchtower:
#   image: containrrr/watchtower:latest
#   ...

Or disable for specific container:

labels:
  - "com.centurylinklabs.watchtower.enable=false"

Comparison: Self-Update vs Watchtower

Feature	Self-Update (Old)	Watchtower (New)
Complexity	Agent pulls images	External service
Docker socket	Required in agent	Only in Watchtower
Update trigger	Agent checks hourly	Watchtower polls
Rollback	Exit code 0	Standard Docker
Notifications	None	Slack, email, etc.
Multi-container	One agent only	All containers
Scheduling	Fixed interval	Cron expressions
Security	Agent has Docker access	Isolated service

Winner: Watchtower - More flexible, secure, and featureful

Security Considerations

Docker Socket Access

Watchtower needs access to Docker socket (/var/run/docker.sock). This gives it full Docker API access.

Mitigation:

• Run Watchtower in separate namespace
• Use label filtering to limit scope
• Monitor Watchtower logs
• Use official Watchtower image only

Image Verification

Watchtower pulls images without signature verification by default.

Options:

• Use Docker Content Trust: export DOCKER_CONTENT_TRUST=1
• Verify image checksums manually
• Use private registry with access controls

Update Testing

Test updates in staging before production:

1. Staging: Use main tag, auto-update
2. Production: Use latest tag, manual update after staging validation

Troubleshooting

Watchtower Not Updating

Check:

1. Watchtower is running: docker ps | grep watchtower
2. Labels are correct: docker inspect towerops-agent | grep watchtower
3. Network access: docker exec watchtower ping -c 1 docker.io
4. Logs for errors: docker logs watchtower

Update Loop

If agent keeps restarting after update:

1. Check agent logs: docker logs towerops-agent
2. Pin to previous version (see Rollback)
3. Report issue on GitHub

High Resource Usage

Watchtower using too much memory:

deploy:
  resources:
    limits:
      memory: 64M  # Reduce from 128M

CI/CD Pipeline

Automatic Builds

On every push to main:

1. GitLab CI builds Docker image
2. Tags with multiple identifiers
3. Pushes to Docker Hub
4. Watchtower detects within 5 minutes
5. Agent updates automatically

View pipeline: https://gitlab.com/towerops/towerops-agent/-/pipelines

Build Time

• Main branch (amd64 only): ~5 minutes
• Git tags (multi-arch): ~20 minutes

Best Practices

1. Use main tag for automatic updates
2. Enable notifications to track updates
3. Set reasonable poll interval (300-1800s)
4. Monitor Watchtower logs regularly
5. Test in staging before production
6. Keep Watchtower updated (docker pull containrrr/watchtower:latest)
7. Use Docker Compose for easier management

Resources

• Watchtower Docs: https://containrrr.dev/watchtower/
• Docker Hub: https://hub.docker.com/r/gmcintire/towerops-agent
• GitLab CI: https://gitlab.com/towerops/towerops-agent/-/blob/main/.gitlab-ci.yml
• Issue Tracker: https://github.com/towerops/towerops-agent/issues

---

Status: ✅ Complete Date: January 16, 2026 Impact: Fully automated zero-touch updates for all deployments