towerops-agent/AUTO_UPDATE_SETUP.md

# Automatic Updates Setup

## Overview

The Towerops agent now supports **automatic updates** using **Watchtower**, eliminating the need for manual updates or self-update code in the agent.

## How It Works

```
┌─────────────────┐
│  Push to main   │
└────────┬────────┘
         │
         ↓
┌─────────────────┐
│  GitLab CI/CD   │ ← Builds Docker image
│  builds image   │   Tags: main, latest, timestamp
└────────┬────────┘
         │
         ↓
┌─────────────────┐
│  Docker Hub     │ ← New image published
│  gmcintire/     │   Tag: main (updated)
│  towerops-agent │
└────────┬────────┘
         │
         ↓
┌─────────────────┐
│  Watchtower     │ ← Polls every 5 minutes
│  (on customer   │   Detects new 'main' tag
│   network)      │   Pulls new image
└────────┬────────┘   Restarts container
         │
         ↓
┌─────────────────┐
│  Agent updated  │ ← Zero downtime
│  automatically  │   New version running
└─────────────────┘
```

## Versioning Strategy

### Git-Based Versioning

The agent version is **automatically generated** from git:

| Scenario | Version Format | Example |
|----------|----------------|---------|
| Exact tag | `X.Y.Z` | `0.2.0` |
| After tag | `X.Y.Z.N.hash` | `0.2.0.5.831588e` |
| No tags | `X.Y.Z.hash` | `0.1.0.831588e` |
| Dirty tree | `X.Y.Z-modified` | `0.2.0-modified` |

**Generated by**: `git describe --tags --always --dirty=-modified`

### Docker Image Tags

Every push to `main` creates **multiple tags**:

| Tag | Purpose | Updated On |
|-----|---------|------------|
| `main` | **Auto-update tracking** | Every push |
| `latest` | Stable/production | Every push |
| `0.1.0.5.831588e` | Specific version | Every push |
| `main-831588e` | Commit reference | Every push |
| `main-20260116-143022` | Timestamp | Every push |

### Release Tags

Git tags trigger **multi-arch releases**:

```bash
git tag v0.2.0
git push --tags
# CI builds: linux/amd64, linux/arm64
# Tags: 0.2.0, v0.2.0, latest
```

## Setup Instructions

### 1. Docker Compose (Recommended)

Copy `docker-compose.example.yml` to `docker-compose.yml`:

```yaml
services:
  towerops-agent:
    image: gmcintire/towerops-agent:main  # ← Track main for auto-updates
    # ... environment vars ...
    labels:
      - "com.centurylinklabs.watchtower.enable=true"

  watchtower:
    image: containrrr/watchtower:latest
    environment:
      - WATCHTOWER_POLL_INTERVAL=300  # Check every 5 minutes
      - WATCHTOWER_LABEL_ENABLE=true
      - WATCHTOWER_CLEANUP=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
```

Start services:

```bash
docker-compose up -d
```

### 2. Docker Run (Manual)

**Agent**:
```bash
docker run -d \
  --name towerops-agent \
  --restart unless-stopped \
  --label com.centurylinklabs.watchtower.enable=true \
  -e TOWEROPS_API_URL=https://app.towerops.com \
  -e TOWEROPS_AGENT_TOKEN=your-token \
  -v $(pwd)/data:/data \
  gmcintire/towerops-agent:main
```

**Watchtower**:
```bash
docker run -d \
  --name watchtower \
  --restart unless-stopped \
  -e WATCHTOWER_POLL_INTERVAL=300 \
  -e WATCHTOWER_LABEL_ENABLE=true \
  -e WATCHTOWER_CLEANUP=true \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower
```

### 3. Kubernetes

Use a Kubernetes CronJob with a Docker image updater like:
- **Keel** (https://keel.sh/)
- **Flux** (https://fluxcd.io/)
- **ArgoCD Image Updater** (https://argocd-image-updater.readthedocs.io/)

Example with Keel:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: towerops-agent
  labels:
    keel.sh/policy: force
    keel.sh/trigger: poll
spec:
  template:
    spec:
      containers:
      - name: agent
        image: gmcintire/towerops-agent:main
```

## Watchtower Configuration

### Update Frequency

**Default**: Every 5 minutes (`WATCHTOWER_POLL_INTERVAL=300`)

**Recommended settings**:
- **Aggressive**: 300s (5 min) - Get updates quickly
- **Balanced**: 1800s (30 min) - Reduce Docker Hub API calls
- **Conservative**: 3600s (1 hour) - Minimize load

### Notifications

Get notified when updates happen:

#### Slack
```yaml
environment:
  - WATCHTOWER_NOTIFICATIONS=slack
  - WATCHTOWER_NOTIFICATION_SLACK_HOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
  - WATCHTOWER_NOTIFICATION_SLACK_IDENTIFIER=Towerops Agent
```

#### Email
```yaml
environment:
  - WATCHTOWER_NOTIFICATIONS=email
  - WATCHTOWER_NOTIFICATION_EMAIL_FROM=watchtower@yourdomain.com
  - WATCHTOWER_NOTIFICATION_EMAIL_TO=alerts@yourdomain.com
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER=smtp.gmail.com
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT=587
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER=your-email@gmail.com
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=your-app-password
```

#### Discord, Telegram, etc.
See: https://containrrr.dev/watchtower/notifications/

### Update Scheduling

Run updates at specific times (cron format):

```yaml
environment:
  # Daily at 4am
  - WATCHTOWER_SCHEDULE=0 0 4 * * *

  # Every Sunday at 2am
  - WATCHTOWER_SCHEDULE=0 0 2 * * 0
```

### Rolling Updates

Update only specific containers:

```yaml
environment:
  - WATCHTOWER_SCOPE=towerops  # Only update containers with this scope label

# On agent:
labels:
  - "com.centurylinklabs.watchtower.scope=towerops"
```

## Tag Selection Strategy

### Production (Stable)

Use **`latest`** tag for stable releases only:

```yaml
image: gmcintire/towerops-agent:latest
```

- Updated only on git tags (`v0.2.0`, `v0.3.0`, etc.)
- Tested releases
- Manual version control

### Auto-Updates (Continuous)

Use **`main`** tag for automatic updates:

```yaml
image: gmcintire/towerops-agent:main
```

- Updated on every push to main branch
- Latest features and bug fixes
- Automated deployment
- **Recommended for most users**

### Pinned Version

Use **specific version** to prevent updates:

```yaml
image: gmcintire/towerops-agent:0.2.0.5.831588e
```

- Never updates automatically
- Full control over upgrades
- Use when stability is critical

## Monitoring Updates

### Check Watchtower Logs

```bash
# Docker Compose
docker-compose logs -f watchtower

# Docker
docker logs -f watchtower
```

### Check Agent Version

```bash
# View agent logs for version on startup
docker logs towerops-agent | grep "Current version"

# Output: Current version: 0.2.0.5.831588e
```

### Manual Update Check

Force Watchtower to check immediately:

```bash
# Send SIGUSR1 to Watchtower
docker kill --signal=SIGUSR1 watchtower
```

## Rollback

### To Previous Version

Find previous image:

```bash
# List available tags
docker image ls gmcintire/towerops-agent

# Use timestamp tag for specific build
docker pull gmcintire/towerops-agent:main-20260116-143022
```

Update docker-compose.yml:

```yaml
image: gmcintire/towerops-agent:main-20260116-143022
```

Restart:

```bash
docker-compose up -d --force-recreate towerops-agent
```

### Disable Auto-Updates

Remove or comment out Watchtower:

```yaml
# watchtower:
#   image: containrrr/watchtower:latest
#   ...
```

Or disable for specific container:

```yaml
labels:
  - "com.centurylinklabs.watchtower.enable=false"
```

## Comparison: Self-Update vs Watchtower

| Feature | Self-Update (Old) | Watchtower (New) |
|---------|-------------------|------------------|
| **Complexity** | Agent pulls images | External service |
| **Docker socket** | Required in agent | Only in Watchtower |
| **Update trigger** | Agent checks hourly | Watchtower polls |
| **Rollback** | Exit code 0 | Standard Docker |
| **Notifications** | None | Slack, email, etc. |
| **Multi-container** | One agent only | All containers |
| **Scheduling** | Fixed interval | Cron expressions |
| **Security** | Agent has Docker access | Isolated service |

**Winner**: Watchtower - More flexible, secure, and featureful

## Security Considerations

### Docker Socket Access

Watchtower needs access to Docker socket (`/var/run/docker.sock`). This gives it full Docker API access.

**Mitigation**:
- Run Watchtower in separate namespace
- Use label filtering to limit scope
- Monitor Watchtower logs
- Use official Watchtower image only

### Image Verification

Watchtower pulls images without signature verification by default.

**Options**:
- Use Docker Content Trust: `export DOCKER_CONTENT_TRUST=1`
- Verify image checksums manually
- Use private registry with access controls

### Update Testing

Test updates in staging before production:

1. **Staging**: Use `main` tag, auto-update
2. **Production**: Use `latest` tag, manual update after staging validation

## Troubleshooting

### Watchtower Not Updating

**Check**:
1. Watchtower is running: `docker ps | grep watchtower`
2. Labels are correct: `docker inspect towerops-agent | grep watchtower`
3. Network access: `docker exec watchtower ping -c 1 docker.io`
4. Logs for errors: `docker logs watchtower`

### Update Loop

If agent keeps restarting after update:

1. Check agent logs: `docker logs towerops-agent`
2. Pin to previous version (see Rollback)
3. Report issue on GitHub

### High Resource Usage

Watchtower using too much memory:

```yaml
deploy:
  resources:
    limits:
      memory: 64M  # Reduce from 128M
```

## CI/CD Pipeline

### Automatic Builds

**On every push to main**:
1. GitLab CI builds Docker image
2. Tags with multiple identifiers
3. Pushes to Docker Hub
4. Watchtower detects within 5 minutes
5. Agent updates automatically

**View pipeline**: https://gitlab.com/towerops/towerops-agent/-/pipelines

### Build Time

- **Main branch** (amd64 only): ~5 minutes
- **Git tags** (multi-arch): ~20 minutes

## Best Practices

1. **Use `main` tag** for automatic updates
2. **Enable notifications** to track updates
3. **Set reasonable poll interval** (300-1800s)
4. **Monitor Watchtower logs** regularly
5. **Test in staging** before production
6. **Keep Watchtower updated** (`docker pull containrrr/watchtower:latest`)
7. **Use Docker Compose** for easier management

## Resources

- **Watchtower Docs**: https://containrrr.dev/watchtower/
- **Docker Hub**: https://hub.docker.com/r/gmcintire/towerops-agent
- **GitLab CI**: https://gitlab.com/towerops/towerops-agent/-/blob/main/.gitlab-ci.yml
- **Issue Tracker**: https://github.com/towerops/towerops-agent/issues

---

**Status**: ✅ Complete
**Date**: January 16, 2026
**Impact**: Fully automated zero-touch updates for all deployments