towerops/DEPLOYMENT.md

# TowerOps Production Deployment Guide

This guide covers deploying TowerOps to production with TimescaleDB, email notifications, and monitoring capabilities.

## Prerequisites

- PostgreSQL 14+ with TimescaleDB extension installed
- Elixir 1.19+ and Erlang 27+
- SSL certificate for HTTPS
- SMTP server for email notifications

## Environment Variables

Create a `.env.prod` file or configure these environment variables in your deployment platform:

### Required

```bash
# Secret key for encryption (generate with: mix phx.gen.secret)
SECRET_KEY_BASE=your-secret-key-here

# Database URL
DATABASE_URL=postgresql://user:password@host:port/towerops_prod

# Application URL
PHX_HOST=towerops.example.com
PORT=4000

# Environment
MIX_ENV=prod
```

### Email Configuration

```bash
# SMTP settings for alert notifications
SMTP_RELAY=smtp.example.com
SMTP_USERNAME=your-smtp-username
SMTP_PASSWORD=your-smtp-password
SMTP_PORT=587
SMTP_TLS=true

# From address for emails
ALERT_FROM_EMAIL=alerts@towerops.example.com
ALERT_FROM_NAME="TowerOps Alerts"
```

### Optional

```bash
# Custom check interval (default: 300 seconds / 5 minutes)
DEFAULT_CHECK_INTERVAL_SECONDS=300

# TimescaleDB retention (default: 90 days)
MONITORING_RETENTION_DAYS=90

# TimescaleDB compression (default: 7 days)
MONITORING_COMPRESSION_DAYS=7
```

## Database Setup

**Important**: TimescaleDB must be installed on your PostgreSQL server. See [TIMESCALEDB.md](TIMESCALEDB.md) for installation instructions.

### 1. Install TimescaleDB Extension

Connect to your PostgreSQL database and enable TimescaleDB:

```sql
CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;
```

### 2. Create Database

```bash
# Development
mix ecto.create

# Production
MIX_ENV=prod mix ecto.create
```

### 3. Run Migrations

```bash
# Development
mix ecto.migrate

# Production
MIX_ENV=prod mix ecto.migrate
```

This will:
- Create all necessary tables
- Convert `monitoring_checks` to TimescaleDB hypertable
- Set up retention policies (auto-delete data older than 90 days)
- Configure compression policies (compress data older than 7 days)
- Create continuous aggregates for dashboard performance

## Application Configuration

### Production Config

Update `config/runtime.exs` with your production settings. Key configurations:

#### Database Pool Size

```elixir
config :towerops, Towerops.Repo,
  pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10")
```

#### Mailer Configuration

The application uses Swoosh for email delivery. Update the mailer configuration:

```elixir
config :towerops, Towerops.Mailer,
  adapter: Swoosh.Adapters.SMTP,
  relay: System.get_env("SMTP_RELAY"),
  username: System.get_env("SMTP_USERNAME"),
  password: System.get_env("SMTP_PASSWORD"),
  port: String.to_integer(System.get_env("SMTP_PORT") || "587"),
  tls: :always
```

## Building for Production

### 1. Install Dependencies

```bash
mix deps.get --only prod
```

### 2. Compile Application

```bash
MIX_ENV=prod mix compile
```

### 3. Build Assets

```bash
MIX_ENV=prod mix assets.deploy
```

This will:
- Build and minify CSS with Tailwind v4
- Bundle JavaScript with esbuild
- Generate digested asset files

### 4. Create Release

```bash
MIX_ENV=prod mix release
```

The release will be created in `_build/prod/rel/towerops/`.

## Deployment

### Using Releases

```bash
# Start the release
_build/prod/rel/towerops/bin/towerops start

# Run as daemon
_build/prod/rel/towerops/bin/towerops daemon

# Check status
_build/prod/rel/towerops/bin/towerops pid

# Stop
_build/prod/rel/towerops/bin/towerops stop
```

### Using systemd

Create `/etc/systemd/system/towerops.service`:

```ini
[Unit]
Description=TowerOps Monitoring Service
After=network.target postgresql.service

[Service]
Type=forking
User=towerops
Group=towerops
WorkingDirectory=/opt/towerops
Environment="PORT=4000"
Environment="MIX_ENV=prod"
ExecStart=/opt/towerops/_build/prod/rel/towerops/bin/towerops daemon
ExecStop=/opt/towerops/_build/prod/rel/towerops/bin/towerops stop
Restart=on-failure
RestartSec=5
SyslogIdentifier=towerops

[Install]
WantedBy=multi-user.target
```

Enable and start:

```bash
sudo systemctl daemon-reload
sudo systemctl enable towerops
sudo systemctl start towerops
sudo systemctl status towerops
```

### Using Docker

Create `Dockerfile`:

```dockerfile
FROM elixir:1.19-alpine AS build

RUN apk add --no-cache build-base git nodejs npm

WORKDIR /app

# Install hex and rebar
RUN mix local.hex --force && \
    mix local.rebar --force

ENV MIX_ENV=prod

# Install dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only prod

# Copy application
COPY config config
COPY lib lib
COPY priv priv
COPY assets assets

# Build assets
RUN mix assets.deploy

# Compile and build release
RUN mix compile
RUN mix release

# Runtime stage
FROM alpine:3.18

RUN apk add --no-cache libstdc++ openssl ncurses-libs

WORKDIR /app

COPY --from=build /app/_build/prod/rel/towerops ./

ENV HOME=/app
ENV PORT=4000
ENV MIX_ENV=prod

CMD ["bin/towerops", "start"]
```

Build and run:

```bash
docker build -t towerops .
docker run -d \
  --name towerops \
  -p 4000:4000 \
  --env-file .env.prod \
  towerops
```

## Monitoring System

The monitoring system starts automatically when the application boots. It will:

1. **Start Equipment Monitors**: Create a GenServer worker for each equipment with `monitoring_enabled: true`
2. **Ping Equipment**: Each worker pings its equipment at the configured interval
3. **Record Checks**: Results are stored in `monitoring_checks` TimescaleDB hypertable
4. **Create Alerts**: Status changes trigger alert creation
5. **Send Emails**: Owners and admins receive email notifications for down/up events
6. **Broadcast Updates**: Real-time UI updates via Phoenix PubSub

### Monitoring Behavior by Environment

- **All Environments** (dev/test/prod):
  - TimescaleDB features enabled (hypertables, retention, compression, aggregates)
  - Monitoring system active

- **Production (`MIX_ENV=prod`)**:
  - Email notifications sent to organization owners/admins

- **Development/Test**:
  - Email notifications disabled (to avoid test database connection issues)
  - Monitoring system runs but emails are suppressed

## SSL/TLS Configuration

### Using Nginx Reverse Proxy

```nginx
upstream towerops {
    server 127.0.0.1:4000;
}

server {
    listen 80;
    server_name towerops.example.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name towerops.example.com;

    ssl_certificate /etc/letsencrypt/live/towerops.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/towerops.example.com/privkey.pem;

    location / {
        proxy_pass http://towerops;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
```

## Production Checklist

### Pre-Deployment

- [ ] Database created with TimescaleDB extension installed
- [ ] All environment variables configured
- [ ] SSL certificate obtained and configured
- [ ] SMTP server configured and tested
- [ ] Assets built and digested
- [ ] Release created successfully

### Post-Deployment

- [ ] Application starts without errors
- [ ] Database migrations ran successfully
- [ ] TimescaleDB hypertables created (`\d+ monitoring_checks` shows hypertable)
- [ ] Continuous aggregates created (`SELECT * FROM timescaledb_information.continuous_aggregates;`)
- [ ] Can access application via HTTPS
- [ ] User registration works
- [ ] Can create organization
- [ ] Can add sites and equipment
- [ ] Monitoring workers start for enabled equipment
- [ ] Ping checks are being recorded
- [ ] Alerts are created on status changes
- [ ] Email notifications are sent
- [ ] LiveView updates work in real-time

### Monitoring

- [ ] Check application logs for errors
- [ ] Monitor database connections
- [ ] Monitor memory usage (GenServer workers)
- [ ] Check email delivery logs
- [ ] Verify TimescaleDB compression is working
- [ ] Verify retention policies are cleaning old data

## Database Maintenance

### Verify TimescaleDB Status

```sql
-- Check hypertables
SELECT * FROM timescaledb_information.hypertables;

-- Check compression
SELECT * FROM timescaledb_information.chunks WHERE is_compressed = true;

-- Check continuous aggregates
SELECT * FROM timescaledb_information.continuous_aggregates;

-- Check retention policies
SELECT * FROM timescaledb_information.jobs WHERE proc_name = 'policy_retention';
```

### Manual Compression

```sql
-- Compress specific chunk
SELECT compress_chunk('_timescaledb_internal._hyper_1_1_chunk');

-- Compress all chunks older than 7 days
SELECT compress_chunk(chunk) FROM (
  SELECT show_chunks('monitoring_checks', older_than => INTERVAL '7 days') AS chunk
) AS chunks;
```

### Backup and Restore

```bash
# Backup (includes TimescaleDB metadata)
pg_dump -Fc -d towerops_prod > towerops_backup.dump

# Restore
pg_restore -d towerops_prod towerops_backup.dump
```

## Scaling Considerations

### Database

- Monitor `monitoring_checks` table size and growth rate
- Adjust retention policy if needed (default: 90 days)
- Consider read replicas for heavy dashboard usage
- TimescaleDB compression reduces storage by 90%+

### Application

- Scale horizontally by running multiple instances
- Use load balancer with sticky sessions for LiveView
- Monitor GenServer worker memory usage
- Consider distributed Erlang if needed

### Email

- Use transactional email service (SendGrid, Postmark, etc.) for reliability
- Implement rate limiting if needed
- Monitor email delivery failures

## Troubleshooting

### Monitoring Not Working

Check logs for GenServer errors:

```bash
tail -f /var/log/towerops/error.log
```

Verify equipment has monitoring enabled:

```sql
SELECT id, name, monitoring_enabled, check_interval_seconds
FROM equipment;
```

### TimescaleDB Not Active

Verify extension is loaded:

```sql
\dx timescaledb
SELECT * FROM timescaledb_information.hypertables;
```

If missing:
1. Ensure TimescaleDB is installed (see TIMESCALEDB.md)
2. Drop and recreate the database: `mix ecto.drop && mix ecto.create && mix ecto.migrate`

### Email Not Sending

Test SMTP connection:

```elixir
# In iex -S mix
Swoosh.Email.new()
|> Swoosh.Email.to("test@example.com")
|> Swoosh.Email.from({"TowerOps", "alerts@towerops.example.com"})
|> Swoosh.Email.subject("Test")
|> Swoosh.Email.text_body("Test email")
|> Towerops.Mailer.deliver()
```

Check environment variables are set correctly.

## Support

For issues or questions:
- Check logs in `_build/prod/rel/towerops/logs/`
- Review TimescaleDB documentation: https://docs.timescale.com/
- Review Phoenix deployment guide: https://hexdocs.pm/phoenix/deployment.html