towerops/DEPLOYMENT.md

TowerOps Production Deployment Guide

This guide covers deploying TowerOps to production with TimescaleDB, email notifications, and monitoring capabilities.

Prerequisites

• PostgreSQL 14+ with TimescaleDB extension installed
• Elixir 1.19+ and Erlang 27+
• SSL certificate for HTTPS
• SMTP server for email notifications

Environment Variables

Create a .env.prod file or configure these environment variables in your deployment platform:

Required

# Secret key for encryption (generate with: mix phx.gen.secret)
SECRET_KEY_BASE=your-secret-key-here

# Database URL
DATABASE_URL=postgresql://user:password@host:port/towerops_prod

# Application URL
PHX_HOST=towerops.example.com
PORT=4000

# Environment
MIX_ENV=prod

Email Configuration

# SMTP settings for alert notifications
SMTP_RELAY=smtp.example.com
SMTP_USERNAME=your-smtp-username
SMTP_PASSWORD=your-smtp-password
SMTP_PORT=587
SMTP_TLS=true

# From address for emails
ALERT_FROM_EMAIL=alerts@towerops.example.com
ALERT_FROM_NAME="TowerOps Alerts"

Optional

# Custom check interval (default: 300 seconds / 5 minutes)
DEFAULT_CHECK_INTERVAL_SECONDS=300

# TimescaleDB retention (default: 90 days)
MONITORING_RETENTION_DAYS=90

# TimescaleDB compression (default: 7 days)
MONITORING_COMPRESSION_DAYS=7

Database Setup

Important: TimescaleDB must be installed on your PostgreSQL server. See TIMESCALEDB.md for installation instructions.

1. Install TimescaleDB Extension

Connect to your PostgreSQL database and enable TimescaleDB:

CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;

2. Create Database

# Development
mix ecto.create

# Production
MIX_ENV=prod mix ecto.create

3. Run Migrations

# Development
mix ecto.migrate

# Production
MIX_ENV=prod mix ecto.migrate

This will:

• Create all necessary tables
• Convert monitoring_checks to TimescaleDB hypertable
• Set up retention policies (auto-delete data older than 90 days)
• Configure compression policies (compress data older than 7 days)
• Create continuous aggregates for dashboard performance

Application Configuration

Production Config

Update config/runtime.exs with your production settings. Key configurations:

Database Pool Size

config :towerops, Towerops.Repo,
  pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10")

Mailer Configuration

The application uses Swoosh for email delivery. Update the mailer configuration:

config :towerops, Towerops.Mailer,
  adapter: Swoosh.Adapters.SMTP,
  relay: System.get_env("SMTP_RELAY"),
  username: System.get_env("SMTP_USERNAME"),
  password: System.get_env("SMTP_PASSWORD"),
  port: String.to_integer(System.get_env("SMTP_PORT") || "587"),
  tls: :always

Building for Production

1. Install Dependencies

mix deps.get --only prod

2. Compile Application

MIX_ENV=prod mix compile

3. Build Assets

MIX_ENV=prod mix assets.deploy

This will:

• Build and minify CSS with Tailwind v4
• Bundle JavaScript with esbuild
• Generate digested asset files

4. Create Release

MIX_ENV=prod mix release

The release will be created in _build/prod/rel/towerops/.

Deployment

Using Releases

# Start the release
_build/prod/rel/towerops/bin/towerops start

# Run as daemon
_build/prod/rel/towerops/bin/towerops daemon

# Check status
_build/prod/rel/towerops/bin/towerops pid

# Stop
_build/prod/rel/towerops/bin/towerops stop

Using systemd

Create /etc/systemd/system/towerops.service:

[Unit]
Description=TowerOps Monitoring Service
After=network.target postgresql.service

[Service]
Type=forking
User=towerops
Group=towerops
WorkingDirectory=/opt/towerops
Environment="PORT=4000"
Environment="MIX_ENV=prod"
ExecStart=/opt/towerops/_build/prod/rel/towerops/bin/towerops daemon
ExecStop=/opt/towerops/_build/prod/rel/towerops/bin/towerops stop
Restart=on-failure
RestartSec=5
SyslogIdentifier=towerops

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable towerops
sudo systemctl start towerops
sudo systemctl status towerops

Using Docker

Create Dockerfile:

FROM elixir:1.19-alpine AS build

RUN apk add --no-cache build-base git nodejs npm

WORKDIR /app

# Install hex and rebar
RUN mix local.hex --force && \
    mix local.rebar --force

ENV MIX_ENV=prod

# Install dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only prod

# Copy application
COPY config config
COPY lib lib
COPY priv priv
COPY assets assets

# Build assets
RUN mix assets.deploy

# Compile and build release
RUN mix compile
RUN mix release

# Runtime stage
FROM alpine:3.18

RUN apk add --no-cache libstdc++ openssl ncurses-libs

WORKDIR /app

COPY --from=build /app/_build/prod/rel/towerops ./

ENV HOME=/app
ENV PORT=4000
ENV MIX_ENV=prod

CMD ["bin/towerops", "start"]

Build and run:

docker build -t towerops .
docker run -d \
  --name towerops \
  -p 4000:4000 \
  --env-file .env.prod \
  towerops

Monitoring System

The monitoring system starts automatically when the application boots. It will:

1. Start Equipment Monitors: Create a GenServer worker for each equipment with monitoring_enabled: true
2. Ping Equipment: Each worker pings its equipment at the configured interval
3. Record Checks: Results are stored in monitoring_checks TimescaleDB hypertable
4. Create Alerts: Status changes trigger alert creation
5. Send Emails: Owners and admins receive email notifications for down/up events
6. Broadcast Updates: Real-time UI updates via Phoenix PubSub

Monitoring Behavior by Environment

• All Environments (dev/test/prod):

  • TimescaleDB features enabled (hypertables, retention, compression, aggregates)
  • Monitoring system active

• Production (MIX_ENV=prod):

  • Email notifications sent to organization owners/admins

• Development/Test:

  • Email notifications disabled (to avoid test database connection issues)
  • Monitoring system runs but emails are suppressed

SSL/TLS Configuration

Using Nginx Reverse Proxy

upstream towerops {
    server 127.0.0.1:4000;
}

server {
    listen 80;
    server_name towerops.example.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name towerops.example.com;

    ssl_certificate /etc/letsencrypt/live/towerops.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/towerops.example.com/privkey.pem;

    location / {
        proxy_pass http://towerops;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Production Checklist

Pre-Deployment

• Database created with TimescaleDB extension installed
• All environment variables configured
• SSL certificate obtained and configured
• SMTP server configured and tested
• Assets built and digested
• Release created successfully

Post-Deployment

• Application starts without errors
• Database migrations ran successfully
• TimescaleDB hypertables created (\d+ monitoring_checks shows hypertable)
• Continuous aggregates created (SELECT * FROM timescaledb_information.continuous_aggregates;)
• Can access application via HTTPS
• User registration works
• Can create organization
• Can add sites and equipment
• Monitoring workers start for enabled equipment
• Ping checks are being recorded
• Alerts are created on status changes
• Email notifications are sent
• LiveView updates work in real-time

Monitoring

• Check application logs for errors
• Monitor database connections
• Monitor memory usage (GenServer workers)
• Check email delivery logs
• Verify TimescaleDB compression is working
• Verify retention policies are cleaning old data

Database Maintenance

Verify TimescaleDB Status

-- Check hypertables
SELECT * FROM timescaledb_information.hypertables;

-- Check compression
SELECT * FROM timescaledb_information.chunks WHERE is_compressed = true;

-- Check continuous aggregates
SELECT * FROM timescaledb_information.continuous_aggregates;

-- Check retention policies
SELECT * FROM timescaledb_information.jobs WHERE proc_name = 'policy_retention';

Manual Compression

-- Compress specific chunk
SELECT compress_chunk('_timescaledb_internal._hyper_1_1_chunk');

-- Compress all chunks older than 7 days
SELECT compress_chunk(chunk) FROM (
  SELECT show_chunks('monitoring_checks', older_than => INTERVAL '7 days') AS chunk
) AS chunks;

Backup and Restore

# Backup (includes TimescaleDB metadata)
pg_dump -Fc -d towerops_prod > towerops_backup.dump

# Restore
pg_restore -d towerops_prod towerops_backup.dump

Scaling Considerations

Database

• Monitor monitoring_checks table size and growth rate
• Adjust retention policy if needed (default: 90 days)
• Consider read replicas for heavy dashboard usage
• TimescaleDB compression reduces storage by 90%+

Application

• Scale horizontally by running multiple instances
• Use load balancer with sticky sessions for LiveView
• Monitor GenServer worker memory usage
• Consider distributed Erlang if needed

Email

• Use transactional email service (SendGrid, Postmark, etc.) for reliability
• Implement rate limiting if needed
• Monitor email delivery failures

Troubleshooting

Monitoring Not Working

Check logs for GenServer errors:

tail -f /var/log/towerops/error.log

Verify equipment has monitoring enabled:

SELECT id, name, monitoring_enabled, check_interval_seconds
FROM equipment;

TimescaleDB Not Active

Verify extension is loaded:

\dx timescaledb
SELECT * FROM timescaledb_information.hypertables;

If missing:

1. Ensure TimescaleDB is installed (see TIMESCALEDB.md)
2. Drop and recreate the database: mix ecto.drop && mix ecto.create && mix ecto.migrate

Email Not Sending

Test SMTP connection:

# In iex -S mix
Swoosh.Email.new()
|> Swoosh.Email.to("test@example.com")
|> Swoosh.Email.from({"TowerOps", "alerts@towerops.example.com"})
|> Swoosh.Email.subject("Test")
|> Swoosh.Email.text_body("Test email")
|> Towerops.Mailer.deliver()

Check environment variables are set correctly.

Support

For issues or questions:

• Check logs in _build/prod/rel/towerops/logs/
• Review TimescaleDB documentation: https://docs.timescale.com/
• Review Phoenix deployment guide: https://hexdocs.pm/phoenix/deployment.html