towerops/TIMESCALEDB.md

TimescaleDB Setup for TowerOps

Overview

TowerOps uses TimescaleDB for efficient time-series storage of monitoring data. While the application will work with standard PostgreSQL, TimescaleDB provides significant performance benefits for production use.

Development vs Production

Development & Test

• Standard PostgreSQL: No TimescaleDB installation required
• Migrations automatically skip TimescaleDB features in dev/test environments
• You'll see messages like: "Development/Test environment - using standard PostgreSQL table"
• Faster setup - developers don't need to install TimescaleDB
• Tests run faster without TimescaleDB overhead

Production

• TimescaleDB Required: Production migrations enable TimescaleDB features
• Automatic data retention (90 days)
• Automatic compression (chunks older than 7 days)
• Continuous aggregates for fast dashboard queries
• Scales to millions of monitoring checks
• You'll see messages like: "Production environment - enabling TimescaleDB features"

Installation

Note: TimescaleDB installation is only required for production deployments. Local development and testing use standard PostgreSQL.

macOS (Homebrew)

# Add TimescaleDB tap
brew tap timescale/tap

# Install TimescaleDB
brew install timescaledb

# Run the setup script
/opt/homebrew/opt/timescaledb/bin/timescaledb-tune --quiet --yes

# Restart PostgreSQL
brew services restart postgresql@17

For Postgres.app users:

# Download TimescaleDB extension for your PostgreSQL version
# Visit: https://docs.timescale.com/self-hosted/latest/install/installation-macos/

# Or use Docker (see below)

Linux (Ubuntu/Debian)

# Add TimescaleDB repository
sudo apt install gnupg postgresql-common apt-transport-https lsb-release wget

sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh

echo "deb https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list

wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/timescaledb.gpg

# Install TimescaleDB
sudo apt update
sudo apt install timescaledb-2-postgresql-17

# Run setup
sudo timescaledb-tune --quiet --yes

# Restart PostgreSQL
sudo systemctl restart postgresql

Docker (Easiest for Development)

# Stop your current PostgreSQL (if running)
brew services stop postgresql@17  # macOS
# or
sudo systemctl stop postgresql    # Linux

# Run TimescaleDB in Docker
docker run -d \
  --name timescaledb \
  -p 5432:5432 \
  -e POSTGRES_PASSWORD=postgres \
  -v timescaledb_data:/var/lib/postgresql/data \
  timescale/timescaledb:latest-pg17

# Update config/dev.exs to use Docker PostgreSQL
# (already configured to use localhost:5432)

Verification

After installing TimescaleDB, verify it's working:

# Connect to PostgreSQL
psql -U postgres -d towerops_dev

# Check if TimescaleDB is available
SELECT * FROM pg_available_extensions WHERE name = 'timescaledb';

# If available, enable it
CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;

# Verify version
SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';

# Exit psql
\q

Running Migrations

Development/Test (Standard PostgreSQL)

# Default behavior - uses standard PostgreSQL
mix ecto.reset

You should see:

Development/Test environment - using standard PostgreSQL table
TimescaleDB features will be enabled in production
Development/Test environment - skipping continuous aggregates
Continuous aggregates will be created in production

Production (TimescaleDB)

# Set MIX_ENV to prod and run migrations
MIX_ENV=prod mix ecto.migrate

You should see:

Production environment - enabling TimescaleDB features
Production environment - creating TimescaleDB continuous aggregates

Environment Detection: Migrations automatically detect the environment via MIX_ENV and only enable TimescaleDB features when MIX_ENV=prod.

TimescaleDB Features Enabled

Hypertable

• monitoring_checks table is automatically partitioned by time
• Optimized for time-range queries
• Efficient INSERTs for high-volume monitoring data

Retention Policy

• Raw monitoring data automatically deleted after 90 days
• Keeps database size manageable
• Configurable in migration file

Compression Policy

• Data older than 7 days is automatically compressed
• ~95% space savings
• Transparent decompression on query

Continuous Aggregates

monitoring_checks_hourly

• Pre-computed hourly statistics per equipment
• Includes: total checks, successful/failed counts, avg/min/max response time
• Refreshed every hour automatically

monitoring_checks_daily

• Pre-computed daily statistics per equipment
• Includes uptime percentage calculation
• Refreshed daily automatically

Querying Time-Series Data

Using Continuous Aggregates

# Get hourly stats for last 24 hours
start_time = DateTime.utc_now() |> DateTime.add(-24 * 60 * 60, :second)
end_time = DateTime.utc_now()
Monitoring.get_hourly_stats(equipment_id, start_time, end_time)

# Get daily stats for last 30 days
start_time = DateTime.utc_now() |> DateTime.add(-30 * 24 * 60 * 60, :second)
Monitoring.get_daily_stats(equipment_id, start_time, DateTime.utc_now())

# Get uptime percentage for last 30 days
Monitoring.get_uptime_percentage(equipment_id, 30)

Direct SQL with TimescaleDB Functions

-- Time-bucketed aggregation
SELECT
  time_bucket('5 minutes', checked_at) AS bucket,
  AVG(response_time_ms) as avg_response_time
FROM monitoring_checks
WHERE equipment_id = 'abc-123'
  AND checked_at > NOW() - INTERVAL '1 hour'
GROUP BY bucket
ORDER BY bucket DESC;

-- Using continuous aggregate for fast queries
SELECT *
FROM monitoring_checks_hourly
WHERE equipment_id = 'abc-123'
  AND bucket > NOW() - INTERVAL '7 days'
ORDER BY bucket DESC;

Monitoring TimescaleDB

-- Check hypertable info
SELECT * FROM timescaledb_information.hypertables;

-- Check compression stats
SELECT * FROM timescaledb_information.compression_settings;

-- Check chunk info
SELECT * FROM timescaledb_information.chunks
WHERE hypertable_name = 'monitoring_checks';

-- Check continuous aggregate policies
SELECT * FROM timescaledb_information.continuous_aggregates;

Troubleshooting

Extension not available

• Make sure TimescaleDB is installed: brew list | grep timescale (macOS)
• Check PostgreSQL config: cat $(pg_config --sysconfdir)/postgresql.conf | grep shared_preload_libraries
• Should include: shared_preload_libraries = 'timescaledb'

Migration fails with TimescaleDB errors

• Drop database and recreate: mix ecto.drop && mix ecto.create && mix ecto.migrate
• Check PostgreSQL logs: tail -f /opt/homebrew/var/log/postgresql@17.log

Continuous aggregates not refreshing

-- Manually refresh
CALL refresh_continuous_aggregate('monitoring_checks_hourly', NULL, NULL);
CALL refresh_continuous_aggregate('monitoring_checks_daily', NULL, NULL);

Production Considerations

Chunk Size

Default chunk interval is 7 days. For high-volume deployments, consider:

-- Set chunk interval to 1 day for higher write throughput
SELECT set_chunk_time_interval('monitoring_checks', INTERVAL '1 day');

Retention Policy

Adjust retention based on compliance requirements:

-- Extend retention to 1 year
SELECT remove_retention_policy('monitoring_checks');
SELECT add_retention_policy('monitoring_checks', INTERVAL '1 year');

Compression

Tune compression for your workload:

-- Compress after 1 day instead of 7 days
SELECT remove_compression_policy('monitoring_checks');
SELECT add_compression_policy('monitoring_checks', INTERVAL '1 day');

Resources

• TimescaleDB Documentation
• TimescaleDB Best Practices
• Continuous Aggregates Guide
• Compression Guide