# TimescaleDB Setup for TowerOps

## Overview

TowerOps uses TimescaleDB for efficient time-series storage of monitoring data. TimescaleDB is **required in all environments** (development, test, and production).

## Why TimescaleDB?

- **Hypertables**: Automatic time-based partitioning for monitoring_checks
- **Compression**: ~95% space savings for historical data
- **Retention**: Automatic cleanup of old data
- **Continuous Aggregates**: Pre-computed hourly/daily statistics for fast dashboard queries
- **Scales**: Handles millions of monitoring checks efficiently

## Installation

TimescaleDB must be installed in all environments (dev, test, prod).

### macOS (Homebrew)

```bash
# Add TimescaleDB tap
brew tap timescale/tap

# Install TimescaleDB
brew install timescaledb

# Run the setup script
/opt/homebrew/opt/timescaledb/bin/timescaledb-tune --quiet --yes

# Restart PostgreSQL
brew services restart postgresql@17
```

**For Postgres.app users:**
```bash
# Download TimescaleDB extension for your PostgreSQL version
# Visit: https://docs.timescale.com/self-hosted/latest/install/installation-macos/

# Or use Docker (see below)
```

### Linux (Ubuntu/Debian)

```bash
# Add TimescaleDB repository
sudo apt install gnupg postgresql-common apt-transport-https lsb-release wget

sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh

echo "deb https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list

wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/timescaledb.gpg

# Install TimescaleDB
sudo apt update
sudo apt install timescaledb-2-postgresql-17

# Run setup
sudo timescaledb-tune --quiet --yes

# Restart PostgreSQL
sudo systemctl restart postgresql
```

### Docker (Easiest for Development)

```bash
# Stop your current PostgreSQL (if running)
brew services stop postgresql@17  # macOS
# or
sudo systemctl stop postgresql    # Linux

# Run TimescaleDB in Docker
docker run -d \
  --name timescaledb \
  -p 5432:5432 \
  -e POSTGRES_PASSWORD=postgres \
  -v timescaledb_data:/var/lib/postgresql/data \
  timescale/timescaledb:latest-pg17

# Update config/dev.exs to use Docker PostgreSQL
# (already configured to use localhost:5432)
```

## Verification

After installing TimescaleDB, verify it's working:

```bash
# Connect to PostgreSQL
psql -U postgres -d towerops_dev

# Check if TimescaleDB is available
SELECT * FROM pg_available_extensions WHERE name = 'timescaledb';

# If available, enable it
CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;

# Verify version
SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';

# Exit psql
\q
```

## Running Migrations

After installing TimescaleDB, run migrations to set up the database:

```bash
# Development
mix ecto.reset

# Test
MIX_ENV=test mix ecto.reset

# Production
MIX_ENV=prod mix ecto.migrate
```

The migrations will:
- Create the `monitoring_checks` hypertable
- Set up retention policy (90 days)
- Configure compression (data older than 7 days)
- Create continuous aggregates (hourly and daily)

## TimescaleDB Features Enabled

### Hypertable
- `monitoring_checks` table is automatically partitioned by time
- Optimized for time-range queries
- Efficient INSERTs for high-volume monitoring data

### Retention Policy
- Raw monitoring data automatically deleted after 90 days
- Keeps database size manageable
- Configurable in migration file

### Compression Policy
- Data older than 7 days is automatically compressed
- ~95% space savings
- Transparent decompression on query

### Continuous Aggregates

#### monitoring_checks_hourly
- Pre-computed hourly statistics per equipment
- Includes: total checks, successful/failed counts, avg/min/max response time
- Refreshed every hour automatically

#### monitoring_checks_daily
- Pre-computed daily statistics per equipment
- Includes uptime percentage calculation
- Refreshed daily automatically

## Querying Time-Series Data

### Using Continuous Aggregates

```elixir
# Get hourly stats for last 24 hours
start_time = DateTime.utc_now() |> DateTime.add(-24 * 60 * 60, :second)
end_time = DateTime.utc_now()
Monitoring.get_hourly_stats(equipment_id, start_time, end_time)

# Get daily stats for last 30 days
start_time = DateTime.utc_now() |> DateTime.add(-30 * 24 * 60 * 60, :second)
Monitoring.get_daily_stats(equipment_id, start_time, DateTime.utc_now())

# Get uptime percentage for last 30 days
Monitoring.get_uptime_percentage(equipment_id, 30)
```

### Direct SQL with TimescaleDB Functions

```sql
-- Time-bucketed aggregation
SELECT
  time_bucket('5 minutes', checked_at) AS bucket,
  AVG(response_time_ms) as avg_response_time
FROM monitoring_checks
WHERE equipment_id = 'abc-123'
  AND checked_at > NOW() - INTERVAL '1 hour'
GROUP BY bucket
ORDER BY bucket DESC;

-- Using continuous aggregate for fast queries
SELECT *
FROM monitoring_checks_hourly
WHERE equipment_id = 'abc-123'
  AND bucket > NOW() - INTERVAL '7 days'
ORDER BY bucket DESC;
```

## Monitoring TimescaleDB

```sql
-- Check hypertable info
SELECT * FROM timescaledb_information.hypertables;

-- Check compression stats
SELECT * FROM timescaledb_information.compression_settings;

-- Check chunk info
SELECT * FROM timescaledb_information.chunks
WHERE hypertable_name = 'monitoring_checks';

-- Check continuous aggregate policies
SELECT * FROM timescaledb_information.continuous_aggregates;
```

## Troubleshooting

### Extension not available
- Make sure TimescaleDB is installed: `brew list | grep timescale` (macOS)
- Check PostgreSQL config: `cat $(pg_config --sysconfdir)/postgresql.conf | grep shared_preload_libraries`
- Should include: `shared_preload_libraries = 'timescaledb'`

### Migration fails with TimescaleDB errors
- Drop database and recreate: `mix ecto.drop && mix ecto.create && mix ecto.migrate`
- Check PostgreSQL logs: `tail -f /opt/homebrew/var/log/postgresql@17.log`

### Continuous aggregates not refreshing
```sql
-- Manually refresh
CALL refresh_continuous_aggregate('monitoring_checks_hourly', NULL, NULL);
CALL refresh_continuous_aggregate('monitoring_checks_daily', NULL, NULL);
```

## Production Considerations

### Chunk Size
Default chunk interval is 7 days. For high-volume deployments, consider:

```sql
-- Set chunk interval to 1 day for higher write throughput
SELECT set_chunk_time_interval('monitoring_checks', INTERVAL '1 day');
```

### Retention Policy
Adjust retention based on compliance requirements:

```sql
-- Extend retention to 1 year
SELECT remove_retention_policy('monitoring_checks');
SELECT add_retention_policy('monitoring_checks', INTERVAL '1 year');
```

### Compression
Tune compression for your workload:

```sql
-- Compress after 1 day instead of 7 days
SELECT remove_compression_policy('monitoring_checks');
SELECT add_compression_policy('monitoring_checks', INTERVAL '1 day');
```

## Resources

- [TimescaleDB Documentation](https://docs.timescale.com/)
- [TimescaleDB Best Practices](https://docs.timescale.com/use-timescale/latest/schema-management/about-schemas/)
- [Continuous Aggregates Guide](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/)
- [Compression Guide](https://docs.timescale.com/use-timescale/latest/compression/)