infra/ansible/postgres-readme.md

# PostgreSQL Server Configuration for APRS.me

This Ansible configuration deploys a high-performance PostgreSQL server on the CM3588 with RAID0 storage configuration, optimized for the APRS.me workload.

## Hardware

- **Server**: CM3588 with RK3588 SoC
- **Storage**: 2x 2TB NVMe SSDs in RAID0 (4TB total capacity)
- **Network**: 2.5 Gigabit Ethernet
- **RAM**: 32GB (assumed)

## Features

- Software RAID0 configuration for maximum performance
- PostgreSQL 16 with PostGIS for spatial data
- PgBouncer for connection pooling
- Optimized for high insert rates and spatial queries
- Automated backups with retention policy
- Health monitoring and alerting
- Kernel and systemd tuning for database workloads

## Prerequisites

1. Ubuntu 22.04 or later installed on the CM3588
2. SSH access to the server with the user 'graham'
3. Ansible installed on your control machine

## Initial Setup

**Note**: This playbook assumes a fresh PostgreSQL installation. All data will be stored on the RAID array from the start.

1. First, create a vault file for sensitive data:
```bash
ansible-vault create group_vars/all/vault.yml
```

Add the following to the vault file:
```yaml
vault_postgresql_aprsme_password: "your-secure-password-here"
```

2. Bootstrap the server (first time only):
```bash
ansible-playbook -i inventory.yml bootstrap.yml --ask-pass --ask-become-pass
```

3. Deploy PostgreSQL:
```bash
ansible-playbook -i inventory.yml postgres-playbook.yml --ask-vault-pass
```

The playbook will:
- Create RAID0 array from both NVMe drives
- Initialize PostgreSQL cluster directly on the RAID array
- Configure all PostgreSQL data, WAL, archives, and backups on the RAID array
- No data migration needed since this is a fresh installation

## Configuration Details

### RAID0 Configuration
- Creates `/dev/md0` from `/dev/nvme0n1` and `/dev/nvme1n1`
- Formatted as ext4 with optimizations for SSDs
- Mounted at `/mnt/pgdata` with noatime,nodiratime options
- All PostgreSQL data stored on RAID array:
  - Data directory: `/mnt/pgdata/16/main`
  - WAL directory: `/mnt/pgdata/wal`
  - Archive directory: `/mnt/pgdata/archive`
  - Backup directory: `/mnt/pgdata/backups`

### PostgreSQL Optimization
- **shared_buffers**: 8GB (25% of RAM)
- **effective_cache_size**: 24GB (75% of RAM)
- **work_mem**: 64MB (for complex spatial queries)
- **max_connections**: 200
- **autovacuum**: Tuned for high insert rate
- **checkpoint**: Optimized for write performance

### PgBouncer Settings
- **pool_mode**: transaction (optimal for APRS.me)
- **default_pool_size**: 25
- **max_client_conn**: 1000

### Security
- PostgreSQL listening on all interfaces (configure firewall accordingly)
- SCRAM-SHA-256 authentication
- Access restricted to 10.0.0.0/8 network (adjust as needed)

## Maintenance

### Check RAID Status
```bash
ssh postgres@10.0.16.230 'sudo mdadm --detail /dev/md0'
```

### PostgreSQL Logs
```bash
ssh postgres@10.0.16.230 'sudo journalctl -u postgresql -f'
```

### Backup Status
```bash
ssh postgres@10.0.16.230 'ls -la /mnt/pgdata/backups/'
```

### Performance Monitoring
```bash
# Connect to PostgreSQL
psql -h 10.0.16.230 -p 5432 -U postgres -d aprsme_prod

# Check slow queries
SELECT * FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10;

# Check table sizes
SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;

# Check index usage
SELECT schemaname, tablename, indexname, idx_scan
FROM pg_stat_user_indexes
ORDER BY idx_scan;
```

## Connecting from APRS.me

Update your APRS.me configuration:

```elixir
# config/runtime.exs
config :aprsme, Aprsme.Repo,
  url: "postgresql://aprsme:password@10.0.16.230:6432/aprsme_prod",
  pool_size: 5
```

Or use environment variable:
```bash
DATABASE_URL=postgresql://aprsme:password@10.0.16.230:6432/aprsme_prod
```

## Disaster Recovery

1. **RAID Failure**: While RAID0 provides no redundancy, the backup script runs nightly
2. **Backup Restore**:
   ```bash
   pg_restore -h localhost -U postgres -d aprsme_prod_restore /mnt/pgdata/backups/aprsme_prod_20240126_020000.sql.gz
   ```
3. **Replication**: Consider setting up streaming replication to a standby server

## Performance Expectations

With this configuration, expect:
- Insert rate: 1000+ packets/second sustained
- Spatial query response: <10ms for indexed queries
- Connection overhead: Minimal with PgBouncer
- Storage capacity: ~4TB usable (years of APRS data)

## Troubleshooting

### RAID Issues
```bash
# Check RAID status
cat /proc/mdstat
mdadm --detail /dev/md0

# Check disk health
smartctl -a /dev/nvme0n1
smartctl -a /dev/nvme1n1
```

### PostgreSQL Performance
```bash
# Check current connections
SELECT count(*) FROM pg_stat_activity;

# Check cache hit ratio
SELECT sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) AS cache_hit_ratio FROM pg_statio_user_tables;
```

### PgBouncer Stats
```bash
psql -h 10.0.16.230 -p 6432 -U postgres pgbouncer -c "SHOW STATS;"
```