infra/ansible/postgres-readme.md

PostgreSQL Server Configuration for APRS.me

This Ansible configuration deploys a high-performance PostgreSQL server on the CM3588 with RAID0 storage configuration, optimized for the APRS.me workload.

Hardware

• Server: CM3588 with RK3588 SoC
• Storage: 2x 2TB NVMe SSDs in RAID0 (4TB total capacity)
• Network: 2.5 Gigabit Ethernet
• RAM: 32GB (assumed)

Features

• Software RAID0 configuration for maximum performance
• PostgreSQL 16 with PostGIS for spatial data
• PgBouncer for connection pooling
• Optimized for high insert rates and spatial queries
• Automated backups with retention policy
• Health monitoring and alerting
• Kernel and systemd tuning for database workloads

Prerequisites

1. Ubuntu 22.04 or later installed on the CM3588
2. SSH access to the server with the user 'graham'
3. Ansible installed on your control machine

Initial Setup

Note: This playbook assumes a fresh PostgreSQL installation. All data will be stored on the RAID array from the start.

1. First, create a vault file for sensitive data:

ansible-vault create group_vars/all/vault.yml

Add the following to the vault file:

vault_postgresql_aprsme_password: "your-secure-password-here"

2. Bootstrap the server (first time only):

ansible-playbook -i inventory.yml bootstrap.yml --ask-pass --ask-become-pass

3. Deploy PostgreSQL:

ansible-playbook -i inventory.yml postgres-playbook.yml --ask-vault-pass

The playbook will:

• Create RAID0 array from both NVMe drives
• Initialize PostgreSQL cluster directly on the RAID array
• Configure all PostgreSQL data, WAL, archives, and backups on the RAID array
• No data migration needed since this is a fresh installation

Configuration Details

RAID0 Configuration

• Creates /dev/md0 from /dev/nvme0n1 and /dev/nvme1n1
• Formatted as ext4 with optimizations for SSDs
• Mounted at /mnt/pgdata with noatime,nodiratime options
• All PostgreSQL data stored on RAID array:
  • Data directory: /mnt/pgdata/16/main
  • WAL directory: /mnt/pgdata/wal
  • Archive directory: /mnt/pgdata/archive
  • Backup directory: /mnt/pgdata/backups

PostgreSQL Optimization

• shared_buffers: 8GB (25% of RAM)
• effective_cache_size: 24GB (75% of RAM)
• work_mem: 64MB (for complex spatial queries)
• max_connections: 200
• autovacuum: Tuned for high insert rate
• checkpoint: Optimized for write performance

PgBouncer Settings

• pool_mode: transaction (optimal for APRS.me)
• default_pool_size: 25
• max_client_conn: 1000

Security

• PostgreSQL listening on all interfaces (configure firewall accordingly)
• SCRAM-SHA-256 authentication
• Access restricted to 10.0.0.0/8 network (adjust as needed)

Maintenance

Check RAID Status

ssh postgres@10.0.16.230 'sudo mdadm --detail /dev/md0'

PostgreSQL Logs

ssh postgres@10.0.16.230 'sudo journalctl -u postgresql -f'

Backup Status

ssh postgres@10.0.16.230 'ls -la /mnt/pgdata/backups/'

Performance Monitoring

# Connect to PostgreSQL
psql -h 10.0.16.230 -p 5432 -U postgres -d aprsme_prod

# Check slow queries
SELECT * FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10;

# Check table sizes
SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;

# Check index usage
SELECT schemaname, tablename, indexname, idx_scan
FROM pg_stat_user_indexes
ORDER BY idx_scan;

Connecting from APRS.me

Update your APRS.me configuration:

# config/runtime.exs
config :aprsme, Aprsme.Repo,
  url: "postgresql://aprsme:password@10.0.16.230:6432/aprsme_prod",
  pool_size: 5

Or use environment variable:

DATABASE_URL=postgresql://aprsme:password@10.0.16.230:6432/aprsme_prod

Disaster Recovery

1. RAID Failure: While RAID0 provides no redundancy, the backup script runs nightly
2. Backup Restore:

   pg_restore -h localhost -U postgres -d aprsme_prod_restore /mnt/pgdata/backups/aprsme_prod_20240126_020000.sql.gz
   

3. Replication: Consider setting up streaming replication to a standby server

Performance Expectations

With this configuration, expect:

• Insert rate: 1000+ packets/second sustained
• Spatial query response: <10ms for indexed queries
• Connection overhead: Minimal with PgBouncer
• Storage capacity: ~4TB usable (years of APRS data)

Troubleshooting

RAID Issues

# Check RAID status
cat /proc/mdstat
mdadm --detail /dev/md0

# Check disk health
smartctl -a /dev/nvme0n1
smartctl -a /dev/nvme1n1

PostgreSQL Performance

# Check current connections
SELECT count(*) FROM pg_stat_activity;

# Check cache hit ratio
SELECT sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) AS cache_hit_ratio FROM pg_statio_user_tables;

PgBouncer Stats

psql -h 10.0.16.230 -p 6432 -U postgres pgbouncer -c "SHOW STATS;"