infra/clusters/aprs/AUTO-HEALING-GUIDE.md

PostgreSQL Auto-Healing Configuration Guide

This guide explains the auto-healing mechanisms implemented to prevent and recover from PostgreSQL connection exhaustion issues.

Overview

The auto-healing system consists of several components working together:

1. Enhanced PostgreSQL Configuration
2. Improved PgBouncer Connection Pooling
3. Health Checks and Automatic Restarts
4. Periodic Connection Cleanup
5. Resource Limits and Scaling

Components

1. PostgreSQL Enhancements (postgis-deployment-enhanced.yaml)

• Increased max_connections: From 100 to 200
• Connection timeouts:
  • idle_in_transaction_session_timeout=30s: Kills idle transactions after 30 seconds
  • statement_timeout=300s: Kills queries running longer than 5 minutes
• Readiness probe: Fails if connection count exceeds 180 (90% of max)
• Increased resources: 2Gi memory, 1 CPU

2. PgBouncer Enhancements (pgbouncer-deployment-enhanced.yaml)

• Connection pooling settings:
  • Pool mode: transaction (releases connections after each transaction)
  • Default pool size: 25 connections per database/user pair
  • Server idle timeout: 10 minutes
  • Query timeout: 5 minutes
• Retry logic:
  • server_login_retry=15: Retries failed logins for 15 seconds
  • server_round_robin=1: Distributes connections evenly
• Enhanced logging: Tracks connections, disconnections, and errors

3. APRS Application Enhancements (aprs-statefulset-enhanced.yaml)

• Reduced connection pool: From 10 to 5 connections per pod
• Connection timeouts:
  • Pool timeout: 60 seconds
  • Connect timeout: 30 seconds
  • Idle timeout: 15 minutes
• Startup probe: Allows up to 120 seconds for application startup
• Increased resources: 1Gi memory, 1 CPU

4. Automatic Connection Cleanup (postgres-cleanup-cronjob.yaml)

Runs every 15 minutes to:

• Kill idle connections older than 30 minutes
• Kill idle-in-transaction connections older than 5 minutes
• Log connection statistics
• Monitor connection usage percentage

5. Pod Disruption Budgets (postgis-pdb.yaml)

Ensures at least 1 replica of PostgreSQL and PgBouncer remain available during:

• Cluster upgrades
• Node maintenance
• Voluntary disruptions

How Auto-Healing Works

Connection Exhaustion Prevention

1. Application-level: Reduced pool size prevents each pod from using too many connections
2. PgBouncer-level: Connection pooling multiplexes many client connections over fewer server connections
3. PostgreSQL-level: Timeouts automatically close stale connections

Automatic Recovery

1. Health checks: Pods restart automatically when:

   • PostgreSQL has too many connections (>180)
   • PgBouncer can't connect to PostgreSQL
   • Application can't connect to the database

2. CronJob cleanup: Every 15 minutes, forcefully closes:

   • Idle connections wasting resources
   • Stuck transactions blocking others

3. Connection retry: PgBouncer retries failed connections for 15 seconds before giving up

Monitoring

Use the provided monitoring script to check system health:

./monitor-postgres-health.sh

This shows:

• Current connection count and states
• Connection usage percentage
• Recent errors from logs
• Last cleanup job execution

Manual Intervention

If automatic healing fails, you can manually intervene:

# Check connection details
kubectl exec -it deployment/postgis -n aprs -- psql -U aprs -d aprs_prod -c "SELECT * FROM pg_stat_activity WHERE datname = 'aprs_prod';"

# Kill all connections (emergency)
kubectl exec -it deployment/postgis -n aprs -- psql -U aprs -d aprs_prod -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'aprs_prod' AND pid <> pg_backend_pid();"

# Restart components
kubectl rollout restart deployment/postgis -n aprs
kubectl rollout restart deployment/pgbouncer -n aprs
kubectl rollout restart statefulset/aprs -n aprs

Applying the Configuration

To apply all auto-healing configurations:

cd /Users/graham/dev/infra/clusters/aprs
./apply-auto-healing.sh

This script:

1. Backs up current configurations
2. Applies all enhanced configurations
3. Waits for rollouts to complete
4. Provides a summary of changes

Tuning Parameters

You may need to adjust these based on your workload:

1. PostgreSQL max_connections: Currently 200, increase if needed
2. PgBouncer pool sizes: Currently 25, adjust based on connection patterns
3. Application POOL_SIZE: Currently 5, can be increased if connections are available
4. Cleanup intervals: Currently 15 minutes, can be more or less frequent

Troubleshooting

Common issues and solutions:

1. "too many clients already": Cleanup job should fix this within 15 minutes
2. "server login has been failing": PgBouncer is in retry mode, will recover in 15 seconds
3. Pods in CrashLoopBackOff: Check logs for specific errors
4. High memory usage: Consider increasing resource limits

Future Improvements

Consider implementing:

1. Prometheus metrics for connection monitoring
2. Alerting when connection usage exceeds 80%
3. Horizontal pod autoscaling based on connection metrics
4. Connection pooling at the application level using a library like poolboy