infra/clusters/aprs/monitoring/MONITORING-GUIDE.md

Monitoring and Observability Guide

This guide explains the comprehensive monitoring stack for the APRS k3s cluster.

Overview

The monitoring stack consists of:

• Prometheus: Metrics collection and storage
• Grafana: Visualization and dashboards
• AlertManager: Alert routing and notifications
• Exporters: PostgreSQL, PgBouncer, Redis, and application metrics

Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   APRS App  │     │ PostgreSQL  │     │  PgBouncer  │
│  /metrics   │     │  Exporter   │     │  Exporter   │
└──────┬──────┘     └──────┬──────┘     └──────┬──────┘
       │                   │                   │
       └───────────────────┴───────────────────┘
                           │
                    ┌──────▼──────┐
                    │ Prometheus  │
                    │  (scraping) │
                    └──────┬──────┘
                           │
                ┌──────────┴──────────┐
                │                     │
         ┌──────▼──────┐      ┌──────▼──────┐
         │   Grafana   │      │ AlertManager│
         │ (dashboards)│      │  (alerts)   │
         └─────────────┘      └─────────────┘

Installation

1. Install the monitoring stack:

cd /path/to/infra/clusters/aprs/monitoring
./install-monitoring.sh

2. Apply the enhanced deployments (if not already done):

cd /path/to/infra/clusters/aprs
./apply-auto-healing.sh

3. Update Redis deployment to include metrics:

kubectl apply -f redis-deployment-with-metrics.yaml

Access

Grafana

kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

Access at: http://localhost:3000

• Username: admin
• Password: changeme (change this!)

Prometheus

kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090

Access at: http://localhost:9090

AlertManager

kubectl port-forward -n monitoring svc/kube-prometheus-stack-alertmanager 9093:9093

Access at: http://localhost:9093

Metrics Available

PostgreSQL Metrics

• pg_connection_stats_total_connections - Current total connections
• pg_connection_stats_max_connections - Maximum allowed connections
• pg_connections_count - Connections by state (active, idle, etc.)
• pg_long_running_queries_count - Queries running > 5 minutes
• pg_database_size_bytes - Database size
• Standard pg_* metrics from postgres_exporter

PgBouncer Metrics

• pgbouncer_pools_server_active_connections - Active server connections
• pgbouncer_pools_client_active_connections - Active client connections
• pgbouncer_pools_pool_size - Configured pool size
• pgbouncer_stats_* - Various PgBouncer statistics

Application Metrics

• phoenix_endpoint_stop_duration_millisecond - HTTP request duration
• aprsme_repo_query_* - Database query metrics
• aprsme_packet_pipeline_* - Packet processing metrics
• aprsme_spatial_pubsub_* - Real-time broadcast metrics
• vm_memory_total - Erlang VM memory usage

Kubernetes Metrics

• container_memory_working_set_bytes - Container memory usage
• container_cpu_usage_seconds_total - Container CPU usage
• kube_pod_container_status_restarts_total - Pod restart count

Dashboards

Pre-installed Dashboards

1. PostgreSQL Dashboard (ID: 9628) - Comprehensive PostgreSQL metrics
2. PgBouncer Dashboard (ID: 15984) - Connection pooling metrics
3. Kubernetes Cluster (ID: 7249) - Cluster overview

Custom Dashboards

1. APRS Application Overview - Application-specific metrics
2. PostgreSQL Connection Analysis - Detailed connection monitoring

Alerts

Critical Alerts

• PostgreSQLConnectionsCritical: > 95% connections used
• PostgreSQLDown: PostgreSQL not responding
• PgBouncerDown: PgBouncer not responding
• APRSPodCrashLooping: Pod restarting frequently

Warning Alerts

• PostgreSQLConnectionsHigh: > 80% connections used
• PostgreSQLIdleTransactionsHigh: > 10 idle transactions
• PostgreSQLLongRunningQueries: Queries > 5 minutes
• APRSHighMemoryUsage: > 90% memory limit
• APRSHighDatabaseConnectionQueueTime: Queue time > 1 second

Troubleshooting

Check Metrics Collection

# Verify all exporters are running
kubectl get pods -n aprs | grep exporter

# Check ServiceMonitors
kubectl get servicemonitor -n aprs

# Verify Prometheus targets
kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090
# Visit http://localhost:9090/targets

Common Issues

1. Metrics not appearing:

   • Check if exporter pods are running
   • Verify ServiceMonitor labels match Prometheus selector
   • Check Prometheus targets page for errors

2. Dashboards empty:

   • Ensure data source is set to "Prometheus"
   • Check metric names in queries
   • Verify time range selection

3. Alerts not firing:

   • Check PrometheusRule is loaded
   • Verify alert expressions in Prometheus
   • Check AlertManager configuration

Adding New Metrics

Application Metrics

Add to /lib/aprsme_web/telemetry.ex:

counter("your_metric_name", 
  description: "Description",
  tags: [:tag1, :tag2]
)

Custom Queries

Add to postgres-exporter ConfigMap:

your_custom_metric:
  query: "SELECT ..."
  metrics:
    - column_name:
        usage: "GAUGE"
        description: "Description"

Performance Tuning

Prometheus Storage

Current settings:

• Retention: 30 days
• Storage: 20Gi

Adjust in kube-prometheus-stack-values.yaml if needed.

Scrape Intervals

Default: 30 seconds Adjust in ServiceMonitors for specific endpoints.

Resource Usage

Monitor Prometheus and Grafana resource usage:

kubectl top pods -n monitoring

Backup and Recovery

Grafana Dashboards

Export dashboards as JSON from Grafana UI for backup.

Prometheus Data

Data is stored in PVC. For backup:

kubectl exec -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -- \
  tar czf /tmp/prometheus-backup.tar.gz /prometheus
kubectl cp monitoring/prometheus-kube-prometheus-stack-prometheus-0:/tmp/prometheus-backup.tar.gz \
  ./prometheus-backup.tar.gz

Integration with CI/CD

Add to your deployment pipeline:

# Check if metrics endpoint is healthy
curl -f http://your-app/metrics || exit 1

# Query Prometheus for deployment validation
curl -G http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=up{job="aprs-app"}' | jq '.data.result[0].value[1]'

Security Considerations

1. Change default passwords in Grafana
2. Configure network policies to restrict access
3. Enable TLS for external access
4. Set up authentication for Prometheus and AlertManager
5. Limit metric cardinality to prevent resource exhaustion

Next Steps

1. Configure AlertManager to send notifications (email, Slack, etc.)
2. Set up long-term storage with Thanos or Cortex
3. Implement SLO/SLI dashboards
4. Add distributed tracing with Jaeger
5. Set up log aggregation with Loki