This commit is contained in:
Graham McIntire 2025-07-30 14:35:11 -05:00
parent 8a5f60ea42
commit 18b85ad512
No known key found for this signature in database
6 changed files with 0 additions and 161 deletions

View file

@ -1,69 +0,0 @@
# Database Migration Strategy for Kubernetes
This document describes how database migrations are handled in the Kubernetes deployment to ensure only one node runs migrations at a time.
## Overview
In a distributed environment with multiple pods, we need to ensure that database migrations only run once, not multiple times concurrently. We use two strategies:
1. **PostgreSQL Advisory Locks** - For runtime migration coordination
2. **Init Containers** - For deployment-time migrations (recommended)
## Implementation
### 1. PostgreSQL Advisory Locks
The `Aprsme.MigrationLock` module uses PostgreSQL advisory locks to ensure only one node can run migrations at a time:
- Uses `pg_try_advisory_lock()` to acquire a non-blocking lock
- Other nodes wait for the lock to be released
- Automatically releases the lock after migrations complete
### 2. Init Container (Recommended)
The StatefulSet can be configured with an init container that runs migrations before the main pods start:
```bash
# Apply the init container patch
kubectl patch statefulset aprs -n aprs --patch-file k8s/statefulset-init-container-patch.yaml
```
This approach:
- Runs migrations sequentially before any pods start
- Prevents race conditions
- Makes migration failures visible in pod events
### 3. Auto-Migration Disabled in Cluster Mode
When `CLUSTER_ENABLED=true`, automatic migrations on startup are disabled to prevent race conditions.
## Manual Migration
To run migrations manually:
```bash
# Run on the first pod
kubectl exec -it aprs-0 -n aprs -- /app/bin/migrate
# Or create a one-off job
kubectl run migrate-job --rm -it --image=ghcr.io/aprsme/aprs.me:latest \
--env="DATABASE_URL=$DATABASE_URL" \
--env="SECRET_KEY_BASE=$SECRET_KEY_BASE" \
--env="MIX_ENV=prod" \
--env="SKIP_DB_CREATE=true" \
--restart=Never \
-- /app/bin/migrate
```
## Configuration
Environment variables:
- `CLUSTER_ENABLED=true` - Enables distributed locking
- `SKIP_DB_CREATE=true` - Skips database creation (for PgBouncer)
## Best Practices
1. **Use Init Containers** for production deployments
2. **Test migrations** in a staging environment first
3. **Monitor migration logs** during deployments
4. **Have rollback plan** ready with `mix ecto.rollback`

View file

@ -1,11 +0,0 @@
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: aprs-pdb
namespace: aprs
spec:
minAvailable: 1
selector:
matchLabels:
app: aprs
unhealthyPodEvictionPolicy: IfHealthyBudget

View file

@ -1,13 +0,0 @@
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: aprs
namespace: aprs
spec:
template:
spec:
containers:
- name: aprs
env:
- name: DEPLOYED_AT
value: "TIMESTAMP_PLACEHOLDER"

View file

@ -1,37 +0,0 @@
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: aprs
namespace: aprs
spec:
template:
spec:
initContainers:
- name: migrate
image: ghcr.io/aprsme/aprs.me:latest
imagePullPolicy: Always
command: ["/app/bin/migrate"]
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
key: database-url-pgbouncer
name: aprs-secret
- name: SECRET_KEY_BASE
valueFrom:
secretKeyRef:
key: secret-key-base
name: aprs-secret
- name: MIX_ENV
value: prod
- name: SKIP_DB_CREATE
value: "true"
- name: CLUSTER_ENABLED
value: "false" # Disable clustering for migration runner
resources:
limits:
cpu: 200m
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi

View file

@ -1,12 +0,0 @@
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: aprs
namespace: aprs
spec:
updateStrategy:
type: RollingUpdate
rollingUpdate:
partition: 0
maxUnavailable: 0 # This ensures pods are only terminated after new ones are ready
minReadySeconds: 30 # Wait 30 seconds after a pod is ready before considering the update successful

View file

@ -1,19 +0,0 @@
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: aprs
namespace: aprs
spec:
template:
spec:
containers:
- name: aprs
startupProbe:
httpGet:
path: /health
port: 4000
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 12 # Gives 60 seconds total to start up