cleanup
This commit is contained in:
parent
8a5f60ea42
commit
18b85ad512
6 changed files with 0 additions and 161 deletions
|
|
@ -1,69 +0,0 @@
|
|||
# Database Migration Strategy for Kubernetes
|
||||
|
||||
This document describes how database migrations are handled in the Kubernetes deployment to ensure only one node runs migrations at a time.
|
||||
|
||||
## Overview
|
||||
|
||||
In a distributed environment with multiple pods, we need to ensure that database migrations only run once, not multiple times concurrently. We use two strategies:
|
||||
|
||||
1. **PostgreSQL Advisory Locks** - For runtime migration coordination
|
||||
2. **Init Containers** - For deployment-time migrations (recommended)
|
||||
|
||||
## Implementation
|
||||
|
||||
### 1. PostgreSQL Advisory Locks
|
||||
|
||||
The `Aprsme.MigrationLock` module uses PostgreSQL advisory locks to ensure only one node can run migrations at a time:
|
||||
|
||||
- Uses `pg_try_advisory_lock()` to acquire a non-blocking lock
|
||||
- Other nodes wait for the lock to be released
|
||||
- Automatically releases the lock after migrations complete
|
||||
|
||||
### 2. Init Container (Recommended)
|
||||
|
||||
The StatefulSet can be configured with an init container that runs migrations before the main pods start:
|
||||
|
||||
```bash
|
||||
# Apply the init container patch
|
||||
kubectl patch statefulset aprs -n aprs --patch-file k8s/statefulset-init-container-patch.yaml
|
||||
```
|
||||
|
||||
This approach:
|
||||
- Runs migrations sequentially before any pods start
|
||||
- Prevents race conditions
|
||||
- Makes migration failures visible in pod events
|
||||
|
||||
### 3. Auto-Migration Disabled in Cluster Mode
|
||||
|
||||
When `CLUSTER_ENABLED=true`, automatic migrations on startup are disabled to prevent race conditions.
|
||||
|
||||
## Manual Migration
|
||||
|
||||
To run migrations manually:
|
||||
|
||||
```bash
|
||||
# Run on the first pod
|
||||
kubectl exec -it aprs-0 -n aprs -- /app/bin/migrate
|
||||
|
||||
# Or create a one-off job
|
||||
kubectl run migrate-job --rm -it --image=ghcr.io/aprsme/aprs.me:latest \
|
||||
--env="DATABASE_URL=$DATABASE_URL" \
|
||||
--env="SECRET_KEY_BASE=$SECRET_KEY_BASE" \
|
||||
--env="MIX_ENV=prod" \
|
||||
--env="SKIP_DB_CREATE=true" \
|
||||
--restart=Never \
|
||||
-- /app/bin/migrate
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variables:
|
||||
- `CLUSTER_ENABLED=true` - Enables distributed locking
|
||||
- `SKIP_DB_CREATE=true` - Skips database creation (for PgBouncer)
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use Init Containers** for production deployments
|
||||
2. **Test migrations** in a staging environment first
|
||||
3. **Monitor migration logs** during deployments
|
||||
4. **Have rollback plan** ready with `mix ecto.rollback`
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: aprs-pdb
|
||||
namespace: aprs
|
||||
spec:
|
||||
minAvailable: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: aprs
|
||||
unhealthyPodEvictionPolicy: IfHealthyBudget
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: aprs
|
||||
namespace: aprs
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: aprs
|
||||
env:
|
||||
- name: DEPLOYED_AT
|
||||
value: "TIMESTAMP_PLACEHOLDER"
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: aprs
|
||||
namespace: aprs
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
initContainers:
|
||||
- name: migrate
|
||||
image: ghcr.io/aprsme/aprs.me:latest
|
||||
imagePullPolicy: Always
|
||||
command: ["/app/bin/migrate"]
|
||||
env:
|
||||
- name: DATABASE_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: database-url-pgbouncer
|
||||
name: aprs-secret
|
||||
- name: SECRET_KEY_BASE
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: secret-key-base
|
||||
name: aprs-secret
|
||||
- name: MIX_ENV
|
||||
value: prod
|
||||
- name: SKIP_DB_CREATE
|
||||
value: "true"
|
||||
- name: CLUSTER_ENABLED
|
||||
value: "false" # Disable clustering for migration runner
|
||||
resources:
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 256Mi
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: aprs
|
||||
namespace: aprs
|
||||
spec:
|
||||
updateStrategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
partition: 0
|
||||
maxUnavailable: 0 # This ensures pods are only terminated after new ones are ready
|
||||
minReadySeconds: 30 # Wait 30 seconds after a pod is ready before considering the update successful
|
||||
|
|
@ -1,19 +0,0 @@
|
|||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: aprs
|
||||
namespace: aprs
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: aprs
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 4000
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 3
|
||||
successThreshold: 1
|
||||
failureThreshold: 12 # Gives 60 seconds total to start up
|
||||
Loading…
Add table
Reference in a new issue