Prod paired the Oban Pro Smart engine + unique workers with core
Oban.Plugins.Lifeline, whose rescue has no unique_violation handling. An
orphaned executing CheckExecutorWorker job rescued back to available collides
with its already-scheduled successor on oban_jobs_unique_index (23505),
crashing the plugin every 60s and leaving ~150 orphans stuck. Switch prod to
Oban.Pro.Plugins.DynamicLifeline, which repairs the conflict via
Smart.clear_uniq_violation. Dev keeps core Lifeline (Basic engine).
Add a shallow /health/live endpoint that does not touch db/redis. k8s liveness
and startup probes now target it so a transient dependency outage can't kill or
block boot of a healthy pod; readiness keeps the deep /health to gate the load
balancer. (deployment.yaml probe/replica change pushed separately, after the
image carrying /health/live is live.)
Also drop unused POSTGRES_* keys from the secrets example.
The towerops-db secret in prod has both DATABASE_URL and individual
POSTGRES_* fields; towerops-redis uses HOST/PORT/PASSWORD instead of
a single REDIS_URL. Update the committed template to match so a fresh
copy of k8s/secrets.example.yaml mirrors what's actually in the
cluster.
Replace the single-secret towerops-llm template with one
multi-document YAML covering every Secret the Deployment references:
towerops-secrets, towerops-db, towerops-aws, towerops-redis,
towerops-billing, towerops-llm.
Workflow: copy k8s/secrets.example.yaml to k8s/secrets.yaml (which is
gitignored), fill in real values, kubectl apply. Real values never land
in git.
