graham/towerops
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix persistent DB connection timeouts in k8s (#93)

Browse source 
- TCP keepalive more aggressive (14s detection vs 30s) so dead
  connections are found before the 15s checkout timeout fires
- Reduce pool size 30→15 per pod (2 pods = 30 total, avoids
  exhaustion during rolling deploys when 4 pods run briefly)
- Add idle_interval: 1s to proactively ping idle connections and
  detect dead SSL sockets before they're handed to real queries
- Simplify SSL config: remove redundant verify_fun and SNI disable
  when verify_none already skips certificate validation

Reviewed-on: graham/towerops-web#93
This commit is contained in:
Graham McIntire 2026-03-20 10:19:43 -05:00 • committed by graham
parent 439c25ba9d
commit b77950aed6
1 changed files with 11 additions and 11 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

22
config/runtime.exs
View file

@ -111,11 +111,7 @@ if config_env() == :prod do
"true" ->
case System.get_env("DATABASE_SSL_VERIFY") do
"false" ->
[
verify: :verify_none,
server_name_indication: :disable,
verify_fun: {fn _, _, _ -> {:valid, :ok} end, :ok}
]
[verify: :verify_none]
_ ->
[verify: :verify_peer]
@ -152,16 +148,16 @@ if config_env() == :prod do
# Default OS keepalive is ~2 hours on Linux, far too slow for k8s where
# load balancers and proxies silently drop idle connections within minutes.
# Linux raw socket options (SOL_TCP=6):
# TCP_KEEPIDLE (4) = 15s - seconds idle before first keepalive probe
# TCP_KEEPINTVL (5) = 5s - seconds between subsequent probes
# TCP_KEEPIDLE (4) = 5s - seconds idle before first keepalive probe
# TCP_KEEPINTVL (5) = 3s - seconds between subsequent probes
# TCP_KEEPCNT (6) = 3 - failed probes before connection is dead
# Total dead-connection detection: 15 + (5 * 3) = 30 seconds
# Total dead-connection detection: 5 + (3 * 3) = 14 seconds
tcp_keepalive_opts =
if System.get_env("KUBERNETES_SERVICE_HOST") do
[
{:keepalive, true},
{:raw, 6, 4, <<15::native-32>>},
{:raw, 6, 5, <<5::native-32>>},
{:raw, 6, 4, <<5::native-32>>},
{:raw, 6, 5, <<3::native-32>>},
{:raw, 6, 6, <<3::native-32>>}
]
else
@ -326,7 +322,11 @@ if config_env() == :prod do
config :towerops, Towerops.Repo,
ssl: ssl_config,
url: database_url,
pool_size: String.to_integer(System.get_env("POOL_SIZE") || "30"),
pool_size: String.to_integer(System.get_env("POOL_SIZE") || "15"),
# Proactively ping idle connections every 1s to detect dead SSL connections
# before they're handed to a real query. Without this, a dead connection
# sits in the pool until a query checks it out and hangs for the full timeout.
idle_interval: 1_000,
# Detect stale connections faster: if a query takes longer than queue_target (2s)
# on average over queue_interval (5s), DBConnection will disconnect idle connections
# to cycle in fresh ones. Helps recover from "ssl recv: closed" errors.