graham/towerops
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Increase Oban Pruner throughput to prevent oban_jobs table bloat (#95)

Browse source 
Default Pruner settings (10k rows/30s = 333/s) can't keep up with job
completion rate (~600+/s from SNMP polling + sync workers), causing
oban_jobs to accumulate millions of completed rows. The bloated table
(47M rows, 29GB) makes Oban.Met.Reporter COUNT queries slow, blocking
on IO and holding DB connections until they timeout at 15s — the actual
root cause of the "ssl recv: closed" connection pool exhaustion.

Increase to 50k rows/5s (10,000/s) to keep the table small.

Reviewed-on: graham/towerops-web#95
This commit is contained in:
Graham McIntire 2026-03-20 11:09:37 -05:00 • committed by graham
parent 071729f65c
commit 96995bd5aa
1 changed files with 5 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
config/runtime.exs
View file

@ -254,8 +254,11 @@ if config_env() == :prod do
# Sync device usage to Stripe daily at 3 AM UTC
{"0 3 * * *", Towerops.Workers.BillingSyncWorker}
]},
# Automatically delete completed jobs after 60 seconds
{Oban.Plugins.Pruner, max_age: 60},
# Automatically delete completed jobs after 60 seconds.
# Default limit of 10k/30s (333/s) can't keep up with job throughput (~600+/s),
# causing oban_jobs to bloat to millions of rows and slow all queries.
# 50k/5s (10,000/s) keeps the table small.
{Oban.Plugins.Pruner, max_age: 60, limit: 50_000, interval: to_timeout(second: 5)},
# Rescue orphaned jobs (when node crashes)
{Oban.Plugins.Reindexer, schedule: "0 2 * * *"},
# Rescue jobs stuck in executing state (mark as cancelled after 10 minutes)