From 96995bd5aaf49a5f8983452606de5d9a512bd63d Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Fri, 20 Mar 2026 11:09:37 -0500 Subject: [PATCH] Increase Oban Pruner throughput to prevent oban_jobs table bloat (#95) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Default Pruner settings (10k rows/30s = 333/s) can't keep up with job completion rate (~600+/s from SNMP polling + sync workers), causing oban_jobs to accumulate millions of completed rows. The bloated table (47M rows, 29GB) makes Oban.Met.Reporter COUNT queries slow, blocking on IO and holding DB connections until they timeout at 15s — the actual root cause of the "ssl recv: closed" connection pool exhaustion. Increase to 50k rows/5s (10,000/s) to keep the table small. Reviewed-on: https://git.mcintire.me/graham/towerops-web/pulls/95 --- config/runtime.exs | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/config/runtime.exs b/config/runtime.exs index 7dcdaa0e..ce0f240d 100644 --- a/config/runtime.exs +++ b/config/runtime.exs @@ -254,8 +254,11 @@ if config_env() == :prod do # Sync device usage to Stripe daily at 3 AM UTC {"0 3 * * *", Towerops.Workers.BillingSyncWorker} ]}, - # Automatically delete completed jobs after 60 seconds - {Oban.Plugins.Pruner, max_age: 60}, + # Automatically delete completed jobs after 60 seconds. + # Default limit of 10k/30s (333/s) can't keep up with job throughput (~600+/s), + # causing oban_jobs to bloat to millions of rows and slow all queries. + # 50k/5s (10,000/s) keeps the table small. + {Oban.Plugins.Pruner, max_age: 60, limit: 50_000, interval: to_timeout(second: 5)}, # Rescue orphaned jobs (when node crashes) {Oban.Plugins.Reindexer, schedule: "0 2 * * *"}, # Rescue jobs stuck in executing state (mark as cancelled after 10 minutes)