fix: prevent Oban index bloat by reindexing primary key and scheduling index (#165)
Severe index bloat discovered on oban_jobs: - oban_jobs_pkey: 82.2 bloat ratio, 3.2GB waste - oban_jobs_state_queue_priority_scheduled_at_id_index: 216.8 bloat, 7.9GB waste Root cause: High DELETE churn from aggressive Pruner config (10k jobs/sec) causes PostgreSQL VACUUM to leave dead index entries. VACUUM only marks space as reusable but cannot shrink indexes. The Oban Reindexer plugin was only reindexing args_index and meta_index by default, missing the primary key and main scheduling index. Solution: Explicitly configure Reindexer to include all high-churn indexes: - oban_jobs_pkey (primary key, used by all queries) - oban_jobs_state_queue_priority_scheduled_at_id_index (job scheduling) - oban_jobs_args_index (default, GIN index) - oban_jobs_meta_index (default, GIN index) Daily REINDEX CONCURRENTLY at 2 AM will prevent future bloat accumulation. Manual immediate reindex required to reclaim 11GB: REINDEX INDEX CONCURRENTLY oban_jobs_pkey; REINDEX INDEX CONCURRENTLY oban_jobs_state_queue_priority_scheduled_at_id_index; Reviewed-on: graham/towerops-web#165
This commit is contained in:
parent
94fd8780de
commit
68354af021
1 changed files with 13 additions and 2 deletions
|
|
@ -261,8 +261,19 @@ if config_env() == :prod do
|
|||
# causing oban_jobs to bloat to millions of rows and slow all queries.
|
||||
# 50k/5s (10,000/s) keeps the table small.
|
||||
{Oban.Plugins.Pruner, max_age: 60, limit: 50_000, interval: to_timeout(second: 5)},
|
||||
# Rescue orphaned jobs (when node crashes)
|
||||
{Oban.Plugins.Reindexer, schedule: "0 2 * * *"},
|
||||
# Rescue orphaned jobs (when node crashes) and prevent index bloat
|
||||
# High DELETE churn (10k jobs/sec pruned) causes severe index bloat on:
|
||||
# - Primary key (82.2 bloat, 3.2GB waste)
|
||||
# - Scheduling index (216.8 bloat, 7.9GB waste)
|
||||
# REINDEX CONCURRENTLY reclaims space without blocking operations.
|
||||
{Oban.Plugins.Reindexer,
|
||||
schedule: "0 2 * * *",
|
||||
indexes: [
|
||||
"oban_jobs_pkey",
|
||||
"oban_jobs_state_queue_priority_scheduled_at_id_index",
|
||||
"oban_jobs_args_index",
|
||||
"oban_jobs_meta_index"
|
||||
]},
|
||||
# Rescue jobs stuck in executing state (mark as cancelled after 10 minutes)
|
||||
{Oban.Plugins.Lifeline, rescue_after: to_timeout(minute: 10)}
|
||||
]
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue