Graham McIntire
664f1353db
Pruning used to only run at the end of a successful PropagationGridWorker pass, so a stretch of failed compute jobs (k8s OOM kills, SIGTERM) stopped prune from running and let the table accumulate ~5h of stale rows. A dedicated PropagationPruneWorker now runs every 15 minutes on its own Oban cron, and PropagationGridWorker also calls prune_old_scores at the start of each run as a second safety net. Bumped the delete timeout from 2m to 5m so the first catch-up pass has enough headroom.
24 lines
794 B
Elixir
24 lines
794 B
Elixir
defmodule Microwaveprop.Workers.PropagationPruneWorker do
|
|
@moduledoc """
|
|
Standalone worker that prunes stale rows from `propagation_scores`.
|
|
|
|
Pruning used to be tacked on to the end of `PropagationGridWorker.perform/1`,
|
|
which meant it only ran after a successful grid compute. When the compute
|
|
worker was being killed mid-run (OOM / SIGTERM), prune never ran and the
|
|
table grew unbounded. Running prune on its own cron keeps the table healthy
|
|
regardless of the compute worker's state.
|
|
"""
|
|
|
|
use Oban.Worker,
|
|
queue: :propagation,
|
|
max_attempts: 3,
|
|
unique: [period: 300, states: [:available, :scheduled, :executing, :retryable]]
|
|
|
|
alias Microwaveprop.Propagation
|
|
|
|
@impl Oban.Worker
|
|
def perform(%Oban.Job{}) do
|
|
Propagation.prune_old_scores()
|
|
:ok
|
|
end
|
|
end
|