4ac606e682
Close 6 cleanup gaps on the shared /data NFS mount:
- HRDPS .hrdps.prop score files: parse_valid_time regex now matches
the .hrdps.prop extension, so these files are found and pruned
- HRDPS scalar dirs (weather_scalars/{iso}.hrdps/): strip .hrdps suffix
before DateTime.from_iso8601 in list_valid_time_dirs
- ScalarFile: prune_older_than was defined but never called from
Propagation.prune_old_scores - now called alongside Scores/Profiles
- Orphan .tmp.* files: sweep all three base dirs (scores/profiles/scalars)
for .tmp.* files left by crashed atomic-write processes
- Building cache (/data/buildings/): add MsFootprints.prune_older_than
with 30-day mtime cutoff
- Canopy cache (/data/canopy/ + staging/): add Canopy.prune_older_than
with 30-day cutoff, cleanup empty staging dir
All cleanup is cutoff-based (older than X time) so if the prune worker
is missed for any period it catches up fully on the next run.
PropagationPruneWorker updated to call all new cleanup functions.
45 lines
1.5 KiB
Elixir
45 lines
1.5 KiB
Elixir
defmodule Microwaveprop.Workers.PropagationPruneWorker do
|
|
@moduledoc """
|
|
Standalone worker that prunes stale rows from `propagation_scores`.
|
|
|
|
Pruning used to be tacked on to the end of `PropagationGridWorker.perform/1`,
|
|
which meant it only ran after a successful grid compute. When the compute
|
|
worker was being killed mid-run (OOM / SIGTERM), prune never ran and the
|
|
table grew unbounded. Running prune on its own cron keeps the table healthy
|
|
regardless of the compute worker's state.
|
|
"""
|
|
|
|
use Oban.Pro.Worker,
|
|
queue: :propagation,
|
|
# Lower priority than PropagationGridWorker so the hourly chain
|
|
# never waits behind a pending prune on the shared :propagation
|
|
# queue (2 slots).
|
|
priority: 5,
|
|
max_attempts: 3,
|
|
unique: [period: 300, states: :incomplete]
|
|
|
|
alias Microwaveprop.Buildings.MsFootprints
|
|
alias Microwaveprop.Canopy
|
|
alias Microwaveprop.Propagation
|
|
|
|
require Logger
|
|
|
|
@impl Oban.Pro.Worker
|
|
def process(%Oban.Job{}) do
|
|
Propagation.prune_old_scores()
|
|
|
|
cutoff_30d = DateTime.add(DateTime.utc_now(), -30, :day)
|
|
buildings_deleted = MsFootprints.prune_older_than(cutoff_30d)
|
|
canopy_deleted = Canopy.prune_older_than(cutoff_30d)
|
|
|
|
if buildings_deleted > 0 do
|
|
Logger.info("MsFootprints: pruned #{buildings_deleted} cached tile files older than #{cutoff_30d}")
|
|
end
|
|
|
|
if canopy_deleted > 0 do
|
|
Logger.info("Canopy: pruned #{canopy_deleted} cached tile/staging files older than #{cutoff_30d}")
|
|
end
|
|
|
|
:ok
|
|
end
|
|
end
|