Graham McIntire
413cc92cab
Partition list for hrrr_profiles + hrdps_profiles was hand-edited in priv/repo/structure.sql; if no human extended it, writes for the next quarter would silently fail at the worker level (Rust hrrr-point-worker marks tasks failed; the calibration pipeline silently joins NULLs). Adds: * Microwaveprop.PartitionManager — runtime DDL helper. Quarter math via a single integer index (year*4 + quarter) for rollover safety. Coverage check via pg_inherits before CREATE so a quarterly target inside an existing wider partition (some 2027 ones are half-year) is treated as :exists rather than colliding. * Microwaveprop.Workers.PartitionMaintenanceWorker — daily cron at 03:15 UTC, 4-quarter lookahead. Idempotent; cheap when nothing's missing. lookback_quarters arg lets ops widen for catch-up. Tests cover the create + idempotent re-run + name format + year rollover paths via a synthetic parent created inside the sandbox transaction (no DDL leaks into the shared test DB).
51 lines
1.8 KiB
Elixir
51 lines
1.8 KiB
Elixir
defmodule Microwaveprop.Workers.PartitionMaintenanceWorker do
|
|
@moduledoc """
|
|
Daily Oban job that keeps the time-partitioned tables ahead of the
|
|
current write window.
|
|
|
|
Calls `Microwaveprop.PartitionManager.ensure_quarterly_partitions/1`
|
|
for the default parents (`hrrr_profiles`, `hrdps_profiles`) with a
|
|
4-quarter lookahead. Idempotent — the partition manager uses
|
|
`CREATE TABLE IF NOT EXISTS`, so a fire that finds nothing missing
|
|
is a no-op.
|
|
|
|
Lookahead choice: 4 quarters ≈ 1 year of runway. The cron runs daily,
|
|
so a single missed fire (or a week of missed fires after an outage)
|
|
still leaves >9 months of partition coverage before writes would hit
|
|
a missing quarter. The safety margin is intentionally large — the
|
|
cost of an extra empty partition is one zero-row table, while the
|
|
cost of a missing partition is silently failed inserts.
|
|
"""
|
|
use Oban.Worker,
|
|
queue: :admin,
|
|
max_attempts: 3,
|
|
unique: [period: 3600, states: [:available, :scheduled, :executing, :retryable]]
|
|
|
|
alias Microwaveprop.PartitionManager
|
|
|
|
require Logger
|
|
|
|
@lookahead_quarters 4
|
|
|
|
@impl Oban.Worker
|
|
def perform(%Oban.Job{args: args}) do
|
|
lookahead = lookahead(args)
|
|
results = PartitionManager.ensure_quarterly_partitions(lookahead)
|
|
|
|
created = Enum.filter(results, fn {_, _, status} -> status == :created end)
|
|
|
|
case created do
|
|
[] ->
|
|
Logger.info("PartitionMaintenanceWorker: all #{length(results)} partitions already exist")
|
|
|
|
list ->
|
|
names = Enum.map_join(list, ", ", fn {_parent, name, _} -> name end)
|
|
Logger.info("PartitionMaintenanceWorker: created partitions: #{names}")
|
|
end
|
|
|
|
:ok
|
|
end
|
|
|
|
defp lookahead(%{"lookahead_quarters" => n}) when is_integer(n) and n >= 0, do: n
|
|
defp lookahead(_), do: @lookahead_quarters
|
|
end
|