Graham McIntire
e9a38623d8
Batched from a system-wide review pass.
**Rate-limit + transient-error log hygiene**
- WeatherFetchWorker: classify IEM HTTP 429 as {:snooze, 300} instead
of {:error}. Stops Oban.PerformError stack-trace spam on every
routine rate-limit during backfill, keeps the retry semantics.
- IonosphereFetchWorker: compress GIRO TLS :unknown_ca error from a
~400-char inspect blob to 'TLS unknown_ca (CA bundle missing)'.
- FreshnessMonitor: log only when an enqueue actually lands (the
Oban unique conflict case was silently dropping jobs but still
producing 'enqueuing grid worker' info lines every 5 minutes).
**Perf**
- Rust grid_level_keys(): pre-format 'TMP:{p} mb' / DPT / HGT keys
once via OnceLock instead of format!()'ing per-cell. Removes ~3.6M
String allocations per f01..f18 chain step across pipeline.rs,
hrrr_points.rs. Same for the wgrib2 :(TMP|DPT|HGT): pattern in
hrrr_points.process_batch (sfc_pattern / prs_pattern OnceLock).
- MapLive preload_forecast: Task.async_stream with max_concurrency=4
replaces the 18-wide Enum.map serial walk. Forecast cache warms
~4× faster after a band change, with ordering preserved.
- grid_tasks: partial composite index on (run_time DESC,
forecast_hour ASC) WHERE status='queued' AND kind='forecast',
matching the Rust claim query's ORDER BY. Drops the old
status-only partial that forced a sort per claim.
**Correctness**
- ContactImportWorker: add Oban unique:[keys: [:import_run_id,
:offset]]. Was missing on a worker whose perform() does a
non-idempotent atomic counter increment — a retry would double-
count imported rows.
- CommonVolumeRadarWorker: x_min..x_max default step is -1 when
x_min > x_max, triggering a runtime deprecation warning. Force
Range.new(_, _, 1) explicitly.
**Cleanup**
- Drop unused tmp_dir parameter from hrrr_point_worker + its
process_batch signature.
32 lines
1.3 KiB
Elixir
32 lines
1.3 KiB
Elixir
defmodule Microwaveprop.Repo.Migrations.TightenGridTasksClaimIndex do
|
|
use Ecto.Migration
|
|
|
|
# CREATE INDEX CONCURRENTLY requires not being inside a transaction,
|
|
# and not grabbing Ecto's migration-level advisory lock (it would
|
|
# serialize with the DDL command we're about to issue).
|
|
@disable_ddl_transaction true
|
|
@disable_migration_lock true
|
|
|
|
@old_idx :grid_tasks_queued_idx
|
|
|
|
# Rust's `claim_next` (rust/prop_grid_rs/src/db.rs) orders by
|
|
# `run_time DESC, forecast_hour ASC` inside the claimable subset
|
|
# (`status='queued' AND forecast_hour > 0 AND kind='forecast'`).
|
|
# The old partial index only indexed `status` — Postgres then had to
|
|
# sort the whole queued tail on every claim. The table is small today
|
|
# (~19 rows/hour, pruned), but ordering inside the index is free to
|
|
# add and future-proofs the worker if we ever expand forecast coverage.
|
|
def change do
|
|
execute(
|
|
"""
|
|
CREATE INDEX CONCURRENTLY IF NOT EXISTS grid_tasks_forecast_claim_idx
|
|
ON grid_tasks (run_time DESC, forecast_hour ASC)
|
|
WHERE status = 'queued' AND forecast_hour > 0 AND kind = 'forecast'
|
|
""",
|
|
"DROP INDEX IF EXISTS grid_tasks_forecast_claim_idx"
|
|
)
|
|
|
|
# Redundant once the covering partial index exists.
|
|
drop_if_exists(index(:grid_tasks, [:status], name: @old_idx))
|
|
end
|
|
end
|