Graham McIntire
46c2d84e5c
Stages 4-5 of the HRDPS plan. Lands the Rust forecast-hour pipeline for source='hrdps' grid_tasks rows. What ships: - `hrdps_fetcher.rs` — sibling of fetcher.rs for ECCC's MSC Datamart shape: date-prefixed URLs, one variable per GRIB2 file, concurrent fetch + byte-concat into a single multi-record blob. wgrib2 reads multi-record files natively so the downstream decoder path is unchanged. Mirrors lib/microwaveprop/weather/hrdps_client.ex. - `grid::hrdps_only_points()` + `grid::hrdps_grid_spec()` — Canadian cells (49-60°N, -141 to -52°W) minus the HRRR overlap. Disjoint from conus_points by construction. - `db::TaskSource` enum + `source` column on ClaimedTask. claim_next surfaces source for forecast lane dispatch; claim_next_analysis filters to source='hrrr' only (HRDPS analysis path is not yet ported — would need native-duct + NEXRAD + commercial merges that Rust doesn't have for HRDPS). - `pipeline::run_chain_step_hrdps` — fetch combined blob, decode with HRDPS grid spec, drop HRRR-overlap cells via a HashSet of hrdps_only_points, back-fill DPT from DEPR (HRDPS publishes DEPR as primary), score every band, write `<band>/<iso>.hrdps.prop`. Skips per-valid_time profile + scalar artifacts since those would clobber HRRR's at the same valid_time — surfacing HRDPS on /weather is a separate stage. - worker.rs dispatches forecast lane on (kind, source) — Hrrr → run_chain_step, Hrdps → run_chain_step_hrdps. Analysis stays HRRR-only. Worker constructs an HrdpsClient alongside HrrrClient. - `scores_file::write_atomic_hrdps` writes to `.hrdps.prop` with the Canadian grid spec via the new encode_with_spec. Coexists with the HRRR `.prop` file for the same (band, valid_time). 163 unit tests pass; backfill_dpt_from_depr handles surface and pressure-level levels with no double-fill when DPT exists.
54 lines
2.1 KiB
Rust
54 lines
2.1 KiB
Rust
//! Rust extraction of the Microwaveprop HRRR propagation pipeline.
|
|
//!
|
|
//! CI retrigger 2026-04-24: Forgejo registry lost the platform-manifest
|
|
//! blob for tag main-1776894678-93b526c, leaving prop-grid-rs stuck in
|
|
//! ImagePullBackOff. Touching this file reruns build-grid-rs.yaml so a
|
|
//! fresh tag is pushed.
|
|
//!
|
|
//! Each module mirrors the Elixir module it ports from so golden-fixture
|
|
//! tests can compare behavior 1:1. Module boundaries chosen so failures
|
|
//! during TDD fall into the smallest possible blast radius.
|
|
//!
|
|
//! Post-Phase-3: f00 analysis and f01..f18 forecasts both live here.
|
|
//! Port targets:
|
|
//! band_config ← lib/microwaveprop/propagation/band_config.ex
|
|
//! region ← lib/microwaveprop/propagation/region.ex
|
|
//! scorer ← lib/microwaveprop/propagation/scorer.ex
|
|
//! grid ← lib/microwaveprop/propagation/grid.ex
|
|
//! decoder ← lib/microwaveprop/weather/grib2/wgrib2.ex
|
|
//! fetcher ← lib/microwaveprop/weather/hrrr_client.ex
|
|
//! scores_file ← lib/microwaveprop/propagation/scores_file.ex
|
|
//! db — new; grid_tasks claim/complete + NOTIFY propagation_ready
|
|
//!
|
|
//! Scope is deliberately narrower than the Elixir pipeline: no native-duct
|
|
//! merge, no NEXRAD, no commercial-link boost. Those are f00-only in
|
|
//! production and Elixir keeps handling f00.
|
|
|
|
// jemalloc for all non-MSVC targets (Linux release builds land here).
|
|
// Set as the global allocator at the library root so every binary in
|
|
// the crate (worker, hrrr_point_worker) inherits it. See Cargo.toml
|
|
// for the rationale behind switching off glibc malloc.
|
|
#[cfg(not(target_env = "msvc"))]
|
|
#[global_allocator]
|
|
static GLOBAL: tikv_jemallocator::Jemalloc = tikv_jemallocator::Jemalloc;
|
|
|
|
pub mod band_config;
|
|
pub mod commercial;
|
|
pub mod db;
|
|
pub mod decoder;
|
|
pub mod duct;
|
|
pub mod fetcher;
|
|
pub mod grid;
|
|
pub mod hrdps_fetcher;
|
|
pub mod hrrr_points;
|
|
pub mod metrics;
|
|
pub mod native_duct;
|
|
pub mod nexrad;
|
|
pub mod pipeline;
|
|
pub mod profiles_file;
|
|
pub mod region;
|
|
pub mod scorer;
|
|
pub mod scores_file;
|
|
pub mod sounding_params;
|
|
pub mod telemetry;
|
|
pub mod weather_scalar_file;
|