Graham McIntire
95976b6b90
Four optimisations targeting the 11.4s chain-step critical path: 1. Drop legacy .mp.gz dual-write (~2.5-2.8s, 24% of step) The Elixir reader prefers .sgrid; the chunked gzip+msgpack write was pure overhead on the Rust write path. Removed scalar_future from both run_chain_step and run_analysis_step. 2. Parallelize wgrib2 decodes (~0.5-1s) sfc and prs decode ran serially (await sfc, then await prs). Both now spawn concurrently on the blocking pool. 3. Overlap sfc decode with prs fetch (~1-2s) sfc (9 messages) resolves before prs (39 messages). Start the sfc wgrib2 subprocess as soon as the sfc blob lands instead of waiting for both blobs. The sfc decode hides behind the remaining prs fetch. 4. Cap grib range retries from 5 to 3 2^n exponential backoff meant 1+2+4+8+16 = 31s worst-case per failed range. Three retries caps at 1+2+4 = 7s. 404s already fail fast. Also: instrument .sgrid write (observe_stage) so it appears in the prop_grid_rs_stage_duration_seconds histogram alongside fetch/decode/derive. Expected mean chain step: ~6-7s (down from 11.4s). |
||
|---|---|---|
| .. | ||
| src | ||
| tests | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||
| Dockerfile | ||