graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
prop/lib/microwaveprop/weather
History
Graham McIntire db57b3a1c7
Some checks failed
Build prop-grid-rs / Test, build, push (push) Successful in 4m20s
Details
Build and Push / Build and Push Docker Image (push) Failing after 4m35s
Details
prop-pipeline: .sgrid dense scalars, HRDPS rotated-pole decode-once, memory trim 
Item 1 — .sgrid dense binary scalar format replacing chunked gzip+msgpack
- New rust/prop_grid_rs/src/sgrid.rs: magic SGRD, same header layout as .pgrid,
  20-field cell-major f32 body, write_atomic (tmp+rename, NFS-safe)
- New lib/microwaveprop/weather/sgrid.ex: Elixir reader modeled on pgrid.ex
  (pread single-cell reads, bounds-filtered viewport reads, NaN→nil sentinel)
- ScalarFile updated to prefer .sgrid reads, chunked .mp.gz as fallback
- Pipeline writes .sgrid alongside existing chunked format (all three paths)

Item 2 — HRDPS decode-once + rotated-pole index, restore 0.125° resolution
- New rust/prop_grid_rs/src/rotated_pole.rs: CF-convention geographic→rotated
  transform, OnceLock-cached GDS params parsed from wgrib2 -grid, precomputed
  Vec<u32> lookup table mapping target cells to native grid indices
- Native decode path in decoder.rs: wgrib2 -no_header -order we:sn -bin
  (raw f32 dump, ~0.32 s/message) + indexing via lookup table
- HRDPS_STEP: 0.5° → 0.125° (4× finer, ~57k Canadian cells vs ~3.5k)

Item 3 — k8s memory limits: 3Gi → 1.5Gi
  (per-task grid footprint: ~200-400 MB HashMap → ~18 MB dense planes)

Item 4 — CLAUDE.md and profiles_file.ex documentation drift fixed:
  .pgrid primary, .mp.gz legacy, .sgrid added, write_atomic protocol doc,
  cleanup gaps reorganized, 'Only f00 is persisted' corrected to f00..f48
2026-08-01 09:11:59 -05:00
..
grib2 fix: resolve all 281 credo issues across source and test files 2026-07-29 09:04:21 -05:00
frontal_analysis.ex fix(prod-build): silence Nx.* undefined warnings in dev/test-only modules 2026-04-30 12:19:16 -05:00
gefs_client.ex fix: resolve 158/183 credo --strict warnings 2026-06-12 15:47:15 -05:00
gefs_profile.ex feat(gefs): scaffold extended-horizon forecast ingestion 2026-04-18 14:28:44 -05:00
grid.ex fix: resolve all 281 credo issues across source and test files 2026-07-29 09:04:21 -05:00
grid_cache.ex fix: resolve all dialyzer type errors across project (46→0 project errors) 2026-06-08 17:51:13 -05:00
hrdps_client.ex fix: resolve 18 bugs across LiveViews, schemas, tests, and logic 2026-07-29 07:46:54 -05:00
hrdps_profile.ex feat(hrdps): hrdps_profiles partitioned table + HrdpsProfile schema 2026-04-29 17:12:41 -05:00
hrrr.ex fix: resolve 27 security, architecture, test, and performance audit findings 2026-07-27 18:19:37 -05:00
hrrr_client.ex feat: remove skippy HRRR proxy + disk cache, fetch directly from NOAA S3 2026-07-28 14:36:54 -05:00
hrrr_climatology.ex fix(logging): suppress /metrics logs and harden scorer hot path 2026-04-21 10:44:25 -05:00
hrrr_native_client.ex fix: resolve 158/183 credo --strict warnings 2026-06-12 15:47:15 -05:00
hrrr_native_profile.ex Extend skew-T plot to cover the full troposphere 2026-04-14 17:25:58 -05:00
hrrr_point_enqueuer.ex fix: resolve 27 security, architecture, test, and performance audit findings 2026-07-27 18:19:37 -05:00
hrrr_profile.ex Add dialyzer specs and types across the codebase 2026-04-12 08:55:04 -05:00
hrrr_profile_lookup.ex fix(skewt): only fall back to profiles with all 13 pressure levels 2026-04-25 14:19:56 -05:00
iem_client.ex fix: resolve all 281 credo issues across source and test files 2026-07-29 09:04:21 -05:00
iem_rate_limiter.ex perf(weather): adaptive IemRateLimiter + rebatch mix task 2026-04-24 12:57:25 -05:00
iemre.ex fix: resolve 27 security, architecture, test, and performance audit findings 2026-07-27 18:19:37 -05:00
iemre_observation.ex Add dialyzer specs and types across the codebase 2026-04-12 08:55:04 -05:00
map_layers.ex feat(weather): add duct cutoff band map layer 2026-05-15 19:51:24 -05:00
narr.ex fix: resolve 27 security, architecture, test, and performance audit findings 2026-07-27 18:19:37 -05:00
narr_client.ex fix: resolve all 281 credo issues across source and test files 2026-07-29 09:04:21 -05:00
narr_profile.ex Rename ERA5 → NARR across the codebase 2026-04-16 09:22:23 -05:00
ncei_metar_client.ex fix: resolve all compiler warnings except framework-generated phoenix_component_verify 2026-05-29 10:18:52 -05:00
nexrad_cache.ex fix: resolve all dialyzer type errors across project (46→0 project errors) 2026-06-08 17:51:13 -05:00
nexrad_client.ex fix: resolve all compiler warnings except framework-generated phoenix_component_verify 2026-05-29 10:18:52 -05:00
nexrad_observation.ex Add dialyzer specs and types across the codebase 2026-04-12 08:55:04 -05:00
profile_lookup.ex fix: sweep bugs across Python MQTT listener, Elixir PSKR/propagation, and Rust 2026-07-20 19:48:45 -05:00
rebatch_asos.ex perf(weather): asos_day — one job per unique (station, UTC date) 2026-04-24 13:19:20 -05:00
scalar_file.ex prop-pipeline: .sgrid dense scalars, HRDPS rotated-pole decode-once, memory trim 2026-08-01 09:11:59 -05:00
sgrid.ex prop-pipeline: .sgrid dense scalars, HRDPS rotated-pole decode-once, memory trim 2026-08-01 09:11:59 -05:00
skewt_params.ex feat(skewt): add wet-bulb / virtual-T traces and right-edge level rail 2026-04-25 14:24:28 -05:00
solar_client.ex fix: resolve all 281 credo issues across source and test files 2026-07-29 09:04:21 -05:00
solar_index.ex Add dialyzer specs and types across the codebase 2026-04-12 08:55:04 -05:00
sounding.ex Add dialyzer specs and types across the codebase 2026-04-12 08:55:04 -05:00
sounding_params.ex fix: resolve all 281 credo issues across source and test files 2026-07-29 09:04:21 -05:00
soundings.ex fix: resolve 18 bugs across LiveViews, schemas, tests, and logic 2026-07-29 07:46:54 -05:00
station.ex Add dialyzer specs and types across the codebase 2026-04-12 08:55:04 -05:00
surface.ex fix: resolve all 281 credo issues across source and test files 2026-07-29 09:04:21 -05:00
surface_observation.ex Add dialyzer specs and types across the codebase 2026-04-12 08:55:04 -05:00
theta_e.ex Guard against zero specific humidity in theta-e computation 2026-04-11 12:52:32 -05:00
uwyo_sounding_client.ex feat(telemetry): wide instrumentation + bump hrrr to 2 per pod 2026-04-18 16:33:34 -05:00
weather_layers.ex fix: resolve all 281 credo issues across source and test files 2026-07-29 09:04:21 -05:00