graham/prop - Forgejo: Beyond coding. We forge.

graham/prop

Fork 0

Commit graph

Author	SHA1	Message	Date
Graham McIntire	e7a7ae073d	Phase 9.3, 9.4, and Phase 3 NEXRAD pipeline Task 9.3 - Weight recalibration via gradient descent: - Recalibrator module fits logistic regression weights using Nx - Trains on QSO positives vs random baseline negatives - Cross-validates by month, normalizes weights to sum to 1.0 - Mix task: mix recalibrate_scorer --sample 5000 --epochs 2000 Task 9.4 - Side-by-side scorer comparison: - ScorerDiff.compare/3 re-scores grid with old vs new weights - Reports mean diff, regressions, improvements, per-band breakdown - Mix task: mix scorer_diff --new-weights '{...}' Phase 3 - NEXRAD ingestion pipeline: - NexradClient fetches IEM n0q composite PNGs, extracts per-point box statistics (mean/max dBZ, texture variance) - NexradObservation schema with unique (lat, lon, observed_at) - NexradWorker on :nexrad queue for background processing - nexrad_texture backtest feature in Features module - mix nexrad_backfill --limit 200 All tasks added to AdminTaskWorker and Release for production use. 1116 tests, 0 failures.	2026-04-10 12:48:36 -05:00
Graham McIntire	3ea2548114	Phase 8: 5-minute METAR ingestion pipeline NCEI ASOS 5-minute data client (Weather.NceiMetarClient): - fetch/3 pulls per-station monthly .dat files from NCEI C00418 - parse/1 decodes the fixed-width METAR format including precise T-group temperatures (T02110094 → 21.1/9.4°C) - metar_5min_observations table: schema-identical to surface_observations, separate table to avoid mixing cadences Weather.recent_surface_obs/3 prefers 5-min data when available, falls back to the hourly surface_observations table. Data URL: https://www.ncei.noaa.gov/data/automated-surface-observing-system-five-minute/access/YYYY/MM/asos-5min-KXXX-YYYYMM.dat Available back to 1996.	2026-04-10 08:53:54 -05:00

Author

SHA1

Message

Date

Graham McIntire

e7a7ae073d

Phase 9.3, 9.4, and Phase 3 NEXRAD pipeline

Task 9.3 - Weight recalibration via gradient descent:
- Recalibrator module fits logistic regression weights using Nx
- Trains on QSO positives vs random baseline negatives
- Cross-validates by month, normalizes weights to sum to 1.0
- Mix task: mix recalibrate_scorer --sample 5000 --epochs 2000

Task 9.4 - Side-by-side scorer comparison:
- ScorerDiff.compare/3 re-scores grid with old vs new weights
- Reports mean diff, regressions, improvements, per-band breakdown
- Mix task: mix scorer_diff --new-weights '{...}'

Phase 3 - NEXRAD ingestion pipeline:
- NexradClient fetches IEM n0q composite PNGs, extracts per-point
  box statistics (mean/max dBZ, texture variance)
- NexradObservation schema with unique (lat, lon, observed_at)
- NexradWorker on :nexrad queue for background processing
- nexrad_texture backtest feature in Features module
- mix nexrad_backfill --limit 200

All tasks added to AdminTaskWorker and Release for production use.
1116 tests, 0 failures.

2026-04-10 12:48:36 -05:00

Graham McIntire

3ea2548114

Phase 8: 5-minute METAR ingestion pipeline

NCEI ASOS 5-minute data client (Weather.NceiMetarClient):
- fetch/3 pulls per-station monthly .dat files from NCEI C00418
- parse/1 decodes the fixed-width METAR format including precise
  T-group temperatures (T02110094 → 21.1/9.4°C)
- metar_5min_observations table: schema-identical to
  surface_observations, separate table to avoid mixing cadences

Weather.recent_surface_obs/3 prefers 5-min data when available,
falls back to the hourly surface_observations table.

Data URL: https://www.ncei.noaa.gov/data/automated-surface-observing-system-five-minute/access/YYYY/MM/asos-5min-KXXX-YYYYMM.dat
Available back to 1996.

2026-04-10 08:53:54 -05:00

2 commits