Graham McIntire
690a3523e2
First step of the disk-backed scores migration from the DuckDB plan doc. Ended up shipping the raw-binary variant instead of Parquet because the data is disposable after ~2h — the ecosystem benefits of Parquet only pay off for long-lived datasets, and the binary path has zero new dependencies. Microwaveprop.Propagation.ScoresFile writes one file per (band_mhz, valid_time) tuple under the configured scores dir, default /data/scores in prod and priv/dev_scores in dev. Layout is a 33-byte header plus a dense n_rows × n_cols uint8 array (255 is the no-data sentinel). The whole CONUS grid serializes to ~93 KB per band, and writes use the temp-then-rename pattern so NFSv4 concurrent readers never see a partial file. Propagation.replace_scores/2 now materializes the score stream once and dual-writes: Postgres on the primary path, then one ScoresFile per band as a best-effort follow-up (any file error is logged but doesn't fail the DB write, so we can verify the file path in prod before cutting readers over). Propagation.prune_old_scores/0 also clears expired score files so the existing 15-minute prune cron covers both storage layers. Dev configuration points at priv/dev_scores/, added to .gitignore. Test configuration points at a per-run tmp directory.
58 lines
1.1 KiB
Text
58 lines
1.1 KiB
Text
# The directory Mix will write compiled artifacts to.
|
|
/_build/
|
|
|
|
# Environment variables (secrets)
|
|
.envrc
|
|
|
|
# If you run "mix test --cover", coverage assets end up here.
|
|
/cover/
|
|
|
|
# The directory Mix downloads your dependencies sources to.
|
|
/deps/
|
|
|
|
# Where 3rd-party dependencies like PHX://priv/static/
|
|
# temporary files for e.g. tests
|
|
/tmp/
|
|
|
|
# Ignore package tarball (built via "mix hex.build").
|
|
microwaveprop-*.tar
|
|
|
|
# Ignore assets that are produced by build tools.
|
|
/priv/static/assets/
|
|
|
|
# Ignore digested assets cache.
|
|
/priv/static/cache_manifest.json
|
|
|
|
# In case you use Node.js/Tailwind standalone binary
|
|
/assets/node_modules/
|
|
|
|
# Ignore database dumps
|
|
*.sql
|
|
*.psql
|
|
*.dump
|
|
|
|
# Dotenv files
|
|
.env
|
|
.env.*
|
|
|
|
# Kubernetes secret manifests (contain plaintext credentials)
|
|
/k8s/secret.yaml
|
|
/k8s/*-secret.yaml
|
|
|
|
# Trained ML model weights — tracked in git for Docker builds
|
|
|
|
# GRIB2 test fixtures (large binary files, downloaded on-demand)
|
|
/test/fixtures/grib2/*.grib2
|
|
|
|
# Dev-only propagation score binaries written by PropagationGridWorker
|
|
/priv/dev_scores/
|
|
|
|
# Dialyzer PLT files
|
|
/priv/plts/
|
|
|
|
# Git worktrees
|
|
/.worktrees/
|
|
|
|
# OS files
|
|
.DS_Store
|
|
Thumbs.db
|