json_weights_golden reads priv/algo/band_weights.json via a bare
relative path. That resolves inside `cargo test` at the crate root but
not inside the image build, whose context is rust/prop_grid_rs and
whose WORKDIR is /src — so the test aborted with "JSON file must
exist" and the build never produced an image.
grid-rs CI has been red since 2fd88a94, the commit that added the
test. Production still runs main-1785606663-95976b6 (95976b6b, the
last green build), so every prop-grid-rs change since then — including
the HRDPS rotated-pole decode fix — has silently failed to ship.
Three parts:
* resolve the fixture from CARGO_MANIFEST_DIR so the path no longer
depends on the cwd
* stage the file into the build context in the workflow
* COPY it to /priv in the builder stage, where
CARGO_MANIFEST_DIR/../../priv resolves from /src
ScalarFile.list_valid_time_dirs/1 stripped only a `.hrdps` suffix
before DateTime.from_iso8601/1, so `<iso>.sgrid` and
`<iso>.hrdps.sgrid` never parsed and were skipped by both
prune_older_than/1 and retain_window/2. Since .sgrid is the format the
pipeline actually writes, nothing on the dense tier was ever deleted —
production held 222 files spanning 8 days against a 48h window.
Strip the known suffixes in order, mirroring ProfilesFile's anchored
parser (which documents this exact failure mode for .hrdps.prop).
Orphaned .tmp.* writes still don't parse, so they stay with the tmp
sweep in Propagation.prune_old_scores/0.
Also corrects the CLAUDE.md entry that recorded this as already fixed —
the call site was present, the suffix parsing was not.
Raising the area cap to 4600 put it above the largest area clamp/1 can
emit (45° lat × 100° lon = 4500 sq deg), so :viewport_too_large became
unreachable and the 413 handlers in both cell controllers became dead
code.
Remove the cap, the error branch, the max_viewport_area_sq_deg/0
accessor and both payload_too_large/1 handlers. The clamp is what
bounds request cost: post-clamp nothing can ask for more than the full
grid (~92k cells, ~1.4 MB), which is materialized once per valid_time
and served from cache, so a global viewport costs no more than a fully
zoomed-out map.
Replaces the cap test with one asserting the clamp never lets absurd
input escape the supported extent — the property the guard was
standing in for.
The map sets minZoom: 4, and a z=4 viewport on a wide window clamps to
the full grid extent (45° lat × 100° lon = 4500 sq deg), which exceeded
the 4000 sq deg cap. The request 413'd and the hook dropped it
silently, so zooming all the way out blanked the overlay.
Raise the cap to 4600 — just above the largest area the clamp can
emit. The clamp, not this cap, is what bounds the work: post-clamp no
request can exceed the full grid (~92k cells, ~1.4 MB), which is
already materialized once per valid_time and served from cache. The
cap stays as defense-in-depth against the clamp window being widened
later.
Consequence: :viewport_too_large is now unreachable via clamp/1, so
the tests that pinned global bounds to a 413 are inverted to assert
the clamped extent is served.
loadLayers did `if (!resp.ok) return`, so a 413 or 400 from
/weather/cells left the map blank with no console error and no UI
feedback — indistinguishable from "no data for this forecast hour".
This is reachable in normal use: the hook sets minZoom: 4, and a z=4
viewport on a wide window clamps to ~4500 sq deg, over GridBounds'
4000 sq deg cap, so zooming all the way out silently drops the
overlay. Zooming back in restores it, which makes the failure look
intermittent and untraceable.
Log the status, requested layers and URL so the next report is
diagnosable from the browser console.
duct_field/2 did d["base"] || d["base_m"], but ProfilesFile.read/1
atomizes both keys via its @mp_atom_keys whitelist. String indexing an
atom-keyed map returns nil, so the Duct Base and Duct Strength layers
were always nil on the cold-derive path and in point-detail popups —
despite the comment directly above claiming both shapes were handled.
Normalize keys before lookup, the way duct_min_freq/1 already did, and
share that normalization between the two.
weather_grid_hrdps_at/2 is the only production caller and backs every
/weather/cells?source=hrdps request, but it read the legacy chunked
.mp.gz dirs exclusively. The pipeline writes <iso>.hrdps.sgrid, so the
snap found a valid_time via list_valid_times_hrdps/0 (which does see
.sgrid) and then read zero chunks for it.
Delegate to read_bounds_hrdps/2, which already does the .sgrid-first,
chunk-fallback read.
Every existing test for this function hand-wrote legacy chunk dirs,
which is how the .sgrid migration slipped past. Added coverage against
a real .sgrid artifact, including the mid-migration case where both
formats exist for one valid_time.
Every HRDPS chain step failed at decode with "could not parse nx×ny
from wgrib2 -grid output", so no .hrdps.prop or .hrdps.sgrid artifact
was ever produced and the Canadian half of the weather overlay was
empty.
parse_wgrib2_grid only handled the verbose grid_template=10 shape
(Rotated Lat/lon Grid:, LatFirst:, Di:, South Pole Location:). The
wgrib2 3.8.0 binary in the pipeline image actually emits the compact
grid_template=1 shape for MSC HRDPS files:
rotated lat-lon grid:(2540 x 1290) units 1e-06 ...
lat -12.302501 to 16.700001 by 0.022500
lon 345.178780 to 42.306283 by 0.022500 #points=3276600
south pole lat=-36.088520 lon=245.305142 angle of rot=0.000000
Accept both, keeping the strict-by-default behaviour so a real MSC grid
change still surfaces as an error. Anisotropic grids are rejected
outright because RotatedPoleParams carries a single isotropic step that
geo_to_rotated applies to both axes.
Test uses the verbatim -grid output captured from the production pod.
Replace fragile sed -i s/-O3/-O0/g with DEBUG=1 env var. EXLA's Makefile
natively checks ifdef DEBUG and uses -g (debug symbols) instead of -O3
(heavy optimization), avoiding the g++ OOM kill on CI runners.
Also set EXLA_CPU_ONLY=1 to skip unnecessary nvcc probing.
mix run -e starts the full OTP application which requires hackney for
Swoosh. Instead, put config :exla, :make_args, [-j1] in the temp
config.exs — EXLA reads it at compile time via Application.get_env.
elixir_make builds a sanitized env map for make — MAKEFLAGS and DEBUG
never reach make. EXLA also sets make_args to nproc-2 parallel jobs.
We must patch the Makefile (-O3→-O0) and override make_args via
Application.put_env to actually control compilation behavior.
The EXLA Makefile uses -O3 by default which causes massive memory usage
during C++ compilation. DEBUG=1 replaces -O3 with -g, drastically
reducing peak RAM. Combined with MAKEFLAGS=-j1 to serialize jobs, this
should fit within CI runner memory limits.
prop-base:latest is required by the production Dockerfile's final stage
but was only built when Dockerfile.base changed. The registry can GC
unused images. Remove paths filter so the image is always kept fresh.
prop-ci:latest on the registry may not have the latest Dockerfile.ci
changes when the test step runs. Build the image locally using
--cache-from for speed, then use the local tag.
Logger.put_module_level suppressed the log output that ObanErrorReporterTest
asserts on via capture_log. The worker-level suppression (NarrFetchWorker)
already handles the noise source.
EXLA's cached_make copies libexla.so but Make still compiles .o files
when they're missing from deps/exla/cache/ (they don't survive
mix deps.get). Save the pre-built cache/ directory at /opt/exla-cache/
in the CI image and restore it after mix deps.get in the test step.
With both .o files and libexla.so present, Make skips all C++ compilation.
EXLA's cached_make needs Mix.Tasks.Compile.ElixirMake (from elixir_make),
which must be compiled first. mix deps.compile exla alone doesn't handle
transitive compile-time deps. Use mix deps.compile without args to
compile in dependency order.
- Dockerfile.ci: compile xla dep before exla — exla's extract_xla
calls XLA.archive_path!/0 which requires xla to be compiled first
- Fold build-ci-image into build.yaml as prerequisite job so tests
always have the precompiled EXLA image available
- Remove standalone build-ci-image.yaml (redundant)
Add Dockerfile.ci that extends the hexpm/elixir image with precompiled
EXLA NIF cached at ~/.cache/xla/. When mix test runs, EXLA's cached_make
finds the precompiled libexla.so and skips the OOM-prone g++ C++
compilation entirely.
- Dockerfile.ci: two-stage build — builder compiles EXLA into cache;
final stage copies the cache and installs test deps (cdo)
- .forgejo/workflows/build-ci-image.yaml: builds and pushes prop-ci
image on mix.exs/mix.lock/vendor changes + weekly cron
- .forgejo/workflows/build.yaml: test step uses prop-ci:latest instead
of raw hexpm/elixir image, removing apt-get + hex/rebar install steps
RoverPathProfileWorker broadcasts {:rover_path_updated, path_id} after
updating a path. handle_info fetches via get_path!/1 and splices into
@paths, but get_path!/1 did not preload :rover_location — causing a
KeyError on %Ecto.Association.NotLoaded{} when the template accessed
location.id. Preload [:rover_location, :station, :mission] matching
the pattern in list_paths/1 and path_live.ex.
The ANALYZE queries run via Task.Supervised which doesn't hold
sandbox ownership. rescue only catches exceptions; sandbox-pool
checkout failures come as exit signals. Add catch :exit so the
Task logs a warning instead of crashing.
- nix/shell.nix: remove export MIX_ENV=dev — it overrode Mix's automatic
env selection, causing mix test to load dev config with no sandbox pool
- config/runtime.exs: use Ecto.Adapters.SQL.Sandbox pool (not
DBConnection.Ownership) for test env — test_helper.exs uses the
Sandbox API which requires the sandbox pool type
- test/test_helper.exs: silence NarrFetchWorker and ObanErrorReporter
log noise at :critical level — expected error-path output when inline
Oban jobs hit stubbed HTTP 500s
- conn_case: remove compile-time alias for AccountsFixtures to fix compilation ordering warning
- hrdps_client_test: clear global hrdps_cycle_available_fn in test setup so injected http_head takes effect (2 previously failing tests)
- test_helper: use Code.ensure_loaded!/1 instead of Code.require_file to avoid redefining module warnings
- mix.exs: add test_ignore_filters to suppress Elixir 1.20 test support file warnings
HrdpsClient already calls Req.get/head with req_options() that
merge :hrdps_req_options from config. Add plug: {Req.Test, HrdpsClient}
to test config so Req.Test.stub intercepts HTTP calls — same pattern
as HrrrClient. Also add hrdps_cycle_available_fn default so the
HTTP probe doesn't fire in tests.
Replace the env-based Application.put_env stubs in DataCase with
Req.Test.stub(HrdpsClient, 404).
Fire-and-forget tasks spawned via async_stream_nolink on the
PartitionSupervisor outlive the test sandbox owner in :manual mode.
When the owner exits, descendant tasks get shutdown on next DB
checkout — the process crash is logged at [error] by Task.Supervised.
Same benign sandbox-cleanup category as the already-suppressed
Postgrex.Protocol, DBConnection.Connection, and DBConnection.Holder.
Add default NarrClient 404 stub to DataCase so incidental inline
NarrFetchWorker execution doesn't log 500 errors in unrelated tests.
Add DBConnection.Holder to the sandbox-cleanup noise suppression list
— same benign 'owner exited' category as the already-suppressed
Postgrex.Protocol and DBConnection.Connection.
Replace the non-idiomatic :auto sandbox mode with :manual, eliminating
the root cause of stale data across test runs. The :auto mode let any
process auto-checkout and commit outside test transactions — Oban in
testing:inline already disables plugins/queues, so :auto was never
needed.
Changes:
- test_helper: Sandbox.mode(Repo, :manual); drop manual app boot (mix
now starts the app idiomatically), drop --no-start alias, drop
table-cleanup loop
- DataCase: single reset_test_state/0 consolidates GridCache.clear,
ScoreCache.clear, score-file wipe, and default HTTP stubs; drop
the :auto-restore on_exit hack; setup_sandbox now just start_owner
- ConnCase: delegates shared reset to DataCase, removes duplicate
cache clears and stub installs
- config: pool Ecto.Adapters.SQL.Sandbox (idiomatic canonical form)
Note: GridCache/ScoreCache are globally-registered GenServers, so
the architecturally clean per-test start_supervised! approach
conflicts with async:true (parallel tests can't register the same
name). The centralized clear in reset_test_state is the practical
compromise — one place, documented, single call.
patch_beacons_json/2 and remove_from_beacons_json/2 were calling
Enum.reject/2 directly on @beacons_json, which is a JSON-encoded
string from Jason.encode!/0 (not a list). This crashed with
Protocol.UndefinedError when no approved beacons existed (string
was "[]") and a PubSub broadcast triggered an update.
Decode with Jason.decode!/1, operate on the list, then re-encode
with Jason.encode!/1 so the template's data-beacons attribute
remains a valid JSON string for the JS hook.
Rows left behind by a crashed/killed prior run persist because
sandbox on_exit callbacks don't fire on BEAM crash or SIGKILL.
Wipe known test-data tables (contacts, hrrr_fetch_tasks,
iemre_observations, oban_jobs) after app boot so every suite
starts clean regardless of previous run state.
Create shared ContactsFixtures module with globally-unique qso_timestamp
seconds to prevent unique-constraint violations when async test modules
run in parallel and insert contacts with identical dedup-key columns.
The qso_timestamp column is timestamp(0), so microsecond offsets were
truncated — use System.unique_integer monotonic seconds instead.
Also make count-asserting tests resilient to sandbox-leaked contacts
from prior tests by using >= assertions or status-based checks rather
than exact counts.
Includes automated DateTime.add → DateTime.shift migration from
mix format.
The Elixir 1.20 code path workaround adds stream_data's ebin but
doesn't start the application. stream_data's macro calls
Application.fetch_env!(:stream_data, :initial_size), which requires
the application to be loaded/started.
198+ concurrent async tests overwhelm the default 50ms queue_target
with ownership pool checkouts, causing frequent :queue_timeout errors.
The 500ms/5000ms values are standard for high-concurrency test suites.
- wgrib2_test: accept both :wgrib2_not_available (no wgrib2) and binary
error message (wgrib2 available via nix) for nonexistent-file tests.
- narr_client_test + narr_fetch_worker_test: add cdo_has_proj? guard that
skips cdo-dependent tests when cdo lacks proj support (e.g. nixpkgs cdo).
Use System.find_executable to avoid ErlangError when cdo not on PATH.
All 4054 tests pass, 6 skipped (fixture/cdo guards).
- nix/wgrib2.nix: build wgrib2 3.8.0 from source with NCEPLIBS-g2c 2.3.0,
matching the Dockerfile.base build.
- flake.nix: wire wgrib2 derivation into dev shell so direnv loads it.
- DataCase: call Req.Test.set_req_test_from_context/1 in setup so
async:false tests get shared Req.Test mode, making stubs visible
to Tasks spawned by async_stream_nolink.
- hrrr_client_test: async:true => async:false so Req.Test stubs are
visible in Task.Supervisor.async_stream_nolink worker processes.
- weather_test: add setup blocks to latest_grid_valid_time/0 and
available_weather_valid_times/0 describe blocks that delete
is_grid_point=true HrrrProfile rows before each test, preventing
cross-contamination from other async test modules.
- contact_map_controller_test: call RateLimiter.reset() in setup so
accumulated ETS counters from prior tests don't trigger 429s.
- Add Phoenix.Ecto.SQL.Sandbox.allow to all start_async closures in ContactLive.Show (8 sites) and PskrSpotsLive (4 sites), fixing silent empty results when async Tasks accessed DB in tests
- Add File.exists? skip-guards to wgrib2_test.exs (14 guards) and hrrr_client_test.exs (3 guards) so tests skip gracefully when GRIB2 fixtures are absent
- Remove leftover debug IO.puts from SandboxHook
- Fix stale homepage assertion in html_module_test.exs
- Add cdo and wgrib2 to nix shell.nix and Forgejo CI apt-get install
Test suite: 4049/4054 passed (99.9%), 5 remaining failures are environment gaps (cdo/wgrib2 not yet in CI runtime)
- Delete Microwaveprop.Aprs, AprsRepo, Aprs.PathParser modules
- Delete Mix.Tasks.Calibrate.Aprs144 mix task
- Remove AprsRepo from application.ex children and config files
- Remove APRS test files and sandbox references in test_helper
- Fix credo nested-module alias suggestions in conn_case.ex
- get_user_by_callsign tests: use unique callsign per test to avoid sandbox collision
- Admin email test: merge into single changeset-level test to avoid shared-sandbox FK constraint violations on rover_locations
- Remove 4 dead {1, nil} = return-value assertions from Repo.update_all calls
- Use DBConnection.Ownership pool (not Ecto.Adapters.SQL.Sandbox) to
satisfy Ecto 3.14's Sandbox.start_owner!/2 pool check
- Set pool in config/test.exs and config/runtime.exs before app start
- mix test --no-start via alias prevents Mix from launching app before
test_helper can configure the sandbox pool
- ExUnit.start() without capture_log: true avoids Logger cascade
- :auto sandbox mode lets background processes (migrations, backfill,
Oban) access the DB during Application boot
- Load test support modules and dep ebin paths in test_helper for
Elixir 1.20 lazy test compilation
