Run the test suite in parallel: 19.8s and 34 failures to 10.3s green #10

graham · 2026-09-09T15:01:02-05:00

graham commented

2026-09-09 15:01:02 -05:00

Takes the suite from 19.8 s with 34 failures to 10.3 s green (4914 passing, 6 skipped). mix precommit exits 0; nine consecutive --repeat-until-failure runs and five seeds (1, 7, 424242, 999983, 31337) pass.

	before	after
wall	19.8 s	10.3 s
async phase	4.8 s	7.5 s
sync phase (fully serial)	14.9 s	2.8 s
non-async modules	105 / 325	47
failures	34	0

The 34 failures

reset_test_state/0 was gated on tags[:test_group] == :scores_dir while no module declared that group, so the score-tree and cache wipe never fired and every empty-state assertion read a previous module's files. The predicate is now tags[:test_group] == :scores_dir or tags[:async] != true — group members are serialized against each other, and sync modules run only after every async module finishes, so both branches are race-free. 23 modules touching the shared :propagation_scores_dir tree, ScoreCache, GridCache, NexradCache or Microwaveprop.Cache now declare group: :scores_dir.

Async triage

58 modules flipped to async: true, 23 into the group, 47 stay sync — each with a one-clause comment above its use line naming the actual VM-global (a config key production reads, a Logger handler, :persistent_term, a named singleton/ETS table, File.cd!, a sandbox mode change, a telemetry handler under a constant id). Without that comment nobody can tell a deliberate sync file from an accident.

Two live-network paths, found by measurement

rover/compute_test.exs restored :rover_road_proximity_enabled to true in its after blocks while config/test.exs sets it false. After that module ran, every later rover compute took the RoadProximity path and dialled Overpass with an 8 s deadline. A serial bisect in an isolated tree: 270 files = 14.8 s, the same 270 minus rover_live_test.exs = 7.0 s.

Closed at three levels: no test writes the key (Compute.run/2 grew a deps[:road_enabled?] seam), RoadProximity threads *_req_options so a future leak fails instantly as "no stub found" instead of live HTTP, and NceiMetarClient plus HrrrNativeGridWorker's bypassing Req.get got the same treatment.

The Ecto sandbox does not isolate PubSub

BeaconLive.Index subscribes to the global beacons topic and stream_inserts the broadcast payload, so MonitorLive.ShowTest's beacon rendered into the page under test and broke a page-global selector about one run in five. A probe confirmed the test's own connection saw only its own row — no commit leak. Fixed with unique fixture identities and identity-scoped selectors, then swept across the LiveView suite (topic → LiveView → handler map in the diff).

Detached tasks

Weather.Grid's cold grid fill and scalar materialize both ran under Task.start/1: no $callers, so their reads escaped the sandbox; they outlived the spawning test, wrote into the score tree after it had been wiped (... materialize failed ... :enoent), and broadcast weather:updated into concurrent LiveViews. Both now sit behind :weather_async_grid_fill — default true, off in test.

Test quality

The rover "calculate" test wrapped its assertions in try/rescue, so neither real assertion ever ran; now two deterministic tests seeding three lattice cells.
PathLive's re-read test mirrors production ordering (NotifyListener invalidates ScoreCache before broadcasting); without it point_forecast/3 served a stale cached score.
Mix.Tasks.UnusedTest 720 ms → ~20 ms: a three-beam fixture dir instead of decoding all 384 _build beams.
Global env writes restore via Application.fetch_env/2. The truthiness check had band_weights_test deleting a legitimately-false :band_weights_json, making every later module load the real priv/algo/band_weights.json.

Infrastructure

max_cases from EXUNIT_MAX_CASES else schedulers_online() * 2 (was pinned to 2); Repo pool sized max_cases + 4 in runtime.exs rather than baked into _build at compile time, with the pool_size >= max_cases invariant documented; Sandbox.mode(Repo, :manual) pinned before tests start; Microwaveprop.Cache memoization off in test; scripts/measure_test_suite.sh records both passes and tolerates a red run so rankings still land. AGENTS.md documents the whole model.

Every production seam added defaults to today's behaviour when its config key is absent, so production is unchanged — which is also why there is no CHANGELOG.md entry: nothing here is observable to a site visitor.

Takes the suite from **19.8 s with 34 failures** to **10.3 s green** (4914 passing, 6 skipped). `mix precommit` exits 0; nine consecutive `--repeat-until-failure` runs and five seeds (1, 7, 424242, 999983, 31337) pass. | | before | after | |---|---|---| | wall | 19.8 s | **10.3 s** | | async phase | 4.8 s | 7.5 s | | sync phase (fully serial) | 14.9 s | 2.8 s | | non-async modules | 105 / 325 | 47 | | failures | 34 | 0 | ## The 34 failures `reset_test_state/0` was gated on `tags[:test_group] == :scores_dir` while no module declared that group, so the score-tree and cache wipe never fired and every empty-state assertion read a previous module's files. The predicate is now `tags[:test_group] == :scores_dir or tags[:async] != true` — group members are serialized against each other, and sync modules run only after every async module finishes, so both branches are race-free. 23 modules touching the shared `:propagation_scores_dir` tree, `ScoreCache`, `GridCache`, `NexradCache` or `Microwaveprop.Cache` now declare `group: :scores_dir`. ## Async triage 58 modules flipped to `async: true`, 23 into the group, 47 stay sync — each with a one-clause comment above its `use` line naming the actual VM-global (a config key production reads, a `Logger` handler, `:persistent_term`, a named singleton/ETS table, `File.cd!`, a sandbox mode change, a telemetry handler under a constant id). Without that comment nobody can tell a deliberate sync file from an accident. ## Two live-network paths, found by measurement `rover/compute_test.exs` restored `:rover_road_proximity_enabled` to `true` in its `after` blocks while `config/test.exs` sets it `false`. After that module ran, every later rover compute took the `RoadProximity` path and dialled Overpass with an 8 s deadline. A serial bisect in an isolated tree: 270 files = 14.8 s, the same 270 minus `rover_live_test.exs` = 7.0 s. Closed at three levels: no test writes the key (`Compute.run/2` grew a `deps[:road_enabled?]` seam), `RoadProximity` threads `*_req_options` so a future leak fails instantly as "no stub found" instead of live HTTP, and `NceiMetarClient` plus `HrrrNativeGridWorker`'s bypassing `Req.get` got the same treatment. ## The Ecto sandbox does not isolate PubSub `BeaconLive.Index` subscribes to the global beacons topic and `stream_insert`s the broadcast payload, so `MonitorLive.ShowTest`'s beacon rendered into the page under test and broke a page-global selector about one run in five. A probe confirmed the test's own connection saw only its own row — no commit leak. Fixed with unique fixture identities and identity-scoped selectors, then swept across the LiveView suite (topic → LiveView → handler map in the diff). ## Detached tasks `Weather.Grid`'s cold grid fill and scalar materialize both ran under `Task.start/1`: no `$callers`, so their reads escaped the sandbox; they outlived the spawning test, wrote into the score tree after it had been wiped (`... materialize failed ... :enoent`), and broadcast `weather:updated` into concurrent LiveViews. Both now sit behind `:weather_async_grid_fill` — default `true`, off in test. ## Test quality - The rover "calculate" test wrapped its assertions in `try/rescue`, so neither real assertion ever ran; now two deterministic tests seeding three lattice cells. - PathLive's re-read test mirrors production ordering (`NotifyListener` invalidates `ScoreCache` *before* broadcasting); without it `point_forecast/3` served a stale cached score. - `Mix.Tasks.UnusedTest` 720 ms → ~20 ms: a three-beam fixture dir instead of decoding all 384 `_build` beams. - Global env writes restore via `Application.fetch_env/2`. The truthiness check had `band_weights_test` *deleting* a legitimately-`false` `:band_weights_json`, making every later module load the real `priv/algo/band_weights.json`. ## Infrastructure `max_cases` from `EXUNIT_MAX_CASES` else `schedulers_online() * 2` (was pinned to 2); Repo pool sized `max_cases + 4` in `runtime.exs` rather than baked into `_build` at compile time, with the `pool_size >= max_cases` invariant documented; `Sandbox.mode(Repo, :manual)` pinned before tests start; `Microwaveprop.Cache` memoization off in test; `scripts/measure_test_suite.sh` records both passes and tolerates a red run so rankings still land. `AGENTS.md` documents the whole model. Every production seam added defaults to today's behaviour when its config key is absent, so production is unchanged — which is also why there is no `CHANGELOG.md` entry: nothing here is observable to a site visitor.

graham added 1 commit

2026-09-09 15:01:02 -05:00

Run the test suite in parallel: 19.8s and 34 failures to 10.3s green

skippy-bot/review Skippy review: 2 blocking findings open — see the PR thread

Details

a7becc34f4

The suite ran 105 of 325 modules synchronously, so wall time was close to
the sum of its parts: 14.9s of the 19.8s run was the serial phase. ExUnit
runs every async module to completion before running sync ones one at a
time, so each sync module's runtime is paid in full.

Reset gate. `reset_test_state/0` was gated on `tags[:test_group] ==
:scores_dir` while no module declared that group, so the score-tree and
cache wipe never fired and every empty-state assertion read a previous
module's files -- the 34 failures. The predicate is now
`tags[:test_group] == :scores_dir or tags[:async] != true`: group members
are serialized against each other, and sync modules run only after every
async module has finished, so both branches are race-free. 23 modules
that write or assert-empty on the shared `:propagation_scores_dir` tree,
`ScoreCache`, `GridCache`, `NexradCache` or `Microwaveprop.Cache` now
declare `group: :scores_dir`.

Async triage. 58 modules flipped to `async: true`, 23 into the group, 47
stay sync -- each with a one-clause comment above its `use` line naming
the actual VM-global (a config key production reads, a Logger handler,
`:persistent_term`, a named singleton or ETS table, `File.cd!`, a
sandbox mode change, a telemetry handler under a constant id).

Live network calls. `rover/compute_test.exs` restored
`:rover_road_proximity_enabled` to `true` in its `after` blocks while
config/test.exs sets it `false`, so every later rover compute took the
RoadProximity path and dialled Overpass with an 8s deadline. Measured
cost: a 270-file run went 14.8s -> 7.0s with `rover_live_test.exs`
removed. No test writes that key now (`Compute.run/2` grew a
`deps[:road_enabled?]` seam), RoadProximity threads `*_req_options` so a
future leak fails instantly as "no stub found", and NceiMetarClient plus
HrrrNativeGridWorker's bypassing `Req.get` got the same treatment.

PubSub is not sandboxed. A LiveView that subscribes to a process-global
topic and streams the broadcast payload renders other modules' records:
BeaconLive.Index's `handle_info` put MonitorLive.ShowTest's beacon into
the page under test, breaking a page-global selector about one run in
five. Fixed with unique fixture identities and identity-scoped
selectors, and swept across the LiveView suite.

Detached tasks. `Weather.Grid`'s cold grid fill and scalar materialize
both ran under `Task.start/1`: no `$callers`, so their reads escaped the
sandbox, they outlived the spawning test, wrote into the score tree
after it had been wiped, and broadcast `weather:updated` into concurrent
LiveViews. Both now sit behind `:weather_async_grid_fill`, default true,
off in test.

Test quality. The rover "calculate" test wrapped its assertions in
try/rescue so neither ever ran; it is now two deterministic tests that
seed three lattice cells. PathLive's re-read test mirrors the production
ordering (NotifyListener invalidates ScoreCache before broadcasting),
without which `point_forecast/3` served a stale cached score.
`Mix.Tasks.UnusedTest` went 720ms -> ~20ms by scanning a three-beam
fixture dir instead of all 384 `_build` beams. Global env writes restore
through `Application.fetch_env/2`, which fixes band_weights_test
deleting a legitimately-`false` key and making every later module load
the real priv/algo/band_weights.json.

Infrastructure. `max_cases` comes from EXUNIT_MAX_CASES or
`schedulers_online() * 2` (was pinned to 2); the Repo pool is sized
`max_cases + 4` in runtime.exs rather than baked into `_build` at
compile time; `Sandbox.mode(Repo, :manual)` is pinned before tests
start; `Microwaveprop.Cache` memoization is off in test.
scripts/measure_test_suite.sh records both passes and tolerates a red
run so rankings still land. AGENTS.md documents the model.

Result: 10.3s (7.5s async, 2.8s sync), 4914 passing, `mix precommit`
clean, nine consecutive `--repeat-until-failure` runs and five seeds
green.

skippy-bot reviewed

2026-09-09 15:52:01 -05:00

skippy-bot left a comment

🤖 Skippy PR review

2 findings — 1 needing attention before merge.

Severity	Location	Issue
🟠 High	`test/support/data_case.ex:48`	Three plain `async: true` modules still touch the shared score tree / ScoreCache the group now writes concurrently
🟡 Warning	`lib/microwaveprop/workers/hrrr_native_grid_worker.ex:195`	`:hrrr_native_http_get` has no test-env default, so this path can still reach NOMADS

_{Reviewed a7becc34f492. Comment skippy review to re-run.}

### 🤖 Skippy PR review **2 findings** — 1 needing attention before merge. | Severity | Location | Issue | | --- | --- | --- | | 🟠 High | `test/support/data_case.ex:48` | Three plain `async: true` modules still touch the shared score tree / ScoreCache the group now writes concurrently | | 🟡 Warning | `lib/microwaveprop/workers/hrrr_native_grid_worker.ex:195` | `:hrrr_native_http_get` has no test-env default, so this path can still reach NOMADS | <sub>Reviewed `a7becc34f492`. Comment `skippy review` to re-run.</sub>

lib/microwaveprop/workers/hrrr_native_grid_worker.ex

					
				@ -191,2 +193,3 @@

				  # the .idx fetch instead of hitting the live NOMADS bucket.

				  defp fetch_idx(url) do

				    case Req.get(url, receive_timeout: 120_000) do

				    runner = Application.get_env(:microwaveprop, :hrrr_native_http_get, &Req.get/2)

🟡 Warning — :hrrr_native_http_get has no test-env default, so this path can still reach NOMADS

The comment promises "a test that reaches this branch stubs the .idx fetch instead of hitting the live NOMADS bucket", but the runner falls back to &Req.get/2 and config/test.exs never sets :hrrr_native_http_get — only individual tests do, via Application.put_env in HrrrNativeClientTest. Every other client in this PR got a config-level guard (ncei_metar_req_options, rover_road_proximity_req_options, …); this one has the seam but not the belt-and-braces. HrrrNativeGridWorkerTest is now async: true, so the first test that does reach fetch_idx/1 spends 120 s of live HTTP inside the concurrent phase — the exact class of leak this PR was built to close.

Add to config/test.exs:

config :microwaveprop, :hrrr_native_http_get, fn url, _opts ->
  {:error, {:unstubbed_http_get, url}}
end

**🟡 Warning — `:hrrr_native_http_get` has no test-env default, so this path can still reach NOMADS** The comment promises "a test that reaches this branch stubs the .idx fetch instead of hitting the live NOMADS bucket", but the runner falls back to `&Req.get/2` and `config/test.exs` never sets `:hrrr_native_http_get` — only individual tests do, via `Application.put_env` in `HrrrNativeClientTest`. Every other client in this PR got a config-level guard (`ncei_metar_req_options`, `rover_road_proximity_req_options`, …); this one has the seam but not the belt-and-braces. `HrrrNativeGridWorkerTest` is now `async: true`, so the first test that does reach `fetch_idx/1` spends 120 s of live HTTP inside the concurrent phase — the exact class of leak this PR was built to close. Add to `config/test.exs`: ```elixir config :microwaveprop, :hrrr_native_http_get, fn url, _opts -> {:error, {:unstubbed_http_get, url}} end ```

skippy-bot marked this conversation as resolved

test/support/data_case.ex

					
				@ -25,2 +46,3 @@

				    _owner = Microwaveprop.DataCase.setup_sandbox(tags)

				    Microwaveprop.DataCase.reset_test_state()

				    if tags[:test_group] == @propagation_group or tags[:async] != true do

🟠 High — Three plain async: true modules still touch the shared score tree / ScoreCache the group now writes concurrently

The gate is right, but the sweep missed three modules, and this PR is what makes them race: the heavy writers (PropagationTest, PathLiveTest, MapLiveTest, ScoresControllerTest, RoverLiveTest, …) used to be async: false, so they only ran after every async module had finished. Now they are async: true, group: :scores_dir and run concurrently with every non-group async module.

Still plain async: true, no group:

test/microwaveprop/propagation/untested_functions_test.exs — asserts the empty state of the shared tree (latest_valid_time() == nil, available_valid_times/1 == [], scores_at/3 == [], latest_scores/2 == [], point_forecast/3 nil, point_detail/4 nil). All of those go through ScoresFile.list_valid_times/1 on :propagation_scores_dir, so any concurrent group member's replace_scores/2 makes them non-empty. This is exactly the failure mode the PR describes, just relocated.
test/microwaveprop_web/live/contact_live/show_coverage_test.exs:77,792 and show_hydration_test.exs:172,340 — ScoreCache.clear/0 in setup, i.e. a global ETS wipe from a plain async module, which the new reset_test_state/0 docs explicitly forbid. Concretely it breaks PropagationTest "cache-hit path does not emit the scores_at.stop span": that test does ScoreCache.put/3 then asserts %{hit: true}; a clear landing in between turns it into a miss and fails both the assert and the refute_received.

Fix: give the first group: :scores_dir (it only reads), and give the two ConnCase modules group: :scores_dir as well so their ScoreCache.clear/0 is serialized against the other members.

**🟠 High — Three plain `async: true` modules still touch the shared score tree / ScoreCache the group now writes concurrently** The gate is right, but the sweep missed three modules, and this PR is what makes them race: the heavy writers (`PropagationTest`, `PathLiveTest`, `MapLiveTest`, `ScoresControllerTest`, `RoverLiveTest`, …) used to be `async: false`, so they only ran after every async module had finished. Now they are `async: true, group: :scores_dir` and run *concurrently with every non-group async module*. Still plain `async: true`, no group: * `test/microwaveprop/propagation/untested_functions_test.exs` — asserts the empty state of the shared tree (`latest_valid_time() == nil`, `available_valid_times/1 == []`, `scores_at/3 == []`, `latest_scores/2 == []`, `point_forecast/3` nil, `point_detail/4` nil). All of those go through `ScoresFile.list_valid_times/1` on `:propagation_scores_dir`, so any concurrent group member's `replace_scores/2` makes them non-empty. This is exactly the failure mode the PR describes, just relocated. * `test/microwaveprop_web/live/contact_live/show_coverage_test.exs:77,792` and `show_hydration_test.exs:172,340` — `ScoreCache.clear/0` in `setup`, i.e. a global ETS wipe from a plain async module, which the new `reset_test_state/0` docs explicitly forbid. Concretely it breaks `PropagationTest` "cache-hit path does not emit the scores_at.stop span": that test does `ScoreCache.put/3` then asserts `%{hit: true}`; a clear landing in between turns it into a miss and fails both the assert and the `refute_received`. Fix: give the first `group: :scores_dir` (it only reads), and give the two ConnCase modules `group: :scores_dir` as well so their `ScoreCache.clear/0` is serialized against the other members.

skippy-bot marked this conversation as resolved

graham added 1 commit

2026-09-09 16:10:52 -05:00

Close two review holes: three ungrouped shared-state modules, native HRRR fetch

skippy-bot/review Skippy review: clean — no open findings

Details

d6e1b74651

Group annotations. Three modules read or mutate the state the
`:scores_dir` group now writes concurrently, and were still plain
`async: true`:

  * propagation/untested_functions_test.exs asserts the empty state of
    the shared tree -- latest_valid_time/0, available_valid_times/1,
    scores_at/3, latest_scores/2, point_forecast/3 and point_detail/4 all
    reach ScoresFile.list_valid_times/1, so any group member's
    replace_scores/2 makes them non-empty.
  * contact_live/show_coverage_test.exs and show_hydration_test.exs call
    ScoreCache.clear/0 from setup -- a wipe of the VM-global ETS table
    that PropagationTest's cache-hit span test seeds and asserts
    %{hit: true} against.

All three now declare `group: :scores_dir`. Before this PR they were
safe only because the heavy writers were `async: false` and therefore
ran after every async module had finished.

Native HRRR idx fetch. The `:hrrr_native_http_get` seam defaults to
`&Req.get/2` and only HrrrNativeClientTest replaced it, so any other
test reaching that branch dialled the live NOMADS bucket with a 120 s
receive_timeout. Both call sites now merge `:hrrr_native_req_options`,
which config/test.exs points at a Req.Test plug -- the same
belt-and-braces convention as hrrr/hrdps/gefs/nexrad/ncei_metar.

That leak was real and load-bearing: with the plug in place,
Mix.Tasks.SimpleTasksTest's HrrrNativeBackfill test immediately raised
"no mock or stub" from fetch_idx/1 -- it had been downloading a real
.idx through inline Oban on every run. Stubbed with a 404 (the task's
contract is the enqueue, not the download), suite wall time went
10.3s -> 7.7s.

Also records why contact_edit_test's fixed-atom Microwaveprop.Cache keys
need no group: memoization is off in test so production never writes
them, and the only other toucher deletes rather than seeds.

Six consecutive --repeat-until-failure runs green at 7.7s; precommit
clean.

graham commented

2026-09-09 16:11:08 -05:00

Both correct, both fixed in d6e1b746.

Three ungrouped shared-state modules. Verified and grouped:

propagation/untested_functions_test.exs — plain async: true, and every assertion is an empty-state read reaching ScoresFile.list_valid_times/1 (lines 25, 31–32, 44, 50, 55, 61, 67, 72, 78, 103, 108). Now group: :scores_dir.
show_coverage_test.exs (ScoreCache.clear/0 at :77, :792) and show_hydration_test.exs (:172, :340) — both now group: :scores_dir, so the ETS wipe is serialized against the members that seed it.

Your framing is the important part: these were safe only because the heavy writers were async: false and therefore ran after every async module. Flipping them into the group is what turned latent into live, and the sweep missed these three.

While in there I swept all 247 plain-async modules for ScoreCache|GridCache|NexradCache|Microwaveprop.Cache|propagation_scores_dir|replace_scores|ScoresFile|ProfilesFile|ScalarFile. Four other hits, all genuinely fine: file_prune_test builds its own System.tmp_dir! path, scores_file_property_test calls extract_points/2 on an in-memory payload, weather_layers_test only names ProfilesFile.read/1 in a comment, and contact_edit_test seeds three fixed-atom Microwaveprop.Cache keys — that last one now carries a comment saying why it needs no group (memoization is off in test, so production never writes those keys, and the only other toucher deletes rather than seeds).

:hrrr_native_http_get. Right diagnosis, and worse than latent — it was firing on every run. Both call sites (hrrr_native_client.ex:206, hrrr_native_grid_worker.ex:194) now merge :hrrr_native_req_options, and config/test.exs points that at {Req.Test, Microwaveprop.Weather.HrrrNativeClient}, matching the hrrr_/hrdps_/gefs_/nexrad_/ncei_metar_ convention. I went with request options rather than a function default so the guard sits on the same axis as every other client, and so the existing Application.put_env transport override in HrrrNativeClientTest still wins.

With the plug in place the suite went red immediately:

** (RuntimeError) no mock or stub for Microwaveprop.Weather.HrrrNativeClient
  lib/microwaveprop/workers/hrrr_native_grid_worker.ex:202: fetch_idx/1
  lib/mix/tasks/hrrr_native_backfill.ex:40
  (Mix.Tasks.SimpleTasksTest) "enqueues jobs for the top hours when contacts exist"

That test had been downloading a real .idx from NOMADS through inline Oban on every single run. Stubbed with a 404 — the task's contract is the enqueue, not the download.

Suite wall time 10.3 s → 7.7 s as a result. Six consecutive --repeat-until-failure runs green, mix precommit exit 0.

Both correct, both fixed in d6e1b746. **Three ungrouped shared-state modules.** Verified and grouped: - `propagation/untested_functions_test.exs` — plain `async: true`, and every assertion is an empty-state read reaching `ScoresFile.list_valid_times/1` (lines 25, 31–32, 44, 50, 55, 61, 67, 72, 78, 103, 108). Now `group: :scores_dir`. - `show_coverage_test.exs` (`ScoreCache.clear/0` at :77, :792) and `show_hydration_test.exs` (:172, :340) — both now `group: :scores_dir`, so the ETS wipe is serialized against the members that seed it. Your framing is the important part: these were safe only *because* the heavy writers were `async: false` and therefore ran after every async module. Flipping them into the group is what turned latent into live, and the sweep missed these three. While in there I swept all 247 plain-async modules for `ScoreCache|GridCache|NexradCache|Microwaveprop.Cache|propagation_scores_dir|replace_scores|ScoresFile|ProfilesFile|ScalarFile`. Four other hits, all genuinely fine: `file_prune_test` builds its own `System.tmp_dir!` path, `scores_file_property_test` calls `extract_points/2` on an in-memory payload, `weather_layers_test` only names `ProfilesFile.read/1` in a comment, and `contact_edit_test` seeds three fixed-atom `Microwaveprop.Cache` keys — that last one now carries a comment saying why it needs no group (memoization is off in test, so production never writes those keys, and the only other toucher deletes rather than seeds). **`:hrrr_native_http_get`.** Right diagnosis, and worse than latent — it was firing on every run. Both call sites (`hrrr_native_client.ex:206`, `hrrr_native_grid_worker.ex:194`) now merge `:hrrr_native_req_options`, and `config/test.exs` points that at `{Req.Test, Microwaveprop.Weather.HrrrNativeClient}`, matching the `hrrr_/hrdps_/gefs_/nexrad_/ncei_metar_` convention. I went with request options rather than a function default so the guard sits on the same axis as every other client, and so the existing `Application.put_env` transport override in `HrrrNativeClientTest` still wins. With the plug in place the suite went red immediately: ``` ** (RuntimeError) no mock or stub for Microwaveprop.Weather.HrrrNativeClient lib/microwaveprop/workers/hrrr_native_grid_worker.ex:202: fetch_idx/1 lib/mix/tasks/hrrr_native_backfill.ex:40 (Mix.Tasks.SimpleTasksTest) "enqueues jobs for the top hours when contacts exist" ``` That test had been downloading a real `.idx` from NOMADS through inline Oban on every single run. Stubbed with a 404 — the task's contract is the enqueue, not the download. **Suite wall time 10.3 s → 7.7 s** as a result. Six consecutive `--repeat-until-failure` runs green, `mix precommit` exit 0.

skippy-bot commented

2026-09-09 16:12:03 -05:00

First-time contributor

Resolved both earlier findings — the three ungrouped shared-state modules (untested_functions_test, show_coverage_test, show_hydration_test) now declare group: :scores_dir, and :hrrr_native_req_options gives the default &Req.get/2 runner a Req.Test plug in config/test.exs so an unstubbed .idx fetch fails instantly instead of burning the 120 s receive_timeout on NOMADS.

0 still open. Nothing new in a7becc34f492..d6e1b746518a — the merge order in both http_get/2 and fetch_idx/1 leaves prod behaviour unchanged (empty options list), and the simple_tasks_test stub is registered in the same process that runs the inline Oban job.

_{Reviewed d6e1b746518a.}

Resolved both earlier findings — the three ungrouped shared-state modules (`untested_functions_test`, `show_coverage_test`, `show_hydration_test`) now declare `group: :scores_dir`, and `:hrrr_native_req_options` gives the default `&Req.get/2` runner a `Req.Test` plug in `config/test.exs` so an unstubbed .idx fetch fails instantly instead of burning the 120 s receive_timeout on NOMADS. 0 still open. Nothing new in `a7becc34f492..d6e1b746518a` — the merge order in both `http_get/2` and `fetch_idx/1` leaves prod behaviour unchanged (empty options list), and the `simple_tasks_test` stub is registered in the same process that runs the inline Oban job. <sub>Reviewed `d6e1b746518a`.</sub>

graham merged commit db926ad94a into main

2026-09-09 16:14:55 -05:00

graham deleted branch perf/async-test-suite

2026-09-09 16:14:55 -05:00

graham referenced this pull request from a commit

2026-09-09 16:14:55 -05:00

Run the test suite in parallel: 19.8s and 34 failures to 10.3s green (#10)

Sign in to join this conversation.

No reviewers

No labels

No milestone

No project