Graham McIntire
07ffcf52d7
Read-side cutover for the binary scores store and a companion cleanup that removes the biggest remaining DB write from the hot path. Propagation.scores_at/3, available_valid_times/1, latest_valid_time/0, latest_valid_time/1, earliest_valid_time/1, point_detail/4, and point_forecast/3 all now prefer ScoresFile and fall back to the propagation_scores table when a file is missing. The map render path reads from /data/scores first; Postgres stays as a safety net while dual-write is on. point_detail still pulls factors from Postgres (analysis-hour rows only) and coalesces nil to an empty map so the JS popup iterates cleanly. replace_scores/2 is now gated by a postgres_writes_enabled? flag (runtime env MICROWAVEPROP_SCORES_POSTGRES=false, or the :propagation_scores_postgres app env key) so the binary-only path can be benchmarked locally without the DB insert. Default stays true. PropagationGridWorker no longer calls store_hrrr_profiles — persisting 92k grid rows × 19 forecast hours of JSONB profiles was ~12 min of wall time per chain for a table only AsosAdjustmentWorker read from. Per-contact HRRR enrichment through HrrrFetchWorker still writes its own (is_grid_point: false) rows. AsosAdjustmentWorker is disabled in all three cron configs since its data source is gone. DataCase resets the scores tree between tests so per-test ScoresFile writes don't leak across cases, and ScoresFileTest switches to async: false because it mutates the global :propagation_scores_dir env.
70 lines
2 KiB
Elixir
70 lines
2 KiB
Elixir
defmodule Microwaveprop.DataCase do
|
|
@moduledoc """
|
|
This module defines the setup for tests requiring
|
|
access to the application's data layer.
|
|
|
|
You may define functions here to be used as helpers in
|
|
your tests.
|
|
|
|
Finally, if the test case interacts with the database,
|
|
we enable the SQL sandbox, so changes done to the database
|
|
are reverted at the end of every test. If you are using
|
|
PostgreSQL, you can even run database tests asynchronously
|
|
by setting `use Microwaveprop.DataCase, async: true`, although
|
|
this option is not recommended for other databases.
|
|
"""
|
|
|
|
use ExUnit.CaseTemplate
|
|
|
|
alias Ecto.Adapters.SQL.Sandbox
|
|
|
|
using do
|
|
quote do
|
|
import Ecto
|
|
import Ecto.Changeset
|
|
import Ecto.Query
|
|
import Microwaveprop.DataCase
|
|
|
|
alias Microwaveprop.Repo
|
|
end
|
|
end
|
|
|
|
setup tags do
|
|
Microwaveprop.DataCase.setup_sandbox(tags)
|
|
Microwaveprop.DataCase.reset_score_files()
|
|
:ok
|
|
end
|
|
|
|
@doc """
|
|
Sets up the sandbox based on the test tags.
|
|
"""
|
|
def setup_sandbox(tags) do
|
|
pid = Sandbox.start_owner!(Microwaveprop.Repo, shared: not tags[:async])
|
|
on_exit(fn -> Sandbox.stop_owner(pid) end)
|
|
end
|
|
|
|
@doc """
|
|
Wipe the propagation ScoresFile tree between tests so files don't
|
|
leak between cases sharing the same tmp dir.
|
|
"""
|
|
def reset_score_files do
|
|
dir = Application.get_env(:microwaveprop, :propagation_scores_dir)
|
|
if is_binary(dir), do: File.rm_rf!(dir)
|
|
end
|
|
|
|
@doc """
|
|
A helper that transforms changeset errors into a map of messages.
|
|
|
|
assert {:error, changeset} = Accounts.create_user(%{password: "short"})
|
|
assert "password is too short" in errors_on(changeset).password
|
|
assert %{password: ["password is too short"]} = errors_on(changeset)
|
|
|
|
"""
|
|
def errors_on(changeset) do
|
|
Ecto.Changeset.traverse_errors(changeset, fn {message, opts} ->
|
|
Regex.replace(~r"%{(\w+)}", message, fn _, key ->
|
|
opts |> Keyword.get(String.to_existing_atom(key), key) |> to_string()
|
|
end)
|
|
end)
|
|
end
|
|
end
|