Graham McIntire
1ec10bec1f
The single Era5MonthBatchWorker pinned an Oban slot for the full 30-45
min a CDS month-tile takes to assemble, and lost the work entirely on a
rolling deploy because the in-flight Task died with the pod. This splits
the flow into two tiny workers and persists the CDS job IDs so deploys
survive.
New pipeline:
1. Era5SubmitWorker (:era5_submit queue) — POSTs both CDS requests in
parallel, writes an `era5_cds_jobs` row with the returned job IDs,
and enqueues an Era5PollWorker scheduled +5 min. ~1s of real work.
Short-circuits when the month-tile is already cached or when a row
already exists (in-flight from a previous attempt).
2. Era5PollWorker (:era5_poll queue) — reads the row, calls
Era5Client.check_status/1 for both CDS job IDs, and:
- returns {:snooze, 300} if either job is still running (Oban
re-schedules without counting an attempt and releases the slot
immediately — a pod can keep dozens of tile-months in flight
without pinning workers)
- streams both GRIB files to disk via Req into: File.stream!,
decodes via Wgrib2.extract_grid_messages_from_file, bulk-inserts
via Era5BatchClient.decode_and_insert/6, deletes the row, and
DELETEs both completed jobs from CDS to free server-side quota
- if either leg CDS-reports failed, deletes the row + both CDS
jobs and returns {:error, reason}
Era5Client gains four testable building blocks:
submit_job/2 (bare POST → {:ok, job_id})
check_status/1 (GET → :running | {:done, src} | {:failed, reason})
download_source_to_file/3 (streams {:url, href} or writes {:body, bin})
delete_job/1 (DELETE /jobs/:id, treats 200/202/204/404 as :ok)
All Req calls now route through `era5_req_options` so tests can stub
CDS responses via Req.Test.stub(Era5Client, fn).
Era5MonthBatchWorker is retained as a thin forwarder to Era5SubmitWorker
so any jobs already in the :era5_batch queue on prod pods drain cleanly
on the next rolling deploy. Safe to delete in a follow-up.
Adds era5_cds_jobs table with a unique index on
(year, month, tile_lat, tile_lon) so duplicate submits collapse.
New queue config in runtime.exs:
era5_submit: local_limit 4, rate_limit 30/hour (burst protection)
era5_poll: local_limit 20 (polls are cheap GETs)
era5_batch: kept at 1 for legacy job drain, delete next cycle
76 lines
3 KiB
Elixir
76 lines
3 KiB
Elixir
import Config
|
|
|
|
alias Microwaveprop.Qrz.Client
|
|
|
|
# Only in tests, remove the complexity from the password hashing algorithm
|
|
config :bcrypt_elixir, :log_rounds, 1
|
|
|
|
# Print only warnings and errors during test
|
|
config :logger, level: :warning
|
|
|
|
config :microwaveprop, Client,
|
|
username: "testuser",
|
|
password: "testpass",
|
|
agent: "microwaveprop-test/1.0",
|
|
plug: {Req.Test, Client},
|
|
retry: false
|
|
|
|
config :microwaveprop, Microwaveprop.Geocoder,
|
|
plug: {Req.Test, Microwaveprop.Geocoder},
|
|
retry: false
|
|
|
|
# In test we don't send emails
|
|
config :microwaveprop, Microwaveprop.Mailer, adapter: Swoosh.Adapters.Test
|
|
config :microwaveprop, Microwaveprop.Qrz, cache_ttl_hours: 0
|
|
|
|
# Configure your database
|
|
#
|
|
# The MIX_TEST_PARTITION environment variable can be used
|
|
# to provide built-in test partitioning in CI environment.
|
|
# Run `mix help test` for more information.
|
|
config :microwaveprop, Microwaveprop.Repo,
|
|
username: "postgres",
|
|
password: "postgres",
|
|
hostname: "localhost",
|
|
database: "microwaveprop_test#{System.get_env("MIX_TEST_PARTITION")}",
|
|
pool: Ecto.Adapters.SQL.Sandbox,
|
|
pool_size: System.schedulers_online() * 2,
|
|
ownership_timeout: 60_000,
|
|
timeout: 60_000
|
|
|
|
# We don't run a server during test. If one is required,
|
|
# you can enable the server option below.
|
|
config :microwaveprop, MicrowavepropWeb.Endpoint,
|
|
http: [ip: {127, 0, 0, 1}, port: 4002],
|
|
secret_key_base: "f7KbnwUezGrPtrv/67l6Wi+W9/+wOqwFG5QJ/ZLs24cVM1gg2lBswZwrKuyWsBAZ",
|
|
server: false
|
|
|
|
# Run Oban jobs inline during tests
|
|
config :microwaveprop, Oban, testing: :inline
|
|
config :microwaveprop, cache_contact_count: false
|
|
config :microwaveprop, cache_contact_map: false
|
|
config :microwaveprop, elevation_req_options: [plug: {Req.Test, Microwaveprop.Terrain.ElevationClient}, retry: false]
|
|
config :microwaveprop, era5_req_options: [plug: {Req.Test, Microwaveprop.Weather.Era5Client}, retry: false]
|
|
config :microwaveprop, hrrr_req_options: [plug: {Req.Test, Microwaveprop.Weather.HrrrClient}, retry: false]
|
|
config :microwaveprop, iem_req_options: [plug: {Req.Test, Microwaveprop.Weather.IemClient}, retry: false]
|
|
|
|
# Route HTTP requests through Req.Test stubs
|
|
config :microwaveprop, nexrad_req_options: [plug: {Req.Test, Microwaveprop.Weather.NexradClient}, retry: false]
|
|
config :microwaveprop, solar_req_options: [plug: {Req.Test, Microwaveprop.Weather.SolarClient}]
|
|
config :microwaveprop, srtm_req_options: [plug: {Req.Test, Microwaveprop.Terrain.Srtm}, retry: false]
|
|
config :microwaveprop, start_freshness_monitor: false
|
|
config :microwaveprop, uwyo_req_options: [plug: {Req.Test, Microwaveprop.Weather.UwyoSoundingClient}, retry: false]
|
|
|
|
# Initialize plugs at runtime for faster test compilation
|
|
config :phoenix, :plug_init_mode, :runtime
|
|
|
|
# Sort query params output of verified routes for robust url comparisons
|
|
config :phoenix,
|
|
sort_verified_routes_query_params: true
|
|
|
|
# Enable helpful, but potentially expensive runtime checks
|
|
config :phoenix_live_view,
|
|
enable_expensive_runtime_checks: true
|
|
|
|
# Disable swoosh api client as it is only required for production adapters
|
|
config :swoosh, :api_client, false
|