Graham McIntire
89eccb7d50
Reproduced in production pod 2026-04-30: wgrib2 3.6.0 crashes with `free(): invalid size` when -lon point extraction is run against any HRDPS rotated lat/lon GRIB2 file, regardless of batch size (failed at N=200, N=1000, N=2000). Output before the abort is denormal f32 garbage (val=4.93944e-41) — wgrib2's internal record buffer is being corrupted. Same files extract cleanly with wgrib2 3.8.0 locally. Bumping the build arg in both Dockerfile (Elixir image, used by hrrr-point-rs/per-QSO HRDPS lookups) and rust/prop_grid_rs/Dockerfile (Rust grid worker) so both consumers get the fix. Also bumped POINT_BATCH 200 → 1000 in decoder.rs. With wgrib2 3.8.0 stable at 1000+ point batches, this drops the chain-step invocation count from 286 → 57 per step. Comment captures the reproduction context so a future bisection finds it quickly.
205 lines
7.7 KiB
Docker
205 lines
7.7 KiB
Docker
# syntax=docker/dockerfile:1.6
|
|
# Find eligible builder and runner images on Docker Hub. We use Ubuntu/Debian
|
|
# instead of Alpine to avoid DNS resolution issues in production.
|
|
#
|
|
# https://hub.docker.com/r/hexpm/elixir/tags?name=ubuntu
|
|
# https://hub.docker.com/_/ubuntu/tags
|
|
#
|
|
# This file is based on these images:
|
|
#
|
|
# - https://hub.docker.com/r/hexpm/elixir/tags - for the build image
|
|
# - https://hub.docker.com/_/debian/tags?name=trixie-20260316-slim - for the release image
|
|
# - https://pkgs.org/ - resource for finding needed packages
|
|
# - Ex: docker.io/hexpm/elixir:1.19.5-erlang-28.4.1-debian-trixie-20260316-slim
|
|
#
|
|
ARG ELIXIR_VERSION=1.19.5
|
|
ARG OTP_VERSION=28.4.1
|
|
ARG DEBIAN_VERSION=trixie-20260316-slim
|
|
|
|
ARG BUILDER_IMAGE="docker.io/hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
|
|
ARG RUNNER_IMAGE="docker.io/debian:${DEBIAN_VERSION}"
|
|
|
|
# ---- wgrib2 build stage (cached independently) ----
|
|
FROM ${RUNNER_IMAGE} AS wgrib2-builder
|
|
|
|
ARG WGRIB2_VERSION=3.8.0
|
|
# wgrib2 3.6.0 has a memory-corruption bug that surfaces on HRDPS
|
|
# rotated lat/lon files via -lon point extraction: emits denormal
|
|
# garbage values, then aborts with `free(): invalid size`. 3.8.0
|
|
# fixes it. Verified by reproducing the failure with wgrib2 3.6.0
|
|
# in the production pod against an HRDPS file on 2026-04-30 and
|
|
# confirming 3.8.0 handles the same file correctly locally. Keep
|
|
# this aligned with rust/prop_grid_rs/Dockerfile.
|
|
# g2c v2.1.0 is the first release where BUILD_G2C defaults ON, which
|
|
# wgrib2 needs at link time (g2c_enc_jpeg2000 / g2c_dec_jpeg2000
|
|
# come from the "new" file-based API).
|
|
ARG G2C_VERSION=2.1.0
|
|
|
|
# BuildKit cache mounts keep the .deb archive + apt-lists out of the
|
|
# image overlay and on the builder's cache storage, so image layers
|
|
# don't pay the disk cost and CI runners with tight /var budgets can
|
|
# still fetch ~100 MB of packages. The `rm -rf lists/*` below targets
|
|
# the cache-mount contents; BuildKit persists whatever remains for
|
|
# the next build.
|
|
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
|
|
--mount=type=cache,target=/var/lib/apt,sharing=locked \
|
|
rm -f /etc/apt/apt.conf.d/docker-clean \
|
|
&& apt-get update \
|
|
&& apt-get install -y --no-install-recommends \
|
|
build-essential gfortran cmake wget ca-certificates pkg-config \
|
|
zlib1g-dev libaec-dev libpng-dev libopenjp2-7-dev
|
|
|
|
# NCEPLIBS-g2c: wgrib2 >= 3.x delegates GRIB2 decompression to this
|
|
# library via find_package whenever any of JASPER / OPENJPEG / PNG
|
|
# is enabled (wgrib2 CMakeLists:94). Build g2c with all the
|
|
# decoders we care about — PNG (MRMS PrecipRate / HRRR native),
|
|
# OpenJPEG (modern JPEG2000 replacement for the CVE-riddled Jasper),
|
|
# and AEC (CCSDS) — and disable Jasper so we don't need libjasper,
|
|
# which Debian dropped years ago.
|
|
WORKDIR /tmp/g2c
|
|
RUN wget -q --content-disposition "https://github.com/NOAA-EMC/NCEPLIBS-g2c/archive/refs/tags/v${G2C_VERSION}.tar.gz" \
|
|
-O g2c.tar.gz \
|
|
&& tar xzf g2c.tar.gz \
|
|
&& cd NCEPLIBS-g2c-${G2C_VERSION} \
|
|
&& mkdir build && cd build \
|
|
&& cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local \
|
|
-DCMAKE_BUILD_TYPE=Release \
|
|
-DUSE_PNG=ON \
|
|
-DUSE_AEC=ON \
|
|
-DUSE_OpenJPEG=ON \
|
|
-DUSE_Jasper=OFF \
|
|
&& make -j$(nproc) \
|
|
&& make install
|
|
|
|
WORKDIR /tmp/wgrib2
|
|
|
|
# wgrib2 flags:
|
|
# * USE_PNG / USE_OPENJPEG / USE_AEC each turn on the matching
|
|
# encoder path inside wgrib2 and add the compression name to the
|
|
# -config output. With g2c v2.1.0 underneath (which exports the
|
|
# g2c_enc_* / g2c_dec_* new-API symbols) all three can be enabled
|
|
# together — the earlier build failure was not about the flag
|
|
# combination, it was g2c v1.9.0 shipping without BUILD_G2C.
|
|
RUN wget -q --content-disposition "https://github.com/NOAA-EMC/wgrib2/archive/refs/tags/v${WGRIB2_VERSION}.tar.gz" \
|
|
-O wgrib2.tar.gz \
|
|
&& tar xzf wgrib2.tar.gz \
|
|
&& cd wgrib2-${WGRIB2_VERSION} \
|
|
&& mkdir build && cd build \
|
|
&& cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local -DCMAKE_BUILD_TYPE=Release \
|
|
-DUSE_PNG=ON -DUSE_OPENJPEG=ON -DUSE_AEC=ON \
|
|
&& make -j$(nproc) \
|
|
&& make install \
|
|
&& strip /usr/local/bin/wgrib2
|
|
|
|
# ---- Elixir build stage ----
|
|
FROM ${BUILDER_IMAGE} AS builder
|
|
|
|
# install build dependencies
|
|
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
|
|
--mount=type=cache,target=/var/lib/apt,sharing=locked \
|
|
rm -f /etc/apt/apt.conf.d/docker-clean \
|
|
&& apt-get update \
|
|
&& apt-get install -y --no-install-recommends build-essential git
|
|
|
|
# prepare build dir
|
|
WORKDIR /app
|
|
|
|
# install hex + rebar
|
|
RUN mix local.hex --force \
|
|
&& mix local.rebar --force
|
|
|
|
# set build ENV
|
|
ENV MIX_ENV="prod"
|
|
|
|
# install mix dependencies (changes only when mix.exs/mix.lock change)
|
|
COPY mix.exs mix.lock ./
|
|
COPY vendor vendor
|
|
RUN mix deps.get --only $MIX_ENV
|
|
|
|
# copy compile-time config before compiling deps
|
|
RUN mkdir config
|
|
COPY config/config.exs config/${MIX_ENV}.exs config/
|
|
RUN mix deps.compile
|
|
|
|
# install esbuild + tailwind binaries (cached until mix.exs changes)
|
|
RUN mix assets.setup
|
|
|
|
# create priv dir so :code.priv_dir works at compile time
|
|
# (actual contents copied after compile to avoid cache busting)
|
|
RUN mkdir -p priv/models priv/static
|
|
|
|
# copy application code and compile
|
|
COPY lib lib
|
|
COPY algo.md algo.md
|
|
# agent-skills markdown is read at compile time by AgentSkillsController
|
|
# (sha256 digests are baked into the module attribute)
|
|
COPY priv/agent-skills priv/agent-skills
|
|
RUN mix compile
|
|
|
|
# copy assets and build (JS/CSS changes don't require recompilation)
|
|
COPY assets assets
|
|
RUN mix assets.deploy
|
|
|
|
# copy priv contents (migrations, ML model, static files)
|
|
# after compile so model retraining doesn't bust the compile cache
|
|
COPY priv priv
|
|
|
|
# runtime config doesn't require recompilation
|
|
COPY config/runtime.exs config/
|
|
|
|
COPY rel rel
|
|
RUN mix release
|
|
|
|
# ---- Final runtime stage ----
|
|
FROM ${RUNNER_IMAGE} AS final
|
|
|
|
# Build timestamp baked in by CI (--build-arg BUILD_TIMESTAMP=...).
|
|
# Only consumed in this final stage, so it never busts the earlier
|
|
# compile cache. Read at runtime by Microwaveprop.Application.build_timestamp/0.
|
|
ARG BUILD_TIMESTAMP=unknown
|
|
|
|
# Runtime deps. BuildKit cache mounts keep the .deb archives and apt
|
|
# lists out of the image layer, so the CI builder's tight /var disk
|
|
# cap doesn't trip during install. cdo is split into its own RUN
|
|
# layer because its dep tree (netCDF/HDF5/ecCodes/Fortran) is ~1 GB
|
|
# and we'd rather not have a single failed cdo install invalidate the
|
|
# base-runtime layer.
|
|
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
|
|
--mount=type=cache,target=/var/lib/apt,sharing=locked \
|
|
rm -f /etc/apt/apt.conf.d/docker-clean \
|
|
&& apt-get update \
|
|
&& apt-get install -y --no-install-recommends \
|
|
libstdc++6 openssl libncurses6 locales ca-certificates snmp \
|
|
libgfortran5 libaec0 zlib1g libpng16-16t64 libopenjp2-7
|
|
|
|
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
|
|
--mount=type=cache,target=/var/lib/apt,sharing=locked \
|
|
rm -f /etc/apt/apt.conf.d/docker-clean \
|
|
&& apt-get update \
|
|
&& apt-get install -y --no-install-recommends cdo gdal-bin
|
|
|
|
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
|
|
|
|
ENV LANG=en_US.UTF-8
|
|
ENV LANGUAGE=en_US:en
|
|
ENV LC_ALL=en_US.UTF-8
|
|
ENV MIX_ENV="prod"
|
|
|
|
WORKDIR "/app"
|
|
RUN chown nobody /app
|
|
|
|
COPY --from=wgrib2-builder /usr/local/bin/wgrib2 /usr/local/bin/wgrib2
|
|
# wgrib2 links dynamically against libg2c.so from the builder stage —
|
|
# carry it across and refresh the loader cache so /usr/local/lib is
|
|
# searched at exec time.
|
|
COPY --from=wgrib2-builder /usr/local/lib/libg2c.so* /usr/local/lib/
|
|
RUN ldconfig
|
|
|
|
COPY --from=builder --chown=nobody:root /app/_build/${MIX_ENV}/rel/microwaveprop ./
|
|
|
|
RUN echo "${BUILD_TIMESTAMP}" > /app/BUILD_TIMESTAMP \
|
|
&& chown nobody /app/BUILD_TIMESTAMP
|
|
|
|
USER nobody
|
|
|
|
CMD ["/app/bin/server"]
|