Commit graph

347 commits

Author SHA1 Message Date
55251df8a0
feat(elixir): OpenTelemetry trace export to cluster OTel Collector
Adds the OTel deps (opentelemetry + exporter + phoenix / ecto / oban /
bandit auto-instrumentation helpers) and attaches them in
Application.start/2 so the existing Phoenix, Ecto, Oban, and Bandit
telemetry events flow as OTLP spans without any call-site changes.

Exporter config is gated on OTEL_EXPORTER_OTLP_ENDPOINT in
config/runtime.exs — set in the k8s manifests to the cluster
collector (otel-collector.observability.svc.cluster.local:4317).
When unset we switch traces_exporter to :none so nothing is shipped;
dev/test stays quiet.

Resource attributes tag spans with service.name=microwaveprop and
service.namespace=prop, matching the Rust workers' attribute shape
so Tempo can group the full hourly chain across both languages.

Both the main prop deployment and the backfill deployment get the
env; backfill is still a full BEAM node running enrichment workers,
so its Oban/Ecto spans are worth seeing too.
2026-04-20 12:02:15 -05:00
FluxCD
a52d2e2b93 chore: update prop image to git.mcintire.me/graham/prop:main-1776694898-8089a16 [skip ci] 2026-04-20 14:26:18 +00:00
FluxCD
48da629147 chore: update prop image to git.mcintire.me/graham/prop:main-1776693990-68910f4 [skip ci] 2026-04-20 14:09:13 +00:00
FluxCD
f05f80ef91 chore: update prop image to git.mcintire.me/graham/prop:main-1776693158-141956b [skip ci] 2026-04-20 13:54:56 +00:00
FluxCD
f671114105 chore: update prop image to git.mcintire.me/graham/prop:main-1776692049-1cbc3e5 [skip ci] 2026-04-20 13:36:51 +00:00
FluxCD
ab653f37ef chore: update prop image to git.mcintire.me/graham/prop:main-1776691261-a3cfff3 [skip ci] 2026-04-20 13:24:35 +00:00
FluxCD
2421bf562a chore: update prop image to git.mcintire.me/graham/prop:main-1776641293-4fefb81 [skip ci] 2026-04-19 23:31:18 +00:00
FluxCD
586222be63 chore: update prop image to git.mcintire.me/graham/prop:main-1776640955-7416583 [skip ci] 2026-04-19 23:27:05 +00:00
FluxCD
03b8c4d5d4 chore: update prop image to git.mcintire.me/graham/prop:main-1776640446-cd7f2fc [skip ci] 2026-04-19 23:16:58 +00:00
FluxCD
47251394c3 chore: update prop image to git.mcintire.me/graham/prop:main-1776640237-65f7963 [skip ci] 2026-04-19 23:14:56 +00:00
FluxCD
6613804bbf chore: update prop image to git.mcintire.me/graham/prop:main-1776639595-1307b1d [skip ci] 2026-04-19 23:02:50 +00:00
FluxCD
9324c92e1a chore: update prop image to git.mcintire.me/graham/prop:main-1776639367-22f5828 [skip ci] 2026-04-19 22:59:47 +00:00
FluxCD
dcb68089db chore: update prop image to git.mcintire.me/graham/prop:main-1776637893-022e692 [skip ci] 2026-04-19 22:34:26 +00:00
FluxCD
ed6b16dc60 chore: update prop image to git.mcintire.me/graham/prop:main-1776637573-4870862 [skip ci] 2026-04-19 22:29:59 +00:00
FluxCD
676999606b chore: update prop image to git.mcintire.me/graham/prop:main-1776637110-0528727 [skip ci] 2026-04-19 22:22:50 +00:00
FluxCD
3572a15c4c chore: update prop image to git.mcintire.me/graham/prop:main-1776636597-234a27b [skip ci] 2026-04-19 22:12:47 +00:00
FluxCD
52c062c90a chore: update prop image to git.mcintire.me/graham/prop:main-1776635907-6c73012 [skip ci] 2026-04-19 22:01:40 +00:00
6c73012d09
fix(k8s): tier label isolates hot pods from backfill endpoints
Both the hot (`prop`) Deployment and the `prop-backfill` Deployment
labelled their pods with `app: prop` so both joined the same libcluster
Erlang cluster. The `prop` Service selector was `app: prop` too — which
meant the backfill pod (PHX_SERVER=false, no listener on :5000) was in
the Service endpoint pool.

With sessionAffinity: ClientIP, Cloudflared's source IP got hashed to
the backfill endpoint after a backfill pod roll and every request
from that client returned 502 (connection refused on :5000).

Fix: add `tier: hot` to hot-pod template labels; narrow the Service
selector to `{app: prop, tier: hot}`. Backfill keeps `tier: backfill`
and is cleanly excluded. libcluster selector (still `app: prop`) is
unchanged so backfill stays in the Erlang cluster for Oban queue
leader election and Oban.Pro's Smart engine.
2026-04-19 16:58:13 -05:00
FluxCD
8ab2d836fb chore: update prop image to git.mcintire.me/graham/prop:main-1776635487-65693ed [skip ci] 2026-04-19 21:54:34 +00:00
FluxCD
54bc46ab60 chore: update prop image to git.mcintire.me/graham/prop:main-1776635090-6d91461 [skip ci] 2026-04-19 21:47:29 +00:00
FluxCD
ae4b0742fb chore: update prop image to git.mcintire.me/graham/prop:main-1776634649-ca33b83 [skip ci] 2026-04-19 21:41:27 +00:00
FluxCD
e7496331ba chore: update prop image to git.mcintire.me/graham/prop:main-1776634019-2d67cf7 [skip ci] 2026-04-19 21:31:30 +00:00
FluxCD
bf370ba1e4 chore: update prop image to git.mcintire.me/graham/prop:main-1776633273-03ba48f [skip ci] 2026-04-19 21:18:02 +00:00
FluxCD
04df743e82 chore: update prop image to git.mcintire.me/graham/prop:main-1776632590-6e64e08 [skip ci] 2026-04-19 21:07:57 +00:00
FluxCD
862c0a915e chore: update prop image to git.mcintire.me/graham/prop:main-1776632295-79bbdf9 [skip ci] 2026-04-19 21:01:53 +00:00
FluxCD
8d3abbc711 chore: update prop image to git.mcintire.me/graham/prop:main-1776632143-8774181 [skip ci] 2026-04-19 20:58:43 +00:00
FluxCD
9a944f40e5 chore: update prop image to git.mcintire.me/graham/prop:main-1776631843-339591c [skip ci] 2026-04-19 20:53:39 +00:00
FluxCD
f0485dd595 chore: update prop image to git.mcintire.me/graham/prop:main-1776631383-b808780 [skip ci] 2026-04-19 20:46:29 +00:00
FluxCD
fd95a82478 chore: update prop image to git.mcintire.me/graham/prop:main-1776628623-f1846c0 [skip ci] 2026-04-19 20:00:04 +00:00
dfc7895197
chore(infra): bump prop to 4 replicas for talos5 (NUC)
talos5 is a NUC7i5 with 32GB RAM and a Samsung 980 Pro NVMe. Adding
a 4th hot-path replica lets the anti-affinity rule place one pod
per physical host across node1/2/3/5 and gives the hourly f00-f18
grid chain 4-way parallel fan-out, with the NVMe-backed pod
handling its share of grib extraction.
2026-04-19 14:37:52 -05:00
FluxCD
3be5480af6 chore: update prop image to git.mcintire.me/graham/prop:main-1776626172-1b2424a [skip ci] 2026-04-19 19:19:50 +00:00
FluxCD
bc153055c6 chore: update prop image to git.mcintire.me/graham/prop:main-1776620433-6f63f3b [skip ci] 2026-04-19 17:43:07 +00:00
FluxCD
0942d0bbfb chore: update prop image to git.mcintire.me/graham/prop:main-1776620227-a6a3d3c [skip ci] 2026-04-19 17:40:04 +00:00
FluxCD
c38f4e2bfe chore: update prop image to git.mcintire.me/graham/prop:main-1776619974-9db9e6a [skip ci] 2026-04-19 17:35:46 +00:00
FluxCD
cad4bfc6f5 chore: update prop image to git.mcintire.me/graham/prop:main-1776619000-ffb14cb [skip ci] 2026-04-19 17:19:36 +00:00
FluxCD
d3a1e93587 chore: update prop image to git.mcintire.me/graham/prop:main-1776618115-01b181b [skip ci] 2026-04-19 17:04:32 +00:00
FluxCD
95a769bf0e chore: update prop image to git.mcintire.me/graham/prop:main-1776612418-0d02239 [skip ci] 2026-04-19 15:30:08 +00:00
FluxCD
543215864c chore: update prop image to git.mcintire.me/graham/prop:main-1776609556-f122eed [skip ci] 2026-04-19 14:41:49 +00:00
FluxCD
e41f84320a chore: update prop image to git.mcintire.me/graham/prop:main-1776608794-5cfb9e6 [skip ci] 2026-04-19 14:28:47 +00:00
FluxCD
ec3251c02f chore: update prop image to git.mcintire.me/graham/prop:main-1776607569-bbf6481 [skip ci] 2026-04-19 14:08:43 +00:00
FluxCD
3ab71cf8fe chore: update prop image to git.mcintire.me/graham/prop:main-1776606930-bb6333b [skip ci] 2026-04-19 13:57:42 +00:00
FluxCD
0e2d04f6d3 chore: update prop image to git.mcintire.me/graham/prop:main-1776606797-160439d [skip ci] 2026-04-19 13:55:40 +00:00
FluxCD
a8e225a92c chore: update prop image to git.mcintire.me/graham/prop:main-1776606153-504f514 [skip ci] 2026-04-19 13:44:38 +00:00
FluxCD
fe0bdd32c2 chore: update prop image to git.mcintire.me/graham/prop:main-1776605224-70e65ba [skip ci] 2026-04-19 13:29:36 +00:00
FluxCD
240bc30df8 chore: update prop image to git.mcintire.me/graham/prop:main-1776604171-28a9eb3 [skip ci] 2026-04-19 13:11:34 +00:00
FluxCD
69f4c81899 chore: update prop image to git.mcintire.me/graham/prop:main-1776551705-4e6c87e [skip ci] 2026-04-18 22:37:31 +00:00
FluxCD
d9ffda43d3 chore: update prop image to git.mcintire.me/graham/prop:main-1776549124-30d5bb7 [skip ci] 2026-04-18 21:55:26 +00:00
FluxCD
80b0dcd5ba chore: update prop image to git.mcintire.me/graham/prop:main-1776548438-c7bc3ed [skip ci] 2026-04-18 21:47:23 +00:00
FluxCD
0833ee44f9 chore: update prop image to git.mcintire.me/graham/prop:main-1776548026-2da74c5 [skip ci] 2026-04-18 21:36:02 +00:00
FluxCD
a69933a589 chore: update prop image to git.mcintire.me/graham/prop:main-1776547343-1a62e51 [skip ci] 2026-04-18 21:24:59 +00:00