fix(otel): switch Elixir exporter to OTLP/HTTP to avoid grpcbox shutdown noise
Every prop pod restart was dumping a pair of ArgumentError stack
traces from grpcbox_channel / grpcbox_subchannel terminate/3:
:ets.select(:gproc, …)
* 1st argument: the table identifier does not refer to an
existing ETS table
This is a known grpcbox shutdown race: the OTLP/gRPC transport's
terminate callback tries to look itself up in gproc's ETS just
after gproc's supervisor has already torn the table down. Noisy,
not harmful — but one ArgumentError per restart × N pods adds up.
The OTLP/HTTP transport (port 4318 on the same otel-collector
Service) has no gproc/ETS channel machinery, so shutdown is
quiet. Spans are identical — our collector already accepts both.
- `config/runtime.exs`: default `otlp_protocol: :http_protobuf`,
with an `OTEL_EXPORTER_OTLP_PROTOCOL=grpc` escape hatch.
- `k8s/deployment.yaml`: point `OTEL_EXPORTER_OTLP_ENDPOINT` at
:4318 instead of :4317. The Rust workers stay on :4317 — their
native opentelemetry-otlp transport doesn't share the grpcbox
bug.
This commit is contained in:
parent
83aa3115df
commit
c46e89a48f
2 changed files with 22 additions and 2 deletions
|
|
@ -366,6 +366,19 @@ if config_env() == :prod do
|
|||
end
|
||||
|
||||
if otlp_endpoint = System.get_env("OTEL_EXPORTER_OTLP_ENDPOINT") do
|
||||
# OTLP/HTTP + protobuf over the collector's 4318 port. We were
|
||||
# previously using the gRPC exporter on 4317, which is transported
|
||||
# by `grpcbox` — that channel's terminate/3 callback races with
|
||||
# gproc's ETS teardown on BEAM shutdown and dumps a pair of
|
||||
# ArgumentError stack traces every pod restart. The HTTP/protobuf
|
||||
# exporter has no channel abstraction to unwind, so shutdown is
|
||||
# quiet. Spans/metrics payload shape is identical.
|
||||
otlp_protocol =
|
||||
case System.get_env("OTEL_EXPORTER_OTLP_PROTOCOL") do
|
||||
"grpc" -> :grpc
|
||||
_ -> :http_protobuf
|
||||
end
|
||||
|
||||
config :opentelemetry, :resource,
|
||||
service: [
|
||||
name: "microwaveprop",
|
||||
|
|
@ -377,7 +390,7 @@ if otlp_endpoint = System.get_env("OTEL_EXPORTER_OTLP_ENDPOINT") do
|
|||
traces_exporter: :otlp
|
||||
|
||||
config :opentelemetry_exporter,
|
||||
otlp_protocol: :grpc,
|
||||
otlp_protocol: otlp_protocol,
|
||||
otlp_endpoint: otlp_endpoint
|
||||
else
|
||||
config :opentelemetry, traces_exporter: :none
|
||||
|
|
|
|||
|
|
@ -67,8 +67,15 @@ spec:
|
|||
# endpoint used by prop-grid-rs / prop-hrrr-point-rs so
|
||||
# Elixir-initiated traces link up with downstream Rust
|
||||
# worker spans in Tempo.
|
||||
# Port 4318 = OTLP/HTTP. We intentionally bypass the
|
||||
# 4317 gRPC path because the Elixir grpcbox transport's
|
||||
# terminate/3 callback races with gproc's ETS teardown on
|
||||
# pod shutdown and dumps a pair of ArgumentError stacks
|
||||
# per restart — HTTP/protobuf has no channel abstraction
|
||||
# to unwind and shuts down quietly. Collector accepts
|
||||
# both.
|
||||
- name: OTEL_EXPORTER_OTLP_ENDPOINT
|
||||
value: "http://otel-collector.observability.svc.cluster.local:4317"
|
||||
value: "http://otel-collector.observability.svc.cluster.local:4318"
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: prop-secrets
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue