Every deploy drops the packets that were still in the ingestion pipeline #49

New issue

Open

opened 2026-11-03 18:52:27 +00:00 by skippy-bot · 0 comments

skippy-bot commented

2026-11-03 18:52:27 +00:00

Problem

On SIGTERM the node marks itself draining and sleeps for 45 seconds. Nothing in that window touches ingestion: the APRS-IS socket stays open, the producer keeps buffering, and each consumer flushes only on its one second timer. Then System.stop(0) halts the VM, so each consumer's pending batch (up to 100 packets times three consumers) and the producer queue (up to 5000 packets while backpressure is active) are discarded with no log line and no count. Because this pod is still the APRS-IS leader for the whole drain, no other pod holds those packets either.

Evidence

At 2f150f66:

lib/aprsme/shutdown_handler.ex:160 schedules :begin_connection_drain after the health grace period, and :109 is Process.sleep(state.drain_timeout), which is the whole of the drain.
lib/aprsme/shutdown_handler.ex:144 calls System.stop(0) with ingestion still running.
lib/aprsme/shutdown_handler.ex:187 shows suspend_acceptors/1 is a no-op under Bandit, so the drain does not even stop HTTP intake.
lib/aprsme/packet_consumer.ex:84 starts the batch timer and :200 flushes on it; the module defines no terminate/2, so a halt loses the current batch.
lib/aprsme/is/is.ex:591 defines terminate/2, which closes the socket at :613 and flushes nothing to the producer.
config/config.exs:85 sets max_buffer_size: 5000 for the producer queue.

Expected

Shutdown stops ingestion first (close the APRS-IS socket, stop the producer), flushes each consumer's pending batch, and logs how many rows were written and how many were dropped before the VM halts. A 45 second sleep that keeps ingesting right up to the halt is worse than no delay.

How to verify

Feed a consumer fewer packets than batch_size, then stop the application (SIGTERM to a local release, or Application.stop(:aprsme) in IEx) and assert those rows exist in packets. Today the batch is gone.

Acceptance criteria

Shutdown closes the APRS-IS connection and stops the producer before the drain completes.
Each consumer flushes its pending batch on shutdown and the flush is covered by a test.
The shutdown logs written and dropped counts.

Effort: M
Lens: reliability

## Problem On SIGTERM the node marks itself draining and sleeps for 45 seconds. Nothing in that window touches ingestion: the APRS-IS socket stays open, the producer keeps buffering, and each consumer flushes only on its one second timer. Then `System.stop(0)` halts the VM, so each consumer's pending batch (up to 100 packets times three consumers) and the producer queue (up to 5000 packets while backpressure is active) are discarded with no log line and no count. Because this pod is still the APRS-IS leader for the whole drain, no other pod holds those packets either. ## Evidence At `2f150f66`: - `lib/aprsme/shutdown_handler.ex:160` schedules `:begin_connection_drain` after the health grace period, and `:109` is `Process.sleep(state.drain_timeout)`, which is the whole of the drain. - `lib/aprsme/shutdown_handler.ex:144` calls `System.stop(0)` with ingestion still running. - `lib/aprsme/shutdown_handler.ex:187` shows `suspend_acceptors/1` is a no-op under Bandit, so the drain does not even stop HTTP intake. - `lib/aprsme/packet_consumer.ex:84` starts the batch timer and `:200` flushes on it; the module defines no `terminate/2`, so a halt loses the current batch. - `lib/aprsme/is/is.ex:591` defines `terminate/2`, which closes the socket at `:613` and flushes nothing to the producer. - `config/config.exs:85` sets `max_buffer_size: 5000` for the producer queue. ## Expected Shutdown stops ingestion first (close the APRS-IS socket, stop the producer), flushes each consumer's pending batch, and logs how many rows were written and how many were dropped before the VM halts. A 45 second sleep that keeps ingesting right up to the halt is worse than no delay. ## How to verify Feed a consumer fewer packets than `batch_size`, then stop the application (SIGTERM to a local release, or `Application.stop(:aprsme)` in IEx) and assert those rows exist in `packets`. Today the batch is gone. ## Acceptance criteria - [ ] Shutdown closes the APRS-IS connection and stops the producer before the drain completes. - [ ] Each consumer flushes its pending batch on shutdown and the flush is covered by a test. - [ ] The shutdown logs written and dropped counts. **Effort:** M **Lens:** reliability

skippy-bot added the

Medium

bug

effort: M

labels

2026-11-03 18:52:27 +00:00