# Ingestion Pipeline

The packet ingestion pipeline moves raw APRS data from the APRS-IS network into Postgres and broadcasts to connected clients.

## Components

### Aprsme.Is (TCP Connection)

- **File:** `lib/aprsme/is/is.ex`
- **Type:** GenServer
- **Role:** Maintains TCP connection to APRS-IS, authenticates with callsign + passcode, receives raw APRS frames
- **Parsing:** Uses `aprs.parse/1` (external Gleam library) to decode APRS frames
- **Filtering:** Configurable APRS-IS filter (by callsign, area, distance) via `Aprsme.Is.LoginParams`
- **Backpressure:** Responds to `{:backpressure, :activate | :deactivate}` signals from `PacketProducer` — sets TCP socket to `:passive` mode when buffer is full

### PacketProducer (GenStage)

- **File:** `lib/aprsme/packet_producer.ex`
- **Type:** GenStage producer
- **Buffer:** Erlang `:queue` with water-mark backpressure
  - **High water** (>80% capacity): signals `Aprsme.Is` to activate backpressure
  - **Low water** (<30% capacity): signals `Aprsme.Is` to deactivate backpressure
- **Demand-driven:** Only emits packets when consumers are ready

### PacketConsumer (GenStage Consumer)

- **File:** `lib/aprsme/packet_consumer.ex`
- **Type:** GenStage consumer (pooled via `PacketConsumerPool`)
- **Processing pipeline per packet:**
  1. Sanitize — `PacketSanitizer.sanitize_packet/1` (overflow prevention)
  2. Normalize — `EncodingUtils` conversions (UTF-8, latin1, floats, decimals)
  3. Extract position — lat/lon → PostGIS geometry
  4. Extract device — `DeviceParser.extract_device_id/1`
  5. Filter fields — `PacketFieldWhitelist` trims to schema columns
  6. Insert — `Repo.insert_all` with `on_conflict: :nothing` (idempotent)
  7. Broadcast — async dispatch to PubSub systems

### Broadcasting After Insert

After successful insert, each packet is broadcast to three PubSub layers:

1. **StreamingPacketsPubSub** — ETS-based, clients subscribe with bounds
2. **SpatialPubSub** — Grid-indexed (1° cells), viewport-filtered delivery
3. **Phoenix.PubSub** — Legacy per-topic broadcasts:
   - `"postgres:aprsme_packets"` — global packet stream
   - `"packets:#{callsign}"` — per-callsign updates
   - `"weather:#{callsign}"` — per-weather-station updates

Broadcast execution is async via `BroadcastTaskSupervisor`.

## Error Handling

Packets that fail parsing or validation are stored in `badpackets` table via `Packets.store_bad_packet/2` with error type/message, viewable at `/badpackets` page.

## Configuration

- APRS-IS server, callsign, passcode set via application config
- Consumer pool size configurable per environment
- Retention period configurable via `CleanupScheduler`

## Performance Notes

- `PreparedQueries` module caches frequently-used query plans
- `InsertOptimizer` batch-tunes insert operations
- `PacketConsumer` uses `insert_all` (not individual inserts) for throughput
- Daily table partitioning via `PartitionManager` keeps index sizes manageable