66 lines
2.9 KiB
Markdown
66 lines
2.9 KiB
Markdown
# Ingestion Pipeline
|
|
|
|
The packet ingestion pipeline moves raw APRS data from the APRS-IS network into Postgres and broadcasts to connected clients.
|
|
|
|
## Components
|
|
|
|
### Aprsme.Is (TCP Connection)
|
|
|
|
- **File:** `lib/aprsme/is/is.ex`
|
|
- **Type:** GenServer
|
|
- **Role:** Maintains TCP connection to APRS-IS, authenticates with callsign + passcode, receives raw APRS frames
|
|
- **Parsing:** Uses `aprs.parse/1` (external Gleam library) to decode APRS frames
|
|
- **Filtering:** Configurable APRS-IS filter (by callsign, area, distance) via `Aprsme.Is.LoginParams`
|
|
- **Backpressure:** Responds to `{:backpressure, :activate | :deactivate}` signals from `PacketProducer` — sets TCP socket to `:passive` mode when buffer is full
|
|
|
|
### PacketProducer (GenStage)
|
|
|
|
- **File:** `lib/aprsme/packet_producer.ex`
|
|
- **Type:** GenStage producer
|
|
- **Buffer:** Erlang `:queue` with water-mark backpressure
|
|
- **High water** (>80% capacity): signals `Aprsme.Is` to activate backpressure
|
|
- **Low water** (<30% capacity): signals `Aprsme.Is` to deactivate backpressure
|
|
- **Demand-driven:** Only emits packets when consumers are ready
|
|
|
|
### PacketConsumer (GenStage Consumer)
|
|
|
|
- **File:** `lib/aprsme/packet_consumer.ex`
|
|
- **Type:** GenStage consumer (pooled via `PacketConsumerPool`)
|
|
- **Processing pipeline per packet:**
|
|
1. Sanitize — `PacketSanitizer.sanitize_packet/1` (overflow prevention)
|
|
2. Normalize — `EncodingUtils` conversions (UTF-8, latin1, floats, decimals)
|
|
3. Extract position — lat/lon → PostGIS geometry
|
|
4. Extract device — `DeviceParser.extract_device_id/1`
|
|
5. Filter fields — `PacketFieldWhitelist` trims to schema columns
|
|
6. Insert — `Repo.insert_all` with `on_conflict: :nothing` (idempotent)
|
|
7. Broadcast — async dispatch to PubSub systems
|
|
|
|
### Broadcasting After Insert
|
|
|
|
After successful insert, each packet is broadcast to three PubSub layers:
|
|
|
|
1. **StreamingPacketsPubSub** — ETS-based, clients subscribe with bounds
|
|
2. **SpatialPubSub** — Grid-indexed (1° cells), viewport-filtered delivery
|
|
3. **Phoenix.PubSub** — Legacy per-topic broadcasts:
|
|
- `"postgres:aprsme_packets"` — global packet stream
|
|
- `"packets:#{callsign}"` — per-callsign updates
|
|
- `"weather:#{callsign}"` — per-weather-station updates
|
|
|
|
Broadcast execution is async via `BroadcastTaskSupervisor`.
|
|
|
|
## Error Handling
|
|
|
|
Packets that fail parsing or validation are stored in `badpackets` table via `Packets.store_bad_packet/2` with error type/message, viewable at `/badpackets` page.
|
|
|
|
## Configuration
|
|
|
|
- APRS-IS server, callsign, passcode set via application config
|
|
- Consumer pool size configurable per environment
|
|
- Retention period configurable via `CleanupScheduler`
|
|
|
|
## Performance Notes
|
|
|
|
- `PreparedQueries` module caches frequently-used query plans
|
|
- `InsertOptimizer` batch-tunes insert operations
|
|
- `PacketConsumer` uses `insert_all` (not individual inserts) for throughput
|
|
- Daily table partitioning via `PartitionManager` keeps index sizes manageable
|