aprs.me/docs/architecture/ingestion-pipeline.md

Ingestion Pipeline

The packet ingestion pipeline moves raw APRS data from the APRS-IS network into Postgres and broadcasts to connected clients.

Components

Aprsme.Is (TCP Connection)

• File: lib/aprsme/is/is.ex
• Type: GenServer
• Role: Maintains TCP connection to APRS-IS, authenticates with callsign + passcode, receives raw APRS frames
• Parsing: Uses aprs.parse/1 (external Gleam library) to decode APRS frames
• Filtering: Configurable APRS-IS filter (by callsign, area, distance) via Aprsme.Is.LoginParams
• Backpressure: Responds to {:backpressure, :activate | :deactivate} signals from PacketProducer — sets TCP socket to :passive mode when buffer is full

PacketProducer (GenStage)

• File: lib/aprsme/packet_producer.ex
• Type: GenStage producer
• Buffer: Erlang :queue with water-mark backpressure
  • High water (>80% capacity): signals Aprsme.Is to activate backpressure
  • Low water (<30% capacity): signals Aprsme.Is to deactivate backpressure
• Demand-driven: Only emits packets when consumers are ready

PacketConsumer (GenStage Consumer)

• File: lib/aprsme/packet_consumer.ex
• Type: GenStage consumer (pooled via PacketConsumerPool)
• Processing pipeline per packet:
  1. Sanitize — PacketSanitizer.sanitize_packet/1 (overflow prevention)
  2. Normalize — EncodingUtils conversions (UTF-8, latin1, floats, decimals)
  3. Extract position — lat/lon → PostGIS geometry
  4. Extract device — DeviceParser.extract_device_id/1
  5. Filter fields — PacketFieldWhitelist trims to schema columns
  6. Insert — Repo.insert_all with on_conflict: :nothing (idempotent)
  7. Broadcast — async dispatch to PubSub systems

Broadcasting After Insert

After successful insert, each packet is broadcast to three PubSub layers:

1. StreamingPacketsPubSub — ETS-based, clients subscribe with bounds
2. SpatialPubSub — Grid-indexed (1° cells), viewport-filtered delivery
3. Phoenix.PubSub — Legacy per-topic broadcasts:
   • "postgres:aprsme_packets" — global packet stream
   • "packets:#{callsign}" — per-callsign updates
   • "weather:#{callsign}" — per-weather-station updates

Broadcast execution is async via BroadcastTaskSupervisor.

Error Handling

Packets that fail parsing or validation are stored in badpackets table via Packets.store_bad_packet/2 with error type/message, viewable at /badpackets page.

Configuration

• APRS-IS server, callsign, passcode set via application config
• Consumer pool size configurable per environment
• Retention period configurable via CleanupScheduler

Performance Notes

• PreparedQueries module caches frequently-used query plans
• InsertOptimizer batch-tunes insert operations
• PacketConsumer uses insert_all (not individual inserts) for throughput
• Daily table partitioning via PartitionManager keeps index sizes manageable