Eliminates redundant work on the hot path:
1. Remove struct_to_map — Map.put on struct already returns plain map,
eliminating O(n) recursive traversal of the entire parser struct tree
per packet. [15-25% gain]
2. Delete data_extended early — moves Map.delete to before recursive
sanitization/DateTime walks, avoiding wasted work on data that is
already extracted to top-level columns. [8-12% gain]
3. Truncate received_at to :second at source — removes microsecond
truncation + the recursive truncate_datetimes_to_second pass.
[5-8% gain]
4. Merge sanitize passes — new sanitize_packet_with_encoding/1 does
truncation + encoding sanitization in one Map.new pass instead of
two sequential traversals. [5-8% gain]
5. Fix raw→raw_packet key — consumer read :raw_packet but dispatch
wrote :raw, silently dropping raw packet data to the DB column.
6. Fold has_weather into extract_additional_data — set when weather
is found during extraction instead of scanning 10 fields per packet.
[2-4% gain]
7. filter_fields with MapSet — O(1) membership check instead of O(n)
list scan. [2-3% gain]
8. Single-pass chunk reduction — builds valid_inserts and valid_bcasts
in one Enum.reduce, saving 2 extra list traversals. [2-3% gain]
Dead code removed: struct_to_map/1, extract_from_mic_e_map/1,
set_has_weather/1, set_received_at/1, truncate_datetimes_to_second/1,
__original_struct__ MicE branch.
Tests: 2481/2490 passing (+3 improvement, remaining 9 are pre-existing)
Drop the pre-insert aprs_messages broadcast in Is.dispatch (subscribers
already get a richer payload via postgres:aprsme_packets after insert).
Switch PacketsLive.CallsignView to the per-callsign packets:<CS> topic
so it only receives relevant packets instead of filtering every one.
In PacketConsumer: reuse the received_at stamped in Is.dispatch instead
of calling DateTime.utc_now/0 per packet; drop the duplicate struct_to_map
pass (Is.dispatch already handled it); fold coordinate validation and
Geo.Point construction into set_lat_lon + create_location_geometry so
coords are normalized once; extract device_identifier from the already-
normalized attrs; pre-build the broadcast payload once (identifier pick,
lat/lon aliases, routing callsign) so the async broadcast task no longer
does Map.drop/Map.merge per packet.
Rewrite PacketSanitizer.sanitize_packet / sanitize_data_map with Map.new/2
instead of Enum.reduce + Map.put — one allocation per packet instead of N.
Also compute has_weather in Packet.changeset/2 so direct-changeset inserts
(tests, backfills) populate it the same way the GenStage pipeline does.
Drop 10 dead parser-compat columns (srccallsign, dstcallsign,
origpacket, body, header, alive, posambiguity, symboltable,
symbolcode, messaging) and move 14 display-only columns into a
single `data` JSONB column (PHG, telemetry, radiorange,
information_field, format, posresolution, position_ambiguity,
luminosity, rain_midnight).
Reduces row width from ~70 to ~48 columns. Table is ephemeral
so migration uses DROP/recreate. Also fixes snow_24h -> snow bug
in weather check query and adds null byte stripping for JSONB
string sanitization.
- Create migration to change string fields from VARCHAR(255) to TEXT type
- Add PacketSanitizer module for UTF-8 safe string truncation
- Update PacketConsumer to sanitize data before database insertion
- Prevents database errors from overly long packet data fields
