Bound batch DISTINCT ON queries by input count. Guard Repo.all(Devices) with exists? check. Add trigram GIN index on base_callsign and (has_weather, received_at DESC) partial index.
Ultraworked with Sisyphus
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
- k8s: 2 replicas, POOL_SIZE 25, cpu 2000m, mem 1Gi (was 1 replica, 5, 500m, 512Mi)
- telemetry poller 10s -> 60s; drop pg_database_size, pg_stat_statements,
pg_table_size and pg_indexes_size (these were ~7100s of cumulative DB time
on their own)
- get_latest_packet_for_callsign: bound by :packet_retention_days so partition
pruning kicks in
- get_nearby_stations_knn: ST_DWithin spatial pre-filter (default 500km) so
the GiST geography index cuts candidates before DISTINCT ON sort
(EXPLAIN: 5793ms -> 400ms)
- New migration: partial functional indexes on upper(object_name) and
upper(item_name), built per-partition CONCURRENTLY then attached to parent.
Lets get_latest_packet_for_callsign use BitmapOr across all 3 upper()
indexes instead of falling back to a scan
- packet_consumer: when a batch exceeds max_batch_size, carry the
excess over to the next cycle instead of dropping it. Previously
any overage was split off into a drop list that was only logged,
losing packets the producer had already acknowledged.
- map_live: cancel the hover_end_timer in terminate/2 so a user
disconnecting while a marker is still highlighted doesn't leave
an orphaned timer behind.
- packet_consumer: rescue/log exceptions inside the async broadcast
task so PubSub serialization bugs or outages are observable
instead of silently killing the Task.
- prepared_queries: swap ST_Y/ST_X BETWEEN predicates for the && /
ST_MakeEnvelope pattern so bounds queries hit the GIST index on
packets.location instead of sequentially scanning.
Updated get_latest_packet_for_callsign to search by sender, object_name,
or item_name. This allows users to search for objects like "RADIOWRLD"
or "147.06-WC" and have the map center on them, just like searching for
a station callsign.
For example:
- Searching "RADIOWRLD" now finds the GPS Central store object
- Searching "147.06-WC" finds the Calgary repeater object
- Searching "CALGRY" still finds the Calgary station
This makes objects first-class searchable entities on the map.
- Replace N+1 get_latest_packet_for_callsign calls in RF path loading
with single get_latest_packets_for_callsigns batch query (DISTINCT ON)
- Change batch packet processing to collect marker removal IDs and push
one remove_markers_batch event instead of individual remove_marker calls
- Add callsignIndex reverse map (callsign_group → Set<markerID>) for O(1)
lookup in new_packet/new_packets handlers instead of scanning all markers
- Batch real-time packet push_events: collect marker data across the
packet batch and send a single "new_packets" WebSocket message instead
of one per packet. JS handler processes all live→historical conversions
before adding new markers to avoid intra-batch conflicts.
- Batch RF path station queries: replace N+1 get_latest_packet_for_callsign
calls with a single DISTINCT ON query selecting only callsign + lat/lng.
- Add read_concurrency: true to streaming_packets_subscribers and :aprsme
ETS tables (read-heavy, rarely written).
- Remove dead GC code in PacketConsumer (threshold > 1000 unreachable with
max batch_size of 100).
