The map only ever shows the newest page of history, so zooming out stops at 500 packets #47

New issue

Open

opened 2026-11-03 18:52:26 +00:00 by skippy-bot · 0 comments

skippy-bot commented

2026-11-03 18:52:26 +00:00

Problem

On any viewport holding more packets than one batch, the map loads the newest batch and then asks for that same page again for every remaining batch. At zoom 10 to 11 the loader plans 1500 packets as 3 batches of 500, at 12 to 14 it plans 3000 as 6 batches, and at 15 to 20 it plans 5000 as 10 batches, so only the newest 500 render. Below zoom 10 the same thing happens at 100, 75 and 50 row batches. The loading indicator finishes normally and the repeated rows merge into the same markers, so nothing tells the user that history stopped at page one.

Evidence

At 2f150f66:

lib/aprsme/packet.ex:15 declares field(:received_at, :utc_datetime), so a row loaded from the database carries a %NaiveDateTime{}.
lib/aprsme_web/live/map_live/historical_loader.ex:270 takes received_at from the last returned row, and :273 assigns historical_cursor only if is_struct(received_at, DateTime) and the id is a binary. A NaiveDateTime fails that check, so no cursor is stored.
lib/aprsme_web/live/map_live/historical_loader.ex:203 then passes cursor: socket.assigns[:historical_cursor], which is still nil, into the next batch.
lib/aprsme/packets.ex:268 applies a cursor only for %{received_at: %DateTime{}}, and :276 silently drops every other cursor shape, so a NaiveDateTime cursor would not work even if it were stored.
lib/aprsme_web/live/map_live/historical_loader.ex:74 computes total_batches = ceil(limit / batch_size), and :240 treats a full page as has_more, so the loop keeps re-querying page one until the planned batch count runs out.
test/aprsme_web/live/map_live/historical_loader_test.exs:360 builds the cursor with DateTime.utc_now(), which is why the mismatch passes the suite.

Expected

The cursor is taken from whatever the query returns, NaiveDateTime included, and applied as received_at < cursor with the id tiebreaker, so a zoom 12 viewport reaches its 3000 packet budget and the duplicate fetches disappear.

How to verify

Add a loader test whose last packet carries received_at: NaiveDateTime.utc_now() and assert assigns.historical_cursor is set; in IEx, Aprsme.Packets.get_recent_packets_for_map(%{limit: 2}) |> List.last() |> Map.get(:received_at) returns a NaiveDateTime, and feeding it back as :cursor currently returns the same rows.

Acceptance criteria

A keyset cursor built from a NaiveDateTime row advances the second batch.
maybe_apply_cursor/2 accepts the timestamp type the schema actually loads, or the loader converts it, with a test on each side.
A zoom 12 viewport returns up to 3000 distinct packets in a test with more than 500 stored rows.
The loader never issues the same page twice for one load cycle.

Effort: M
Lens: data

## Problem On any viewport holding more packets than one batch, the map loads the newest batch and then asks for that same page again for every remaining batch. At zoom 10 to 11 the loader plans 1500 packets as 3 batches of 500, at 12 to 14 it plans 3000 as 6 batches, and at 15 to 20 it plans 5000 as 10 batches, so only the newest 500 render. Below zoom 10 the same thing happens at 100, 75 and 50 row batches. The loading indicator finishes normally and the repeated rows merge into the same markers, so nothing tells the user that history stopped at page one. ## Evidence At `2f150f66`: - `lib/aprsme/packet.ex:15` declares `field(:received_at, :utc_datetime)`, so a row loaded from the database carries a `%NaiveDateTime{}`. - `lib/aprsme_web/live/map_live/historical_loader.ex:270` takes `received_at` from the last returned row, and `:273` assigns `historical_cursor` only `if is_struct(received_at, DateTime)` and the id is a binary. A NaiveDateTime fails that check, so no cursor is stored. - `lib/aprsme_web/live/map_live/historical_loader.ex:203` then passes `cursor: socket.assigns[:historical_cursor]`, which is still nil, into the next batch. - `lib/aprsme/packets.ex:268` applies a cursor only for `%{received_at: %DateTime{}}`, and `:276` silently drops every other cursor shape, so a NaiveDateTime cursor would not work even if it were stored. - `lib/aprsme_web/live/map_live/historical_loader.ex:74` computes `total_batches = ceil(limit / batch_size)`, and `:240` treats a full page as `has_more`, so the loop keeps re-querying page one until the planned batch count runs out. - `test/aprsme_web/live/map_live/historical_loader_test.exs:360` builds the cursor with `DateTime.utc_now()`, which is why the mismatch passes the suite. ## Expected The cursor is taken from whatever the query returns, NaiveDateTime included, and applied as `received_at < cursor` with the id tiebreaker, so a zoom 12 viewport reaches its 3000 packet budget and the duplicate fetches disappear. ## How to verify Add a loader test whose last packet carries `received_at: NaiveDateTime.utc_now()` and assert `assigns.historical_cursor` is set; in IEx, `Aprsme.Packets.get_recent_packets_for_map(%{limit: 2}) |> List.last() |> Map.get(:received_at)` returns a NaiveDateTime, and feeding it back as `:cursor` currently returns the same rows. ## Acceptance criteria - [ ] A keyset cursor built from a NaiveDateTime row advances the second batch. - [ ] `maybe_apply_cursor/2` accepts the timestamp type the schema actually loads, or the loader converts it, with a test on each side. - [ ] A zoom 12 viewport returns up to 3000 distinct packets in a test with more than 500 stored rows. - [ ] The loader never issues the same page twice for one load cycle. **Effort:** M **Lens:** data

skippy-bot added the

High

bug

effort: M

labels

2026-11-03 18:52:26 +00:00