Refactor and Hardening Implementation Handoff

Date: 2026-07-26

Purpose

This file is the continuation point for the repository-wide architecture, security, performance, SQL, and maintainability audit. A substantial first tranche has been implemented, but enough work remains that it should be handled as a separate, reviewed change set.

Do not use npm. This is a Phoenix application whose JavaScript is built with esbuild through Mix.

Current working tree

The working tree contains a large, uncommitted implementation. Preserve and review those changes before continuing. In particular, vendor/aprs is dirty and contains source/test changes. Those changes appeared during this implementation and must be reviewed as part of the work; do not blindly reset the submodule.

Useful first commands:

git status --short
git diff --check
git diff --stat
git -C vendor/aprs status --short
git diff --submodule=short -- vendor/aprs

Implemented

Replaced the split packet-streaming paths with Aprsme.SpatialPubSub for both LiveView and mobile clients.
Removed per-packet broadcast tasks and made subscriptions PID-aware and idempotent.
Removed obsolete packet receiver, streaming pubsub, replay, DB optimizer, insert optimizer, and sequence-counter infrastructure, together with their dead tests.
Added a stale-packet guard to the LiveView packet batcher.
Fixed the status page to use the correct task supervisor.
Added callsign/transport-identifier validation at ingestion and made client-side map marker construction safe from HTML injection.
Added user roles, an admin role task, and admin authorization for operational dashboards, error tracking, and bad-packet pages.
Hardened session and remember-me cookies and enabled production HTTPS/HSTS enforcement.
Upgraded Bandit and plug_crypto.
Made packet-retention partition deletion use an exact timestamp cutoff.
Added a partition-aware geometry GiST index migration.
Replaced the broken sequence-based packet count with an atomic row counter and reconciled the existing value.
Made partition deletion adjust the packet counter transactionally.
Corrected database telemetry for partitioned tables and removed fabricated pool metrics.
Changed cumulative PromEx values from counters to last-value metrics.
Made failed release migrations fail application startup.
Removed duplicate release initialization and supervised deployment notification.
Made PartitionManager the sole owner of packet retention.
Removed unused registries and duplicate/empty asset build targets and loaders.
Added Sobelow and a GitHub Actions workflow for format, compile, Credo, tests, Sobelow, Hex audit, and Dialyzer.
Updated affected architecture documentation.

Validation already completed

Full suite: 2447 passed (35 doctests, 31 properties, 2381 tests).
MIX_ENV=test mix compile --warnings-as-errors passed.
MIX_ENV=dev mix esbuild default passed.
git diff --check passed before the final documentation/runtime edits.

These checks must be rerun after all remaining work:

MIX_ENV=dev mix format
git diff --check
MIX_ENV=dev mix compile --warnings-as-errors
MIX_ENV=test mix compile --warnings-as-errors
MIX_ENV=dev mix credo --strict
MIX_ENV=dev mix sobelow --config
MIX_ENV=dev mix hex.audit
MIX_ENV=dev mix dialyzer
MIX_ENV=test mix test
MIX_ENV=dev mix esbuild default
MIX_ENV=prod mix assets.deploy

Remaining high-priority security work

Audit RemoteIp/forwarded-header handling. Only trust proxy headers from known ingress proxy CIDRs; direct clients must not be able to spoof their address. Add tests for trusted and untrusted peers.
Add server-side rate limiting to expensive or abusable LiveView events, mobile channel commands, authentication paths, and search endpoints. Do not rely only on controller plugs.
Finish Content Security Policy hardening. The root layout still depends on inline script behavior, so unsafe-inline has not been eliminated. Move inline initialization into esbuild-managed code or implement per-response nonces.
Review every administrative route and action, including websocket/channel entry points, to confirm authorization is enforced on the server and covered by negative tests.
Run Sobelow and triage the existing skip/fingerprint configuration. Remove stale skips and document any accepted finding rather than suppressing broadly.
Review secrets and credentials in Kubernetes manifests. Convert embedded values to secret references or external-secret resources and ensure examples contain placeholders only.
Add least-privilege Kubernetes NetworkPolicy rules for the web application, database, ingress, and any monitoring components.

Remaining reliability and performance work

Bound all ingress and client-facing buffers:
- APRS connection receive/reconnect buffers.
- Mobile channel pending packet lists.
- LiveView queues and task result accumulation. Define overflow behavior and expose drop/backpressure telemetry.
Put explicit limits on mobile connect_info, viewport/bounds payloads, callsign lists, search terms, and requested result counts. Reject malformed or oversized payloads before database work.
Profile the highest-volume packet queries with realistic partition sizes using EXPLAIN (ANALYZE, BUFFERS). Verify partition pruning and the new spatial index.
Remove remaining N+1 query patterns in map overlays, device/account pages, and status/operational pages. Prefer bounded batch queries and preloads.
Audit mobile and map search for unbounded scans. Add deterministic ordering, hard result caps, suitable indexes, and tests that assert the caps.
Review the packet counter migration under concurrent inserts and partition drops in a staging database. Exercise rollback/failed-migration behavior.
Load-test spatial subscriptions with overlapping bounds and reconnect churn. Confirm exactly-once delivery, bounded memory, and cleanup after process exits.

Remaining maintainability work

Split AprsmeWeb.MapLive.Index into focused state, event, subscription, and rendering modules. Preserve LiveView behavior with integration tests first.
Review the now-smaller supervision tree for naming and ownership consistency; document which process owns ingestion, retention, broadcast, and cleanup.
Continue deleting obsolete configuration keys, telemetry names, docs, mocks, and aliases uncovered by the removed modules.
Review callsign naming. Ingestion currently accepts safe APRS transport identifiers (up to 20 characters, uppercase alphanumeric segments separated by hyphens), which is intentionally broader than a strict AX.25 callsign. Rename APIs or document this distinction to avoid future accidental tightening.
Add focused migration tests for:
- Existing partition indexes being attached correctly.
- Counter reconciliation on an existing populated database.
- Exact cutoff behavior around partition boundaries.
Review the deleted test count. The suite decreased because tests for removed infrastructure were deleted; ensure important end-to-end ingestion coverage was not lost when packet_pipeline_integration_test.exs was removed.

Review notes

The geometry-index migration creates and attaches indexes per partition in production. Test behavior uses a simpler direct index path. Validate this on a production-like PostgreSQL version and data volume.
The packet-count trigger approach prioritizes exactness. Measure its write contention at expected ingestion rates; if it is too expensive, replace it with an explicitly approximate metric rather than a misleading exact API.
The CSP work should retain the early theme selection behavior to avoid a flash of the wrong theme.
Do not reintroduce StreamingPacketsPubSub or per-packet spawned broadcast tasks; extend SpatialPubSub if delivery behavior needs adjustment.

Suggested sequence

Review and isolate the dirty vendor/aprs changes.
Run all static gates and fix existing findings.
Complete proxy trust, rate limiting, CSP, and authorization review.
Bound buffers and payloads, then load-test delivery.
Profile and fix SQL query/index issues.
Refactor the large LiveView only after behavioral and performance coverage is stable.
Finish Kubernetes hardening and rerun the full release build.

8.3 KiB Raw Blame History