This commit introduces several major performance improvements and adds
robust error handling for malformed HTTP requests:
Performance Optimizations:
- Add database migration with BRIN indexes for time-series data and partial
indexes for recent queries, significantly improving query performance
- Create global StreamingPacketsPubSub system for real-time packet distribution
with geographic bounds filtering using ETS for fast lookups
- Refactor PacketConsumer to use Stream module for memory-efficient processing,
preventing memory accumulation during batch operations
- Implement dedicated BroadcastTaskSupervisor pool for async broadcast operations,
preventing GenServer blocking on I/O
- Increase database connection pool from 25 to 45 (production) and 15 to 30 (dev)
for better concurrency support
Error Handling:
- Add SentryFilter to prevent Bandit.HTTPError from missing Host headers from
cluttering Sentry (common with bots/scanners)
- Configure custom 400 Bad Request error handling
- Filter out common bot/scanner paths from error reporting
All changes include comprehensive test coverage following TDD practices.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add aggressive autovacuum settings for packet_counters table
- Replace table-based counter with PostgreSQL sequence to eliminate bloat
- Sequences are designed for high-frequency counter operations
- Maintains backward compatibility through get_packet_count() function
The packet_counters table was experiencing severe bloat due to 350K+ updates
on a single row. PostgreSQL sequences avoid this issue entirely.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Remove duplicate received_at indexes (packets_received_at_asc_idx and packets_recent_hour_idx)
that were consuming 423 MB, keeping only packets_received_at_idx
- Fix KeyError by initializing missing :all_packets and :visible_packets in socket assigns
- All tests now pass successfully
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
