The memory monitoring was incorrectly using total VM memory instead of
per-process memory, causing false positives whenever any process in the
system allocated memory.
Changes:
- Changed memory monitoring to per-process instead of total VM memory
- Reduced thresholds to reasonable per-process values (50MB diff, 100MB total)
- Increased minor GC trigger from 100 to 1000 packets to prevent GC after
every normal batch
This prevents detecting memory changes from other processes and eliminates
the constant GC warnings while still maintaining proper memory management.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Reduced batch processing parameters to prevent excessive GC:
- Reduced batch_size from 500 to 100 packets
- Adjusted batch_timeout from 2000ms to 1000ms for faster processing
- Reduced max_demand from 750 to 300 to match smaller batches
- Updated hardcoded fallback from 500 to 100 in packet_consumer.ex
- Changed minor GC trigger from 500 to 100 packets to match batch size
These smaller batch sizes were proven to work well in production and
should significantly reduce the frequency of garbage collection while
maintaining good performance.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Increased memory thresholds from 10MB/20MB to 500MB/1GB since the server
has 15GB available memory. This prevents constant GC warnings while
maintaining memory safety.
Configured VM garbage collection settings in vm.args.eex:
- Enabled concurrent ports/sockets limit (+Q 65536)
- Set heap sizes and GC parameters optimized for better performance
- Added scheduler and binary heap settings
These changes will reduce unnecessary GC overhead and improve overall
application performance.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The PacketPipelineSetup was attempting to subscribe to a single
PacketConsumer process that doesn't exist. The application uses
PacketConsumerPool which automatically handles subscriptions for
multiple consumer processes, making PacketPipelineSetup redundant
and causing startup failures.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Create PacketConsumerPool to run 3 parallel consumers (configurable)
- Increase max_demand from 250 to 750 total (250 per consumer)
- Fix timer management to properly cancel/restart on batch processing
- Configure proper GenStage subscriptions with backpressure control
- Allow unnamed consumers for pool usage
This 3x increase in processing capacity prevents the "Packet buffer full"
warnings by ensuring consumers can keep up with incoming packet rate.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit introduces several major performance improvements and adds
robust error handling for malformed HTTP requests:
Performance Optimizations:
- Add database migration with BRIN indexes for time-series data and partial
indexes for recent queries, significantly improving query performance
- Create global StreamingPacketsPubSub system for real-time packet distribution
with geographic bounds filtering using ETS for fast lookups
- Refactor PacketConsumer to use Stream module for memory-efficient processing,
preventing memory accumulation during batch operations
- Implement dedicated BroadcastTaskSupervisor pool for async broadcast operations,
preventing GenServer blocking on I/O
- Increase database connection pool from 25 to 45 (production) and 15 to 30 (dev)
for better concurrency support
Error Handling:
- Add SentryFilter to prevent Bandit.HTTPError from missing Host headers from
cluttering Sentry (common with bots/scanners)
- Configure custom 400 Bad Request error handling
- Filter out common bot/scanner paths from error reporting
All changes include comprehensive test coverage following TDD practices.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Rename Gleam module from encoding_simple to encoding
- Move Gleam files from src/aprs/ to src/aprsme/ to match project namespace
- Create custom Mix task for Gleam compilation (lib/mix/tasks/gleam_compile.ex)
- Update EncodingUtils to wrap Gleam implementation instead of pure Elixir
- Add Gleam dependencies to mix.exs and configure build paths
- Update Mix aliases to include Gleam compilation in test and compile tasks
- Add Gleam support to GitHub Actions CI workflow with caching
- Add GLEAM_INTEGRATION.md documentation
- All 357 tests passing with Gleam integration
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replace bracket notation (packet_data[key]) with Map.get/2 to avoid
UndefinedFunctionError when processing ParseError structs that don't
implement the Access behavior.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Restored original PacketConsumer implementation from before optimizations
- Reverted to fixed batch size of 50 packets (original value)
- Removed SystemMonitor and InsertOptimizer from application startup
- Restored original config with buffer size 1000 and batch timeout 1000ms
- Removed all dynamic batch sizing and optimization logic
The recent performance optimizations were causing packet buffer overflows
because the dynamic batch sizing was actually slowing down processing.
Reverting to the simpler, working implementation that processes packets
consistently without buffer overflows.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Increased packet buffer size from 1,000 to 10,000 to handle traffic spikes
- Removed InsertOptimizer dynamic batch sizing that was slowing processing
- Reverted to fixed batch size of 200 for consistent performance
- Simplified insert options to use static configuration
- Added telemetry metrics for buffer overflow and utilization monitoring
- Fixed unused variable warning in packet_consumer.ex
The root cause was the recent performance optimizations that introduced
dynamic batch sizing, which actually degraded performance and caused
the producer buffer to overflow when the consumer couldn't keep up.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fixed InsertOptimizer returning map instead of keyword list for Ecto
- Increased batch size ranges to handle high load (100-800 packets)
- Improved consumer responsiveness:
- Reduced batch timeout from 1000ms to 500ms
- Process batches at 80% capacity for better throughput
- Reduced adjustment interval from 10s to 5s
- Added comprehensive logging for debugging buffer status
- Created integration tests for packet pipeline
- Coordinated batch sizing between SystemMonitor and InsertOptimizer
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Renamed predicate functions to follow Elixir conventions:
- is_finite? → finite?
- is_finite_number? → finite_number?
- is_finite_float? → finite_float?
- Fixed all function returns to ensure proper nil handling
- Applied mix format to all files
- All tests pass with no compilation warnings
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Security improvements:
- Added sanitize_numeric_string to remove dangerous characters
- Limited input string length to prevent DoS attacks (20 chars for numbers)
- Added is_finite? checks to prevent infinity/NaN values
- Validate coordinate ranges (lat: -90 to 90, lng: -180 to 180)
- Added safe_parse_coordinate with proper validation
- Updated to_float in EncodingUtils with security validations
- Protected against integer overflow in coordinate conversions
- Added sanitize_path_string for APRS path validation
All coordinate inputs from users are now:
1. Sanitized to remove injection characters
2. Length-limited to prevent resource exhaustion
3. Validated for finite values (no infinity/NaN)
4. Checked against valid geographic ranges
5. Given safe fallback defaults
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Key improvements:
- Fix spatial query order: Apply bounds filtering BEFORE limit/offset
- Add zoom-level based batch sizing (20-100 packets per batch)
- Dynamic batch count based on zoom level (2-5 batches)
- Faster loading delays for high zoom levels (25ms vs 50ms)
- Include zoom level in cache keys for better cache efficiency
Performance benefits:
- Zoomed in far (zoom 15+): 2 batches of 20 packets each
- Moderately zoomed (zoom 12-14): 3 batches of 35 packets each
- Medium zoom (zoom 8-11): 4 batches of 50 packets each
- Zoomed out (zoom <8): 5 batches of 75-100 packets each
This ensures packets are loaded from the current viewport first,
dramatically reducing load times when zoomed in.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
