Revert to smaller batch sizes to reduce garbage collection frequency

Reduced batch processing parameters to prevent excessive GC:
- Reduced batch_size from 500 to 100 packets
- Adjusted batch_timeout from 2000ms to 1000ms for faster processing
- Reduced max_demand from 750 to 300 to match smaller batches
- Updated hardcoded fallback from 500 to 100 in packet_consumer.ex
- Changed minor GC trigger from 500 to 100 packets to match batch size

These smaller batch sizes were proven to work well in production and
should significantly reduce the frequency of garbage collection while
maintaining good performance.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Graham McIntire 2025-07-19 17:46:04 -05:00
parent 2629589b7f
commit a0c2970b42
No known key found for this signature in database
2 changed files with 9 additions and 9 deletions

View file

@ -62,12 +62,12 @@ config :aprsme,
packet_pipeline: [
# Larger buffer since inserts are async
max_buffer_size: 5000,
# ~32KB per packet, stays within work_mem
batch_size: 500,
# 2 seconds - longer timeout for larger batches
batch_timeout: 2000,
# Higher demand to keep pipeline flowing
max_demand: 750,
# Smaller batch size to reduce memory pressure
batch_size: 100,
# 1 second timeout for smaller batches
batch_timeout: 1000,
# Adjust demand for smaller batches
max_demand: 300,
# Number of parallel consumers for better throughput
num_consumers: 3
]

View file

@ -122,8 +122,8 @@ defmodule Aprsme.PacketConsumer do
start_time = System.monotonic_time(:millisecond)
# Chunk size optimized for PostgreSQL work_mem=16MB
# With ~32KB per packet, we can fit ~500 packets in work_mem
chunk_size = Application.get_env(:aprsme, :packet_pipeline)[:batch_size] || 500
# Using smaller batches to reduce memory pressure
chunk_size = Application.get_env(:aprsme, :packet_pipeline)[:batch_size] || 100
# Use Stream for memory-efficient processing
results =
@ -167,7 +167,7 @@ defmodule Aprsme.PacketConsumer do
end
# Always do minor GC after large batches to prevent memory accumulation
if length(packets) > 500 do
if length(packets) > 100 do
:erlang.garbage_collect(self(), type: :minor)
end