Revert to smaller batch sizes to reduce garbage collection frequency
Reduced batch processing parameters to prevent excessive GC: - Reduced batch_size from 500 to 100 packets - Adjusted batch_timeout from 2000ms to 1000ms for faster processing - Reduced max_demand from 750 to 300 to match smaller batches - Updated hardcoded fallback from 500 to 100 in packet_consumer.ex - Changed minor GC trigger from 500 to 100 packets to match batch size These smaller batch sizes were proven to work well in production and should significantly reduce the frequency of garbage collection while maintaining good performance. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
2629589b7f
commit
a0c2970b42
2 changed files with 9 additions and 9 deletions
|
|
@ -62,12 +62,12 @@ config :aprsme,
|
|||
packet_pipeline: [
|
||||
# Larger buffer since inserts are async
|
||||
max_buffer_size: 5000,
|
||||
# ~32KB per packet, stays within work_mem
|
||||
batch_size: 500,
|
||||
# 2 seconds - longer timeout for larger batches
|
||||
batch_timeout: 2000,
|
||||
# Higher demand to keep pipeline flowing
|
||||
max_demand: 750,
|
||||
# Smaller batch size to reduce memory pressure
|
||||
batch_size: 100,
|
||||
# 1 second timeout for smaller batches
|
||||
batch_timeout: 1000,
|
||||
# Adjust demand for smaller batches
|
||||
max_demand: 300,
|
||||
# Number of parallel consumers for better throughput
|
||||
num_consumers: 3
|
||||
]
|
||||
|
|
|
|||
|
|
@ -122,8 +122,8 @@ defmodule Aprsme.PacketConsumer do
|
|||
start_time = System.monotonic_time(:millisecond)
|
||||
|
||||
# Chunk size optimized for PostgreSQL work_mem=16MB
|
||||
# With ~32KB per packet, we can fit ~500 packets in work_mem
|
||||
chunk_size = Application.get_env(:aprsme, :packet_pipeline)[:batch_size] || 500
|
||||
# Using smaller batches to reduce memory pressure
|
||||
chunk_size = Application.get_env(:aprsme, :packet_pipeline)[:batch_size] || 100
|
||||
|
||||
# Use Stream for memory-efficient processing
|
||||
results =
|
||||
|
|
@ -167,7 +167,7 @@ defmodule Aprsme.PacketConsumer do
|
|||
end
|
||||
|
||||
# Always do minor GC after large batches to prevent memory accumulation
|
||||
if length(packets) > 500 do
|
||||
if length(packets) > 100 do
|
||||
:erlang.garbage_collect(self(), type: :minor)
|
||||
end
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue