Changes:
- Add Oban configuration to config/dev.exs (was only in runtime.exs)
- Replace Exq telemetry metrics with Oban equivalents
- Update publish_exq_stats to publish_oban_stats using Oban.Job queries
- Track queue sizes, executing jobs, and available jobs
This fixes the startup error where Oban config was not available in dev.
Wrapped all Exq.Api calls in individual try/rescue/catch blocks to handle
GenServer exits and exceptions when Exq hasn't connected to Redis yet.
The "GenServer Exq terminating" errors in logs are from Exq itself crashing
when it receives calls before it's ready, not from our code. These will still
appear in logs as Exq restarts, but our telemetry code won't crash.
Changes:
- Added try/rescue/catch around each queue_size call in the loop
- Added try/rescue/catch around processes/busy calls
- Kept outer try/rescue/catch as a final safety net
- rescue blocks must come before catch blocks in Elixir
Changed pattern matching to proper error handling in telemetry functions
to prevent crashes when Exq or Redis connections aren't available.
Issues fixed:
1. Exq.Api.queue_size/2 calls were using pattern matching {:ok, size} =
which crashed with FunctionClauseError when Exq wasn't connected to Redis
2. Redix.start_link/1 and Redix.command/2 calls were using pattern matching
which crashed when Redis wasn't available
Changes:
- publish_exq_stats: Use case statements for queue_size calls
- publish_exq_stats: Use with statement for processes/busy calls
- publish_redis_stats: Use with statement for all Redis operations
This allows the app to start and run even when Redis/Exq connections
aren't immediately available, which is especially important during
container startup and rolling deployments.
The `not` operator in Elixir only works with booleans, not nil values.
When Exq process doesn't exist, Process.whereis(Exq) returns nil, causing
`:badarg` error when used with `not`.
Error:
Class=:error
Reason=:badarg
{:erlang, :not, [nil], [error_info: %{module: :erl_erts_errors}]}
Changed from:
not Process.whereis(Exq)
To:
is_nil(Process.whereis(Exq))
This properly checks if the process exists without causing runtime errors.
- Check if Exq process is running before attempting to collect metrics
- Change rescue to catch to handle process exits (:noproc)
- Prevents error logs in environments where Exq is not configured
Added telemetry metrics for monitoring:
Exq Metrics:
- Queue sizes for all queues (default, discovery, polling, monitoring, maintenance)
- Number of busy worker processes
- Total number of worker processes
Redis/Valkey Metrics:
- Connected clients
- Memory usage
- Total commands processed
Metrics are collected every 10 seconds via telemetry_poller and displayed
in LiveDashboard at /dashboard
