- Add 'error' to valid sensor reading status values
- SensorReading changeset was rejecting 'error' status
- PollWorker creates error readings when SNMP fails
- Fixed validation to include: ok, warning, critical, error
- Replace log capture tests with functional assertions
- ExUnit.CaptureLog not reliably capturing worker logs
- Changed to verify actual behavior instead of log messages
- Tests now check created records and state changes
- Add debug assertions to sensor error test
- Verify SNMP device exists and has sensors
- Better error messages for test failures
All 1045 tests now passing (0 failures, 2 skipped)
Configures Redix (Exq's Redis client) with better connection handling:
- sync_connect: true - fail fast if Redis unavailable at startup
- exit_on_disconnection: false - don't crash process on disconnect
- backoff_initial: 500ms, backoff_max: 30s - retry with exponential backoff
This helps Exq survive temporary Redis disconnections without crashing,
reducing the frequency of Protocol.UndefinedError crashes in Exq.Node.Server.
Fixes crashes in Exq.Node.Server when Redis connections fail by adding
health checks before startup and wrapping Exq in a custom supervisor
with better resilience.
Problem:
- Valkey pod restarting frequently (24 times in 28h) due to K8s Flannel CNI issues
- When Redis disconnects, Exq.Node.Server crashes trying to process_signals/2
- Gets nil from Redis instead of expected list, crashes on Enum.each/2
- Rapid crash/restart cycles reduce application stability
Changes:
1. RedisHealthCheck module (lib/towerops/redis_health_check.ex)
- Waits for Redis availability with exponential backoff before starting Exq
- Prevents Exq from starting when Redis is unavailable
- Handles connection errors gracefully without crashing caller processes
- Supports configurable retry attempts, timeouts, and backoff intervals
2. ExqSupervisor module (lib/towerops/exq_supervisor.ex)
- Custom supervisor that wraps Exq with health checks
- Uses one_for_one strategy with limited restarts (3 per 60s)
- Prevents rapid crash loops when Redis is unstable
- Increased Exq retry/backoff settings for better Redis recovery
- Allows application to continue functioning without background jobs if Redis unavailable
3. Updated Application.ex
- Replaced direct Exq start with ExqSupervisor
- Removed inline exq_config/0 function (moved to supervisor)
4. Tests
- Comprehensive tests for RedisHealthCheck (health checks, retries, timeouts)
- Tests for ExqSupervisor (configuration, restart strategy)
- All 1006 tests passing
Benefits:
- Prevents Exq crashes from propagating when Redis fails
- Application continues running even when Redis is temporarily unavailable
- Better logging of Redis connection issues
- More graceful degradation during infrastructure problems
Infrastructure Issue (to be addressed separately):
- K8s Flannel CNI plugin failing: /run/flannel/subnet.env not found
- Causing Valkey pod network issues and restarts
- Needs investigation of Flannel daemonset and node configuration
🤖 Generated with Claude Code
- Update MibTranslator to automatically expand configured MIB directories
- snmptranslate doesn't search recursively, so we explicitly list all subdirs
- Simplify config to only specify root priv/mibs directory
- Update both config.exs (production) and dev.exs (development)
- Subdirectories like mikrotik/, cisco/, net-snmp/ are auto-discovered
- Remove PVC volume mount from k8s/deployment.yaml
- Delete k8s/mib-pvc.yaml (no longer needed)
- Update .gitignore to commit MIB files to git
- Add mix import_mibs task to copy MIBs from LibreNMS
- Add all MIB files from priv/mibs/ to git
This fixes the multi-attach volume error in Kubernetes where new pods
couldn't start because the RWO (ReadWriteOnce) PVC was attached to the
old pod. MIBs are now part of the Docker image and can be used by all
pods simultaneously.
- Add monitoring_enabled and check_interval_seconds fields to Device and SnmpDevice protobuf messages
- Add MonitoringCheck protobuf message for ping results
- Update AgentChannel to handle monitoring_check events from agents
- Configure monitoring settings in job payloads sent to agents
- Created new on_mount hook :load_default_organization that automatically loads user's first organization
- Moved sites and devices routes to root path (/ sites, /devices) instead of /orgs/:org_slug
- Updated all navigation links and route references throughout the app
- Dashboard, alerts, and agents still use org-specific routes (/orgs/:org_slug)
- Users can still switch organizations via org selector for other pages
The neighbor discovery was using snmp_device.id instead of the actual
Equipment device.id when creating neighbor records. This caused all
neighbor inserts to fail because the foreign key constraint expected
the Equipment ID, not the SNMP Device ID.
Fixed in two places:
- Discovery: Use snmp_device.device_id (FK to Equipment)
- PollerWorker: Use device.id (Equipment ID) instead of snmp_device.id
Neighbors should now be properly discovered and saved during both
initial discovery and ongoing polling.
- Pass range parameter to SensorChart hook via data-range attribute
- Calculate x-axis min/max dynamically based on selected range (1h, 6h, 12h, 24h, 7d, 30d)
- Show date + time on x-axis labels for ranges > 24 hours
- Show time only for ranges <= 24 hours
- Fixes issue where 7d and 30d graphs showed incorrect time range
- Changed ping module to use microsecond timer resolution instead of millisecond
- Prevents 0ms readings for very fast pings (localhost, local network)
- Updated response_time_ms to float type to store decimal precision
- Migration converts existing integer values to float
- Add timezone to socket assigns in mount_current_scope
- Detect user timezone from browser using Intl.DateTimeFormat
- Auto-save detected timezone to user profile on first connection
- Update all templates to use TimeHelpers with timezone parameter
- Update agent helper functions to accept timezone parameter
- Templates updated:
- Alert list (triggered, acknowledged, resolved)
- Device list (last checked)
- Agent list and show (created, last seen, updated)
- Admin dashboard (impersonation logs)
- Admin org/user lists (created/joined dates)
- Main dashboard (alert times)
Wrapped all Exq.Api calls in individual try/rescue/catch blocks to handle
GenServer exits and exceptions when Exq hasn't connected to Redis yet.
The "GenServer Exq terminating" errors in logs are from Exq itself crashing
when it receives calls before it's ready, not from our code. These will still
appear in logs as Exq restarts, but our telemetry code won't crash.
Changes:
- Added try/rescue/catch around each queue_size call in the loop
- Added try/rescue/catch around processes/busy calls
- Kept outer try/rescue/catch as a final safety net
- rescue blocks must come before catch blocks in Elixir
Fixed two profile import issues:
1. Skip sensors without OIDs:
- Some profiles have sensors with null/missing OID fields
- These can't be used for SNMP polling anyway
- Added check to skip importing sensors without oid or num_oid
- Prevents Ecto.Changeset validation errors
2. Truncate MIB values to 255 characters:
- ERROR 22001 (string_data_right_truncation)
- MIB field is varchar(255) but some values exceed this
- Added truncation to first 255 chars before saving
- Location: lib/towerops/device_profiles/importer.ex:229-236
Both errors were causing profile imports to fail with crashes.
Changed pattern matching to proper error handling in telemetry functions
to prevent crashes when Exq or Redis connections aren't available.
Issues fixed:
1. Exq.Api.queue_size/2 calls were using pattern matching {:ok, size} =
which crashed with FunctionClauseError when Exq wasn't connected to Redis
2. Redix.start_link/1 and Redix.command/2 calls were using pattern matching
which crashed when Redis wasn't available
Changes:
- publish_exq_stats: Use case statements for queue_size calls
- publish_exq_stats: Use with statement for processes/busy calls
- publish_redis_stats: Use with statement for all Redis operations
This allows the app to start and run even when Redis/Exq connections
aren't immediately available, which is especially important during
container startup and rolling deployments.
Fixed two import-related errors:
1. DateTime microseconds error when updating API token last_used_at:
- Error: :utc_datetime expects microseconds to be empty
- Fix: Use DateTime.truncate(:second) to remove microseconds
- Location: lib/towerops/api_tokens.ex:179
2. Float parsing error for integer strings like "0":
- Error: :erlang.binary_to_float("0") - not a textual representation of float
- Fix: Use Float.parse/1 instead of String.to_float/1
- Float.parse/1 handles both "0" and "0.0" correctly
- Returns {float, remainder} on success, :error on failure
- Location: lib/towerops/device_profiles/importer.ex:380
Both errors were causing profile imports to fail with crashes.
The `not` operator in Elixir only works with booleans, not nil values.
When Exq process doesn't exist, Process.whereis(Exq) returns nil, causing
`:badarg` error when used with `not`.
Error:
Class=:error
Reason=:badarg
{:erlang, :not, [nil], [error_info: %{module: :erl_erts_errors}]}
Changed from:
not Process.whereis(Exq)
To:
is_nil(Process.whereis(Exq))
This properly checks if the process exists without causing runtime errors.
The token success modal was missing z-index classes that caused the modal
content to appear behind the grey backdrop. Users could only see the
checkmark icon and couldn't interact with the modal properly.
Changes:
- Added z-0 class to backdrop div
- Added relative z-10 classes to span and modal content div
- Added unique ID to modal (token-success-modal)
- Added unique ID to modal title (token-success-modal-title)
This matches the z-index structure used in the "add token" modal.
API token creation fix:
- Handle both nested (%{"token" => %{...}}) and flat params
- Support URL-encoded form data from LiveView
- Add validation for required name and organization_id fields
Rust agent raw ICMP ping implementation:
- New ping module with async ICMP echo request/reply
- Build ICMP packets manually with proper checksum calculation
- Use socket2 for raw socket creation (requires CAP_NET_RAW)
- Parse both raw ICMP and IP-wrapped responses
- Include unit tests for checksum and packet building
- Dependencies: socket2 v0.5, rand v0.8
- Compiles successfully with release profile optimizations
AutoDismissFlash hook fix:
- Move hook definitions before LiveSocket initialization
- Ensures hooks are defined when referenced in LiveSocket constructor
- Remove duplicate Hooks object
Raw ICMP ping implementation:
- Replace system ping command with pure Elixir implementation
- Use raw ICMP sockets for echo request/reply
- Build ICMP packets manually with proper checksum calculation
- Parse IP addresses using :inet.parse_address/1
- Handle both raw ICMP and IP-wrapped responses
- Requires CAP_NET_RAW capability in production containers
- Suitable for containerized environments without ping command
