The plug was incorrectly placed in the endpoint, causing it to run on
all requests including APIs. Moved to :browser pipeline where it belongs,
and removed redundant fetch_session call since the pipeline handles it.
When Kubernetes performs rolling deployments, the old pod gracefully shuts down
and Oban's internal processes receive EXIT :shutdown signals. These are logged
as "unexpected" even though they're normal during shutdown.
Added custom logger filter to drop these messages in production:
- Oban.Queue.Watchman/Producer receiving EXIT :shutdown
- Oban plugin processes receiving EXIT :shutdown
- Reduces log noise during deployments without hiding real errors
The filter is production-only to preserve full debugging in development.
Replaced GenServer-based periodic workers with Oban Cron jobs to improve
pod rollover resilience and simplify architecture.
Worker Changes:
- NeighborCleanupWorker: GenServer → Oban Cron (hourly)
- Cleans stale neighbors, ARP entries, and MAC addresses
- Runs every hour via Oban.Plugins.Cron
- StaleAgentWorker: GenServer → Oban Cron (every minute)
- Detects agents that haven't checked in for 10+ minutes
- Refactored to reduce nesting (extracted helper functions)
- Removed stateful tracking (now stateless, re-evaluates each run)
- AgentLatencyEvaluator: GenServer → Oban Cron (every 5 minutes)
- Latency-based agent reassignment with 20% threshold
- Removed trigger_evaluation/0 (no longer needed)
- JobHealthCheckWorker: NEW Oban Cron worker (every 10 minutes)
- Safety net to recover missing monitor/poller jobs
- Auto-creates jobs for devices with monitoring/SNMP enabled
Infrastructure Changes:
- Removed Monitoring.Supervisor (no longer needed)
- Updated application.ex to remove GenServer workers from supervision tree
- Added Oban.Plugins.Cron to dev.exs and runtime.exs
- All workers now run cluster-wide via PostgreSQL-backed coordination
Test Updates:
- Updated all worker tests to call perform(%Oban.Job{args: %{}})
- Removed GenServer lifecycle tests (start_link, send messages, etc.)
- Removed async sleep calls (no longer needed)
Benefits:
- Better pod rollover resilience (Cron jobs run cluster-wide)
- Simpler architecture (no GenServers for periodic tasks)
- Better observability (all jobs visible in Oban dashboard)
- Safety net for missing jobs (JobHealthCheckWorker)
- Stateless workers (easier to reason about and test)
Documentation:
- Updated CLAUDE.md with Background Job Architecture section
- Documented job types, queues, and resilience features
All tests passing (3,686 tests, 0 failures).
Changes:
- Add Oban configuration to config/dev.exs (was only in runtime.exs)
- Replace Exq telemetry metrics with Oban equivalents
- Update publish_exq_stats to publish_oban_stats using Oban.Job queries
- Track queue sizes, executing jobs, and available jobs
This fixes the startup error where Oban config was not available in dev.
Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue.
This simplifies the architecture by eliminating the need for a separate etcd cluster
while providing better reliability and observability.
Changes:
- Add Oban dependency and migration (oban_jobs table)
- Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers
- Remove EtcdCoordinator and EtcdLock modules
- Update application supervisor to start Oban
- Configure Oban with pollers (50 workers) and monitors (50 workers) queues
- Remove etcd StatefulSet from Kubernetes manifests
- Update monitoring supervisor documentation
Benefits:
- Simpler architecture (no etcd cluster to manage)
- PostgreSQL-based (uses existing database)
- Built-in uniqueness prevents duplicate jobs cluster-wide
- Better observability with Oban Web UI
- Automatic job rescue on node crashes
- Easier local development (no etcd required)
What was removed:
- etcd StatefulSet (3 pods)
- EtcdCoordinator module (320 lines)
- EtcdLock module (158 lines)
- eetcd dependency
All 3,686 tests passing.
Redis authentication was failing with "NOAUTH Authentication required"
because the password wasn't being read from the environment variable.
Changes:
- Read REDIS_PASSWORD from environment in runtime.exs
- Add password to :redis config keyword list if present
- Phoenix.PubSub.Redis and Exq now read password from Application config
The application.ex and exq_supervisor.ex already had password support
from System.get_env("REDIS_PASSWORD"), but runtime.exs wasn't passing
the password through to the :redis config, so it was never available
to the application code.
This enables connection to the new Proxmox-hosted Valkey instance at
10.0.15.21 with authentication.
- Update MibTranslator to automatically expand configured MIB directories
- snmptranslate doesn't search recursively, so we explicitly list all subdirs
- Simplify config to only specify root priv/mibs directory
- Update both config.exs (production) and dev.exs (development)
- Subdirectories like mikrotik/, cisco/, net-snmp/ are auto-discovered
- Update config.exs to point to /app/priv/mibs in production
- Update dev.exs to include priv/mibs root for standard MIBs
- Remove obsolete :mib_dir (singular) config
- MIB files are now part of the release and included in Docker image
- Paths work in both development and production environments
- Add exq ~> 0.23 dependency with redix and elixir_uuid
- Configure Exq in application supervisor with 4 queues (default, discovery, polling, maintenance)
- Set Exq as included_application to prevent auto-start
- Only start Exq in dev/prod environments, skip in test
- Use :env config to determine runtime environment
- Configure 10 concurrent workers with exq namespace
- Connect to Redis/Valkey sidecar via existing :redis config
- Add Valkey 8.0 alpine container as sidecar in k8s deployment
- Configure with appropriate security context (non-root, no privilege escalation)
- Add resource limits (256Mi memory, 200m CPU) and requests (128Mi, 50m)
- Add health probes using valkey-cli ping command
- Configure Redis connection via REDIS_HOST and REDIS_PORT environment variables
- Add Redis config to runtime.exs, dev.exs, and test.exs
- Valkey runs on localhost:6379 within the same pod as Phoenix app
- Change default agent image to gmcintire/towerops-agent:main
- Add Watchtower service to docker-compose instructions
- Configure 15-minute update check interval for development
- Add DOCKER_API_VERSION=1.44 to prevent API compatibility errors
- Remove deprecated docker-compose version field
Bandit 1.10.1 does not support read_timeout/write_timeout in http_1_options
or stream_idle_timeout in http_2_options. These invalid options caused the
application to crash on startup with 'Unsupported key(s) in http_1_options'.
Moved read_timeout to thousand_island_options where it belongs, and removed
the unsupported write_timeout and stream_idle_timeout options.
Also fixed UserSettingsHTMLTest to include the missing mobile_sessions assign
that is required by the edit.html.heex template.
- Add comprehensive type definitions for vendor libraries (Chart.js, topbar)
- Add TypeScript types for LiveView hooks and WebAuthn interfaces
- Convert all JS files to TypeScript with proper typing:
- app.js → app.ts
- webauthn.js → webauthn.ts
- webauthn_utils.js → webauthn_utils.ts
- passkey_settings.js → passkey_settings.ts
- passkey_login.js → passkey_login.ts
- passkey_discoverable.js → passkey_discoverable.ts
- Update tsconfig.json with modern ES2022 and strict type checking
- Update esbuild config to use app.ts as entry point
- All assets compile successfully with esbuild's native TypeScript support
- No node_modules required, following Phoenix/Elixir conventions
Added log filter to suppress noisy errors from:
- HTTP/0.9 requests (obsolete protocol from 1991)
- Invalid HTTP version errors from port scanners and bots
- Malformed HTTP requests from automated tools
These errors are expected on public-facing servers and don't indicate
application problems. The filter uses Elixir's logger handler filters
to suppress them at the logging layer.
Resolves sporadic 'Bandit.HTTPError: Invalid HTTP version: {0, 9}' errors
in production logs.
The :accepts plug in the agent_api pipeline was only allowing JSON,
which caused 406 errors when the agent sent protobuf requests.
Changes:
- Register application/x-protobuf MIME type in config
- Add protobuf to agent_api pipeline accepts list
- Agent can now successfully communicate using protobuf format
Fixed two critical issues preventing agent communication:
1. Fix Mix.env() runtime error in production
- Replace Mix.env() with Application.get_env(:towerops, :env)
- Add :env config to test.exs
- Mix module not available in production releases
2. Add Protocol Buffers support to agent API endpoints
- GET /api/v1/agent/config now accepts application/x-protobuf
- POST /api/v1/agent/heartbeat now accepts application/x-protobuf
- Added conversion functions for config/equipment/sensors/interfaces
- Maintains JSON backward compatibility as fallback
All agent controller tests passing (14 tests, 0 failures)
- Added agent_docker_image config option (defaults to towerops/agent:latest)
- Updated agent UI modal to use configurable image URL
- Can override in runtime.exs or environment config for production
- Updated AGENT_NEXT_STEPS.md success criteria with completion status
- 8/12 success criteria now complete
SNMPKit library logs complete SNMP messages at debug level, which
includes the community string (credential). Setting log level to :info
prevents these debug messages from appearing in development logs.
Production and test environments already use :info and :warning levels.
