- Update UserNotifier to use t_email() helper for all email content
- Extract 4 email subjects and 4 email bodies to emails domain:
* Update email instructions
* Log in instructions
* Confirmation instructions
* Reset password instructions
- Replace string interpolation with Gettext bindings (%{email}, %{url})
- Auto-populate English translations (msgstr = msgid)
- All emails now ready for future internationalization
First module migration complete per design plan.
The plug was incorrectly placed in the endpoint, causing it to run on
all requests including APIs. Moved to :browser pipeline where it belongs,
and removed redundant fetch_session call since the pipeline handles it.
When Kubernetes performs rolling deployments, the old pod gracefully shuts down
and Oban's internal processes receive EXIT :shutdown signals. These are logged
as "unexpected" even though they're normal during shutdown.
Added custom logger filter to drop these messages in production:
- Oban.Queue.Watchman/Producer receiving EXIT :shutdown
- Oban plugin processes receiving EXIT :shutdown
- Reduces log noise during deployments without hiding real errors
The filter is production-only to preserve full debugging in development.
Replaced GenServer-based periodic workers with Oban Cron jobs to improve
pod rollover resilience and simplify architecture.
Worker Changes:
- NeighborCleanupWorker: GenServer → Oban Cron (hourly)
- Cleans stale neighbors, ARP entries, and MAC addresses
- Runs every hour via Oban.Plugins.Cron
- StaleAgentWorker: GenServer → Oban Cron (every minute)
- Detects agents that haven't checked in for 10+ minutes
- Refactored to reduce nesting (extracted helper functions)
- Removed stateful tracking (now stateless, re-evaluates each run)
- AgentLatencyEvaluator: GenServer → Oban Cron (every 5 minutes)
- Latency-based agent reassignment with 20% threshold
- Removed trigger_evaluation/0 (no longer needed)
- JobHealthCheckWorker: NEW Oban Cron worker (every 10 minutes)
- Safety net to recover missing monitor/poller jobs
- Auto-creates jobs for devices with monitoring/SNMP enabled
Infrastructure Changes:
- Removed Monitoring.Supervisor (no longer needed)
- Updated application.ex to remove GenServer workers from supervision tree
- Added Oban.Plugins.Cron to dev.exs and runtime.exs
- All workers now run cluster-wide via PostgreSQL-backed coordination
Test Updates:
- Updated all worker tests to call perform(%Oban.Job{args: %{}})
- Removed GenServer lifecycle tests (start_link, send messages, etc.)
- Removed async sleep calls (no longer needed)
Benefits:
- Better pod rollover resilience (Cron jobs run cluster-wide)
- Simpler architecture (no GenServers for periodic tasks)
- Better observability (all jobs visible in Oban dashboard)
- Safety net for missing jobs (JobHealthCheckWorker)
- Stateless workers (easier to reason about and test)
Documentation:
- Updated CLAUDE.md with Background Job Architecture section
- Documented job types, queues, and resilience features
All tests passing (3,686 tests, 0 failures).
Changes:
- Add Oban configuration to config/dev.exs (was only in runtime.exs)
- Replace Exq telemetry metrics with Oban equivalents
- Update publish_exq_stats to publish_oban_stats using Oban.Job queries
- Track queue sizes, executing jobs, and available jobs
This fixes the startup error where Oban config was not available in dev.
Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue.
This simplifies the architecture by eliminating the need for a separate etcd cluster
while providing better reliability and observability.
Changes:
- Add Oban dependency and migration (oban_jobs table)
- Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers
- Remove EtcdCoordinator and EtcdLock modules
- Update application supervisor to start Oban
- Configure Oban with pollers (50 workers) and monitors (50 workers) queues
- Remove etcd StatefulSet from Kubernetes manifests
- Update monitoring supervisor documentation
Benefits:
- Simpler architecture (no etcd cluster to manage)
- PostgreSQL-based (uses existing database)
- Built-in uniqueness prevents duplicate jobs cluster-wide
- Better observability with Oban Web UI
- Automatic job rescue on node crashes
- Easier local development (no etcd required)
What was removed:
- etcd StatefulSet (3 pods)
- EtcdCoordinator module (320 lines)
- EtcdLock module (158 lines)
- eetcd dependency
All 3,686 tests passing.
Redis authentication was failing with "NOAUTH Authentication required"
because the password wasn't being read from the environment variable.
Changes:
- Read REDIS_PASSWORD from environment in runtime.exs
- Add password to :redis config keyword list if present
- Phoenix.PubSub.Redis and Exq now read password from Application config
The application.ex and exq_supervisor.ex already had password support
from System.get_env("REDIS_PASSWORD"), but runtime.exs wasn't passing
the password through to the :redis config, so it was never available
to the application code.
This enables connection to the new Proxmox-hosted Valkey instance at
10.0.15.21 with authentication.
- Update MibTranslator to automatically expand configured MIB directories
- snmptranslate doesn't search recursively, so we explicitly list all subdirs
- Simplify config to only specify root priv/mibs directory
- Update both config.exs (production) and dev.exs (development)
- Subdirectories like mikrotik/, cisco/, net-snmp/ are auto-discovered
- Update config.exs to point to /app/priv/mibs in production
- Update dev.exs to include priv/mibs root for standard MIBs
- Remove obsolete :mib_dir (singular) config
- MIB files are now part of the release and included in Docker image
- Paths work in both development and production environments
- Add exq ~> 0.23 dependency with redix and elixir_uuid
- Configure Exq in application supervisor with 4 queues (default, discovery, polling, maintenance)
- Set Exq as included_application to prevent auto-start
- Only start Exq in dev/prod environments, skip in test
- Use :env config to determine runtime environment
- Configure 10 concurrent workers with exq namespace
- Connect to Redis/Valkey sidecar via existing :redis config
- Add Valkey 8.0 alpine container as sidecar in k8s deployment
- Configure with appropriate security context (non-root, no privilege escalation)
- Add resource limits (256Mi memory, 200m CPU) and requests (128Mi, 50m)
- Add health probes using valkey-cli ping command
- Configure Redis connection via REDIS_HOST and REDIS_PORT environment variables
- Add Redis config to runtime.exs, dev.exs, and test.exs
- Valkey runs on localhost:6379 within the same pod as Phoenix app
- Change default agent image to gmcintire/towerops-agent:main
- Add Watchtower service to docker-compose instructions
- Configure 15-minute update check interval for development
- Add DOCKER_API_VERSION=1.44 to prevent API compatibility errors
- Remove deprecated docker-compose version field
Bandit 1.10.1 does not support read_timeout/write_timeout in http_1_options
or stream_idle_timeout in http_2_options. These invalid options caused the
application to crash on startup with 'Unsupported key(s) in http_1_options'.
Moved read_timeout to thousand_island_options where it belongs, and removed
the unsupported write_timeout and stream_idle_timeout options.
Also fixed UserSettingsHTMLTest to include the missing mobile_sessions assign
that is required by the edit.html.heex template.
- Add comprehensive type definitions for vendor libraries (Chart.js, topbar)
- Add TypeScript types for LiveView hooks and WebAuthn interfaces
- Convert all JS files to TypeScript with proper typing:
- app.js → app.ts
- webauthn.js → webauthn.ts
- webauthn_utils.js → webauthn_utils.ts
- passkey_settings.js → passkey_settings.ts
- passkey_login.js → passkey_login.ts
- passkey_discoverable.js → passkey_discoverable.ts
- Update tsconfig.json with modern ES2022 and strict type checking
- Update esbuild config to use app.ts as entry point
- All assets compile successfully with esbuild's native TypeScript support
- No node_modules required, following Phoenix/Elixir conventions
Added log filter to suppress noisy errors from:
- HTTP/0.9 requests (obsolete protocol from 1991)
- Invalid HTTP version errors from port scanners and bots
- Malformed HTTP requests from automated tools
These errors are expected on public-facing servers and don't indicate
application problems. The filter uses Elixir's logger handler filters
to suppress them at the logging layer.
Resolves sporadic 'Bandit.HTTPError: Invalid HTTP version: {0, 9}' errors
in production logs.
The :accepts plug in the agent_api pipeline was only allowing JSON,
which caused 406 errors when the agent sent protobuf requests.
Changes:
- Register application/x-protobuf MIME type in config
- Add protobuf to agent_api pipeline accepts list
- Agent can now successfully communicate using protobuf format
Fixed two critical issues preventing agent communication:
1. Fix Mix.env() runtime error in production
- Replace Mix.env() with Application.get_env(:towerops, :env)
- Add :env config to test.exs
- Mix module not available in production releases
2. Add Protocol Buffers support to agent API endpoints
- GET /api/v1/agent/config now accepts application/x-protobuf
- POST /api/v1/agent/heartbeat now accepts application/x-protobuf
- Added conversion functions for config/equipment/sensors/interfaces
- Maintains JSON backward compatibility as fallback
All agent controller tests passing (14 tests, 0 failures)
- Added agent_docker_image config option (defaults to towerops/agent:latest)
- Updated agent UI modal to use configurable image URL
- Can override in runtime.exs or environment config for production
- Updated AGENT_NEXT_STEPS.md success criteria with completion status
- 8/12 success criteria now complete
