No description
Graham McIntire
ee8a3220c4
Replaced GenServer-based periodic workers with Oban Cron jobs to improve
pod rollover resilience and simplify architecture.
Worker Changes:
- NeighborCleanupWorker: GenServer → Oban Cron (hourly)
- Cleans stale neighbors, ARP entries, and MAC addresses
- Runs every hour via Oban.Plugins.Cron
- StaleAgentWorker: GenServer → Oban Cron (every minute)
- Detects agents that haven't checked in for 10+ minutes
- Refactored to reduce nesting (extracted helper functions)
- Removed stateful tracking (now stateless, re-evaluates each run)
- AgentLatencyEvaluator: GenServer → Oban Cron (every 5 minutes)
- Latency-based agent reassignment with 20% threshold
- Removed trigger_evaluation/0 (no longer needed)
- JobHealthCheckWorker: NEW Oban Cron worker (every 10 minutes)
- Safety net to recover missing monitor/poller jobs
- Auto-creates jobs for devices with monitoring/SNMP enabled
Infrastructure Changes:
- Removed Monitoring.Supervisor (no longer needed)
- Updated application.ex to remove GenServer workers from supervision tree
- Added Oban.Plugins.Cron to dev.exs and runtime.exs
- All workers now run cluster-wide via PostgreSQL-backed coordination
Test Updates:
- Updated all worker tests to call perform(%Oban.Job{args: %{}})
- Removed GenServer lifecycle tests (start_link, send messages, etc.)
- Removed async sleep calls (no longer needed)
Benefits:
- Better pod rollover resilience (Cron jobs run cluster-wide)
- Simpler architecture (no GenServers for periodic tasks)
- Better observability (all jobs visible in Oban dashboard)
- Safety net for missing jobs (JobHealthCheckWorker)
- Stateless workers (easier to reason about and test)
Documentation:
- Updated CLAUDE.md with Background Job Architecture section
- Documented job types, queues, and resilience features
All tests passing (3,686 tests, 0 failures).
|
||
|---|---|---|
| .gitlab/agents | ||
| assets | ||
| config | ||
| docs | ||
| k8s | ||
| lib | ||
| mibs | ||
| priv | ||
| rel/overlays/bin | ||
| scripts | ||
| test | ||
| towerops | ||
| .credo.exs | ||
| .dialyzer_ignore.exs | ||
| .dockerignore | ||
| .formatter.exs | ||
| .gitignore | ||
| .gitlab-ci.yml | ||
| .sourceignore | ||
| .test-watch.exs | ||
| .tool-versions | ||
| AGENTS.md | ||
| CLAUDE.md | ||
| DEPLOYMENT.md | ||
| Dockerfile | ||
| LIBRENMS_INTEGRATION.md | ||
| mix.exs | ||
| mix.lock | ||
| MOBILE_API.md | ||
| PROFILES.md | ||
| PROFILES_README.md | ||
| README.md | ||
| tail_logs.sh | ||
TowerOps
Network monitoring and alerting platform built with Phoenix LiveView.
Features
- Multi-tenant architecture - Organizations with role-based permissions
- Site hierarchy - Organize equipment across multiple sites
- Automated monitoring - Real-time ping monitoring with configurable intervals
- Time-series data - Efficient storage with TimescaleDB (optional)
- Real-time updates - LiveView dashboard with PubSub
- Equipment tracking - Monitor network devices by IP address
Quick Start
Prerequisites
- Elixir 1.14+
- PostgreSQL 14+
- (Optional) TimescaleDB for production-grade time-series performance
Setup
# Install dependencies
mix setup
# Start the server
mix phx.server
Visit localhost:4000 from your browser.
TimescaleDB (Production Only)
Development: Uses standard PostgreSQL (no TimescaleDB required).
Production: TimescaleDB is automatically enabled for optimal performance with time-series data.
# Production deployment - install TimescaleDB first
brew tap timescale/tap && brew install timescaledb # macOS
# Then run migrations with MIX_ENV=prod
MIX_ENV=prod mix ecto.migrate
See TIMESCALEDB.md for detailed installation and configuration.
How it works: Migrations detect the environment (MIX_ENV) and only enable TimescaleDB features (hypertables, compression, retention policies, continuous aggregates) in production.
Development
Database
mix ecto.create # Create database
mix ecto.migrate # Run migrations
mix ecto.reset # Drop, create, and migrate
Testing
mix test # Run all tests
mix test --trace # Run with detailed output
Code Quality
mix format # Format code with Styler
mix compile --warnings-as-errors
Learn more
- Official website: https://www.phoenixframework.org/
- Guides: https://hexdocs.pm/phoenix/overview.html
- Docs: https://hexdocs.pm/phoenix
- Forum: https://elixirforum.com/c/phoenix-forum
- Source: https://github.com/phoenixframework/phoenix