When 50%+ of the access points at one site simultaneously have Preseem QoE score <60, root cause is almost always a shared upstream problem (backhaul, transit, DNS, DHCP) — not per-AP RF. This rule emits one site-level insight that points the operator at the right investigation layer instead of letting them chase per-AP RF symptoms across many APs at the same site. - Towerops.Recommendations.Rules.UpstreamDegradation — single query pulls all matched Preseem APs grouped by site_id. Critical when 100% of APs at the site are degraded; warning otherwise. Lists every degraded AP with its QoE score in metadata. - Insight.@valid_types extended with upstream_degradation. - LLM prompt hint tells the model to instruct the operator to investigate upstream FIRST and to NOT chase per-AP RF symptoms. - UI: indigo card with degraded/total AP counts, an "Investigate upstream first" badge, and a clickable list of affected APs. - Wired into RecommendationsRunWorker FIRST in the @rules list so the upstream signal surfaces before per-AP RF rules. Also fixes two flakes: - device_live/show_test.exs is now async: false. It inserts to InterfaceStat which is a TimescaleDB hypertable; concurrent inserts deadlock on chunk-table locks under parallel SQL-sandbox tests (ERROR 40P01 deadlock_detected in setup). - Cleared an unused-variable warning in device_overheating_test.exs. 8 new tests for UpstreamDegradation. |
||
|---|---|---|
| .claude | ||
| .forgejo | ||
| assets | ||
| bin | ||
| c_src | ||
| config | ||
| docs | ||
| e2e | ||
| k8s | ||
| lib | ||
| mibs | ||
| nix | ||
| policies | ||
| priv | ||
| rel | ||
| scripts | ||
| test | ||
| tools | ||
| vendor | ||
| .credo.exs | ||
| .dialyzer_ignore.exs | ||
| .dockerignore | ||
| .envrc.example | ||
| .formatter.exs | ||
| .gitignore | ||
| .stride.md | ||
| .test-watch.exs | ||
| .tool-versions | ||
| AGENTS.md | ||
| CHANGELOG.txt | ||
| CLAUDE.md | ||
| Dockerfile | ||
| flake.lock | ||
| flake.nix | ||
| Makefile | ||
| mix.exs | ||
| mix.lock | ||
| README.md | ||
| renovate.json | ||
| setup_stripe_meter.exs | ||
| setup_stripe_production.sh | ||
| shell.nix | ||
| tail_logs.sh | ||
TowerOps
Network monitoring and alerting platform built with Phoenix LiveView.
Features
- Multi-tenant architecture - Organizations with role-based permissions
- Site hierarchy - Organize equipment across multiple sites
- Automated monitoring - Real-time ping monitoring with configurable intervals
- Time-series data - Efficient storage with TimescaleDB (optional)
- Real-time updates - LiveView dashboard with PubSub
- Equipment tracking - Monitor network devices by IP address
Quick Start
Prerequisites
- Elixir 1.14+
- PostgreSQL 14+
- (Optional) TimescaleDB for production-grade time-series performance
Setup
# Install dependencies
mix setup
# Start the server
mix phx.server
Visit localhost:4000 from your browser.
TimescaleDB (Production Only)
Development: Uses standard PostgreSQL (no TimescaleDB required).
Production: TimescaleDB is automatically enabled for optimal performance with time-series data.
# Production deployment - install TimescaleDB first
brew tap timescale/tap && brew install timescaledb # macOS
# Then run migrations with MIX_ENV=prod
MIX_ENV=prod mix ecto.migrate
See TIMESCALEDB.md for detailed installation and configuration.
How it works: Migrations detect the environment (MIX_ENV) and only enable TimescaleDB features (hypertables, compression, retention policies, continuous aggregates) in production.
Encryption Setup (Production)
TowerOps uses AES-256-GCM encryption for sensitive data (SNMP communities, MikroTik API passwords, etc.).
Development/Test
Encryption keys are pre-configured in config/dev.exs and config/test.exs. No action required.
Production
Set the CLOAK_KEY environment variable with a base64-encoded 32-byte key:
# Generate encryption key
openssl rand -base64 32
Important:
- Store the generated key securely in 1Password or your secrets manager
- Never commit the production key to version control
- Losing the encryption key makes encrypted data unrecoverable
Kubernetes Deployment
If secret doesn't exist yet (new deployment):
# Generate CLOAK_KEY (store in 1Password first!)
CLOAK_KEY=$(openssl rand -base64 32)
# Create towerops-secrets with all required keys
kubectl create secret generic towerops-secrets \
--from-literal=RELEASE_COOKIE=$(openssl rand -base64 32) \
--from-literal=SECRET_KEY_BASE=$(mix phx.gen.secret) \
--from-literal=CLOAK_KEY="$CLOAK_KEY" \
-n towerops
If secret already exists (add CLOAK_KEY to existing secret):
# Store new key in 1Password first!
# Bash/Zsh:
CLOAK_KEY=$(openssl rand -base64 32)
# Fish shell:
set CLOAK_KEY (openssl rand -base64 32)
# Method 1: Using kubectl create with dry-run and apply
kubectl create secret generic towerops-secrets \
--from-literal=CLOAK_KEY="$CLOAK_KEY" \
--dry-run=client -o yaml | \
kubectl apply -f - -n towerops
# Method 2: Direct inline generation (works in all shells)
kubectl create secret generic towerops-secrets \
--from-literal=CLOAK_KEY="$(openssl rand -base64 32)" \
--dry-run=client -o yaml | \
kubectl apply -f - -n towerops
# Restart pods to pick up new key
kubectl rollout restart deployment/towerops -n towerops
Development
Database
mix ecto.create # Create database
mix ecto.migrate # Run migrations
mix ecto.reset # Drop, create, and migrate
Testing
mix test # Run all tests
mix test --trace # Run with detailed output
Code Quality
mix format # Format code with Styler
mix compile --warnings-as-errors
Firmware Version Tracking
The system automatically checks for latest firmware versions daily (2 AM dev, 4 AM prod). To manually trigger a firmware check:
# Start IEx console
iex -S mix phx.server
# Manually trigger firmware version fetch
Oban.insert(Towerops.Workers.FirmwareVersionFetcherWorker.new(%{}))
The worker will:
- Fetch the latest MikroTik RouterOS version from RSS feed
- Store version information in the database
- Enable firmware update indicators on device detail pages
Check the logs for fetch results:
# View recent Oban jobs
Towerops.Repo.all(Oban.Job) |> Enum.take(5)
Learn more
- Official website: https://www.phoenixframework.org/
- Guides: https://hexdocs.pm/phoenix/overview.html
- Docs: https://hexdocs.pm/phoenix
- Forum: https://elixirforum.com/c/phoenix-forum
- Source: https://github.com/phoenixframework/phoenix