No description
Graham McIntire
5ee241fc16
Addresses production Redis disconnection issues by implementing a highly
resilient Valkey (Redis) setup with automatic failover capabilities.
Infrastructure Changes:
- Add Valkey ConfigMap with optimized connection and memory settings
- TCP keepalive (60s), connection limits (10k clients)
- Memory management (256MB with LRU eviction)
- Separate configs for master, replica, and sentinel
- Update Valkey StatefulSet to 3 replicas (1 master + 2 replicas)
- Auto-configuration via init container (master vs replica)
- Increased memory limits: 256Mi → 1Gi
- Improved readiness probes with replication status checks
- Add Valkey Sentinel StatefulSet (3 instances for quorum)
- Automatic failover detection (5s down-after-milliseconds)
- Fast failover execution (10s timeout)
- Monitors master and promotes replicas automatically
- Add Sentinel headless service for pod discovery
Application Resilience:
- Update Phoenix.PubSub.Redis with TCP keepalive and reconnection
- Connection timeout: 5s
- Exponential backoff: 500ms → 30s
- exit_on_disconnection: false
- Update Exq background jobs with same resilience settings
- TCP keepalive enabled
- Better connection pool management
Benefits:
- Automatic failover when Valkey pod dies (no manual intervention)
- Zero data loss with replica synchronization
- Fast failure detection and recovery (5-10s total)
- Survives Flannel CNI networking issues
- Apps reconnect automatically during disconnections
Testing:
- All 3,686 tests passing
- Kustomize manifest validated
🤖 Generated with Claude Code
|
||
|---|---|---|
| .gitlab/agents | ||
| assets | ||
| config | ||
| docs | ||
| k8s | ||
| lib | ||
| mibs | ||
| priv | ||
| rel/overlays/bin | ||
| scripts | ||
| test | ||
| towerops | ||
| .credo.exs | ||
| .dialyzer_ignore.exs | ||
| .dockerignore | ||
| .formatter.exs | ||
| .gitignore | ||
| .gitlab-ci.yml | ||
| .sourceignore | ||
| .test-watch.exs | ||
| .tool-versions | ||
| AGENTS.md | ||
| CLAUDE.md | ||
| DEPLOYMENT.md | ||
| Dockerfile | ||
| LIBRENMS_INTEGRATION.md | ||
| mix.exs | ||
| mix.lock | ||
| MOBILE_API.md | ||
| PROFILES.md | ||
| PROFILES_README.md | ||
| README.md | ||
| tail_logs.sh | ||
TowerOps
Network monitoring and alerting platform built with Phoenix LiveView.
Features
- Multi-tenant architecture - Organizations with role-based permissions
- Site hierarchy - Organize equipment across multiple sites
- Automated monitoring - Real-time ping monitoring with configurable intervals
- Time-series data - Efficient storage with TimescaleDB (optional)
- Real-time updates - LiveView dashboard with PubSub
- Equipment tracking - Monitor network devices by IP address
Quick Start
Prerequisites
- Elixir 1.14+
- PostgreSQL 14+
- (Optional) TimescaleDB for production-grade time-series performance
Setup
# Install dependencies
mix setup
# Start the server
mix phx.server
Visit localhost:4000 from your browser.
TimescaleDB (Production Only)
Development: Uses standard PostgreSQL (no TimescaleDB required).
Production: TimescaleDB is automatically enabled for optimal performance with time-series data.
# Production deployment - install TimescaleDB first
brew tap timescale/tap && brew install timescaledb # macOS
# Then run migrations with MIX_ENV=prod
MIX_ENV=prod mix ecto.migrate
See TIMESCALEDB.md for detailed installation and configuration.
How it works: Migrations detect the environment (MIX_ENV) and only enable TimescaleDB features (hypertables, compression, retention policies, continuous aggregates) in production.
Development
Database
mix ecto.create # Create database
mix ecto.migrate # Run migrations
mix ecto.reset # Drop, create, and migrate
Testing
mix test # Run all tests
mix test --trace # Run with detailed output
Code Quality
mix format # Format code with Styler
mix compile --warnings-as-errors
Learn more
- Official website: https://www.phoenixframework.org/
- Guides: https://hexdocs.pm/phoenix/overview.html
- Docs: https://hexdocs.pm/phoenix
- Forum: https://elixirforum.com/c/phoenix-forum
- Source: https://github.com/phoenixframework/phoenix