Prevents FailedCreatePodSandBox errors when talos-worker1 reboots and pods
try to start before Flannel writes /run/flannel/subnet.env.
Init container polls DNS resolution until network is ready before starting Valkey.
Prevents race condition where towerops pods try to start before Flannel
CNI is fully initialized during node restarts.
Same fix as applied to Valkey StatefulSet (commit 30a0b9a). Both
critical infrastructure components now have proper priority classes to
ensure CNI readiness before pod scheduling.
Benefits:
- Eliminates FailedCreatePodSandBox errors during node restarts
- Reduces unnecessary pod restarts from 24+ to 0
- Ensures stable startup even during Flannel restarts
- Combined with Redis health checks for complete resilience
Verified working:
- Deployment rolled out successfully with 0 issues
- Priority class applied: system-cluster-critical
- Redis health check confirmed: 'Redis is available, starting Exq'
🤖 Generated with Claude Code
Fixes race condition where Valkey pod tries to start before Flannel CNI
is fully initialized during node restarts.
Problem:
- Valkey pod had 24 restarts due to FailedCreatePodSandBox errors
- During node restarts, pods would start before Flannel wrote /run/flannel/subnet.env
- CNI plugin would fail: 'failed to load flannel subnet.env file'
- Pod would retry until Flannel finished initialization
Solution:
- Added priorityClassName: system-cluster-critical to Valkey StatefulSet
- This gives Valkey same priority level as etcd, coredns, and other cluster services
- Scheduler will ensure CNI (Flannel) is ready before starting Valkey
- Reduces race condition window during node restarts
Priority Classes:
- system-node-critical (2000001000): Flannel, kube-proxy
- system-cluster-critical (2000000000): Valkey, etcd, coredns
- default (0): Regular application pods
Combined with previous commit's Redis health checks and Exq supervisor
improvements, the application is now resilient to both infrastructure
issues and Redis connection failures.
🤖 Generated with Claude Code
- Remove PVC volume mount from k8s/deployment.yaml
- Delete k8s/mib-pvc.yaml (no longer needed)
- Update .gitignore to commit MIB files to git
- Add mix import_mibs task to copy MIBs from LibreNMS
- Add all MIB files from priv/mibs/ to git
This fixes the multi-attach volume error in Kubernetes where new pods
couldn't start because the RWO (ReadWriteOnce) PVC was attached to the
old pod. MIBs are now part of the Docker image and can be used by all
pods simultaneously.
- Add timezone to socket assigns in mount_current_scope
- Detect user timezone from browser using Intl.DateTimeFormat
- Auto-save detected timezone to user profile on first connection
- Update all templates to use TimeHelpers with timezone parameter
- Update agent helper functions to accept timezone parameter
- Templates updated:
- Alert list (triggered, acknowledged, resolved)
- Device list (last checked)
- Agent list and show (created, last seen, updated)
- Admin dashboard (impersonation logs)
- Admin org/user lists (created/joined dates)
- Main dashboard (alert times)
Changed Valkey service from headless to ClusterIP and updated
REDIS_HOST to use full FQDN for more reliable DNS resolution.
Changes:
- valkey-service.yaml: Changed from headless (clusterIP: None) to ClusterIP
- deployment.yaml: Changed REDIS_HOST from "valkey" to "valkey.towerops.svc.cluster.local"
This ensures stable ClusterIP and explicit DNS resolution for Redis connections.
Previously Valkey ran as a sidecar container in the main deployment,
causing it to restart every time the web app deployed. This resulted
in connection errors and cache loss during deployments.
Changes:
- Created valkey-statefulset.yaml with StatefulSet resource (1 replica)
- Created valkey-service.yaml with headless service
- Removed Valkey sidecar container from deployment.yaml
- Updated REDIS_HOST from 127.0.0.1 to valkey-0.valkey.towerops.svc.cluster.local
- Added new resources to kustomization.yaml
Benefits:
- Valkey persists across web app deployments
- No connection errors during rolling updates
- Cache data preserved
- Independent scaling and resource management
With Horde's distributed supervision, devices will automatically
distribute ~25% per pod across all 4 replicas.
Benefits:
- 4x horizontal scaling for device monitoring and SNMP polling
- Better fault tolerance (can lose 3 pods and still operate)
- Automatic load balancing via consistent hashing
- Each device still polls from exactly ONE pod (no duplicates)
Script checks:
- Container status in pod
- Valkey logs
- Network connectivity from towerops to Valkey
- Environment variables
- Valkey health
Usage: ./k8s/debug-redis.sh [namespace]
Changes:
- Added startupProbe to towerops container to allow up to 70s for startup
(gives Valkey sidecar time to be ready)
- Changed REDIS_HOST from 'localhost' to '127.0.0.1' for explicit loopback
- Removed initialDelaySeconds from liveness/readiness probes (handled by startup)
- Added comment clarifying Redis connects to Valkey sidecar
This ensures the app waits for Valkey to be ready before accepting traffic.
- Add Valkey 8.0 alpine container as sidecar in k8s deployment
- Configure with appropriate security context (non-root, no privilege escalation)
- Add resource limits (256Mi memory, 200m CPU) and requests (128Mi, 50m)
- Add health probes using valkey-cli ping command
- Configure Redis connection via REDIS_HOST and REDIS_PORT environment variables
- Add Redis config to runtime.exs, dev.exs, and test.exs
- Valkey runs on localhost:6379 within the same pod as Phoenix app
Prevent WebSocket disconnections during deployments by:
1. Rolling Update Strategy:
- maxSurge: 1 (allow 1 extra pod during rollout)
- maxUnavailable: 0 (keep all pods running)
- minReadySeconds: 10 (wait before continuing rollout)
2. Graceful Shutdown:
- terminationGracePeriodSeconds: 30 (allow Phoenix to drain connections)
3. PodDisruptionBudget:
- minAvailable: 1 (ensure at least 1 pod always available)
- Prevents all pods from being terminated simultaneously
4. Improved Health Checks:
- Faster readiness probe (5s initial, 3s period)
- More aggressive success/failure thresholds
- New pods marked ready faster
This ensures new pods are fully ready and serving traffic before old
pods are terminated, maintaining WebSocket connections and preventing
'something went wrong' errors during deployments.
- Update GitLab CI deploy job to set DEPLOY_TIMESTAMP when deploying
- Use 'kubectl set env' to update all pods with same timestamp
- Change deployment.yaml to use static placeholder value
- Update documentation to explain GitLab CI timestamp approach
This ensures all pods show the same deployment time, regardless of:
- Which pod handles the request (with replicas=2)
- When individual pods were created or restarted
- Pod restarts from crashes or rolling updates
The timestamp now represents when the deployment was initiated by GitLab CI,
not when individual pods were created.
- Read DEPLOY_TIMESTAMP environment variable for actual deploy time
- Fall back to compile time in development when env var not set
- Add Kubernetes Downward API to inject metadata.creationTimestamp
- Update k8s/deployment.yaml to set DEPLOY_TIMESTAMP env var
- Document deployment timestamp mechanism in k8s/README.md
This ensures the footer shows when the pod was deployed to the cluster,
not when the Docker image was built in CI/CD.
Secrets are now managed directly in the cluster rather than generated
from .envrc files. This fixes FluxCD reconciliation errors since .envrc
is gitignored and cannot be used in GitOps workflows.
All secrets have been backed up to 1Password for disaster recovery.
Use Plug.Telemetry's dynamic log level feature to disable logging for
/health requests. This is the proper Phoenix way to exclude specific
endpoints from request logs.
- Add log_level/1 function to endpoint that returns false for /health
- Configure Plug.Telemetry to use dynamic log level
- Remove log: false from router (handled by endpoint now)
This prevents Kubernetes health probe spam in application logs.
- Migrations now run in Application.start/2 before starting services
- Ecto's advisory locks prevent concurrent migrations in clustered setup
- Simplified deployment - no separate migration job needed
- Removed migrate-job.yaml manifest
- Set runAsNonRoot and runAsUser to 65534 (nobody)
- Disable privilege escalation
- Drop all Linux capabilities
- Use RuntimeDefault seccomp profile
- Applies to both migration job and main deployment
- Create Kubernetes Job to run migrations before deployment
- Update GitLab CI to execute migration job and wait for completion
- Ensures migrations run once per deployment in clustered environment
- Migration job uses PostgreSQL advisory locks to prevent race conditions
