These CRDs are not available for FluxCD dry-run validation, causing the
kustomization to fail. Apply certificate.yaml and ingressroute.yaml
manually instead.
Switch deployment pipeline from GitLab Agent kubectl to FluxCD image
automation. CI now only builds and pushes tagged images to Forgejo
registry; FluxCD detects new tags and updates the deployment manifest.
Fix Ecto.MultipleResultsError on /dashboard when a device has multiple
Preseem access point records by using limit(1) instead of Repo.get_by.
Reverted k8s/Dockerfile back to using official hexpm/elixir and debian
images directly instead of custom gmcintire/towerops-base images.
This restores the original Dockerfile behavior where all dependencies
are installed during each build. The base image work in k8s/base-image/
remains available but is not used by the main application build.
Changed image naming scheme from separate builder/runtime images to a
single image name with different tags:
- docker.io/gmcintire/towerops-base:builder
- docker.io/gmcintire/towerops-base:runtime
- docker.io/gmcintire/towerops-base:latest (alias for runtime)
This simplifies the naming and makes it clearer that both images are
part of the same towerops-base image family.
Changes:
- build.sh: Use IMAGE_NAME with BUILDER_TAG/RUNTIME_TAG
- Makefile: Update all targets to use new naming
- k8s/Dockerfile: Point to gmcintire/towerops-base:builder and :runtime
Instead of installing system packages and tools on every deploy, we now
have two base images that pre-bake all the slow setup steps.
Base Images Created:
1. elixir-builder:latest - Builder stage base
- hexpm/elixir with build-essential and git pre-installed
- hex and rebar pre-installed
- Saves ~40 seconds per deploy
2. debian-runtime:latest - Runtime stage base
- Debian with all runtime packages pre-installed
- Locale pre-configured (en_US.UTF-8)
- /app directory pre-created
- Saves ~60 seconds per deploy
Main Dockerfile Changes:
- Use elixir-builder:latest instead of hexpm/elixir
- Use debian-runtime:latest instead of debian:trixie-slim
- Remove apt-get install steps (now in base images)
- Remove hex/rebar install (now in builder base)
- Remove locale setup (now in runtime base)
Build System:
- build.sh builds both images with podman/docker auto-detect
- Pushes to both GitLab registry and Docker Hub
- Makefile targets for build/test/push/clean
Total Time Saved Per Deploy: ~100 seconds (1m40s)
Benefits:
- Faster CI/CD builds (no repeated apt-get install)
- Faster local development builds
- Consistent build environment across all deploys
- Security updates centralized in base image rebuilds
The hexpm images install Erlang and Elixir in /usr/local instead of /usr/lib.
Changes:
- Copy from /usr/local/lib/erlang instead of /usr/lib/erlang
- Copy all Erlang/Elixir binaries from /usr/local/bin
- Remove manual symlink creation (binaries already exist)
- Binaries are symlinks to ../lib/erlang/bin and ../lib/elixir/bin
The Erlang Solutions APT repository was experiencing 504 Gateway Timeout
errors during builds, making the image build process unreliable.
Changes:
- Use official hexpm/elixir Docker image as builder stage
- More reliable (official Docker Hub images)
- Faster builds (pre-built binaries, no package installation)
- Still produces minimal runtime-only final image
- Updated documentation to reflect new approach
Benefits:
- No dependency on erlang-solutions CDN availability
- Faster build times (no apt-get install of large packages)
- Same minimal final image size (~150-200 MB)
- Easier to specify exact Erlang/Elixir versions
Enhanced base image build system with:
Container Engine Support:
- Auto-detect podman or docker (prefers podman if both installed)
- All scripts work with either engine seamlessly
- No configuration needed
Docker Hub Integration:
- Push to both GitLab and Docker Hub registries
- GitLab: registry.gitlab.com/towerops/towerops/elixir-runtime
- Docker Hub: docker.io/gmcintire/elixir-runtime
- Both versioned and :latest tags pushed to each
Build Script:
- Detect and use $CONTAINER_CMD for all operations
- Show which container engine is being used in output
- Tag and push to both registries automatically
Makefile:
- Add CONTAINER_CMD variable with auto-detection
- Add DOCKERHUB_REGISTRY configuration
- Update all targets to use detected container engine
- Add login-dockerhub target for Docker Hub authentication
- Update push target to push to both registries
Documentation:
- Document podman/docker auto-detection
- Update login instructions for both registries
- Update troubleshooting with examples for both engines
This makes the build system more flexible and accessible to users
who prefer podman over docker, while also publishing to the public
Docker Hub registry for easier access.
Created build system for minimal Debian 13 (Trixie) image with Erlang/OTP
and Elixir runtime pre-installed. This will speed up CI/CD builds by caching
the runtime environment.
Features:
- Multi-stage Dockerfile for minimal final image (~150-200 MB)
- Build script with automatic tagging (versioned, latest, dated)
- Update script for easy security patch rebuilds
- Makefile for common operations
- Comprehensive documentation (README + QUICKSTART)
Benefits:
- Faster CI/CD: Saves 30-60s per build (no apt-get install runtime deps)
- Smaller images: Only essential runtime packages included
- Security: Easy to rebuild weekly/monthly with latest patches
- Consistency: Same runtime across all environments
Usage:
cd k8s/base-image
make build # Build the image
make test # Verify it works
make push # Push to registry
Then update k8s/Dockerfile to use the custom base image.
This replaces the previous Dockerfile.base approach with a more
comprehensive and maintainable build system.
Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue.
This simplifies the architecture by eliminating the need for a separate etcd cluster
while providing better reliability and observability.
Changes:
- Add Oban dependency and migration (oban_jobs table)
- Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers
- Remove EtcdCoordinator and EtcdLock modules
- Update application supervisor to start Oban
- Configure Oban with pollers (50 workers) and monitors (50 workers) queues
- Remove etcd StatefulSet from Kubernetes manifests
- Update monitoring supervisor documentation
Benefits:
- Simpler architecture (no etcd cluster to manage)
- PostgreSQL-based (uses existing database)
- Built-in uniqueness prevents duplicate jobs cluster-wide
- Better observability with Oban Web UI
- Automatic job rescue on node crashes
- Easier local development (no etcd required)
What was removed:
- etcd StatefulSet (3 pods)
- EtcdCoordinator module (320 lines)
- EtcdLock module (158 lines)
- eetcd dependency
All 3,686 tests passing.
Removing all Valkey (Redis) resources from Kubernetes due to instability
caused by Flannel CNI networking issues. Redis will now run on Proxmox
hosts for better stability and performance.
Changes:
- Delete Valkey StatefulSet (master + 2 replicas)
- Delete Valkey Sentinel StatefulSet (3 instances)
- Delete Valkey services (headless and sentinel)
- Delete Valkey ConfigMap
- Remove Valkey resources from kustomization.yaml
- Update deployment to use towerops-redis secret for connection
Next Steps:
- Set up Redis Sentinel on 3 Proxmox hosts/LXC containers
- Create towerops-redis secret with REDIS_HOST and REDIS_PORT
- Test failover and application connectivity
Benefits:
- Not affected by K8s networking issues (Flannel failures)
- More stable (no restarts from node issues)
- Better performance (no K8s overhead)
- Independent lifecycle from K8s cluster
Sentinel requires individual pod DNS names to work correctly.
Making the valkey service headless enables DNS resolution for:
- valkey-0.valkey.towerops.svc.cluster.local
- valkey-1.valkey.towerops.svc.cluster.local
- valkey-2.valkey.towerops.svc.cluster.local
This fixes CrashLoopBackOff in valkey-sentinel pods.
- Rewrote Ping module to use system ping binary instead of raw ICMP
sockets which required CAP_NET_RAW privileges
- Added iputils-ping package to Docker images for production
- Added proper IP address validation before executing ping command
- Updated tests to account for 1 second minimum timeout of system ping
- System ping binary is setuid root and works without elevated privileges
Prevents FailedCreatePodSandBox errors when talos-worker1 reboots and pods
try to start before Flannel writes /run/flannel/subnet.env.
Init container polls DNS resolution until network is ready before starting Valkey.
Prevents race condition where towerops pods try to start before Flannel
CNI is fully initialized during node restarts.
Same fix as applied to Valkey StatefulSet (commit 30a0b9a). Both
critical infrastructure components now have proper priority classes to
ensure CNI readiness before pod scheduling.
Benefits:
- Eliminates FailedCreatePodSandBox errors during node restarts
- Reduces unnecessary pod restarts from 24+ to 0
- Ensures stable startup even during Flannel restarts
- Combined with Redis health checks for complete resilience
Verified working:
- Deployment rolled out successfully with 0 issues
- Priority class applied: system-cluster-critical
- Redis health check confirmed: 'Redis is available, starting Exq'
🤖 Generated with Claude Code
Fixes race condition where Valkey pod tries to start before Flannel CNI
is fully initialized during node restarts.
Problem:
- Valkey pod had 24 restarts due to FailedCreatePodSandBox errors
- During node restarts, pods would start before Flannel wrote /run/flannel/subnet.env
- CNI plugin would fail: 'failed to load flannel subnet.env file'
- Pod would retry until Flannel finished initialization
Solution:
- Added priorityClassName: system-cluster-critical to Valkey StatefulSet
- This gives Valkey same priority level as etcd, coredns, and other cluster services
- Scheduler will ensure CNI (Flannel) is ready before starting Valkey
- Reduces race condition window during node restarts
Priority Classes:
- system-node-critical (2000001000): Flannel, kube-proxy
- system-cluster-critical (2000000000): Valkey, etcd, coredns
- default (0): Regular application pods
Combined with previous commit's Redis health checks and Exq supervisor
improvements, the application is now resilient to both infrastructure
issues and Redis connection failures.
🤖 Generated with Claude Code
- Remove PVC volume mount from k8s/deployment.yaml
- Delete k8s/mib-pvc.yaml (no longer needed)
- Update .gitignore to commit MIB files to git
- Add mix import_mibs task to copy MIBs from LibreNMS
- Add all MIB files from priv/mibs/ to git
This fixes the multi-attach volume error in Kubernetes where new pods
couldn't start because the RWO (ReadWriteOnce) PVC was attached to the
old pod. MIBs are now part of the Docker image and can be used by all
pods simultaneously.
- Add timezone to socket assigns in mount_current_scope
- Detect user timezone from browser using Intl.DateTimeFormat
- Auto-save detected timezone to user profile on first connection
- Update all templates to use TimeHelpers with timezone parameter
- Update agent helper functions to accept timezone parameter
- Templates updated:
- Alert list (triggered, acknowledged, resolved)
- Device list (last checked)
- Agent list and show (created, last seen, updated)
- Admin dashboard (impersonation logs)
- Admin org/user lists (created/joined dates)
- Main dashboard (alert times)
