Replace the single-secret towerops-llm template with one
multi-document YAML covering every Secret the Deployment references:
towerops-secrets, towerops-db, towerops-aws, towerops-redis,
towerops-billing, towerops-llm.
Workflow: copy k8s/secrets.example.yaml to k8s/secrets.yaml (which is
gitignored), fill in real values, kubectl apply. Real values never land
in git.
Two new multi-source rules using data we already collect, both wired
into the existing hourly RecommendationsRunWorker.
SectorOverload — fires when a Preseem-monitored AP has <25% free
airtime AND active subscribers. Critical urgency when free airtime
drops below 15%, OR when free airtime <25% AND QoE <50. Reuses the
airtime/subscriber/QoE data already pulled by PreseemSyncWorker so
no new collection is needed. UI: orange evidence card with free
airtime, subscriber count, QoE score, and AP model.
CpeRealign — fires when a wireless client has BOTH signal_strength
<=-78 dBm AND SNR <=18 dB seen in the last 2 hours. This is the
classic alignment / obstruction signature, distinct from the
existing single-metric wireless_signal_weak / wireless_snr_low
alerts produced by WirelessInsightWorker. One insight per (AP, CPE)
pair via a metadata.dedup_key. Critical when signal <=-88 OR SNR
<=10. UI: rose evidence card with signal, SNR, TX/RX rate, distance,
hostname.
Insights.insert_insight_if_new/1 dedup logic now prefers a
metadata.dedup_key over device_id when explicitly set, allowing
per-CPE insights without collapsing multiple CPEs on the same AP
into a single insight. Existing rules without dedup_key are
unaffected.
Insight.@valid_types extended with sector_overload and cpe_realign.
LLM enrichment automatically applies to both new types via the
existing Phase 1 worker — no extra wiring.
Also adds k8s/secrets.yaml to .gitignore so operators can drop a
local Secret manifest with real values, kubectl apply manually, and
never accidentally commit it. Documented in k8s/README.md.
Adds an optional plain-language summary and recommended action to every
active insight. A new Oban cron worker runs every 5 minutes, picks up
unenriched insights, and asks the configured LLM to restate the finding
and suggest one specific action grounded in the structured metadata.
- Migration: adds llm_summary, recommended_action, llm_model,
llm_enriched_at columns to preseem_insights
- Towerops.LLM context with swappable behaviour; real client uses Req
with the standard Req.Test plug for test isolation
- MockClient + InsightPrompt module (builds chat messages, parses JSON
with code-fence stripping and a raw-text fallback)
- Worker logs and skips on rate limits / missing key / parse errors so
insights still render without an AI summary when the LLM is unavailable
- Optional towerops-llm k8s secret with example template; deployment.yaml
references it as optional in both init and main containers
- UI renders summary + recommended action callout on /insights and on
the org-level insights page
- Update GitLab CI deploy job to set DEPLOY_TIMESTAMP when deploying
- Use 'kubectl set env' to update all pods with same timestamp
- Change deployment.yaml to use static placeholder value
- Update documentation to explain GitLab CI timestamp approach
This ensures all pods show the same deployment time, regardless of:
- Which pod handles the request (with replicas=2)
- When individual pods were created or restarted
- Pod restarts from crashes or rolling updates
The timestamp now represents when the deployment was initiated by GitLab CI,
not when individual pods were created.
- Read DEPLOY_TIMESTAMP environment variable for actual deploy time
- Fall back to compile time in development when env var not set
- Add Kubernetes Downward API to inject metadata.creationTimestamp
- Update k8s/deployment.yaml to set DEPLOY_TIMESTAMP env var
- Document deployment timestamp mechanism in k8s/README.md
This ensures the footer shows when the pod was deployed to the cluster,
not when the Docker image was built in CI/CD.
Secrets are now managed directly in the cluster rather than generated
from .envrc files. This fixes FluxCD reconciliation errors since .envrc
is gitignored and cannot be used in GitOps workflows.
All secrets have been backed up to 1Password for disaster recovery.
