Adds an optional plain-language summary and recommended action to every
active insight. A new Oban cron worker runs every 5 minutes, picks up
unenriched insights, and asks the configured LLM to restate the finding
and suggest one specific action grounded in the structured metadata.
- Migration: adds llm_summary, recommended_action, llm_model,
llm_enriched_at columns to preseem_insights
- Towerops.LLM context with swappable behaviour; real client uses Req
with the standard Req.Test plug for test isolation
- MockClient + InsightPrompt module (builds chat messages, parses JSON
with code-fence stripping and a raw-text fallback)
- Worker logs and skips on rate limits / missing key / parse errors so
insights still render without an AI summary when the LLM is unavailable
- Optional towerops-llm k8s secret with example template; deployment.yaml
references it as optional in both init and main containers
- UI renders summary + recommended action callout on /insights and on
the org-level insights page
Add PromEx with Application/BEAM/Phoenix/LiveView/Ecto/Oban plugins plus
a custom plugin that bridges existing Towerops Oban/Redis telemetry events.
The metrics HTTP server runs isolated on port 9568 so scrape traffic never
traverses the public Traefik IngressRoute.
The pod template carries prometheus.io annotations (scrape, port, path, job)
so the external Prometheus on 10.0.15.31 discovers each pod through the
existing kubernetes-pods scrape job (apiserver-proxy).
The repo got renamed towerops-web → towerops on the Codeberg side,
so the published image follows: image is now
codeberg.org/gmcintire/towerops with the same main-<ts>-<sha> tag
pattern. Login URL is the hardcoded env.REGISTRY rather than
secrets.REGISTRY_URL so a stale URL secret can't push to the wrong
registry.
With 1 replica, the old pod was torn down too quickly after the new pod
passed its first health check. Increase successThreshold to 3 (15s of
consecutive checks) and minReadySeconds to 30 to ensure Traefik has fully
propagated the new endpoint before the old pod is terminated.
Resolver
- Resolver.resolve_range/3: returns the contiguous on-call segments for a
schedule across [start_at, end_at). Walks layer handoffs + override
boundaries, drops gap segments, and merges adjacent same-user segments.
UserColor module
- New Towerops.OnCall.UserColor with deterministic id -> color mapping
(Tailwind class + matching #RRGGBB hex). Refactored ScheduleLive.Show
to use it instead of its own per-page index palette so the same person
gets the same color across the schedules list and detail pages.
ScheduleLive.Index
- Date navigation: Today / prev / next + 1/2/4-week range selector.
- URL-driven: ?start=YYYY-MM-DD&range=N for shareable views; falls back
to defaults on malformed values.
- Per-row card: schedule name link, on-call-now avatar swatch, day-of-week
header strip, gantt strip rendered as a CSS grid with one column per day,
Today marker overlay.
Tests: 5 new resolver tests, 9 new UserColor tests, 5 new ScheduleLive
integration tests. Full suite: 10,210 / 0 failures. Format/dialyzer clean.
Also: cleaned up two stale inline comments in k8s/deployment.yaml that
disagreed with the values they sat next to (replicas/maxSurge).
- k8s/deployment.yaml: bump startup-probe failureThreshold from 12 to 24
(70s -> 130s max startup time) to give the app more headroom on cold start.
- lib/towerops/profiles/yaml_profiles.ex: defer YAML profile loading from
init/1 to a handle_info(:load_profiles, ...) so the supervisor (and Bandit)
come up immediately and the profiles populate in the background within ~60s.
- e2e/Makefile: convenience target wrappers around the existing npm scripts.
Without these comments the ImageUpdateAutomation with strategy=Setters
has nothing to rewrite, so new image tags detected by ImagePolicy never
get committed back to main and the cluster never rolls out new builds.
Reviewed-on: graham/towerops-web#238
Every timeout stack trace pointed at ssl_gen_statem.call/2 — the OTP 27
SSL gen_statem hanging during recv. PG is on private network (10.0.15.20)
and SSL verify was already disabled (verify_none), providing no security
benefit while adding CPU overhead and the gen_statem layer that caused
all connection timeouts.
Removed DATABASE_SSL and DATABASE_SSL_VERIFY env vars from both the
init container and main container. Also disabled SSL on the PG server
(ssl=off in postgresql.conf, hostssl→host in pg_hba.conf).
Reviewed-on: graham/towerops-web#97
Pods were running at 500m/500m (100% of limit) with 44-47% of scheduling
periods being CPU-throttled. SSL decryption is CPU-intensive — when the
BEAM gets throttled mid ssl_gen_statem:call/2, the SSL recv stalls until
the cgroup allows CPU again, causing the 15s DBConnection timeout.
This was the actual root cause of "client timed out because it queued
and checked out the connection for longer than 15000ms" errors. Previous
fixes (TCP keepalive, pool tuning, oban pruner) were treating symptoms.
- CPU limit: 500m → 2000m (4 cores burst capacity)
- CPU request: 100m → 500m (guarantee adequate baseline)
Reviewed-on: graham/towerops-web#96
- Add stripe_meter_id to dev, runtime, and test configs
- Add STRIPE_METER_ID to k8s deployment secret references
- Use real Stripe meter and price IDs from Stripe setup
- Meter: mtr_test_61UHPU067veEZ7bhl41S77kvnTfgyODY
- Price: price_1T81XBS77kvnTfgyPlw1jF8N ($1/device/month)
- Include setup script for Stripe billing configuration
- Changed from hardcoded placeholder values to secretKeyRef
- References towerops-billing secret (with optional: true)
- Won't crash if secret doesn't exist yet
- User can create secret later with real Stripe keys
- Add Stripe test API keys to dev.exs for local development
- Add placeholder Stripe env vars to k8s deployment to prevent startup crashes
- Includes secret_key, webhook_secret, and price_id configuration
- Add init container to run migrations before app starts
- Uses /app/bin/migrate script (runs Towerops.Release.migrate())
- Migrations run once per deployment, before any pods start
- Update CI/CD to set both migrate and towerops container images
- Prevents issues like missing severity column in production
