Three changes targeting the root cause of production up/down alerting:
1. Skip BruteForceProtection for /health and /health/live paths
(removes unnecessary Repo.get_by(IpBlock) DB query from every
k8s probe, matching the existing /socket/agent exemption).
2. Relax k8s readiness probe: periodSeconds 5→10, timeoutSeconds
2→5, failureThreshold 2→3, successThreshold 3→2 so transient
DB/Redis blips don't cascade into full service unavailability.
3. Use the persistent Towerops.Redix connection for health checks
instead of opening a brand-new TCP connection (handshake→AUTH→
PING→close) on every probe. Towerops.Redix.Fake gains a
set_ping_response/2 toggle for testing failure paths.
Now that the image serving /health/live is live, switch liveness and startup
probes to the shallow endpoint so transient db/redis blips can't kill or block
a healthy pod; readiness stays on the deep /health. Run 2 replicas so a single
pod restart no longer means full downtime.
Prod paired the Oban Pro Smart engine + unique workers with core
Oban.Plugins.Lifeline, whose rescue has no unique_violation handling. An
orphaned executing CheckExecutorWorker job rescued back to available collides
with its already-scheduled successor on oban_jobs_unique_index (23505),
crashing the plugin every 60s and leaving ~150 orphans stuck. Switch prod to
Oban.Pro.Plugins.DynamicLifeline, which repairs the conflict via
Smart.clear_uniq_violation. Dev keeps core Lifeline (Basic engine).
Add a shallow /health/live endpoint that does not touch db/redis. k8s liveness
and startup probes now target it so a transient dependency outage can't kill or
block boot of a healthy pod; readiness keeps the deep /health to gate the load
balancer. (deployment.yaml probe/replica change pushed separately, after the
image carrying /health/live is live.)
Also drop unused POSTGRES_* keys from the secrets example.
SESSION_SIGNING_SALT, SESSION_ENCRYPTION_SALT, and LIVE_VIEW_SIGNING_SALT
are now loaded from environment variables in production. Dev/test keep the
previous defaults via config.exs. k8s/secrets.yaml has placeholder entries;
the user fills in real values before applying.
The towerops-db secret in prod has both DATABASE_URL and individual
POSTGRES_* fields; towerops-redis uses HOST/PORT/PASSWORD instead of
a single REDIS_URL. Update the committed template to match so a fresh
copy of k8s/secrets.example.yaml mirrors what's actually in the
cluster.
Replace the single-secret towerops-llm template with one
multi-document YAML covering every Secret the Deployment references:
towerops-secrets, towerops-db, towerops-aws, towerops-redis,
towerops-billing, towerops-llm.
Workflow: copy k8s/secrets.example.yaml to k8s/secrets.yaml (which is
gitignored), fill in real values, kubectl apply. Real values never land
in git.
Two new multi-source rules using data we already collect, both wired
into the existing hourly RecommendationsRunWorker.
SectorOverload — fires when a Preseem-monitored AP has <25% free
airtime AND active subscribers. Critical urgency when free airtime
drops below 15%, OR when free airtime <25% AND QoE <50. Reuses the
airtime/subscriber/QoE data already pulled by PreseemSyncWorker so
no new collection is needed. UI: orange evidence card with free
airtime, subscriber count, QoE score, and AP model.
CpeRealign — fires when a wireless client has BOTH signal_strength
<=-78 dBm AND SNR <=18 dB seen in the last 2 hours. This is the
classic alignment / obstruction signature, distinct from the
existing single-metric wireless_signal_weak / wireless_snr_low
alerts produced by WirelessInsightWorker. One insight per (AP, CPE)
pair via a metadata.dedup_key. Critical when signal <=-88 OR SNR
<=10. UI: rose evidence card with signal, SNR, TX/RX rate, distance,
hostname.
Insights.insert_insight_if_new/1 dedup logic now prefers a
metadata.dedup_key over device_id when explicitly set, allowing
per-CPE insights without collapsing multiple CPEs on the same AP
into a single insight. Existing rules without dedup_key are
unaffected.
Insight.@valid_types extended with sector_overload and cpe_realign.
LLM enrichment automatically applies to both new types via the
existing Phase 1 worker — no extra wiring.
Also adds k8s/secrets.yaml to .gitignore so operators can drop a
local Secret manifest with real values, kubectl apply manually, and
never accidentally commit it. Documented in k8s/README.md.
Adds an optional plain-language summary and recommended action to every
active insight. A new Oban cron worker runs every 5 minutes, picks up
unenriched insights, and asks the configured LLM to restate the finding
and suggest one specific action grounded in the structured metadata.
- Migration: adds llm_summary, recommended_action, llm_model,
llm_enriched_at columns to preseem_insights
- Towerops.LLM context with swappable behaviour; real client uses Req
with the standard Req.Test plug for test isolation
- MockClient + InsightPrompt module (builds chat messages, parses JSON
with code-fence stripping and a raw-text fallback)
- Worker logs and skips on rate limits / missing key / parse errors so
insights still render without an AI summary when the LLM is unavailable
- Optional towerops-llm k8s secret with example template; deployment.yaml
references it as optional in both init and main containers
- UI renders summary + recommended action callout on /insights and on
the org-level insights page
Add PromEx with Application/BEAM/Phoenix/LiveView/Ecto/Oban plugins plus
a custom plugin that bridges existing Towerops Oban/Redis telemetry events.
The metrics HTTP server runs isolated on port 9568 so scrape traffic never
traverses the public Traefik IngressRoute.
The pod template carries prometheus.io annotations (scrape, port, path, job)
so the external Prometheus on 10.0.15.31 discovers each pod through the
existing kubernetes-pods scrape job (apiserver-proxy).
Splits the runtime apt installs (gdal-bin, snmp, libsnmp40, locales,
BEAM runtime libs) into k8s/Dockerfile.base, hosted at
codeberg.org/gmcintire/towerops-base:latest. The app Dockerfile now
does FROM that base instead of re-installing gdal (~500 MB) on every
push.
The new build-base workflow rebuilds the base image only when
k8s/Dockerfile.base or the workflow itself changes, weekly via cron
(Sundays 06:00 UTC, with CACHE_BUST=<ISO week> to force apt-get update
on a week boundary), or via workflow_dispatch.
Production workflow now uses buildx + does docker login before the
build so it can pull the private base image.
The repo got renamed towerops-web → towerops on the Codeberg side,
so the published image follows: image is now
codeberg.org/gmcintire/towerops with the same main-<ts>-<sha> tag
pattern. Login URL is the hardcoded env.REGISTRY rather than
secrets.REGISTRY_URL so a stale URL secret can't push to the wrong
registry.
Catalog-only system for LIDAR-derived 1m DEMs covering Texas. We do not
mirror raster data — the DB stores tile metadata (URL + footprint
geometry) and elevation values are streamed on-demand from public USGS
3DEP / TNRIS Cloud-Optimized GeoTIFFs via GDAL /vsicurl/ HTTP byte-range
reads. Used for future tower coverage line-of-sight calculations.
- Enable PostGIS extension; tile footprints indexed via GiST
- Catalog sync from USGS 3DEP (primary) and TNRIS (fallback for gaps)
- Tile dedup by ST_Contains so TNRIS doesn't duplicate 3DEP coverage
- Single-point reads via gdallocationinfo with tile fallthrough on nodata
- Grid retrieval via gdal_translate AAIGrid with multi-tile mosaicing
- Monthly Oban cron worker keeps catalog fresh
- gdal-bin added to runtime image; geo_postgis dep added
With 1 replica, the old pod was torn down too quickly after the new pod
passed its first health check. Increase successThreshold to 3 (15s of
consecutive checks) and minReadySeconds to 30 to ensure Traefik has fully
propagated the new endpoint before the old pod is terminated.
Resolver
- Resolver.resolve_range/3: returns the contiguous on-call segments for a
schedule across [start_at, end_at). Walks layer handoffs + override
boundaries, drops gap segments, and merges adjacent same-user segments.
UserColor module
- New Towerops.OnCall.UserColor with deterministic id -> color mapping
(Tailwind class + matching #RRGGBB hex). Refactored ScheduleLive.Show
to use it instead of its own per-page index palette so the same person
gets the same color across the schedules list and detail pages.
ScheduleLive.Index
- Date navigation: Today / prev / next + 1/2/4-week range selector.
- URL-driven: ?start=YYYY-MM-DD&range=N for shareable views; falls back
to defaults on malformed values.
- Per-row card: schedule name link, on-call-now avatar swatch, day-of-week
header strip, gantt strip rendered as a CSS grid with one column per day,
Today marker overlay.
Tests: 5 new resolver tests, 9 new UserColor tests, 5 new ScheduleLive
integration tests. Full suite: 10,210 / 0 failures. Format/dialyzer clean.
Also: cleaned up two stale inline comments in k8s/deployment.yaml that
disagreed with the values they sat next to (replicas/maxSurge).
- k8s/deployment.yaml: bump startup-probe failureThreshold from 12 to 24
(70s -> 130s max startup time) to give the app more headroom on cold start.
- lib/towerops/profiles/yaml_profiles.ex: defer YAML profile loading from
init/1 to a handle_info(:load_profiles, ...) so the supervisor (and Bandit)
come up immediately and the profiles populate in the background within ~60s.
- e2e/Makefile: convenience target wrappers around the existing npm scripts.
Without these comments the ImageUpdateAutomation with strategy=Setters
has nothing to rewrite, so new image tags detected by ImagePolicy never
get committed back to main and the cluster never rolls out new builds.
Reviewed-on: graham/towerops-web#238
Incremental compilation can miss schema field removals in dependent
modules, causing undefined_column errors in production.
Reviewed-on: graham/towerops-web#119
Set up mix_gleam archive, gleam_stdlib, and gleeunit deps. Add Gleam
compiler to all Dockerfiles, CI workflows, nix shell, and .tool-versions.
Rewrite Numeric, QueryHelpers, and URLValidator from Elixir to Gleam with
full ExUnit test coverage. Update all callers to use the Gleam modules
directly via :towerops@module syntax. Remove duplicate sanitize_like/1
private functions in snmp.ex and trace.ex.
Reviewed-on: graham/towerops-web#98
Every timeout stack trace pointed at ssl_gen_statem.call/2 — the OTP 27
SSL gen_statem hanging during recv. PG is on private network (10.0.15.20)
and SSL verify was already disabled (verify_none), providing no security
benefit while adding CPU overhead and the gen_statem layer that caused
all connection timeouts.
Removed DATABASE_SSL and DATABASE_SSL_VERIFY env vars from both the
init container and main container. Also disabled SSL on the PG server
(ssl=off in postgresql.conf, hostssl→host in pg_hba.conf).
Reviewed-on: graham/towerops-web#97
Pods were running at 500m/500m (100% of limit) with 44-47% of scheduling
periods being CPU-throttled. SSL decryption is CPU-intensive — when the
BEAM gets throttled mid ssl_gen_statem:call/2, the SSL recv stalls until
the cgroup allows CPU again, causing the 15s DBConnection timeout.
This was the actual root cause of "client timed out because it queued
and checked out the connection for longer than 15000ms" errors. Previous
fixes (TCP keepalive, pool tuning, oban pruner) were treating symptoms.
- CPU limit: 500m → 2000m (4 cores burst capacity)
- CPU request: 100m → 500m (guarantee adequate baseline)
Reviewed-on: graham/towerops-web#96
