- Fix doctests for Accounts, Agents.Stats, Snmp to match actual behavior
- Fix dynamic_extra_test vendor post-processing tests to seed sensor data
- Fix activity_controller_test to seed devices for feed data
- Fix session_manager_test to create browser session for test
- Fix topology_test link creation for connection test
- Fix device_monitor/driver_worker tests for unique job constraints
- Fix accounts_test expired_tokens assertion (magic link token is expired)
- Fix happy_path_test and show_events_test to seed monitor data
- Fix admin user_live_test user.name -> user.email (no name field)
- Fix schema_test to seed activity data
- Fix mobile_qr_live_test to match actual template text
- Fix SnmpKit.MIB doctests and tests for enriched return values
- Fix onboarding_live, mobile_controller, mib_test weak assertions
- Remove dead code and fix credo warnings
All checks set to exit_status: 0 initially to avoid blocking on
pre-existing violations. Remove exit_status overrides as each
check category is cleaned up incrementally.
Dependency added to dev/test only.
All checks set to exit_status: 0 initially to avoid blocking on
pre-existing violations. Remove exit_status overrides as each
check category is cleaned up incrementally.
Three changes targeting the root cause of production up/down alerting:
1. Skip BruteForceProtection for /health and /health/live paths
(removes unnecessary Repo.get_by(IpBlock) DB query from every
k8s probe, matching the existing /socket/agent exemption).
2. Relax k8s readiness probe: periodSeconds 5→10, timeoutSeconds
2→5, failureThreshold 2→3, successThreshold 3→2 so transient
DB/Redis blips don't cascade into full service unavailability.
3. Use the persistent Towerops.Redix connection for health checks
instead of opening a brand-new TCP connection (handshake→AUTH→
PING→close) on every probe. Towerops.Redix.Fake gains a
set_ping_response/2 toggle for testing failure paths.
Production code (lib/): all Process.sleep calls replaced with receive
after timeout - retry backoff, exponential backoff, batch delays,
poll intervals, and settle waits.
Test code: poll_until helpers across 4 agent channel test files now
use receive after instead of Process.sleep. Various standalone
Process.sleep(N) calls replaced with :sys.get_state sync barriers
or receive after. deferred_discovery test simulation sleeps reduced
500ms -> 30ms.
Test takes ~16s on this hardware to resolve/error on invalid
charlists. 1000ms timeout was causing CI pipeline failure, preventing
image build and deployment.
Backhaul radios (ePMP, AirOS, PMP, Cambium, MikroTik, RouterOS) now
monitor their connected client count with a low threshold of 1. When
the count drops to 0 the SnmpSensorExecutor marks it CRITICAL, which
triggers an alert through the existing check → alert pipeline.
- Adds field to sensor_def type in Vendor behaviour
- Sets on all clients-type wireless sensors in
backhaul-capable vendor profiles
- Adds default executor handling: any clients sensor at 0 → status 2
- 3 new tests for clients sensor behavior
Prod paired the Oban Pro Smart engine + unique workers with core
Oban.Plugins.Lifeline, whose rescue has no unique_violation handling. An
orphaned executing CheckExecutorWorker job rescued back to available collides
with its already-scheduled successor on oban_jobs_unique_index (23505),
crashing the plugin every 60s and leaving ~150 orphans stuck. Switch prod to
Oban.Pro.Plugins.DynamicLifeline, which repairs the conflict via
Smart.clear_uniq_violation. Dev keeps core Lifeline (Basic engine).
Add a shallow /health/live endpoint that does not touch db/redis. k8s liveness
and startup probes now target it so a transient dependency outage can't kill or
block boot of a healthy pod; readiness keeps the deep /health to gate the load
balancer. (deployment.yaml probe/replica change pushed separately, after the
image carrying /health/live is live.)
Also drop unused POSTGRES_* keys from the secrets example.
CheckExecutorWorker had no Oban `unique` constraint, so every insert path
created a fresh job row: self-scheduling, JobHealthCheckWorker, the check
form/API/GraphQL, and — the main amplifier — Monitoring.schedule_check/1
called unconditionally by snmp.ex discovery after create_check/1, which is
an upsert that returns {:ok, check} even when the check already exists.
Every SNMP discovery pass re-enqueued a job for every existing check, so
the check_executors queue grew to ~70k jobs for ~5.7k checks and stopped
draining.
Add `unique: [keys: [:check_id], states: [:available, :scheduled,
:retryable], period: :infinity]`. `:executing` is excluded so perform/1's
self-reschedule does not conflict with its own running job.
Also add the `checks` and `check_executors` queues to config/dev.exs —
they were missing from dev entirely (prod's runtime.exs already had them).
SyncErrors.transient?/1 only had explicit clauses for the {:http_error, ...}
shape, but vendor API clients return {:unexpected_status, status, body}. A 502
was retried only by accident via the catch-all, and a 4xx in that shape would
have been retried 20 times. Add explicit {:unexpected_status, ...} clauses: 5xx
is transient (Oban backs off and retries), 4xx is permanent.
- M20: Add check_no_hardlinks and check_no_special_files to MIB upload
archive extraction to prevent hard link, device node, and FIFO attacks
- M21: Change RateLimit ETS table from :public to :protected; route hit/get/reset
through GenServer.call instead of direct ETS access
- M22: Add 8-hour impersonation timeout — store impersonated_at in session and
auto-revoke impersonation when expired
- L14: Add default limit (500) and optional offset to list_checks for pagination
Use Repo.get_by(schema, id: id, organization_id: organization_id) instead of
Repo.get + pattern match so that resources from wrong orgs return :not_found
instead of :forbidden, preventing org membership discovery.
- M3: Replace blacklist-based valid_return_path? with whitelist of known app
route prefixes, plus URI decoding to prevent %2f encoding bypasses
- M6: Replace full-org device/link load in get_node_detail with targeted
join query filtering by discovered node fields
- M7: Add partial unique index on (organization_id, email) for pending
invitations (WHERE accepted_at IS NULL)
- M12: Add on_mount superuser verification to all 7 admin LiveViews for
defense-in-depth
Sonar, Splynx, VISP, NetBox, and Gaiia sync workers all followed the same
"cron runs, then Enum.map syncs every integration in the same job" shape.
A single slow tenant — anything from an upstream rate limit to a stuck
GraphQL call — would hold the maintenance queue slot for ten minutes
while every other integration waited in line.
Each worker now uses the dispatcher/per-integration pattern that
UispSyncWorker already had:
- the cron `perform(%Oban.Job{args: %{}})` fans out one job per eligible
integration via `PollingOffset.calculate_offset/2`, staggered across a
5- to 10-minute window with `unique: [period: window_seconds]`
- `perform(%Oban.Job{args: %{"integration_id" => id}})` does the actual
sync, classifying failures via `Towerops.Workers.SyncErrors.transient?/1`
so permanent errors (auth, 4xx) don't get retried forever
Worker-specific defaults preserved:
- gaiia / netbox: 30/600s window, longer interval
- sonar / splynx / visp: 10/300s window
Tests updated to match the two-stage dispatcher pattern — `perform_job/2`
with empty args now asserts the per-integration jobs got enqueued and
the actual sync is invoked via a separate `perform_job/2` call with the
integration_id arg.
- M17: default SNMPv3 auth protocol on the four SNMP executors
(interface/storage/processor/sensor) is now SHA-256 instead of MD5.
Devices that explicitly set MD5 still use it; only the fallback for
unconfigured devices changes.
- M23: permissions check now logs a warning when an org's memberships
aren't preloaded so a silent denial is visible at the call site
instead of looking identical to a real authz failure.
- L9: extracted the duplicated should_sync?/1 logic from gaiia, sonar,
splynx, visp, and netbox sync workers into
Towerops.Integrations.due_for_sync?/2. Each worker now passes its
own default interval (10 or 30 minutes) to a single shared helper.
Tests updated to cover the helper directly.
- H19: /api/v1/mobile/auth/qr/verify no longer returns user_email. Knowing
a QR token now only tells the caller the token is valid; the email is
only revealed by /complete which consumes the token.
- H20: CoverageWorker max_attempts dropped from 3 to 1. The fail/2 path
already returns :ok and writes the failure onto the coverage record,
so the 3-attempt retry policy was never reachable and was misleading.
- H25: ChecksController.create rejects device_ids that don't belong to
the caller's organization (via ScopedResource.fetch). Without this an
API token could attach service checks to devices in another tenant.
Also strip bug ID references from in-code comments (per feedback that
H/M/L numbers don't survive past their bugs.md removal); commit history
keeps the audit trail.
- H12: session and remember-me cookies get http_only + secure (prod-only).
Cookie can no longer be read via document.cookie (XSS exfil defense)
and the Secure flag is set in production via config/prod.exs.
- L2: 404 tracker uses EXPIRE … NX so a sustained probe can't keep
refreshing the 60s window and dodge the threshold ban.
- L5: vault only reads CLOAK_KEY when :env == :prod — a developer with
a prod env var set in their shell won't accidentally encrypt local
data with the production key.
- L6: health endpoint no longer leaks the app version.
- L8: Preseem.dismiss_insight/2 + InsightsLive uses it — dismissing now
requires the insight to belong to the user's org (closes IDOR).
- L10: SidebarCollapse JS hook stores its click handler and removes it
in destroyed(); listeners no longer accumulate across LV navigation.
- L11: WebMCP navigate tool rejects anything that isn't a same-origin
absolute path (blocks javascript:, data:, off-site URLs).
- Add organization membership check before API token creation
- Fix admin security allowlist form reading current_user instead of current_scope.user
- Use recursive File.ls instead of Path.wildcard to include hidden files in MIB validation
- Add upload size check before ZIP extraction in MIB controller
- Centralize CSP in SecurityHeaders plug with per-request nonces instead of 'unsafe-inline'
Warning: 65°C → 80°C, Critical: 75°C → 90°C. Outdoor gear in Texas
routinely runs at 70-75°C ambient. The old thresholds generated useless
AI insights like "Verona to Altoga at 72°C may be overheating" — that's
normal for an enclosure in summer, not actionable.
All 4 webhook controllers now verify the signature, insert an Oban job,
and return immediately. Previously, Stripe and PagerDuty did DB queries +
writes inline; Gaiia ran upserts; Agent Release broadcast to all connected
WebSockets — all in the request path.
New workers:
- StripeWebhookWorker — delegates to WebhookProcessor.process/1
- GaiiaWebhookWorker — delegates to Webhooks.process_event/3
- PagerdutyWebhookWorker — resolve/acknowledge alerts by dedup key
- AgentReleaseWebhookWorker — broadcast mass update to agents
Also fix: log error when resolve_alert fails in agent_channel (was silently
discarded, causing "Resolving stuck device_down alert" to repeat in logs).
Adds a parallel LLM path that reviews the whole org snapshot — Preseem
APs, SNMP signals, backhaul, agents, and the insight types the rules
already covered — and emits novel observations as ai_observation
insights (source=ai). Existing rule-based insights and per-insight
enrichment continue unchanged.
- lib/towerops/llm/network_snapshot.ex builds the bounded snapshot
- lib/towerops/llm/network_insight_prompt.ex builds the prompt and
parses the LLM's JSON observations array
- lib/towerops/workers/ai_network_insight_worker.ex runs nightly at
03:30 UTC (after the rule pass + per-insight enrichment) and on
the superuser regenerate button
- preseem/insight.ex extends @valid_types with "ai_observation" and
@valid_sources with "ai"
- insights_live/index.ex includes the new worker in the regen list
and adds an "AI" source badge style
Bumps the SnmpKit Manager timeout-options test threshold from 500ms
to 3000ms (was flaking on loaded dev machines), and scopes
list_unenriched_insights/1 assertion in insights_test:573 to the
test's org so cross-suite leaks don't fail the count.
- ToweropsWeb.AgentChannel now runs sensor readings through
Towerops.Snmp.SensorScale.normalize/2 in both resolve_sensor_value/2
and the GraphQL publisher. Without this, MikroTik mtxrHl* deci-degree
readings (e.g. Culleoka reporting 294 -> 29.4°C) were stored and
alerted on at 294°C. The DevicePollerWorker path already did this;
only the agent path was missing it.
- Demote "Received SNMP result from agent" and "Processing polling
result for X" from info to debug — they fire every poll cycle on
every device and were dominating prod logs.
- /insights now shows a "Regenerate insights" button for superusers
that enqueues all seven insight workers (Preseem baselines,
capacity, device-health, Gaiia, system, wireless, LLM enrichment).
Non-superusers neither see the button nor can trigger the event.
- Dialyzer is clean: added :tools to plt_add_apps, gave
Preseem.Insight a @type t, pinned a few unmatched returns, and
suppressed a known MapSet opaque false-positive in Towerops.Unused.
- e2e: NODE_OPTIONS=--disable-warning=DEP0205 silences the Node 25
deprecation noise from Playwright's internal ESM loader.
Token-usage doesn't belong on the per-insight row — every consumer of
Towerops.LLM (insight enrichment today; billing-anomaly summaries,
monitoring narratives, fleet weekly reports tomorrow) needs to record
into one queryable place. Per-insight columns also made it impossible
to track LLM calls that aren't tied to an insight.
Schema: `llm_usage` keyed (organization_id, day, model, purpose) with a
unique index. organization_id is nullable for system-level analyses.
Each row carries running totals plus a request_count — callers UPSERT
and increment atomically.
API: `Towerops.LLM.Usage.record/1` for write, `summarize/2` for
read-side rollups. The worker now calls `Usage.record(%{... purpose:
"insight_enrichment"})` after each successful enrichment.
Removed: `llm_prompt_tokens` / `llm_completion_tokens` from
preseem_insights — same migration drops the columns and creates
llm_usage in one go. `apply_llm_enrichment` reverts to /3 (no usage
arg).
If a rule keeps re-firing the underlying condition, the dedup query
suppresses duplicate inserts but `inserted_at` of the live row stays
old — so a real 6-month-old issue and a one-off blip both linger
forever. Auto-resolving any active insight older than 7 days clears
the dashboard. Still-real issues will be re-created by the next rule
pass (dedup only checks `status: "active"`), so the operator sees
fresh evidence rather than stale text.
- Towerops.Workers.InsightExpiryWorker (queue: :maintenance) flips
`status` to "resolved" on rows where `inserted_at < now - 7d`.
- Window is configurable via `:max_age_days` for ops who want a
different cadence per env.
- Oban cron: 04:00 UTC nightly, after the 02:00 rule pass and 03:00
LLM enrichment.
UI: the AI-generated footer dropped " · deepseek-v4-pro" — model
identity is irrelevant to operators reviewing the recommendation.
Parser: stop letting raw JSON leak into the summary text when the LLM
returns an almost-but-not-quite-valid response.
- Strip ```json fences (already done) PLUS try a balanced-brace
extraction so a "Here's the JSON: {…}" preamble doesn't break
Jason.decode
- When all decode paths fail, regex-extract `"summary"` and `"action"`
fields from the text rather than dumping the whole blob with braces
intact
- Last-ditch sanitiser strips remaining brace/quote noise so the
rendered summary never shows literal `"summary"` keys
Maintenance: `Towerops.Maintenance.RepairLlmSummaries.run()` clears
`llm_enriched_at` on already-stored rows whose summaries contain JSON
artifacts so the next worker cycle re-enriches them with the new
parser.
Test: insights_live_events_test now asserts the AI-generated label is
present *and* that the model name does not leak into the rendered HTML.
Some MikroTik models report `mtxrGaugeValue` (mtxrGaugeUnit=1, °C) in
deci-degrees, while others — like the CRS317 in upstream test data —
return whole degrees. The MIB doesn't disclose which, so the YAML
profile can't pick one literal divisor. Result: a sensor that should
read 56.92 °C was being stored and alerted on as 5,692 °C.
- Towerops.Snmp.SensorScale.normalize/2: divide-by-10 a temperature
reading until it falls below 150 °C, capped at 3 iterations. Other
sensor types pass through untouched.
- DevicePollerWorker.poll_simple_sensor wires the normaliser in after
the YAML divisor is applied, so live polling stops storing bogus
values immediately.
- DeviceOverheating rule now bounds the query at 150 °C as defense in
depth so a future scaling glitch never produces a 5,000 °C alert.
- mix towerops.fix_mikrotik_temperature_scaling backfills any rows
already in the DB by re-running normalize/2 over them.
- routeros.yaml profile gets a comment pointing at the runtime
normaliser instead of a wrong unconditional divisor.
Adds an "ai" pill to the source-filter strip on /insights and a small
blue "AI" badge on every card whose `llm_enriched_at` is set. Selecting
the pill returns every LLM-enriched insight regardless of underlying
source (preseem / snmp / gaiia / system).
`source: "ai"` is a virtual filter — `list_insights/2` translates it to
`WHERE llm_enriched_at IS NOT NULL` rather than equality on the source
column, so existing real source semantics stay intact.
Track DeepSeek token spend per enriched insight so cost can be rolled
up by SQL — `SELECT date_trunc('day', llm_enriched_at), organization_id,
SUM(llm_prompt_tokens), SUM(llm_completion_tokens) FROM preseem_insights
GROUP BY 1, 2`.
- Migration adds llm_prompt_tokens + llm_completion_tokens to
preseem_insights
- DeepSeek client surfaces the API's `usage` block (prompt/completion/
total tokens) on the success response
- apply_llm_enrichment/4 takes a usage map and persists the two int
columns
- Worker plumbs the usage from the response through to the context
Replaces the previous :requires_redis skip with a real test double so
the suite stays self-contained — no external Redis service needed in
CI, but the call paths through RedisHealthCheck and FourOhFourTracker
still get exercised end-to-end.
- Towerops.Redix.Fake: GenServer that speaks PING / SADD / SCARD /
SISMEMBER / SMEMBERS / EXPIRE / DEL / FLUSHDB. start_link refuses
anything other than localhost:6379 so failure-path tests still fail
the way they did against real Redix.
- Application supervisor starts the fake under name Towerops.Redix in
test env (instead of the dummy Agent stub).
- RedisHealthCheck and FourOhFourTracker now route Redix calls through
Application.get_env(:towerops, :redix_module, Redix). Test config
points it at the fake.
- Drops the test_helper :requires_redis exclusion and the @moduletag
on the two test files; FourOhFourTrackerTest setup just resets the
fake between runs.
The Forgejo CI test gate has no Redis service, so FourOhFourTracker
and RedisHealthCheck tests fail with Redix.ConnectionError. They've
always been broken in CI but the failures got noticed now.
Mirrors the existing :requires_gdal pattern: probe localhost:6379 in
test_helper.exs and add :requires_redis to extra_excludes when nothing
answers. Locally (where direnv runs Redis) the tag isn't excluded so
the tests still run.
