- M17: default SNMPv3 auth protocol on the four SNMP executors
(interface/storage/processor/sensor) is now SHA-256 instead of MD5.
Devices that explicitly set MD5 still use it; only the fallback for
unconfigured devices changes.
- M23: permissions check now logs a warning when an org's memberships
aren't preloaded so a silent denial is visible at the call site
instead of looking identical to a real authz failure.
- L9: extracted the duplicated should_sync?/1 logic from gaiia, sonar,
splynx, visp, and netbox sync workers into
Towerops.Integrations.due_for_sync?/2. Each worker now passes its
own default interval (10 or 30 minutes) to a single shared helper.
Tests updated to cover the helper directly.
- M2: get_user_by_email/1 uses lower(email) = lower(?) so User@Example.com
and user@example.com resolve to the same account; closes a lookalike-
registration / password-reset confusion risk.
- M8: webhook auth plug no longer echoes "Webhook authentication not
configured" — the misconfiguration is logged server-side and the caller
gets a generic Internal server error so endpoints can't be probed.
- M9: coverages controller logs the underlying KMZ build error and
returns a generic "Failed to build KMZ" string — filesystem paths and
zip internals no longer leak.
- M10: account-data controller no longer crashes with MatchError when an
org has zero or multiple memberships; defaults to "member" and treats
any owner membership as owner.
- M11: ActivityController switches to a hard whitelist
(Atom.to_string/1 comparison) instead of String.to_existing_atom/1;
unknown filter types are dropped silently and the atom table can't be
grown from API input.
- M16: title-tracking MutationObserver is stored on `this` so destroyed()
actually disconnects it — fixes a memory leak per LV navigation.
- H19: /api/v1/mobile/auth/qr/verify no longer returns user_email. Knowing
a QR token now only tells the caller the token is valid; the email is
only revealed by /complete which consumes the token.
- H20: CoverageWorker max_attempts dropped from 3 to 1. The fail/2 path
already returns :ok and writes the failure onto the coverage record,
so the 3-attempt retry policy was never reachable and was misleading.
- H25: ChecksController.create rejects device_ids that don't belong to
the caller's organization (via ScopedResource.fetch). Without this an
API token could attach service checks to devices in another tenant.
Also strip bug ID references from in-code comments (per feedback that
H/M/L numbers don't survive past their bugs.md removal); commit history
keeps the audit trail.
- H12: session and remember-me cookies get http_only + secure (prod-only).
Cookie can no longer be read via document.cookie (XSS exfil defense)
and the Secure flag is set in production via config/prod.exs.
- L2: 404 tracker uses EXPIRE … NX so a sustained probe can't keep
refreshing the 60s window and dodge the threshold ban.
- L5: vault only reads CLOAK_KEY when :env == :prod — a developer with
a prod env var set in their shell won't accidentally encrypt local
data with the production key.
- L6: health endpoint no longer leaks the app version.
- L8: Preseem.dismiss_insight/2 + InsightsLive uses it — dismissing now
requires the insight to belong to the user's org (closes IDOR).
- L10: SidebarCollapse JS hook stores its click handler and removes it
in destroyed(); listeners no longer accumulate across LV navigation.
- L11: WebMCP navigate tool rejects anything that isn't a same-origin
absolute path (blocks javascript:, data:, off-site URLs).
- H23: introduce Towerops.Workers.SyncErrors.transient?/1 classifier;
uisp/preseem/cn_maestro sync workers now retry transient failures (5xx,
timeouts, rate limits) via Oban while still discarding permanent errors
(401/403/404) as :ok. Stale monitoring data is the bigger risk than an
extra retry on a known-bad credential.
- H26: add Monitoring.get_latency_data_for_devices/2 batch counterpart
(currently a stub map but called from SiteLive.Show so any future ping
implementation lands in a single query, not one-per-device).
- H28: Nominatim search popups in the coverage map now bind via a text
node instead of an HTML string, so a compromised/poisoned response
can't execute as JS in the popup.
- H29: yaml_profiles dot-boundary OID prefix match (handles optional
trailing dot from YAML profile keys). Stops 1.3.6.1.4.1.9 from
incorrectly matching 1.3.6.1.4.1.99.
- H6: batch interface stats query in get_site_capacity_summary, replacing
per-interface SELECT with a single grouped query
- H8: explicit expires_at check in MobileAuth and GraphQLAuth plugs as
defense in depth — the query already filters expired rows, but a future
refactor dropping the filter can't quietly re-enable revoked sessions
- H17: Reports LiveView (toggle/delete/run_now) now uses
Reports.get_organization_report/2 to scope by organization_id, fixing
IDOR where a user could manipulate other tenants' reports by ID
- H18: ToweropsWeb.Api.ParamFilter strips identity fields (id,
organization_id, user_id, created_by_id, inserted_at, updated_at) from
user-supplied API params; applied to devices, sites, checks, and
escalation_policies create/update paths to prevent mass-assignment of
tenant ownership fields if a changeset cast list ever drifts
- H1: batch count queries in MobileController (org sites/devices/alerts, site
device counts) — reduces 1+3N+2M queries to 4 + 2
- H2: batch Oban cancel for stop_device_checks — single ANY(?) query instead
of one per check
- H3: resolve_active_alerts_for_device uses update_all + deduped PubSub
broadcasts (was N updates + N broadcasts)
- H4: atomic ON CONFLICT upsert for auto-discovery checks; restore partial
unique index dropped in 20260404000002 (dedup migration included)
- H5: wrap apply_agent_to_all_equipment delete+insert in transaction so a
failed insert_all rolls back the prior delete
- H14: replace String.to_integer/1 on untrusted SNMP OID components with
Integer.parse/1 + validation (neighbor_discovery, discovery storage index,
printer supply index)
- H15: ActivityFeedLive whitelists activity types against @all_types instead
of String.to_existing_atom/1
- H16: defensive page param parsing in user_settings and admin dashboard
LiveViews
- H27: search sanitize/1 delegates to QueryHelpers.sanitize_like/1 which
escapes backslash first
- H30: JobCleanupTask no longer cancels jobs in "executing" state so
in-flight polls complete naturally
- Add organization membership check before API token creation
- Fix admin security allowlist form reading current_user instead of current_scope.user
- Use recursive File.ls instead of Path.wildcard to include hidden files in MIB validation
- Add upload size check before ZIP extraction in MIB controller
- Centralize CSP in SecurityHeaders plug with per-request nonces instead of 'unsafe-inline'
- Enforce upload size limit (100 MB), max extracted files (2000), max
uncompressed size (500 MB), and reject symlinks in MIB archives
- Replace unbounded String.to_atom calls in SNMP tokenizer with
safe_atom/1 that warns when approaching the atom table limit
- Cache MIB vendor listing with persistent_term, invalidated on
upload/delete to avoid blocking synchronous Path.wildcard scans
- Client IP: only trust X-Forwarded-For from RFC 1918 proxy IPs
- Webhook auth: handle nil/blank secret with controlled error, not 500 crash
- Sudo redirect: reuse validated return_path? from login to prevent open redirect
- Map live: remove redundant inline script (ensureLeaflet hook handles loading)
- Bang calls: convert crash-prone exact matches to case in QR live and API controllers
Fix auto_match return tuple order (mac/ip/site were swapped). Add ghost
devices tab for cleaning up stale mappings to deleted devices. Add "Link"
button on untracked devices to manually link to existing Gaiia items.
Add manual refresh button for on-demand reconciliation.
Adds MAC address matching as the primary matching phase:
- Normalizes MACs (strips colons/dots/dashes) from SNMP interface data
- Matches against gaiia_inventory_items.mac_address (synced from Gaiia)
- Phase 1: MAC → Phase 2: IP → Phase 3: Site
Flash now shows breakdown: "Auto-matched 47 devices (12 by MAC, 34 by
IP, 1 by site). 17 remaining."
A bare <select phx-change="..."> sends %{"value" => ...}, not the name
attribute. Wrapping in <form phx-change="..."> fixes this so the handler
receives %{"manufacturer_id" => value}.
- Handle empty manufacturer_id selection (select placeholder)
- Log and flash when model fetch fails or returns empty
- Add require Logger to reconciliation live view
Replaces auto-matching with explicit two-step selection:
1. Select manufacturer from Gaiia's inventory model library (dropdown)
2. Select model for that manufacturer (dropdown, filtered live)
3. Click Create
Manufacturers and models are fetched from Gaiia on demand and cached
in socket assigns. Only one device form is open at a time.
ngettext is a macro that already calls dngettext internally. The
previous code was calling dngettext a second time on the result string,
passing nil as the count — causing a FunctionClauseError.
Adds `Towerops.Gaiia.InventoryCreator` which:
- Matches a device's SNMP-discovered manufacturer/model against Gaiia's
inventory model library to find the right modelId
- Calls `startAddInventoryItemsJob` to create the item, assigned to the
correct Gaiia network site with MAC address from SNMP interfaces
A "Create in Gaiia" button now appears next to each untracked device on
the reconciliation page.
Two improvements to Gaiia reconciliation:
1. Auto-match now uses two-phase matching:
- Phase 1: IP match (high confidence, same as before)
- Phase 2: Site match — if a device is at a Towerops site linked to a
Gaiia network site, match it to unmapped inventory assigned to that site
2. Gaiia sync now automatically maps network sites to Towerops sites by
name during sync. When a Gaiia network site name matches a Towerops
site name, `site_id` is set on the `gaiia_network_sites` row. This
enables the site-based matching in phase 2.
Flash message now shows breakdown: "Auto-matched 47 devices (34 by IP,
13 by site). 17 remaining."
Adds `Gaiia.auto_match_untracked/1` which finds unmapped Gaiia inventory
items whose IP matches an untracked Towerops device and links them.
Button on the reconciliation page's Untracked tab runs the match
inline and shows a flash with matched/remaining counts.
Warning: 65°C → 80°C, Critical: 75°C → 90°C. Outdoor gear in Texas
routinely runs at 70-75°C ambient. The old thresholds generated useless
AI insights like "Verona to Altoga at 72°C may be overheating" — that's
normal for an enclosure in summer, not actionable.
The LLM can now copy device_id from the snapshot into its observation
output. The worker wires it into the insight's device_id FK so the UI
can link directly to the referenced device.
Adds a wireless section to the NetworkSnapshot with per-AP radio state
(channel, frequency, channel width, noise floor, RF/QoE scores), per-channel
occupancy (own-fleet + foreign neighbor counts), and APs with the worst
foreign interference. Updates the system prompt to describe this data and
removes device_id from the expected output schema — the LLM now references
devices by name.
deepseek-v4-flash is still a reasoning variant and burns tokens on
chain-of-thought before producing content. deepseek-chat produces output
directly — 82 tokens all went to content in testing, vs 85-102 tokens
that produced empty observations on flash.
Info-level logs for: org count at start, observation count per org,
successful persistence, and dedup. Makes the worker observable without
needing database access.
deepseek-v4-pro is a reasoning model that burns tokens on chain-of-thought
before producing content. The flash variant is faster, cheaper, and doesn't
starve the content output — a better fit for summarization and observation
generation tasks.
deepseek-v4-pro is a reasoning model that burns tokens on
reasoning_content before producing content. When max_tokens was too low
(400/1500), all tokens went to reasoning and content was empty.
- parse_response now falls back to reasoning_content when content is blank
- max_tokens raised to 20_000 across all call sites and default
- improved parse logging shows raw/cleaned byte sizes
The network insight prompt is large and max_tokens is 1500 — DeepSeek
takes longer than 30s. Also plumb opts[:receive_timeout] through so
workers can tune it.
The parse function silently returned {:ok, []} on any decode failure or
observation rejection. Log warnings for both cases so we can see what the
LLM is actually returning.
All 4 webhook controllers now verify the signature, insert an Oban job,
and return immediately. Previously, Stripe and PagerDuty did DB queries +
writes inline; Gaiia ran upserts; Agent Release broadcast to all connected
WebSockets — all in the request path.
New workers:
- StripeWebhookWorker — delegates to WebhookProcessor.process/1
- GaiiaWebhookWorker — delegates to Webhooks.process_event/3
- PagerdutyWebhookWorker — resolve/acknowledge alerts by dedup key
- AgentReleaseWebhookWorker — broadcast mass update to agents
Also fix: log error when resolve_alert fails in agent_channel (was silently
discarded, causing "Resolving stuck device_down alert" to repeat in logs).
The Repo.transaction wrapping Repo.stream() held a checkout for the
entire iteration, including the LLM HTTP call. When the LLM took >15s,
Postgrex killed the connection, cascading to all subsequent orgs.
Collect org IDs first, then process each independently.
Adds a parallel LLM path that reviews the whole org snapshot — Preseem
APs, SNMP signals, backhaul, agents, and the insight types the rules
already covered — and emits novel observations as ai_observation
insights (source=ai). Existing rule-based insights and per-insight
enrichment continue unchanged.
- lib/towerops/llm/network_snapshot.ex builds the bounded snapshot
- lib/towerops/llm/network_insight_prompt.ex builds the prompt and
parses the LLM's JSON observations array
- lib/towerops/workers/ai_network_insight_worker.ex runs nightly at
03:30 UTC (after the rule pass + per-insight enrichment) and on
the superuser regenerate button
- preseem/insight.ex extends @valid_types with "ai_observation" and
@valid_sources with "ai"
- insights_live/index.ex includes the new worker in the regen list
and adds an "AI" source badge style
Bumps the SnmpKit Manager timeout-options test threshold from 500ms
to 3000ms (was flaking on loaded dev machines), and scopes
list_unenriched_insights/1 assertion in insights_test:573 to the
test's org so cross-suite leaks don't fail the count.
- ToweropsWeb.AgentChannel now runs sensor readings through
Towerops.Snmp.SensorScale.normalize/2 in both resolve_sensor_value/2
and the GraphQL publisher. Without this, MikroTik mtxrHl* deci-degree
readings (e.g. Culleoka reporting 294 -> 29.4°C) were stored and
alerted on at 294°C. The DevicePollerWorker path already did this;
only the agent path was missing it.
- Demote "Received SNMP result from agent" and "Processing polling
result for X" from info to debug — they fire every poll cycle on
every device and were dominating prod logs.
- /insights now shows a "Regenerate insights" button for superusers
that enqueues all seven insight workers (Preseem baselines,
capacity, device-health, Gaiia, system, wireless, LLM enrichment).
Non-superusers neither see the button nor can trigger the event.
- Dialyzer is clean: added :tools to plt_add_apps, gave
Preseem.Insight a @type t, pinned a few unmatched returns, and
suppressed a known MapSet opaque false-positive in Towerops.Unused.
- e2e: NODE_OPTIONS=--disable-warning=DEP0205 silences the Node 25
deprecation noise from Playwright's internal ESM loader.
Token-usage doesn't belong on the per-insight row — every consumer of
Towerops.LLM (insight enrichment today; billing-anomaly summaries,
monitoring narratives, fleet weekly reports tomorrow) needs to record
into one queryable place. Per-insight columns also made it impossible
to track LLM calls that aren't tied to an insight.
Schema: `llm_usage` keyed (organization_id, day, model, purpose) with a
unique index. organization_id is nullable for system-level analyses.
Each row carries running totals plus a request_count — callers UPSERT
and increment atomically.
API: `Towerops.LLM.Usage.record/1` for write, `summarize/2` for
read-side rollups. The worker now calls `Usage.record(%{... purpose:
"insight_enrichment"})` after each successful enrichment.
Removed: `llm_prompt_tokens` / `llm_completion_tokens` from
preseem_insights — same migration drops the columns and creates
llm_usage in one go. `apply_llm_enrichment` reverts to /3 (no usage
arg).
If a rule keeps re-firing the underlying condition, the dedup query
suppresses duplicate inserts but `inserted_at` of the live row stays
old — so a real 6-month-old issue and a one-off blip both linger
forever. Auto-resolving any active insight older than 7 days clears
the dashboard. Still-real issues will be re-created by the next rule
pass (dedup only checks `status: "active"`), so the operator sees
fresh evidence rather than stale text.
- Towerops.Workers.InsightExpiryWorker (queue: :maintenance) flips
`status` to "resolved" on rows where `inserted_at < now - 7d`.
- Window is configurable via `:max_age_days` for ops who want a
different cadence per env.
- Oban cron: 04:00 UTC nightly, after the 02:00 rule pass and 03:00
LLM enrichment.
