feat(insights): device-overheating recommendation
Operators frequently misdiagnose thermal throttling as RF problems and waste truck rolls. This rule reads the hottest temperature sensor per device and, when the device is linked to a Preseem AP with degraded QoE, makes the thermal-correlation explicit so the fix is ventilation/shading rather than antenna alignment. - Towerops.Recommendations.Rules.DeviceOverheating — single query pulls the hottest temperature sensor per device above 65 °C; one insight per device. Critical at 75 °C+. Includes the linked Preseem qoe_score in metadata when present. - Insight.@valid_types extended with device_overheating. - LLM prompt hint tells the model to recommend ventilation/fan/ shading inspection BEFORE any RF action when QoE is also low. - UI: red card with sensor name, temperature, linked QoE score, and an "Above manufacturer spec" badge for >=75 °C. Wired into RecommendationsRunWorker. 8 new tests.
This commit is contained in:
parent
727a743c11
commit
5eb1acf9d4
8 changed files with 286 additions and 2 deletions
|
|
@ -1,3 +1,20 @@
|
||||||
|
2026-05-09
|
||||||
|
feat(insights): device-overheating recommendation
|
||||||
|
Towerops.Recommendations.Rules.DeviceOverheating — flags devices
|
||||||
|
whose hottest temperature sensor reads >=65 °C (warning) or
|
||||||
|
>=75 °C (critical). Picks the hottest sensor per device so
|
||||||
|
chassis vs CPU vs PHY all roll up to one actionable insight.
|
||||||
|
When the device is linked to a Preseem AccessPoint with
|
||||||
|
qoe_score < 60, the description and LLM prompt hint at thermal
|
||||||
|
throttling — operators frequently misdiagnose thermal as RF and
|
||||||
|
waste truck rolls.
|
||||||
|
Insight.@valid_types extended with device_overheating.
|
||||||
|
Wired into RecommendationsRunWorker.
|
||||||
|
LLM prompt for device_overheating tells the model to recommend
|
||||||
|
ventilation/fan/shading inspection BEFORE RF action.
|
||||||
|
UI: red card with hottest sensor temp, sensor name, linked QoE
|
||||||
|
score, and "Above manufacturer spec" badge above 75 °C.
|
||||||
|
|
||||||
2026-05-09
|
2026-05-09
|
||||||
feat(llm): per-insight-type prompt hints
|
feat(llm): per-insight-type prompt hints
|
||||||
Towerops.LLM.InsightPrompt now appends a type-specific expert hint
|
Towerops.LLM.InsightPrompt now appends a type-specific expert hint
|
||||||
|
|
|
||||||
|
|
@ -98,6 +98,17 @@ defmodule Towerops.LLM.InsightPrompt do
|
||||||
Rule context: a Preseem AP has more subscribers than its baseline
|
Rule context: a Preseem AP has more subscribers than its baseline
|
||||||
p95. Metadata includes current_count and baseline_p95. Recommend
|
p95. Metadata includes current_count and baseline_p95. Recommend
|
||||||
rebalancing or capacity management.
|
rebalancing or capacity management.
|
||||||
|
""",
|
||||||
|
"device_overheating" => """
|
||||||
|
Rule context: a device has at least one temperature sensor above
|
||||||
|
65 °C (warning) or 75 °C (critical). Metadata fields:
|
||||||
|
temperature_c, sensor_descr, qoe_score, warning_threshold_c,
|
||||||
|
critical_threshold_c.
|
||||||
|
If qoe_score is also low (<60), highlight that the QoE drop is
|
||||||
|
likely thermal — operators frequently misdiagnose thermal
|
||||||
|
throttling as RF and waste truck rolls. Recommend ventilation,
|
||||||
|
fan, or shading inspection BEFORE any RF action. Mention the
|
||||||
|
specific sensor name when available.
|
||||||
"""
|
"""
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,7 @@ defmodule Towerops.Preseem.Insight do
|
||||||
@primary_key {:id, :binary_id, autogenerate: true}
|
@primary_key {:id, :binary_id, autogenerate: true}
|
||||||
@foreign_key_type :binary_id
|
@foreign_key_type :binary_id
|
||||||
|
|
||||||
@valid_types ~w(qoe_degradation capacity_saturation firmware_opportunity model_underperforming subscriber_growth config_drift snmp_cpu_high snmp_memory_high snmp_disk_high device_poll_gap agent_offline firmware_mismatch reconciliation_finding subscriber_growth_trend suspect_config_change backhaul_over_capacity backhaul_near_capacity wireless_signal_weak wireless_snr_low wireless_ap_overloaded wireless_client_missing wireless_coverage_gap ap_frequency_change sector_overload cpe_realign own_fleet_channel_conflict)
|
@valid_types ~w(qoe_degradation capacity_saturation firmware_opportunity model_underperforming subscriber_growth config_drift snmp_cpu_high snmp_memory_high snmp_disk_high device_poll_gap agent_offline firmware_mismatch reconciliation_finding subscriber_growth_trend suspect_config_change backhaul_over_capacity backhaul_near_capacity wireless_signal_weak wireless_snr_low wireless_ap_overloaded wireless_client_missing wireless_coverage_gap ap_frequency_change sector_overload cpe_realign own_fleet_channel_conflict device_overheating)
|
||||||
@valid_urgencies ~w(critical warning info)
|
@valid_urgencies ~w(critical warning info)
|
||||||
@valid_statuses ~w(active dismissed resolved)
|
@valid_statuses ~w(active dismissed resolved)
|
||||||
@valid_channels ~w(proactive contextual passive)
|
@valid_channels ~w(proactive contextual passive)
|
||||||
|
|
|
||||||
102
lib/towerops/recommendations/rules/device_overheating.ex
Normal file
102
lib/towerops/recommendations/rules/device_overheating.ex
Normal file
|
|
@ -0,0 +1,102 @@
|
||||||
|
defmodule Towerops.Recommendations.Rules.DeviceOverheating do
|
||||||
|
@moduledoc """
|
||||||
|
Flags devices whose hottest temperature sensor crosses the
|
||||||
|
manufacturer-typical "warm" or "hot" thresholds. When the device is
|
||||||
|
linked to a Preseem AccessPoint with a degraded QoE score, the
|
||||||
|
insight description hints at thermal-throttling correlation so the
|
||||||
|
operator can avoid an unnecessary RF truck-roll.
|
||||||
|
|
||||||
|
Thresholds are intentionally band-agnostic:
|
||||||
|
* 65 °C → warning (warm enclosure / chassis)
|
||||||
|
* 75 °C → critical (most outdoor radio specs cap at 70-80 °C)
|
||||||
|
|
||||||
|
Pulls the *hottest* current temperature sensor per device — chassis
|
||||||
|
vs CPU vs PHY don't matter for the operator's first-pass action;
|
||||||
|
whichever is hottest is the actionable one.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import Ecto.Query
|
||||||
|
|
||||||
|
alias Towerops.Devices.Device
|
||||||
|
alias Towerops.Preseem.AccessPoint
|
||||||
|
alias Towerops.Repo
|
||||||
|
alias Towerops.Snmp.Device, as: SnmpDevice
|
||||||
|
alias Towerops.Snmp.Sensor
|
||||||
|
|
||||||
|
@warning_threshold_c 65.0
|
||||||
|
@critical_threshold_c 75.0
|
||||||
|
@qoe_concern_threshold 60.0
|
||||||
|
|
||||||
|
@spec evaluate(Ecto.UUID.t()) :: [map()]
|
||||||
|
def evaluate(organization_id) when is_binary(organization_id) do
|
||||||
|
organization_id
|
||||||
|
|> load_hottest_temperatures()
|
||||||
|
|> Enum.map(&build_insight(&1, organization_id))
|
||||||
|
end
|
||||||
|
|
||||||
|
defp load_hottest_temperatures(organization_id) do
|
||||||
|
# Fetch device + the hottest temperature sensor per device above the
|
||||||
|
# warning threshold. Done as a single query to avoid N+1.
|
||||||
|
from(d in Device,
|
||||||
|
join: sd in SnmpDevice,
|
||||||
|
on: sd.device_id == d.id,
|
||||||
|
join: s in Sensor,
|
||||||
|
on: s.snmp_device_id == sd.id,
|
||||||
|
where:
|
||||||
|
d.organization_id == ^organization_id and
|
||||||
|
s.sensor_type == "temperature" and
|
||||||
|
not is_nil(s.last_value) and
|
||||||
|
s.last_value >= ^@warning_threshold_c,
|
||||||
|
order_by: [asc: d.id, desc: s.last_value],
|
||||||
|
select: {d, s}
|
||||||
|
)
|
||||||
|
|> Repo.all()
|
||||||
|
|> Enum.uniq_by(fn {d, _s} -> d.id end)
|
||||||
|
end
|
||||||
|
|
||||||
|
defp build_insight({device, sensor}, organization_id) do
|
||||||
|
temp = sensor.last_value
|
||||||
|
qoe = preseem_qoe(device.id)
|
||||||
|
|
||||||
|
%{
|
||||||
|
organization_id: organization_id,
|
||||||
|
device_id: device.id,
|
||||||
|
type: "device_overheating",
|
||||||
|
urgency: urgency_for(temp),
|
||||||
|
channel: "proactive",
|
||||||
|
source: "snmp",
|
||||||
|
title: "Device #{device.name || sensor.sensor_descr} running hot: #{format_temp(temp)}",
|
||||||
|
description: description_for(temp, qoe),
|
||||||
|
metadata: %{
|
||||||
|
"temperature_c" => temp,
|
||||||
|
"sensor_descr" => sensor.sensor_descr,
|
||||||
|
"warning_threshold_c" => @warning_threshold_c,
|
||||||
|
"critical_threshold_c" => @critical_threshold_c,
|
||||||
|
"qoe_score" => qoe,
|
||||||
|
"qoe_concern_threshold" => @qoe_concern_threshold
|
||||||
|
}
|
||||||
|
}
|
||||||
|
end
|
||||||
|
|
||||||
|
defp preseem_qoe(device_id) do
|
||||||
|
Repo.one(from(ap in AccessPoint, where: ap.device_id == ^device_id, select: ap.qoe_score, limit: 1))
|
||||||
|
end
|
||||||
|
|
||||||
|
defp urgency_for(temp) when is_number(temp) and temp >= @critical_threshold_c, do: "critical"
|
||||||
|
defp urgency_for(_), do: "warning"
|
||||||
|
|
||||||
|
defp description_for(temp, qoe) when is_number(qoe) and qoe < @qoe_concern_threshold do
|
||||||
|
"Hottest temperature sensor reads #{format_temp(temp)}. Linked Preseem QoE " <>
|
||||||
|
"score is #{qoe} — possible thermal-throttling correlation. Inspect " <>
|
||||||
|
"enclosure ventilation, fan, and shading before assuming an RF cause."
|
||||||
|
end
|
||||||
|
|
||||||
|
defp description_for(temp, _qoe) do
|
||||||
|
"Hottest temperature sensor reads #{format_temp(temp)}. Above the " <>
|
||||||
|
"manufacturer-typical warning threshold; check enclosure ventilation, " <>
|
||||||
|
"fan, and direct-sun exposure."
|
||||||
|
end
|
||||||
|
|
||||||
|
defp format_temp(temp) when is_number(temp), do: "#{Float.round(temp / 1.0, 1)} °C"
|
||||||
|
defp format_temp(_), do: "?°C"
|
||||||
|
end
|
||||||
|
|
@ -18,6 +18,7 @@ defmodule Towerops.Workers.RecommendationsRunWorker do
|
||||||
alias Towerops.Organizations.Organization
|
alias Towerops.Organizations.Organization
|
||||||
alias Towerops.Preseem.Insights
|
alias Towerops.Preseem.Insights
|
||||||
alias Towerops.Recommendations.Rules.CpeRealign
|
alias Towerops.Recommendations.Rules.CpeRealign
|
||||||
|
alias Towerops.Recommendations.Rules.DeviceOverheating
|
||||||
alias Towerops.Recommendations.Rules.FrequencyChange
|
alias Towerops.Recommendations.Rules.FrequencyChange
|
||||||
alias Towerops.Recommendations.Rules.OwnFleetChannelConflict
|
alias Towerops.Recommendations.Rules.OwnFleetChannelConflict
|
||||||
alias Towerops.Recommendations.Rules.SectorOverload
|
alias Towerops.Recommendations.Rules.SectorOverload
|
||||||
|
|
@ -25,7 +26,7 @@ defmodule Towerops.Workers.RecommendationsRunWorker do
|
||||||
|
|
||||||
require Logger
|
require Logger
|
||||||
|
|
||||||
@rules [FrequencyChange, OwnFleetChannelConflict, SectorOverload, CpeRealign]
|
@rules [FrequencyChange, OwnFleetChannelConflict, SectorOverload, CpeRealign, DeviceOverheating]
|
||||||
|
|
||||||
@impl Oban.Worker
|
@impl Oban.Worker
|
||||||
def perform(%Oban.Job{}) do
|
def perform(%Oban.Job{}) do
|
||||||
|
|
|
||||||
|
|
@ -480,6 +480,41 @@
|
||||||
</div>
|
</div>
|
||||||
<% end %>
|
<% end %>
|
||||||
|
|
||||||
|
<%= if insight.type == "device_overheating" do %>
|
||||||
|
<div class="mt-3 rounded-md border border-red-200 bg-red-50 p-3 dark:border-red-800 dark:bg-red-900/20">
|
||||||
|
<div class="flex flex-wrap items-center gap-4 text-sm">
|
||||||
|
<div class="flex flex-col">
|
||||||
|
<span class="text-xs text-red-700 dark:text-red-300">
|
||||||
|
{t("Hottest sensor")}
|
||||||
|
</span>
|
||||||
|
<span class="font-mono text-base font-semibold text-red-900 dark:text-red-100">
|
||||||
|
{insight.metadata["temperature_c"]} °C
|
||||||
|
</span>
|
||||||
|
<%= if insight.metadata["sensor_descr"] do %>
|
||||||
|
<span class="text-xs text-red-700 dark:text-red-300">
|
||||||
|
{insight.metadata["sensor_descr"]}
|
||||||
|
</span>
|
||||||
|
<% end %>
|
||||||
|
</div>
|
||||||
|
<%= if insight.metadata["qoe_score"] do %>
|
||||||
|
<div class="flex flex-col">
|
||||||
|
<span class="text-xs text-red-700 dark:text-red-300">
|
||||||
|
{t("Linked QoE")}
|
||||||
|
</span>
|
||||||
|
<span class="font-mono text-base font-semibold text-red-900 dark:text-red-100">
|
||||||
|
{Float.round(insight.metadata["qoe_score"] / 1.0, 1)}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
<% end %>
|
||||||
|
<%= if is_number(insight.metadata["temperature_c"]) and insight.metadata["temperature_c"] >= 75 do %>
|
||||||
|
<span class="ml-auto rounded-md bg-red-200 px-2 py-1 text-xs font-medium text-red-900 dark:bg-red-800 dark:text-red-100">
|
||||||
|
{t("Above manufacturer spec")}
|
||||||
|
</span>
|
||||||
|
<% end %>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<% end %>
|
||||||
|
|
||||||
<%= if insight.type == "own_fleet_channel_conflict" do %>
|
<%= if insight.type == "own_fleet_channel_conflict" do %>
|
||||||
<div class="mt-3 rounded-md border border-amber-200 bg-amber-50 p-3 dark:border-amber-800 dark:bg-amber-900/20">
|
<div class="mt-3 rounded-md border border-amber-200 bg-amber-50 p-3 dark:border-amber-800 dark:bg-amber-900/20">
|
||||||
<div class="flex items-center gap-3 text-sm">
|
<div class="flex items-center gap-3 text-sm">
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
2026-05-09 — Smarter recommendations + AI-generated insight summaries
|
2026-05-09 — Smarter recommendations + AI-generated insight summaries
|
||||||
|
* New recommendation: device overheating — flags any device with a temperature sensor at or above 65 °C, with the linked QoE score so you can tell whether a slowdown is thermal vs RF before sending a truck
|
||||||
* New recommendation: same-channel conflicts within your own network — when two or more of your APs at the same site sit on the same channel, the system flags the conflict with a clickable evidence table showing noise floor, RF score, average CPE SNR, and client count per AP, and escalates to critical when the RF environment is also contaminated
|
* New recommendation: same-channel conflicts within your own network — when two or more of your APs at the same site sit on the same channel, the system flags the conflict with a clickable evidence table showing noise floor, RF score, average CPE SNR, and client count per AP, and escalates to critical when the RF environment is also contaminated
|
||||||
* New recommendation: sector overload detection — APs with low free airtime and active subscribers get flagged with subscriber count, QoE score, and AP model so you can decide whether to split the sector or upgrade the radio
|
* New recommendation: sector overload detection — APs with low free airtime and active subscribers get flagged with subscriber count, QoE score, and AP model so you can decide whether to split the sector or upgrade the radio
|
||||||
* New recommendation: CPE re-aim candidates — wireless clients with both weak signal and low SNR (the alignment-issue signature) get individual cards with signal, SNR, TX/RX rate, and distance, so each truck-roll candidate is actionable on its own
|
* New recommendation: CPE re-aim candidates — wireless clients with both weak signal and low SNR (the alignment-issue signature) get individual cards with signal, SNR, TX/RX rate, and distance, so each truck-roll candidate is actionable on its own
|
||||||
|
|
|
||||||
117
test/towerops/recommendations/rules/device_overheating_test.exs
Normal file
117
test/towerops/recommendations/rules/device_overheating_test.exs
Normal file
|
|
@ -0,0 +1,117 @@
|
||||||
|
defmodule Towerops.Recommendations.Rules.DeviceOverheatingTest do
|
||||||
|
@moduledoc """
|
||||||
|
Tests the rule that flags devices with elevated enclosure / chassis
|
||||||
|
temperature, optionally correlating with Preseem QoE degradation.
|
||||||
|
"""
|
||||||
|
use Towerops.DataCase, async: true
|
||||||
|
|
||||||
|
import Towerops.AccountsFixtures
|
||||||
|
import Towerops.DevicesFixtures
|
||||||
|
import Towerops.OrganizationsFixtures
|
||||||
|
|
||||||
|
alias Towerops.Preseem.AccessPoint
|
||||||
|
alias Towerops.Recommendations.Rules.DeviceOverheating
|
||||||
|
alias Towerops.Snmp.Device, as: SnmpDevice
|
||||||
|
alias Towerops.Snmp.Sensor
|
||||||
|
|
||||||
|
setup do
|
||||||
|
user = user_fixture()
|
||||||
|
org = organization_fixture(user.id)
|
||||||
|
device = device_fixture(%{organization_id: org.id, name: "AP-Hot"})
|
||||||
|
|
||||||
|
{:ok, snmp} =
|
||||||
|
%SnmpDevice{}
|
||||||
|
|> SnmpDevice.changeset(%{device_id: device.id, sys_name: "AP-Hot"})
|
||||||
|
|> Repo.insert()
|
||||||
|
|
||||||
|
%{org: org, device: device, snmp: snmp}
|
||||||
|
end
|
||||||
|
|
||||||
|
defp insert_temp(snmp, value, opts \\ []) do
|
||||||
|
%Sensor{}
|
||||||
|
|> Sensor.changeset(%{
|
||||||
|
snmp_device_id: snmp.id,
|
||||||
|
sensor_type: Keyword.get(opts, :type, "temperature"),
|
||||||
|
sensor_index: "t_#{System.unique_integer([:positive])}",
|
||||||
|
sensor_oid: "1.2.3.#{System.unique_integer([:positive])}",
|
||||||
|
sensor_unit: "C",
|
||||||
|
sensor_divisor: 1,
|
||||||
|
sensor_descr: Keyword.get(opts, :descr, "Enclosure"),
|
||||||
|
last_value: value
|
||||||
|
})
|
||||||
|
|> Repo.insert!()
|
||||||
|
end
|
||||||
|
|
||||||
|
test "no insight when no temperature sensors", %{org: org} do
|
||||||
|
assert DeviceOverheating.evaluate(org.id) == []
|
||||||
|
end
|
||||||
|
|
||||||
|
test "no insight when temperature is healthy", %{org: org, snmp: snmp} do
|
||||||
|
insert_temp(snmp, 45.0)
|
||||||
|
assert DeviceOverheating.evaluate(org.id) == []
|
||||||
|
end
|
||||||
|
|
||||||
|
test "warning insight at 65–75°C", %{org: org, device: device, snmp: snmp} do
|
||||||
|
insert_temp(snmp, 68.0)
|
||||||
|
|
||||||
|
assert [insight] = DeviceOverheating.evaluate(org.id)
|
||||||
|
assert insight.type == "device_overheating"
|
||||||
|
assert insight.urgency == "warning"
|
||||||
|
assert insight.device_id == device.id
|
||||||
|
assert insight.metadata["temperature_c"] == 68.0
|
||||||
|
assert insight.metadata["sensor_descr"] == "Enclosure"
|
||||||
|
assert insight.title =~ "AP-Hot"
|
||||||
|
assert insight.title =~ "68"
|
||||||
|
end
|
||||||
|
|
||||||
|
test "critical insight at 75°C+", %{org: org, snmp: snmp} do
|
||||||
|
insert_temp(snmp, 78.0)
|
||||||
|
|
||||||
|
assert [insight] = DeviceOverheating.evaluate(org.id)
|
||||||
|
assert insight.urgency == "critical"
|
||||||
|
end
|
||||||
|
|
||||||
|
test "uses the hottest sensor when multiple are present", %{org: org, snmp: snmp} do
|
||||||
|
insert_temp(snmp, 60.0, descr: "CPU")
|
||||||
|
insert_temp(snmp, 70.0, descr: "Enclosure")
|
||||||
|
|
||||||
|
assert [insight] = DeviceOverheating.evaluate(org.id)
|
||||||
|
assert insight.metadata["temperature_c"] == 70.0
|
||||||
|
assert insight.metadata["sensor_descr"] == "Enclosure"
|
||||||
|
end
|
||||||
|
|
||||||
|
test "correlates with Preseem QoE degradation when AP is linked",
|
||||||
|
%{org: org, device: device, snmp: snmp} do
|
||||||
|
insert_temp(snmp, 72.0)
|
||||||
|
|
||||||
|
{:ok, _ap} =
|
||||||
|
%AccessPoint{}
|
||||||
|
|> AccessPoint.changeset(%{
|
||||||
|
organization_id: org.id,
|
||||||
|
preseem_id: "ap-thermal-#{System.unique_integer([:positive])}",
|
||||||
|
name: "Linked",
|
||||||
|
device_id: device.id,
|
||||||
|
match_confidence: "auto_ip",
|
||||||
|
qoe_score: 45.0
|
||||||
|
})
|
||||||
|
|> Repo.insert()
|
||||||
|
|
||||||
|
assert [insight] = DeviceOverheating.evaluate(org.id)
|
||||||
|
assert insight.metadata["qoe_score"] == 45.0
|
||||||
|
# Description should hint at thermal throttling correlation
|
||||||
|
assert insight.description =~ "QoE"
|
||||||
|
end
|
||||||
|
|
||||||
|
test "scopes to organization", %{org: org, snmp: snmp} do
|
||||||
|
insert_temp(snmp, 78.0)
|
||||||
|
|
||||||
|
other_user = user_fixture()
|
||||||
|
other_org = organization_fixture(other_user.id)
|
||||||
|
assert DeviceOverheating.evaluate(other_org.id) == []
|
||||||
|
end
|
||||||
|
|
||||||
|
test "ignores non-temperature sensors", %{org: org, snmp: snmp} do
|
||||||
|
insert_temp(snmp, 78.0, type: "voltage")
|
||||||
|
assert DeviceOverheating.evaluate(org.id) == []
|
||||||
|
end
|
||||||
|
end
|
||||||
Loading…
Add table
Reference in a new issue