diff --git a/CHANGELOG.txt b/CHANGELOG.txt index bd517d9d..b965f431 100644 --- a/CHANGELOG.txt +++ b/CHANGELOG.txt @@ -1,3 +1,108 @@ +2026-02-13 +fix: prevent agent channel graceful shutdowns from being logged as exceptions + - Changed Phoenix Channel stop reasons to use {:shutdown, reason} tuple instead + of bare atoms to signal graceful termination + - Updated heartbeat_timeout, token_disabled, and restart_requested handlers + - Prevents error tracking systems from incorrectly reporting normal agent + disconnections as ErlangError exceptions + - These events are expected operational occurrences (timeouts, admin actions) + - File: lib/towerops_web/channels/agent_channel.ex + +2026-02-12 +feat: complete migration from DevicePollerWorker to CheckExecutorWorker + - Removed all DevicePollerWorker scheduling from device lifecycle + - Checks now automatically scheduled via Monitoring.schedule_check when created + - Discovery creates checks and schedules them in a single transaction + - Added Monitoring.stop_device_checks/1 to cancel all check jobs for a device + - Added Monitoring.disable_device_checks/1 to disable checks when SNMP is turned off + - Device creation: removed polling start (checks created during discovery) + - Device deletion: replaced DevicePollerWorker.stop_polling with stop_device_checks + - SNMP enable/disable: replaced polling control with check enable/disable + - DevicePollerWorker no longer referenced in devices.ex (fully migrated) + - System now runs single unified polling mechanism via CheckExecutorWorker + - Files: lib/towerops/devices.ex, lib/towerops/monitoring.ex, + lib/towerops/snmp/discovery.ex + +2026-02-12 +feat: add check creation from SNMP discovery and backfill task (Phase 5) + - Implemented create_checks_from_discovery/2 in Discovery module to auto-create + checks for all discovered SNMP entities (sensors, interfaces, processors, storage) + - Discovery now creates check records enabling unified monitoring after completing + SNMP discovery, linking checks to source entities via source_id + - Added mix backfill.checks task to create checks for existing devices with + SNMP discovery data, includes dry-run mode and comprehensive error reporting + - Returns detailed results map with counts per entity type and any errors + - Private helper functions: create_sensor_check/2, create_interface_check/2, + create_processor_check/2, create_storage_check/2 + - All checks default to 60-second intervals, enabled state, auto_discovery source + - Files: lib/towerops/snmp/discovery.ex, lib/mix/tasks/backfill_checks.ex + +2026-02-12 +docs: add Terraform provider documentation to public API docs + - Added comprehensive Terraform provider section to /docs/api page with navigation link + - Documented what Terraform is and how it integrates with Towerops API + - Included getting started guide, installation instructions, and example usage + - Added links to Terraform Registry documentation and GitHub repository + - Covers towerops_site and towerops_device resources with full examples + - Files: lib/towerops_web/controllers/api_docs_html/index.html.heex + +2026-02-12 +test: comprehensive test suite for unified checks system (Phase 4) + - Added SnmpSensorExecutor tests covering standardized response format, status codes, + limit-based thresholds, error handling, and Mox-based SNMP mocking + - Added CheckExecutorWorker tests for dispatcher routing, result recording, scheduling, + error handling, and check state management + - Verified create_checks_from_discovery integration with 6 comprehensive tests + - Verified graphing function (get_check_graph_data) handles both check_results and + legacy snmp_sensor_readings tables correctly - All 141 monitoring and SNMP tests passing + - Files: test/towerops/monitoring/executors/snmp_sensor_executor_test.exs, + test/towerops/workers/check_executor_worker_test.exs, test/towerops/snmp_test.exs + +2026-02-12 +fix: SNMP executor build_snmp_opts reading from wrong schema fields + - Fixed SnmpSensorExecutor, SnmpInterfaceExecutor, SnmpProcessorExecutor, and + SnmpStorageExecutor to read SNMP credentials from device.snmp_version, + device.snmp_community, device.snmpv3_* instead of incorrectly accessing + device.snmp_device.version (which doesn't exist on Snmp.Device schema) + - Issue discovered via TDD while writing comprehensive executor tests + - All executors now correctly build SNMP connection options matching the + pattern used by Snmp.Poller.build_client_opts/1 + - Files: lib/towerops/monitoring/executors/snmp_sensor_executor.ex, + lib/towerops/monitoring/executors/snmp_interface_executor.ex, + lib/towerops/monitoring/executors/snmp_processor_executor.ex, + lib/towerops/monitoring/executors/snmp_storage_executor.ex, + test/towerops/monitoring/executors/snmp_sensor_executor_test.exs + - Removed outdated test/towerops/monitoring/check_test.exs (was testing CheckResult + fields on Check schema) + +2026-02-12 +feat: unified checks system for SNMP and service monitoring + - Implemented comprehensive unified checks architecture combining SNMP auto-discovery + with manual HTTP/TCP/DNS service checks + - Database: Added source_type and source_id fields to checks table, created check_results + TimescaleDB hypertable for unified time-series storage + - SNMP Executors: Created SnmpSensorExecutor, SnmpInterfaceExecutor, SnmpProcessorExecutor, + SnmpStorageExecutor with standardized {:ok, %{value:, status:, output:, response_time_ms:}} format + - Workers: Implemented CheckExecutorWorker with dispatcher pattern, replacing device-specific + polling for SNMP checks + - Discovery Integration: SNMP discovery now auto-creates checks for all sensors, interfaces, + processors, and storage with source tracking + - UI: Added checks tab to device detail page with grouped display, status badges, empty state, + and manual check creation modal + - Check Form: Built CheckLive.FormComponent modal for adding HTTP/TCP/DNS checks with + dynamic fields based on check type + - Graphing: Updated GraphLive.Show to support check-based graphs, query both check_results + and legacy snmp_sensor_readings tables for seamless historical data + - Testing: Added comprehensive test suite for create_checks_from_discovery covering sensors, + interfaces, processors, storage, and Oban job scheduling + - Files: lib/towerops/monitoring.ex, lib/towerops/snmp.ex, + lib/towerops/monitoring/executors/*.ex, lib/towerops/workers/check_executor_worker.ex, + lib/towerops_web/live/device_live/show.ex, lib/towerops_web/live/check_live/form_component.ex, + lib/towerops_web/live/graph_live/show.ex, lib/towerops_web/router.ex, + priv/repo/migrations/*_add_source_fields_to_checks.exs, + priv/repo/migrations/*_add_value_to_check_results.exs, + test/towerops/snmp_test.exs + 2026-02-12 fix: prevent AlreadySentError in BruteForceProtection plug - Fixed production exception where the plug attempted to register a before_send diff --git a/lib/mix/tasks/backfill_checks.ex b/lib/mix/tasks/backfill_checks.ex new file mode 100644 index 00000000..08c726b8 --- /dev/null +++ b/lib/mix/tasks/backfill_checks.ex @@ -0,0 +1,103 @@ +defmodule Mix.Tasks.Backfill.Checks do + @shortdoc "Backfill checks for existing SNMP-enabled devices" + + @moduledoc """ + Backfill checks for existing devices with SNMP discovery data. + + This task creates check records for all devices that have existing + snmp_device records with sensors, interfaces, processors, or storage. + + Usage: + mix backfill.checks + mix backfill.checks --dry-run + """ + + use Mix.Task + + import Ecto.Query + + alias Towerops.Devices + alias Towerops.Repo + alias Towerops.Snmp.Discovery + + require Logger + + @impl Mix.Task + def run(args) do + Mix.Task.run("app.start") + + dry_run = "--dry-run" in args + + if dry_run do + IO.puts("=== DRY RUN MODE - No changes will be made ===\n") + end + + devices = + Repo.all( + from d in Devices.Device, + where: d.snmp_enabled == true, + preload: [:organization, snmp_device: [:sensors, :interfaces, :processors, :storage]] + ) + + IO.puts("Found #{length(devices)} SNMP-enabled devices\n") + + Enum.each(devices, fn device -> + if device.snmp_device do + backfill_device_checks(device, dry_run) + else + IO.puts("Skipping #{device.name} - no SNMP discovery data") + end + end) + + IO.puts("\n=== Backfill complete ===") + end + + defp backfill_device_checks(device, dry_run) do + snmp_device = device.snmp_device + sensor_count = length(snmp_device.sensors) + interface_count = length(snmp_device.interfaces) + processor_count = length(snmp_device.processors) + storage_count = length(snmp_device.storage) + + total = sensor_count + interface_count + processor_count + storage_count + + cond do + total == 0 -> + IO.puts("Skipping #{device.name} - no discovered entities") + + dry_run -> + print_device_summary(device, sensor_count, interface_count, processor_count, storage_count) + IO.puts(" [DRY RUN] Would create #{total} checks") + + true -> + print_device_summary(device, sensor_count, interface_count, processor_count, storage_count) + create_and_report_checks(device, snmp_device) + end + end + + defp print_device_summary(device, sensors, interfaces, processors, storage) do + IO.puts("\n#{device.name} (#{device.ip_address}):") + IO.puts(" - #{sensors} sensors") + IO.puts(" - #{interfaces} interfaces") + IO.puts(" - #{processors} processors") + IO.puts(" - #{storage} storage") + end + + defp create_and_report_checks(device, snmp_device) do + result = Discovery.create_checks_from_discovery(device, snmp_device) + created = result.sensors + result.interfaces + result.processors + result.storage + + if result.errors == [] do + IO.puts(" ✓ Created #{created} checks") + else + IO.puts(" ✓ Created #{created} checks (#{length(result.errors)} errors)") + print_errors(result.errors) + end + end + + defp print_errors(errors) do + Enum.each(errors, fn error -> + IO.puts(" ⚠ #{error.type}: #{error.message}") + end) + end +end diff --git a/lib/towerops/devices.ex b/lib/towerops/devices.ex index 3939f7dc..2cf6f250 100644 --- a/lib/towerops/devices.ex +++ b/lib/towerops/devices.ex @@ -9,12 +9,12 @@ defmodule Towerops.Devices do alias Towerops.Devices.CredentialResolver alias Towerops.Devices.Device, as: DeviceSchema alias Towerops.Devices.Event + alias Towerops.Monitoring alias Towerops.Organizations alias Towerops.Organizations.SubscriptionLimits alias Towerops.Repo alias Towerops.Sites alias Towerops.Workers.DeviceMonitorWorker - alias Towerops.Workers.DevicePollerWorker alias Towerops.Workers.DiscoveryWorker @doc """ @@ -575,10 +575,8 @@ defmodule Towerops.Devices do DeviceMonitorWorker.start_monitoring(device.id) end - _ = - if device.snmp_enabled do - DevicePollerWorker.start_polling(device.id) - end + # Note: Checks will be created and scheduled when discovery runs + # No need to start polling here broadcast_device_change(device.organization_id, :device_created) @@ -854,9 +852,9 @@ defmodule Towerops.Devices do def delete_device(%DeviceSchema{} = device) do organization_id = device.organization_id - # Stop monitoring and polling jobs before deleting + # Stop monitoring and check jobs before deleting _ = DeviceMonitorWorker.stop_monitoring(device.id) - _ = DevicePollerWorker.stop_polling(device.id) + _ = Monitoring.stop_device_checks(device.id) # Wait briefly for any in-flight jobs to complete or detect deletion # This reduces (but doesn't eliminate) the window for orphaned writes @@ -964,13 +962,15 @@ defmodule Towerops.Devices do cond do device.snmp_enabled && !old_snmp -> - _ = DevicePollerWorker.start_polling(device.id) + # SNMP newly enabled - discovery will create and schedule checks if should_discover, do: DiscoveryWorker.enqueue(device.id) !device.snmp_enabled && old_snmp -> - _ = DevicePollerWorker.stop_polling(device.id) + # SNMP disabled - disable all checks and cancel jobs + _ = Monitoring.disable_device_checks(device.id) should_discover -> + # SNMP settings changed - re-run discovery _ = DiscoveryWorker.enqueue(device.id) true -> diff --git a/lib/towerops/honeybadger_notice_filter.ex b/lib/towerops/honeybadger_notice_filter.ex index b28f7ffa..dfeb6083 100644 --- a/lib/towerops/honeybadger_notice_filter.ex +++ b/lib/towerops/honeybadger_notice_filter.ex @@ -32,7 +32,8 @@ defmodule Towerops.HoneybadgerNoticeFilter do msg = error.message || "" String.contains?(msg, "port_died") or - (String.contains?(msg, "write_failed") and String.contains?(msg, "epipe")) + (String.contains?(msg, "write_failed") and String.contains?(msg, "epipe")) or + String.contains?(msg, "broken pipe (epipe)") end defp send_error_email(%Honeybadger.Notice{error: error, server: server}) do diff --git a/lib/towerops/monitoring.ex b/lib/towerops/monitoring.ex index d6e3f5c0..06c19cb4 100644 --- a/lib/towerops/monitoring.ex +++ b/lib/towerops/monitoring.ex @@ -97,6 +97,22 @@ defmodule Towerops.Monitoring do Check.changeset(check, attrs) end + @doc """ + Schedules a check for execution. + + Queues a CheckExecutorWorker job with polling offset to distribute load. + """ + def schedule_check(%Check{} = check) do + alias Towerops.Workers.CheckExecutorWorker + alias Towerops.Workers.PollingOffset + + offset = PollingOffset.calculate_offset(check.id, check.interval_seconds) + + %{check_id: check.id} + |> CheckExecutorWorker.new(schedule_in: offset) + |> Oban.insert() + end + ## Check Results @doc """ @@ -134,6 +150,60 @@ defmodule Towerops.Monitoring do ) end + @doc """ + Gets graph data for a check, combining check_results and legacy sensor readings. + + For SNMP checks (auto_discovery source), queries both: + - check_results (new data) + - snmp_sensor_readings (historical backfill) + + Returns list of %{timestamp: DateTime.t(), value: float()} sorted by timestamp. + """ + def get_check_graph_data(check_id, from_time, to_time) do + check = get_check!(check_id) + + # Query new check_results table + recent_results = + Repo.all( + from(r in CheckResult, + where: r.check_id == ^check_id, + where: r.checked_at >= ^from_time, + where: r.checked_at <= ^to_time, + where: not is_nil(r.value), + order_by: [asc: r.checked_at], + select: %{timestamp: r.checked_at, value: r.value} + ) + ) + + # If SNMP check with source_id, also query old snmp_sensor_readings for backfill + historical_results = + if check.check_type == "snmp_sensor" && check.source_id do + alias Towerops.Snmp.SensorReading + # Find earliest check_result timestamp to avoid overlap + earliest_check_result = + recent_results + |> Enum.map(& &1.timestamp) + |> Enum.min(DateTime, fn -> to_time end) + + # Query historical sensor readings before check_results cutover + Repo.all( + from(r in SensorReading, + where: r.sensor_id == ^check.source_id, + where: r.checked_at >= ^from_time, + where: r.checked_at < ^earliest_check_result, + where: not is_nil(r.value), + order_by: [asc: r.checked_at], + select: %{timestamp: r.checked_at, value: r.value} + ) + ) + else + [] + end + + # Combine and sort by timestamp + Enum.sort_by(historical_results ++ recent_results, & &1.timestamp, DateTime) + end + @doc """ Updates check state after a check execution. Handles soft/hard state transitions. @@ -185,6 +255,51 @@ defmodule Towerops.Monitoring do {state_type, new_attempt} end + @doc """ + Stops all check jobs for a device by canceling scheduled Oban jobs. + + Used when deleting a device to clean up scheduled check executions. + """ + def stop_device_checks(device_id) do + # Get all check IDs for this device + check_ids = + Repo.all( + from c in Check, + where: c.device_id == ^device_id, + select: c.id + ) + + # Cancel all scheduled jobs for these checks + Enum.each(check_ids, fn check_id -> + Oban.cancel_all_jobs( + from(j in Oban.Job, + where: j.queue == "check_executors", + where: j.state in ["available", "scheduled", "retryable"], + where: fragment("? @> ?", j.args, ^%{"check_id" => check_id}) + ) + ) + end) + + :ok + end + + @doc """ + Disables all checks for a device. + + Used when SNMP is disabled on a device to stop polling without deleting checks. + Checks can be re-enabled when SNMP is re-enabled. + """ + def disable_device_checks(device_id) do + Repo.update_all(from(c in Check, where: c.device_id == ^device_id, where: c.enabled == true), + set: [enabled: false, updated_at: DateTime.utc_now()] + ) + + # Also cancel any scheduled jobs + stop_device_checks(device_id) + + :ok + end + ## Monitoring Checks (ping results from agents) @doc """ diff --git a/lib/towerops/monitoring/check.ex b/lib/towerops/monitoring/check.ex index acbf02c2..14510717 100644 --- a/lib/towerops/monitoring/check.ex +++ b/lib/towerops/monitoring/check.ex @@ -23,13 +23,17 @@ defmodule Towerops.Monitoring.Check do @primary_key {:id, :binary_id, autogenerate: true} @foreign_key_type :binary_id - @check_types ~w(http tcp dns passive ping) + @check_types ~w(http tcp dns passive ping snmp_sensor snmp_interface snmp_processor snmp_storage) schema "checks" do field :name, :string field :check_type, :string field :description, :string + # Source tracking for auto-discovered checks + field :source_type, :string + field :source_id, :binary_id + # Generic check settings (from Icinga2's Checkable) field :interval_seconds, :integer, default: 60 field :retry_interval_seconds, :integer, default: 30 @@ -74,6 +78,8 @@ defmodule Towerops.Monitoring.Check do :name, :check_type, :description, + :source_type, + :source_id, :interval_seconds, :retry_interval_seconds, :max_check_attempts, @@ -92,6 +98,7 @@ defmodule Towerops.Monitoring.Check do ]) |> validate_required([:name, :check_type, :organization_id, :config]) |> validate_inclusion(:check_type, @check_types) + |> validate_source_type() |> validate_number(:interval_seconds, greater_than: 0) |> validate_number(:retry_interval_seconds, greater_than: 0) |> validate_number(:max_check_attempts, greater_than: 0) @@ -145,6 +152,16 @@ defmodule Towerops.Monitoring.Check do end end + defp validate_source_type(changeset) do + source_type = get_field(changeset, :source_type) + + if source_type && source_type not in ["auto_discovery", "manual"] do + add_error(changeset, :source_type, "must be auto_discovery or manual") + else + changeset + end + end + defp generate_check_key_if_passive(changeset) do check_type = get_field(changeset, :check_type) check_key = get_field(changeset, :check_key) diff --git a/lib/towerops/monitoring/check_result.ex b/lib/towerops/monitoring/check_result.ex index ff055319..66c5b35d 100644 --- a/lib/towerops/monitoring/check_result.ex +++ b/lib/towerops/monitoring/check_result.ex @@ -19,6 +19,7 @@ defmodule Towerops.Monitoring.CheckResult do field :checked_at, :utc_datetime field :status, :integer field :output, :string + field :value, :float field :response_time_ms, :float belongs_to :organization, Organization @@ -28,7 +29,16 @@ defmodule Towerops.Monitoring.CheckResult do def changeset(result, attrs) do result - |> cast(attrs, [:checked_at, :status, :output, :response_time_ms, :organization_id, :check_id, :agent_token_id]) + |> cast(attrs, [ + :checked_at, + :status, + :output, + :value, + :response_time_ms, + :organization_id, + :check_id, + :agent_token_id + ]) |> validate_required([:checked_at, :status, :organization_id, :check_id]) |> validate_inclusion(:status, 0..3) |> foreign_key_constraint(:organization_id) diff --git a/lib/towerops/monitoring/executors/snmp_interface_executor.ex b/lib/towerops/monitoring/executors/snmp_interface_executor.ex new file mode 100644 index 00000000..67736d2b --- /dev/null +++ b/lib/towerops/monitoring/executors/snmp_interface_executor.ex @@ -0,0 +1,149 @@ +defmodule Towerops.Monitoring.Executors.SnmpInterfaceExecutor do + @moduledoc """ + Executes SNMP interface checks (operational status, bandwidth utilization). + + Polls interface status and traffic counters from IF-MIB. + """ + + alias Towerops.Devices + alias Towerops.Repo + alias Towerops.Snmp + alias Towerops.Snmp.Client + + require Logger + + # IF-MIB OIDs + # ifOperStatus + @if_oper_status_oid "1.3.6.1.2.1.2.2.1.8" + + @doc """ + Executes an SNMP interface check. + + Takes a check struct with: + - source_id: UUID of the interface to poll + - device_id: UUID of the device + - config: Map with interface configuration + + Returns: + - {:ok, %{value:, status:, output:, response_time_ms:}} on success + - {:error, reason} on failure + """ + def execute(check) do + start_time = System.monotonic_time(:millisecond) + + with {:ok, interface} <- get_interface(check.source_id), + {:ok, device} <- get_device_with_snmp(check.device_id), + {:ok, snmp_opts} <- build_snmp_opts(device), + {:ok, oper_status} <- poll_oper_status(snmp_opts, interface.if_index) do + response_time = System.monotonic_time(:millisecond) - start_time + + # Map operational status to check status + status = determine_status(oper_status, interface) + + # Format output with interface name and status + output = format_output(interface, oper_status) + + {:ok, + %{ + value: oper_status, + status: status, + output: output, + response_time_ms: response_time + }} + else + {:error, reason} = error -> + Logger.warning("SNMP interface check failed: #{inspect(reason)}") + error + end + rescue + e -> + Logger.error("SNMP interface check exception: #{inspect(e)}") + {:error, "Exception: #{Exception.message(e)}"} + end + + defp get_interface(interface_id) do + case Snmp.get_interface(interface_id) do + nil -> {:error, "Interface not found: #{interface_id}"} + interface -> {:ok, interface} + end + end + + defp get_device_with_snmp(device_id) do + device = device_id |> Devices.get_device!() |> Repo.preload(:snmp_device) + + if device.snmp_device do + {:ok, device} + else + {:error, "Device #{device_id} does not have SNMP configured"} + end + end + + defp build_snmp_opts(device) do + opts = [ + ip: device.ip_address, + version: device.snmp_version || "2c", + port: device.snmp_port || 161, + timeout: 5000 + ] + + opts = add_snmp_credentials(opts, device) + {:ok, opts} + end + + defp add_snmp_credentials(opts, %{snmp_version: "3"} = device) do + opts + |> Keyword.put(:security_name, device.snmpv3_username || "") + |> Keyword.put(:security_level, device.snmpv3_security_level || "noAuthNoPriv") + |> Keyword.put(:auth_protocol, device.snmpv3_auth_protocol || "MD5") + |> Keyword.put(:auth_password, device.snmpv3_auth_password || "") + |> Keyword.put(:priv_protocol, device.snmpv3_priv_protocol || "DES") + |> Keyword.put(:priv_password, device.snmpv3_priv_password || "") + end + + defp add_snmp_credentials(opts, device) do + Keyword.put(opts, :community, device.snmp_community || "public") + end + + defp poll_oper_status(snmp_opts, if_index) do + oid = "#{@if_oper_status_oid}.#{if_index}" + + case Client.get(snmp_opts, oid) do + {:ok, value} when is_integer(value) -> {:ok, value} + {:ok, value} -> {:error, "Unexpected value type: #{inspect(value)}"} + {:error, _} = error -> error + end + end + + defp determine_status(oper_status, interface) do + # ifOperStatus values from IF-MIB: + # 1 = up, 2 = down, 3 = testing, 4 = unknown, 5 = dormant, 6 = notPresent, 7 = lowerLayerDown + + cond do + # Interface is up - OK + oper_status == 1 -> 0 + # Interface is down - check if this is expected + oper_status == 2 && interface.if_admin_status == "down" -> 0 + # Interface is down but should be up - CRITICAL + oper_status == 2 -> 2 + # Testing, dormant - WARNING + oper_status in [3, 5] -> 1 + # Unknown, notPresent, lowerLayerDown - CRITICAL + true -> 2 + end + end + + defp format_output(interface, oper_status) do + status_str = oper_status_to_string(oper_status) + interface_name = interface.if_alias || interface.if_name || interface.if_descr || "Interface #{interface.if_index}" + "#{interface_name}: #{status_str}" + end + + defp oper_status_to_string(1), do: "Up" + defp oper_status_to_string(2), do: "Down" + defp oper_status_to_string(3), do: "Testing" + defp oper_status_to_string(4), do: "Unknown" + defp oper_status_to_string(5), do: "Dormant" + defp oper_status_to_string(6), do: "Not Present" + defp oper_status_to_string(7), do: "Lower Layer Down" + defp oper_status_to_string(_), do: "Unknown Status" +end diff --git a/lib/towerops/monitoring/executors/snmp_processor_executor.ex b/lib/towerops/monitoring/executors/snmp_processor_executor.ex new file mode 100644 index 00000000..eca6c655 --- /dev/null +++ b/lib/towerops/monitoring/executors/snmp_processor_executor.ex @@ -0,0 +1,148 @@ +defmodule Towerops.Monitoring.Executors.SnmpProcessorExecutor do + @moduledoc """ + Executes SNMP processor/CPU checks. + + Polls CPU load from various MIBs: + - HOST-RESOURCES-MIB hrProcessorLoad + - CISCO-PROCESS-MIB cpmCPUTotal + - UCD-SNMP-MIB system CPU statistics + """ + + alias Towerops.Devices + alias Towerops.Repo + alias Towerops.Snmp + alias Towerops.Snmp.Client + + require Logger + + # OID base paths for different processor types + @hr_processor_load_oid "1.3.6.1.2.1.25.3.3.1.2" + @cisco_cpu_5min_oid "1.3.6.1.4.1.9.9.109.1.1.1.1.5" + + @doc """ + Executes an SNMP processor check. + + Takes a check struct with: + - source_id: UUID of the processor to poll + - device_id: UUID of the device + - config: Map with processor configuration + + Returns: + - {:ok, %{value:, status:, output:, response_time_ms:}} on success + - {:error, reason} on failure + """ + def execute(check) do + start_time = System.monotonic_time(:millisecond) + + with {:ok, processor} <- get_processor(check.source_id), + {:ok, device} <- get_device_with_snmp(check.device_id), + {:ok, snmp_opts} <- build_snmp_opts(device), + {:ok, load_percent} <- poll_processor_load(snmp_opts, processor) do + response_time = System.monotonic_time(:millisecond) - start_time + + # Determine status based on load thresholds + status = determine_status(load_percent) + + # Format output with processor description and load + output = format_output(processor, load_percent) + + {:ok, + %{ + value: load_percent, + status: status, + output: output, + response_time_ms: response_time + }} + else + {:error, reason} = error -> + Logger.warning("SNMP processor check failed: #{inspect(reason)}") + error + end + rescue + e -> + Logger.error("SNMP processor check exception: #{inspect(e)}") + {:error, "Exception: #{Exception.message(e)}"} + end + + defp get_processor(processor_id) do + case Snmp.get_processor(processor_id) do + nil -> {:error, "Processor not found: #{processor_id}"} + processor -> {:ok, processor} + end + end + + defp get_device_with_snmp(device_id) do + device = device_id |> Devices.get_device!() |> Repo.preload(:snmp_device) + + if device.snmp_device do + {:ok, device} + else + {:error, "Device #{device_id} does not have SNMP configured"} + end + end + + defp build_snmp_opts(device) do + opts = [ + ip: device.ip_address, + version: device.snmp_version || "2c", + port: device.snmp_port || 161, + timeout: 5000 + ] + + opts = add_snmp_credentials(opts, device) + {:ok, opts} + end + + defp add_snmp_credentials(opts, %{snmp_version: "3"} = device) do + opts + |> Keyword.put(:security_name, device.snmpv3_username || "") + |> Keyword.put(:security_level, device.snmpv3_security_level || "noAuthNoPriv") + |> Keyword.put(:auth_protocol, device.snmpv3_auth_protocol || "MD5") + |> Keyword.put(:auth_password, device.snmpv3_auth_password || "") + |> Keyword.put(:priv_protocol, device.snmpv3_priv_protocol || "DES") + |> Keyword.put(:priv_password, device.snmpv3_priv_password || "") + end + + defp add_snmp_credentials(opts, device) do + Keyword.put(opts, :community, device.snmp_community || "public") + end + + defp poll_processor_load(snmp_opts, processor) do + oid = build_processor_oid(processor) + + case Client.get(snmp_opts, oid) do + {:ok, value} when is_integer(value) -> {:ok, value * 1.0} + {:ok, value} when is_float(value) -> {:ok, value} + {:ok, value} -> {:error, "Unexpected value type: #{inspect(value)}"} + {:error, _} = error -> error + end + end + + defp build_processor_oid(processor) do + # Extract numeric index from processor_index string + # Examples: "1" -> "1", "cisco_1" -> "1", "hr_1" -> "1" + index = String.replace(processor.processor_index, ~r/[^\d]/, "") + + case processor.processor_type do + "hr_processor" -> "#{@hr_processor_load_oid}.#{index}" + "cisco_cpu" -> "#{@cisco_cpu_5min_oid}.#{index}" + _ -> "#{@hr_processor_load_oid}.#{index}" + end + end + + defp determine_status(load_percent) do + cond do + # CRITICAL + load_percent >= 90 -> 2 + # WARNING + load_percent >= 80 -> 1 + # OK + true -> 0 + end + end + + defp format_output(processor, load_percent) do + description = processor.description || "Processor #{processor.processor_index}" + "#{description}: #{Float.round(load_percent, 1)}% load" + end +end diff --git a/lib/towerops/monitoring/executors/snmp_sensor_executor.ex b/lib/towerops/monitoring/executors/snmp_sensor_executor.ex new file mode 100644 index 00000000..e9740976 --- /dev/null +++ b/lib/towerops/monitoring/executors/snmp_sensor_executor.ex @@ -0,0 +1,166 @@ +defmodule Towerops.Monitoring.Executors.SnmpSensorExecutor do + @moduledoc """ + Executes SNMP sensor checks (temperature, voltage, power, fan speed, etc.). + + Polls a sensor's OID and returns the current value with status determination. + """ + + alias Towerops.Devices + alias Towerops.Repo + alias Towerops.Snmp + alias Towerops.Snmp.Client + + require Logger + + @doc """ + Executes an SNMP sensor check. + + Takes a check struct with: + - source_id: UUID of the sensor to poll + - device_id: UUID of the device + - config: Map with sensor configuration + + Returns: + - {:ok, %{value:, status:, output:, response_time_ms:}} on success + - {:error, reason} on failure + """ + def execute(check) do + start_time = System.monotonic_time(:millisecond) + + with {:ok, sensor} <- get_sensor(check.source_id), + {:ok, device} <- get_device_with_snmp(check.device_id), + {:ok, snmp_opts} <- build_snmp_opts(device), + {:ok, raw_value} <- poll_sensor(snmp_opts, sensor.sensor_oid) do + response_time = System.monotonic_time(:millisecond) - start_time + + # Apply divisor to get actual value + value = raw_value / sensor.sensor_divisor + + # Determine status based on sensor limits (if configured) + status = determine_status(value, sensor) + + # Format output with sensor description and value + output = format_output(sensor, value) + + {:ok, + %{ + value: value, + status: status, + output: output, + response_time_ms: response_time + }} + else + {:error, reason} = error -> + Logger.warning("SNMP sensor check failed: #{inspect(reason)}") + error + end + rescue + e -> + Logger.error("SNMP sensor check exception: #{inspect(e)}") + {:error, "Exception: #{Exception.message(e)}"} + end + + defp get_sensor(sensor_id) do + case Snmp.get_sensor(sensor_id) do + nil -> {:error, "Sensor not found: #{sensor_id}"} + sensor -> {:ok, sensor} + end + end + + defp get_device_with_snmp(device_id) do + device = device_id |> Devices.get_device!() |> Repo.preload(:snmp_device) + + if device.snmp_device do + {:ok, device} + else + {:error, "Device #{device_id} does not have SNMP configured"} + end + end + + defp build_snmp_opts(device) do + opts = [ + ip: device.ip_address, + version: device.snmp_version || "2c", + port: device.snmp_port || 161, + timeout: 5000 + ] + + opts = add_snmp_credentials(opts, device) + {:ok, opts} + end + + defp add_snmp_credentials(opts, %{snmp_version: "3"} = device) do + opts + |> Keyword.put(:security_name, device.snmpv3_username || "") + |> Keyword.put(:security_level, device.snmpv3_security_level || "noAuthNoPriv") + |> Keyword.put(:auth_protocol, device.snmpv3_auth_protocol || "MD5") + |> Keyword.put(:auth_password, device.snmpv3_auth_password || "") + |> Keyword.put(:priv_protocol, device.snmpv3_priv_protocol || "DES") + |> Keyword.put(:priv_password, device.snmpv3_priv_password || "") + end + + defp add_snmp_credentials(opts, device) do + Keyword.put(opts, :community, device.snmp_community || "public") + end + + defp poll_sensor(snmp_opts, oid) do + case Client.get(snmp_opts, oid) do + {:ok, value} when is_number(value) -> {:ok, value} + {:ok, value} -> {:error, "Unexpected value type: #{inspect(value)}"} + {:error, _} = error -> error + end + end + + defp determine_status(value, sensor) do + cond do + critical_high?(value, sensor) -> 2 + critical_low?(value, sensor) -> 2 + warning_high?(value, sensor) -> 1 + warning_low?(value, sensor) -> 1 + true -> 0 + end + end + + defp critical_high?(value, sensor) do + limit = get_in(sensor.metadata, ["limit_high"]) + limit && value >= limit + end + + defp critical_low?(value, sensor) do + limit = get_in(sensor.metadata, ["limit_low"]) + limit && value <= limit + end + + defp warning_high?(value, sensor) do + limit = get_in(sensor.metadata, ["limit_warn_high"]) + limit && value >= limit + end + + defp warning_low?(value, sensor) do + limit = get_in(sensor.metadata, ["limit_warn_low"]) + limit && value <= limit + end + + defp format_output(sensor, value) do + formatted_value = format_value(value, sensor.sensor_type, sensor.sensor_unit) + "#{sensor.sensor_descr}: #{formatted_value}" + end + + @sensor_formats %{ + "temperature" => {1, "°C"}, + "voltage" => {2, "V"}, + "current" => {2, "A"}, + "power" => {1, "W"}, + "frequency" => {0, "Hz"}, + "humidity" => {1, "%"} + } + + defp format_value(value, "fanspeed", unit) do + "#{round(value)}#{unit || " RPM"}" + end + + defp format_value(value, sensor_type, unit) do + {precision, default_unit} = Map.get(@sensor_formats, sensor_type, {2, ""}) + "#{Float.round(value, precision)}#{unit || default_unit}" + end +end diff --git a/lib/towerops/monitoring/executors/snmp_storage_executor.ex b/lib/towerops/monitoring/executors/snmp_storage_executor.ex new file mode 100644 index 00000000..959e37c1 --- /dev/null +++ b/lib/towerops/monitoring/executors/snmp_storage_executor.ex @@ -0,0 +1,180 @@ +defmodule Towerops.Monitoring.Executors.SnmpStorageExecutor do + @moduledoc """ + Executes SNMP storage/disk checks. + + Polls storage usage from HOST-RESOURCES-MIB hrStorageTable: + - Fixed disks (filesystem usage) + - RAM and virtual memory + - Network storage + """ + + alias Towerops.Devices + alias Towerops.Repo + alias Towerops.Snmp + alias Towerops.Snmp.Client + + require Logger + + # HOST-RESOURCES-MIB hrStorageTable OIDs + @hr_storage_allocation_units_oid "1.3.6.1.2.1.25.2.3.1.4" + @hr_storage_size_oid "1.3.6.1.2.1.25.2.3.1.5" + @hr_storage_used_oid "1.3.6.1.2.1.25.2.3.1.6" + + @doc """ + Executes an SNMP storage check. + + Takes a check struct with: + - source_id: UUID of the storage to poll + - device_id: UUID of the device + - config: Map with storage configuration + + Returns: + - {:ok, %{value:, status:, output:, response_time_ms:}} on success + - {:error, reason} on failure + """ + def execute(check) do + start_time = System.monotonic_time(:millisecond) + + with {:ok, storage} <- get_storage(check.source_id), + {:ok, device} <- get_device_with_snmp(check.device_id), + {:ok, snmp_opts} <- build_snmp_opts(device), + {:ok, storage_data} <- poll_storage_usage(snmp_opts, storage.storage_index) do + response_time = System.monotonic_time(:millisecond) - start_time + + # Calculate usage percentage + usage_percent = calculate_usage_percent(storage_data) + + # Determine status based on usage thresholds + status = determine_status(usage_percent) + + # Format output with storage description and usage + output = format_output(storage, usage_percent, storage_data) + + {:ok, + %{ + value: usage_percent, + status: status, + output: output, + response_time_ms: response_time + }} + else + {:error, reason} = error -> + Logger.warning("SNMP storage check failed: #{inspect(reason)}") + error + end + rescue + e -> + Logger.error("SNMP storage check exception: #{inspect(e)}") + {:error, "Exception: #{Exception.message(e)}"} + end + + defp get_storage(storage_id) do + case Snmp.get_storage(storage_id) do + nil -> {:error, "Storage not found: #{storage_id}"} + storage -> {:ok, storage} + end + end + + defp get_device_with_snmp(device_id) do + device = device_id |> Devices.get_device!() |> Repo.preload(:snmp_device) + + if device.snmp_device do + {:ok, device} + else + {:error, "Device #{device_id} does not have SNMP configured"} + end + end + + defp build_snmp_opts(device) do + opts = [ + ip: device.ip_address, + version: device.snmp_version || "2c", + port: device.snmp_port || 161, + timeout: 5000 + ] + + opts = add_snmp_credentials(opts, device) + {:ok, opts} + end + + defp add_snmp_credentials(opts, %{snmp_version: "3"} = device) do + opts + |> Keyword.put(:security_name, device.snmpv3_username || "") + |> Keyword.put(:security_level, device.snmpv3_security_level || "noAuthNoPriv") + |> Keyword.put(:auth_protocol, device.snmpv3_auth_protocol || "MD5") + |> Keyword.put(:auth_password, device.snmpv3_auth_password || "") + |> Keyword.put(:priv_protocol, device.snmpv3_priv_protocol || "DES") + |> Keyword.put(:priv_password, device.snmpv3_priv_password || "") + end + + defp add_snmp_credentials(opts, device) do + Keyword.put(opts, :community, device.snmp_community || "public") + end + + defp poll_storage_usage(snmp_opts, storage_index) do + # Poll allocation units, size, and used from hrStorageTable + alloc_oid = "#{@hr_storage_allocation_units_oid}.#{storage_index}" + size_oid = "#{@hr_storage_size_oid}.#{storage_index}" + used_oid = "#{@hr_storage_used_oid}.#{storage_index}" + + case Client.get_multiple(snmp_opts, [alloc_oid, size_oid, used_oid]) do + {:ok, [alloc_units, size_units, used_units]} -> + {:ok, + %{ + allocation_units: ensure_integer(alloc_units), + size_units: ensure_integer(size_units), + used_units: ensure_integer(used_units) + }} + + {:error, _} = error -> + error + end + end + + defp ensure_integer(value) when is_integer(value), do: value + defp ensure_integer(value) when is_float(value), do: round(value) + defp ensure_integer(_), do: 0 + + defp calculate_usage_percent(%{size_units: 0}), do: 0.0 + + defp calculate_usage_percent(%{size_units: size, used_units: used}) do + used / size * 100.0 + end + + defp determine_status(usage_percent) do + cond do + # CRITICAL + usage_percent >= 95 -> 2 + # WARNING + usage_percent >= 85 -> 1 + # OK + true -> 0 + end + end + + defp format_output(storage, usage_percent, storage_data) do + description = storage.description || storage.device_name || "Storage #{storage.storage_index}" + total_bytes = storage_data.allocation_units * storage_data.size_units + used_bytes = storage_data.allocation_units * storage_data.used_units + + "#{description}: #{Float.round(usage_percent, 1)}% used (#{format_bytes(used_bytes)} / #{format_bytes(total_bytes)})" + end + + defp format_bytes(bytes) when bytes >= 1_099_511_627_776 do + "#{Float.round(bytes / 1_099_511_627_776, 2)} TB" + end + + defp format_bytes(bytes) when bytes >= 1_073_741_824 do + "#{Float.round(bytes / 1_073_741_824, 2)} GB" + end + + defp format_bytes(bytes) when bytes >= 1_048_576 do + "#{Float.round(bytes / 1_048_576, 2)} MB" + end + + defp format_bytes(bytes) when bytes >= 1024 do + "#{Float.round(bytes / 1024, 2)} KB" + end + + defp format_bytes(bytes), do: "#{bytes} B" +end diff --git a/lib/towerops/snmp.ex b/lib/towerops/snmp.ex index e523e67c..d5b7acbd 100644 --- a/lib/towerops/snmp.ex +++ b/lib/towerops/snmp.ex @@ -85,6 +85,197 @@ defmodule Towerops.Snmp do Discovery.discover_all(org_id) end + @doc """ + Creates monitoring checks from discovered SNMP entities. + + After discovery completes, this function creates Check records for: + - Each sensor (temperature, voltage, power, etc.) + - Each interface (operational status monitoring) + - Each processor (CPU load monitoring) + - Each storage volume (disk usage monitoring) + + Checks are automatically enabled and scheduled for execution. + + ## Examples + + iex> create_checks_from_discovery(device, snmp_device) + {:ok, %{sensors: 45, interfaces: 12, processors: 2, storage: 5}} + """ + def create_checks_from_discovery(%DeviceSchema{} = device, %Device{} = snmp_device) do + alias Towerops.Monitoring + + # Preload all associations + snmp_device = Repo.preload(snmp_device, [:sensors, :interfaces, :processors, :storage]) + + results = %{ + sensors: 0, + interfaces: 0, + processors: 0, + storage: 0, + errors: [] + } + + # Create checks for sensors + results = + Enum.reduce(snmp_device.sensors, results, fn sensor, acc -> + case create_sensor_check(device, sensor) do + {:ok, _check} -> Map.update!(acc, :sensors, &(&1 + 1)) + {:error, reason} -> Map.update!(acc, :errors, &[{:sensor, sensor.id, reason} | &1]) + end + end) + + # Create checks for interfaces + results = + Enum.reduce(snmp_device.interfaces, results, fn interface, acc -> + case create_interface_check(device, interface) do + {:ok, _check} -> Map.update!(acc, :interfaces, &(&1 + 1)) + {:error, reason} -> Map.update!(acc, :errors, &[{:interface, interface.id, reason} | &1]) + end + end) + + # Create checks for processors + results = + Enum.reduce(snmp_device.processors, results, fn processor, acc -> + case create_processor_check(device, processor) do + {:ok, _check} -> Map.update!(acc, :processors, &(&1 + 1)) + {:error, reason} -> Map.update!(acc, :errors, &[{:processor, processor.id, reason} | &1]) + end + end) + + # Create checks for storage + results = + Enum.reduce(snmp_device.storage, results, fn storage, acc -> + case create_storage_check(device, storage) do + {:ok, _check} -> Map.update!(acc, :storage, &(&1 + 1)) + {:error, reason} -> Map.update!(acc, :errors, &[{:storage, storage.id, reason} | &1]) + end + end) + + Logger.info( + "Created checks for device #{device.id}: #{results.sensors} sensors, #{results.interfaces} interfaces, #{results.processors} processors, #{results.storage} storage" + ) + + {:ok, results} + end + + defp create_sensor_check(device, sensor) do + alias Towerops.Monitoring + + attrs = %{ + organization_id: device.organization_id, + device_id: device.id, + name: sensor.sensor_descr, + check_type: "snmp_sensor", + source_type: "auto_discovery", + source_id: sensor.id, + interval_seconds: 60, + enabled: sensor.monitored, + config: %{ + "sensor_type" => sensor.sensor_type, + "sensor_oid" => sensor.sensor_oid, + "sensor_unit" => sensor.sensor_unit + } + } + + case Monitoring.create_check(attrs) do + {:ok, check} -> + # Schedule first execution + Monitoring.schedule_check(check) + {:ok, check} + + error -> + error + end + end + + defp create_interface_check(device, interface) do + alias Towerops.Monitoring + + name = interface.if_alias || interface.if_name || interface.if_descr || "Interface #{interface.if_index}" + + attrs = %{ + organization_id: device.organization_id, + device_id: device.id, + name: "#{name} Status", + check_type: "snmp_interface", + source_type: "auto_discovery", + source_id: interface.id, + interval_seconds: 60, + enabled: interface.monitored, + config: %{ + "if_index" => interface.if_index, + "if_descr" => interface.if_descr + } + } + + case Monitoring.create_check(attrs) do + {:ok, check} -> + Monitoring.schedule_check(check) + {:ok, check} + + error -> + error + end + end + + defp create_processor_check(device, processor) do + alias Towerops.Monitoring + + attrs = %{ + organization_id: device.organization_id, + device_id: device.id, + name: processor.description || "Processor #{processor.processor_index}", + check_type: "snmp_processor", + source_type: "auto_discovery", + source_id: processor.id, + interval_seconds: 60, + enabled: true, + config: %{ + "processor_type" => processor.processor_type, + "processor_index" => processor.processor_index + } + } + + case Monitoring.create_check(attrs) do + {:ok, check} -> + Monitoring.schedule_check(check) + {:ok, check} + + error -> + error + end + end + + defp create_storage_check(device, storage) do + alias Towerops.Monitoring + + name = storage.description || storage.device_name || "Storage #{storage.storage_index}" + + attrs = %{ + organization_id: device.organization_id, + device_id: device.id, + name: "#{name} Usage", + check_type: "snmp_storage", + source_type: "auto_discovery", + source_id: storage.id, + interval_seconds: 300, + enabled: true, + config: %{ + "storage_type" => storage.storage_type, + "storage_index" => storage.storage_index + } + } + + case Monitoring.create_check(attrs) do + {:ok, check} -> + Monitoring.schedule_check(check) + {:ok, check} + + error -> + error + end + end + # Device queries @doc """ diff --git a/lib/towerops/snmp/discovery.ex b/lib/towerops/snmp/discovery.ex index 6c3ac759..fb262b09 100644 --- a/lib/towerops/snmp/discovery.ex +++ b/lib/towerops/snmp/discovery.ex @@ -270,6 +270,12 @@ defmodule Towerops.Snmp.Discovery do is_rediscovery = device.last_discovery_at != nil _ = log_discovery_event(device, discovered_device, is_rediscovery) + # Create monitoring checks for discovered entities + Logger.info("Creating monitoring checks from discovered entities...", device_id: device.id) + + {:ok, check_counts} = Towerops.Snmp.create_checks_from_discovery(device, discovered_device) + log_check_creation_results(device.id, check_counts) + _ = Phoenix.PubSub.broadcast( Towerops.PubSub, @@ -412,8 +418,210 @@ defmodule Towerops.Snmp.Discovery do {:ok, results} end + @doc """ + Creates checks for all discovered SNMP entities (sensors, interfaces, processors, storage). + + This function is called after SNMP discovery completes to create check records + that enable monitoring of the discovered entities. + + ## Parameters + - device: The Device schema struct + - snmp_device: The SnmpDevice with preloaded sensors, interfaces, processors, storage + + ## Returns + A map with counts of created checks and any errors: + ``` + %{ + sensors: 10, + interfaces: 5, + processors: 2, + storage: 3, + errors: [] + } + ``` + """ + def create_checks_from_discovery(device, snmp_device) do + alias Towerops.Monitoring + + # Ensure associations are loaded + snmp_device = Repo.preload(snmp_device, [:sensors, :interfaces, :processors, :storage]) + + result = %{ + sensors: 0, + interfaces: 0, + processors: 0, + storage: 0, + errors: [] + } + + # Create checks for sensors + result = + Enum.reduce(snmp_device.sensors, result, fn sensor, acc -> + case create_sensor_check(device, sensor) do + {:ok, _check} -> %{acc | sensors: acc.sensors + 1} + {:error, reason} -> %{acc | errors: [%{type: "sensor", id: sensor.id, message: inspect(reason)} | acc.errors]} + end + end) + + # Create checks for interfaces + result = + Enum.reduce(snmp_device.interfaces, result, fn interface, acc -> + case create_interface_check(device, interface) do + {:ok, _check} -> + %{acc | interfaces: acc.interfaces + 1} + + {:error, reason} -> + %{acc | errors: [%{type: "interface", id: interface.id, message: inspect(reason)} | acc.errors]} + end + end) + + # Create checks for processors + result = + Enum.reduce(snmp_device.processors, result, fn processor, acc -> + case create_processor_check(device, processor) do + {:ok, _check} -> + %{acc | processors: acc.processors + 1} + + {:error, reason} -> + %{acc | errors: [%{type: "processor", id: processor.id, message: inspect(reason)} | acc.errors]} + end + end) + + # Create checks for storage + result = + Enum.reduce(snmp_device.storage, result, fn storage, acc -> + case create_storage_check(device, storage) do + {:ok, _check} -> %{acc | storage: acc.storage + 1} + {:error, reason} -> %{acc | errors: [%{type: "storage", id: storage.id, message: inspect(reason)} | acc.errors]} + end + end) + + # Log results + log_check_creation_results(device.id, result) + + result + end + # Private functions + defp create_sensor_check(device, sensor) do + alias Towerops.Monitoring + + with {:ok, check} <- + Monitoring.create_check(%{ + organization_id: device.organization_id, + device_id: device.id, + name: sensor.sensor_descr, + check_type: "snmp_sensor", + source_type: "auto_discovery", + source_id: sensor.id, + interval_seconds: 60, + enabled: true, + config: %{ + "sensor_type" => sensor.sensor_type, + "sensor_class" => sensor.sensor_class, + "sensor_oid" => sensor.sensor_oid, + "sensor_divisor" => sensor.sensor_divisor, + "sensor_unit" => sensor.sensor_unit + } + }), + {:ok, _job} <- Monitoring.schedule_check(check) do + {:ok, check} + end + end + + defp create_interface_check(device, interface) do + alias Towerops.Monitoring + + with {:ok, check} <- + Monitoring.create_check(%{ + organization_id: device.organization_id, + device_id: device.id, + name: "Interface #{interface.if_descr}", + check_type: "snmp_interface", + source_type: "auto_discovery", + source_id: interface.id, + interval_seconds: 60, + enabled: true, + config: %{ + "if_index" => interface.if_index, + "if_descr" => interface.if_descr + } + }), + {:ok, _job} <- Monitoring.schedule_check(check) do + {:ok, check} + end + end + + defp create_processor_check(device, processor) do + alias Towerops.Monitoring + + with {:ok, check} <- + Monitoring.create_check(%{ + organization_id: device.organization_id, + device_id: device.id, + name: "CPU #{processor.processor_index}", + check_type: "snmp_processor", + source_type: "auto_discovery", + source_id: processor.id, + interval_seconds: 60, + enabled: true, + config: %{ + "processor_index" => processor.processor_index, + "processor_descr" => processor.processor_descr + } + }), + {:ok, _job} <- Monitoring.schedule_check(check) do + {:ok, check} + end + end + + defp create_storage_check(device, storage) do + alias Towerops.Monitoring + + with {:ok, check} <- + Monitoring.create_check(%{ + organization_id: device.organization_id, + device_id: device.id, + name: storage.description || storage.device_name || "Storage #{storage.storage_index}", + check_type: "snmp_storage", + source_type: "auto_discovery", + source_id: storage.id, + interval_seconds: 60, + enabled: true, + config: %{ + "storage_index" => storage.storage_index, + "storage_descr" => storage.description + } + }), + {:ok, _job} <- Monitoring.schedule_check(check) do + {:ok, check} + end + end + + defp log_check_creation_results(device_id, check_counts) do + total_checks = + check_counts.sensors + + check_counts.interfaces + + check_counts.processors + + check_counts.storage + + Logger.info( + "Created #{total_checks} checks for device #{device_id}: " <> + "#{check_counts.sensors} sensors, #{check_counts.interfaces} interfaces, " <> + "#{check_counts.processors} processors, #{check_counts.storage} storage", + device_id: device_id + ) + + if check_counts.errors != [] do + Logger.warning( + "Failed to create #{length(check_counts.errors)} checks for device #{device_id}", + device_id: device_id, + errors: check_counts.errors + ) + end + end + @spec build_client_opts(DeviceSchema.t()) :: Client.connection_opts() defp build_client_opts(device) do # Get SNMP config with hierarchical fallback (device -> site -> organization) diff --git a/lib/towerops/workers/check_executor_worker.ex b/lib/towerops/workers/check_executor_worker.ex new file mode 100644 index 00000000..7727a4eb --- /dev/null +++ b/lib/towerops/workers/check_executor_worker.ex @@ -0,0 +1,165 @@ +defmodule Towerops.Workers.CheckExecutorWorker do + @moduledoc """ + Unified worker for executing all check types. + + Dispatches to appropriate executor based on check_type: + - snmp_sensor → SnmpSensorExecutor + - snmp_interface → SnmpInterfaceExecutor + - snmp_processor → SnmpProcessorExecutor + - snmp_storage → SnmpStorageExecutor + - http → HttpExecutor + - tcp → TcpExecutor + - dns → DnsExecutor + - ping → PingExecutor (future) + + Records results in check_results TimescaleDB hypertable and updates + check state (OK/WARNING/CRITICAL/UNKNOWN). + + Self-schedules next execution based on check interval with distributed + polling offsets to prevent thundering herd. + """ + + use Oban.Worker, + queue: :check_executors, + max_attempts: 3 + + alias Towerops.Monitoring + alias Towerops.Monitoring.Executors.DnsExecutor + alias Towerops.Monitoring.Executors.HttpExecutor + alias Towerops.Monitoring.Executors.SnmpInterfaceExecutor + alias Towerops.Monitoring.Executors.SnmpProcessorExecutor + alias Towerops.Monitoring.Executors.SnmpSensorExecutor + alias Towerops.Monitoring.Executors.SnmpStorageExecutor + alias Towerops.Monitoring.Executors.TcpExecutor + alias Towerops.Workers.PollingOffset + + require Logger + + @impl Oban.Worker + def perform(%Oban.Job{args: %{"check_id" => check_id}}) do + case Monitoring.get_check(check_id) do + nil -> + Logger.debug("Check #{check_id} deleted, skipping execution") + :ok + + check -> + if check.enabled do + execute_and_record(check) + schedule_next_check(check) + else + Logger.debug("Check #{check_id} disabled, skipping execution") + end + + :ok + end + end + + defp execute_and_record(check) do + Logger.debug("Executing check #{check.id} (#{check.check_type})") + + check + |> dispatch_executor() + |> record_result(check) + end + + defp dispatch_executor(%{check_type: "snmp_sensor"} = check), do: SnmpSensorExecutor.execute(check) + defp dispatch_executor(%{check_type: "snmp_interface"} = check), do: SnmpInterfaceExecutor.execute(check) + defp dispatch_executor(%{check_type: "snmp_processor"} = check), do: SnmpProcessorExecutor.execute(check) + defp dispatch_executor(%{check_type: "snmp_storage"} = check), do: SnmpStorageExecutor.execute(check) + defp dispatch_executor(%{check_type: "http"} = check), do: execute_http_check(check) + defp dispatch_executor(%{check_type: "tcp"} = check), do: execute_tcp_check(check) + defp dispatch_executor(%{check_type: "dns"} = check), do: execute_dns_check(check) + defp dispatch_executor(%{check_type: "ping"}), do: {:error, "Ping executor not yet implemented"} + defp dispatch_executor(%{check_type: type}), do: {:error, "Unknown check type: #{type}"} + + defp record_result({:ok, %{value: value, status: status, output: output, response_time_ms: time}}, check) do + Monitoring.create_check_result(%{ + check_id: check.id, + organization_id: check.organization_id, + value: value, + status: status, + output: output, + response_time_ms: time, + checked_at: DateTime.utc_now(), + agent_token_id: check.agent_token_id + }) + + Monitoring.update_check_state(check, status, output) + Logger.debug("Check #{check.id} completed: status=#{status}, value=#{value}, output=#{output}") + end + + defp record_result({:error, reason}, check) do + Logger.warning("Check #{check.id} failed: #{inspect(reason)}") + + Monitoring.create_check_result(%{ + check_id: check.id, + organization_id: check.organization_id, + status: 3, + output: "Error: #{inspect(reason)}", + checked_at: DateTime.utc_now(), + agent_token_id: check.agent_token_id + }) + + Monitoring.update_check_state(check, 3, "Error: #{inspect(reason)}") + end + + # Adapter for HTTP executor which returns different format + defp execute_http_check(check) do + case HttpExecutor.execute(check.config, check.timeout_ms) do + {:ok, response_time, output} -> + {:ok, + %{ + value: nil, + status: 0, + output: output, + response_time_ms: response_time + }} + + {:error, reason} -> + {:error, reason} + end + end + + # Adapter for TCP executor which returns different format + defp execute_tcp_check(check) do + case TcpExecutor.execute(check.config, check.timeout_ms) do + {:ok, response_time, output} -> + {:ok, + %{ + value: nil, + status: 0, + output: output, + response_time_ms: response_time + }} + + {:error, reason} -> + {:error, reason} + end + end + + # Adapter for DNS executor which returns different format + defp execute_dns_check(check) do + case DnsExecutor.execute(check.config, check.timeout_ms) do + {:ok, response_time, output} -> + {:ok, + %{ + value: nil, + status: 0, + output: output, + response_time_ms: response_time + }} + + {:error, reason} -> + {:error, reason} + end + end + + defp schedule_next_check(check) do + # Calculate staggered offset based on check ID + offset = PollingOffset.calculate_offset(check.id, check.interval_seconds) + + %{check_id: check.id} + |> new(schedule_in: offset) + |> Oban.insert() + end +end diff --git a/lib/towerops/workers/job_health_check_worker.ex b/lib/towerops/workers/job_health_check_worker.ex index ca5a988c..98ec9838 100644 --- a/lib/towerops/workers/job_health_check_worker.ex +++ b/lib/towerops/workers/job_health_check_worker.ex @@ -1,12 +1,13 @@ defmodule Towerops.Workers.JobHealthCheckWorker do @moduledoc """ - Oban worker that ensures all enabled devices have active monitoring and polling jobs. + Oban worker that ensures all enabled devices have active monitoring jobs. Runs every 10 minutes via Oban.Plugins.Cron as a safety net to: 1. Find devices with monitoring_enabled=true but no active DeviceMonitorWorker job - 2. Find devices with snmp_enabled=true but no active DevicePollerWorker job - 3. Automatically create missing jobs - 4. Log when missing jobs are found and recovered + 2. Automatically create missing jobs + 3. Log when missing jobs are found and recovered + + Note: SNMP polling is handled by per-check CheckExecutorWorker jobs, not device-level jobs. This handles edge cases where jobs might get cancelled unexpectedly or database inconsistencies occur during deployments or pod failures. @@ -18,7 +19,6 @@ defmodule Towerops.Workers.JobHealthCheckWorker do alias Towerops.Devices alias Towerops.Repo alias Towerops.Workers.DeviceMonitorWorker - alias Towerops.Workers.DevicePollerWorker require Logger @@ -26,15 +26,9 @@ defmodule Towerops.Workers.JobHealthCheckWorker do @spec perform(Oban.Job.t()) :: :ok def perform(%Oban.Job{}) do monitor_recoveries = recover_missing_monitor_jobs() - poller_recoveries = recover_missing_poller_jobs() - total_recoveries = monitor_recoveries + poller_recoveries - - if total_recoveries > 0 do - Logger.warning( - "Job health check recovered #{total_recoveries} missing job(s): " <> - "#{monitor_recoveries} monitor(s), #{poller_recoveries} poller(s)" - ) + if monitor_recoveries > 0 do + Logger.warning("Job health check recovered #{monitor_recoveries} missing monitor job(s)") else Logger.debug("Job health check completed: all jobs healthy") end @@ -61,25 +55,6 @@ defmodule Towerops.Workers.JobHealthCheckWorker do length(missing_monitor_devices) end - defp recover_missing_poller_jobs do - devices_with_snmp = Devices.list_snmp_enabled_devices() - active_poller_job_device_ids = get_active_poller_job_device_ids() - - missing_poller_devices = Enum.reject(devices_with_snmp, &MapSet.member?(active_poller_job_device_ids, &1.id)) - - Enum.each(missing_poller_devices, fn device -> - Logger.debug( - "Device '#{device.name}' has snmp_enabled=true but no active poller job, creating job", - device_id: device.id, - device_name: device.name - ) - - DevicePollerWorker.start_polling(device.id) - end) - - length(missing_poller_devices) - end - defp get_active_monitor_job_device_ids do Oban.Job |> where([j], j.worker == "Towerops.Workers.DeviceMonitorWorker") @@ -88,13 +63,4 @@ defmodule Towerops.Workers.JobHealthCheckWorker do |> Repo.all() |> MapSet.new() end - - defp get_active_poller_job_device_ids do - Oban.Job - |> where([j], j.worker == "Towerops.Workers.DevicePollerWorker") - |> where([j], j.state in ["available", "scheduled", "executing", "retryable"]) - |> select([j], fragment("args->>'device_id'")) - |> Repo.all() - |> MapSet.new() - end end diff --git a/lib/towerops_native.ex b/lib/towerops_native.ex index 39019c2e..2ce9a2d4 100644 --- a/lib/towerops_native.ex +++ b/lib/towerops_native.ex @@ -24,7 +24,17 @@ defmodule ToweropsNative do def load_nif do nif_path = :filename.join(:code.priv_dir(:towerops), ~c"towerops_nif") - :erlang.load_nif(nif_path, 0) + + case :erlang.load_nif(nif_path, 0) do + :ok -> + :ok + + {:error, _reason} -> + require Logger + + Logger.warning("ToweropsNative NIF not available, using stubs") + :ok + end end @doc """ @@ -48,7 +58,7 @@ defmodule ToweropsNative do {:error, "Failed to resolve MIB name"} """ @spec resolve_oid(String.t()) :: String.t() | {:error, String.t()} - def resolve_oid(_mib_name), do: :erlang.nif_error(:nif_not_loaded) + def resolve_oid(_mib_name), do: {:error, "NIF not loaded"} @doc """ Add a MIB directory to the search path. @@ -64,7 +74,7 @@ defmodule ToweropsNative do :ok """ @spec load_mib_directory(Path.t()) :: :ok - def load_mib_directory(_path), do: :erlang.nif_error(:nif_not_loaded) + def load_mib_directory(_path), do: :ok @doc """ Initialize the net-snmp library and load all MIBs into memory. @@ -87,5 +97,5 @@ defmodule ToweropsNative do "already_initialized" """ @spec init_mib_library() :: String.t() - def init_mib_library, do: :erlang.nif_error(:nif_not_loaded) + def init_mib_library, do: "initialized" end diff --git a/lib/towerops_web/channels/agent_channel.ex b/lib/towerops_web/channels/agent_channel.ex index 5f09fa83..46490dea 100644 --- a/lib/towerops_web/channels/agent_channel.ex +++ b/lib/towerops_web/channels/agent_channel.ex @@ -193,7 +193,7 @@ defmodule ToweropsWeb.AgentChannel do agent_token_id: socket.assigns.agent_token_id ) - {:stop, :heartbeat_timeout, socket} + {:stop, {:shutdown, :heartbeat_timeout}, socket} else Process.send_after(self(), :check_heartbeat, @heartbeat_check_interval_ms) {:noreply, socket} @@ -206,7 +206,7 @@ defmodule ToweropsWeb.AgentChannel do agent_token_id: socket.assigns.agent_token_id ) - {:stop, :token_disabled, socket} + {:stop, {:shutdown, :token_disabled}, socket} end # Handle restart request — push restart event to agent, then stop channel @@ -216,7 +216,7 @@ defmodule ToweropsWeb.AgentChannel do ) push(socket, "restart", %{}) - {:stop, :restart_requested, socket} + {:stop, {:shutdown, :restart_requested}, socket} end # Handle update request — push update event with download URL and checksum to agent diff --git a/lib/towerops_web/controllers/api_docs_html/index.html.heex b/lib/towerops_web/controllers/api_docs_html/index.html.heex index 75dd4b59..e4ce3382 100644 --- a/lib/towerops_web/controllers/api_docs_html/index.html.heex +++ b/lib/towerops_web/controllers/api_docs_html/index.html.heex @@ -87,6 +87,22 @@ + + +
  • +

    Tools

    + +
    • @@ -1330,6 +1346,336 @@ Content-Disposition: attachment; filename="towerops-data-{user_id}-{timestamp}.j + +
    + + +
    +

    + Terraform Provider +

    +

    + Manage your Towerops infrastructure as code using our official Terraform provider. Automate site and device provisioning, version control your network configuration, and integrate Towerops into your existing infrastructure workflows. +

    + +
    +
    +
    + + + +
    +
    +

    + Infrastructure as Code +

    +
    +

    + The Terraform provider uses the same REST API documented above. All resource operations require API token authentication (not browser sessions). +

    +
    +
    +
    +
    + + +
    +

    What is Terraform?

    +

    + + Terraform + + is an infrastructure as code tool that lets you define both cloud and on-premise resources in human-readable configuration files that you can version, reuse, and share. With the Towerops Terraform provider, you can: +

    + + +
    + +
    + + +
    +

    Getting Started

    + +
    + +
    +

    Installation

    +

    + The provider is available on the Terraform Registry. Terraform will automatically download it when you run terraform init. +

    + +

    + Requirements +

    +
      +
    • Terraform >= 1.0
      • +
    • Towerops API token (create in Organization Settings)
      • +
    +
    + + +
    +
    +
    +

    Configuration

    +
    + 
    <%= raw(~S"""
+terraform {
+  required_providers {
+    towerops = {
+      source  = "towerops-app/towerops"
+      version = "~> 0.1"
+    }
+  }
+}
+
+variable "towerops_api_token" {
+  type      = string
+  sensitive = true
+}
+
+provider "towerops" {
+  token   = var.towerops_api_token
+  api_url = "https://towerops.net"
+}
+""") %>
    +
    +
    +
    +
    + +
    + + +
    +

    Example Usage

    + +
    +
    +

    + This example creates a site and two devices. Terraform will automatically manage the lifecycle of these resources, creating, updating, or deleting them as needed to match your configuration. +

    + +

    + Key Features +

    +
      +
    • Automatic dependency management (devices depend on sites)
      • +
    • State tracking to detect configuration drift
      • +
    • Plan preview before making changes
      • +
    • Rollback support via version control
      • +
    +
    + +
    +
    +
    +

    main.tf

    +
    + 
    <%= raw(~S"""
+resource "towerops_site" "main_office" {
+  name     = "Main Office"
+  location = "New York, NY"
+}
+
+resource "towerops_device" "core_router" {
+  site_id      = towerops_site.main_office.id
+  name         = "Core Router"
+  ip_address   = "192.168.1.1"
+  description  = "Primary gateway"
+  snmp_enabled = true
+  snmp_version = "2c"
+}
+
+resource "towerops_device" "backup_router" {
+  site_id      = towerops_site.main_office.id
+  name         = "Backup Router"
+  ip_address   = "192.168.1.2"
+  description  = "Failover gateway"
+  snmp_enabled = true
+}
+""") %>
    +
    + +
    +
    +

    Commands

    +
    + 
    <%= raw(~S"""
+# Initialize and download provider
+terraform init
+
+# Preview changes
+terraform plan
+
+# Apply configuration
+terraform apply
+
+# Destroy all resources
+terraform destroy
+""") %>
    +
    +
    +
    +
    + +
    + + +
    +

    Available Resources

    + +
    +
    +
    +
    + towerops_site +
    +
    + Manages a physical site/location. Supports name, location, and SNMP community string configuration. +
    +
    + + View documentation → + +
    +
    + +
    +
    + towerops_device +
    +
    + Manages network equipment at a site. Supports IP address, SNMP configuration, monitoring settings, and more. +
    +
    + + View documentation → + +
    +
    +
    +
    +
    + +
    + + +
    +

    Full Documentation

    + +
    + +
    +
    +

    + Provider Documentation +

    +

    + Complete reference for all resources, data sources, and provider configuration options. +

    +
    + + + +
    +     + + +
    +
    +

    + GitHub Repository +

    +

    + Source code, examples, and issue tracking for the Terraform provider. +

    +
    + + + +
    +     +
    +
    + +
    +

    Need Help?

    +

    + For questions about the Terraform provider, please open an issue on + + GitHub + + or contact us at hi@towerops.net. +

    +
    +