The test was using 75ms timeout which was too tight:
- 5 broadcasts with 5ms delays = 25ms
- Debounce delay = 50ms
- Processing overhead = variable
Increased to 200ms to match other similar tests in the file and provide
adequate buffer for debouncing + processing.
Also reformatted proto erlang_compat files to fix line length issues.
Reviewed-on: graham/towerops-web#210
Phase 1: Foundation Types (100% complete)
- query_helpers: SQL LIKE sanitization with pipe operators
- numeric: Integer parsing with pattern matching guards
- result: Pure Elixir Result monad (map, and_then, unwrap_or)
Phase 2: Ecto Domain Types (100% complete)
- ip_address: IPv4/IPv6 validation using :inet directly
- mac_address: Multi-format MAC parsing (colon/hyphen/dot/compact)
- snmp_oid: OID parsing/manipulation with recursive pattern matching
All 198 tests passing across converted modules.
API changed from Gleam-style {:some/:none to idiomatic {:ok/:error.
Refactored parse_numeric_oid to use with statement, reducing nesting depth.
Reviewed-on: graham/towerops-web#196
When an agent joins, we update last_seen_at in the database but don't set
last_heartbeat_db_update in the socket. This causes the heartbeat throttling
logic to be out of sync:
- On join: DB updated, but last_heartbeat_db_update not set (nil)
- First heartbeat: Updates DB again because last_heartbeat_db_update is nil
- Subsequent heartbeats: Check if >30s since last DB update
- Problem: last_heartbeat_db_update tracks the first heartbeat, not the join DB update
This manifests when Phoenix recompiles during development - the channel process
continues running with old socket assigns, and if last_heartbeat_db_update is
recent, heartbeats won't update the DB. After 10+ minutes without DB updates,
the agent appears offline even though it's still connected.
Fix: Set last_heartbeat_db_update when we update the DB on join, so the
throttling logic accurately tracks the most recent DB update.
Reviewed-on: graham/towerops-web#142
The Check protobuf has a `oneof config` field which generates separate
fields in the Elixir struct: :http, :tcp, :dns, :ssl. However,
check_type_config/2 was returning [config: {:http, ...}] which doesn't
match the struct definition, causing a KeyError crash in
build_and_push_check_jobs/1.
This bug was originally fixed in e0a74e2f but was accidentally
reintroduced in f703e61b when attempting to use "tagged tuple config".
Fix: Return field-specific keyword lists (e.g., [http: %HttpCheckConfig{}])
instead of [config: {:http, ...}].
Also added comprehensive test that creates all 4 check types and verifies
no KeyError crash occurs when sending check jobs to agent.
Reviewed-on: graham/towerops-web#141
Replace the protobuf hex package (agent.pb.ex) and hand-written validator
(validator.ex) with a pure Gleam implementation: wire format primitives,
all 23 message types, encode/decode functions, and validation merged into
decode. Thin Elixir wrappers in agent.ex preserve the existing struct API.
- Wire format: varint, length-delimited, double (IEEE 754 via FFI), tag encode/decode
- Types: all proto3 message types as Gleam custom types with enums and oneof
- Decode: field-level parsing with integrated validation (UUID, IP, hostname, limits)
- Encode: proto3 default omission, bytes_tree concatenation, map/repeated fields
- Wrappers: Elixir structs with encode/decode delegating to Gleam, enum atom conversion
- Remove {:protobuf, "~> 0.12"} dependency
Reviewed-on: graham/towerops-web#104
Protobuf oneof fields use a single :config key with tagged tuples
like {:dns, %DnsCheckConfig{}} rather than individual keys. The
check_type_config functions were setting e.g. dns: %DnsCheckConfig{}
directly, causing KeyError crashes in Check.__struct__/1.
Reviewed-on: graham/towerops-web#101
- add MobileSocket for token-authenticated mobile WebSocket connections
- add MobileChannel with org-level and device-level real-time events
- add /mobile/socket endpoint for iOS app connections
- add "Link Mobile App" to user dropdown menu for discoverability
- improve QR code page with numbered steps and clearer instructions
- add socket/channel tests (17 total)
The server only sent polling/ping jobs to agents once on WebSocket join
and never re-dispatched them. After the first batch completed (~5-10s),
devices were never polled again until the agent reconnected.
:send_jobs now schedules a follow-up dispatch every 60s (configurable).
Unifies the timer management with :assignments_changed to prevent
accumulation.
- Fix ApplicationSetting duplicate key constraint errors in BillingTest and AdminTest
by deleting existing records before inserting new ones in tests
- Increase timeouts for agent_channel debounce tests from 75ms to 200ms
to prevent flaky failures on slower systems (50ms debounce + processing time)
All 7633 tests now pass consistently across multiple runs.
Reduce various test timeouts that were unnecessarily long:
- assert_push timeouts: 150ms → 75ms (6 tests in agent_channel_test)
- refute_receive timeouts: 200ms → 50ms (3 tests in worker tests)
With 50ms channel debounce, 75ms is sufficient for assert_push.
For refute_receive (expecting no message), 50ms is plenty.
Estimated savings: ~600ms across affected tests
- Remove all 25ms and 50ms sleeps from agent_channel_test
- Add poll_until helper for async DB operations (early exit on success)
- Reduce timeout test sleeps from 100ms to 60ms
- Reduce polling delays from 10ms to 5ms in multiple tests
- Remove unnecessary timestamp and background task sleeps
- Reduce circuit breaker recovery sleep from 150ms to 110ms
Performance: 30.3s → 28.3s (6.6% faster)
Total improvement from baseline: 52s → 28.3s (45.6% faster)
My changes added database queries for stuck alert resolution on every poll,
which slightly increased processing time. Increased timeouts from 500ms to 1000ms
and 100ms to 200ms to account for this.
When agents poll devices via SNMP and get successful results, the code was
only processing sensor/interface data without updating device status or
resolving alerts. This caused device_down alerts to persist even after
devices came back online.
This fix:
- Updates device status to 'up' when SNMP polling succeeds
- Auto-resolves any active device_down alerts
- Creates device_up alert (matching DeviceMonitorWorker behavior)
- Broadcasts device_status_changed via PubSub
Also fixes device_up alert creation to include resolved_at timestamp
(matching DeviceMonitorWorker behavior).
Test coverage: Added test verifying auto-resolution when device recovers.
Test Performance Improvements:
- Reduce SNMP test timeouts from 2000ms to 100ms (walk and bulk tests)
- Reduce AgentChannelTest assert_push timeouts from 2000ms to 1000ms
- Reduce Process.sleep delays: 200ms → 50ms, 100ms → 25ms
- Results: Top 10 slowest 14.1s → 10.5s (26% faster)
CI Test Fixes:
- MIBStubsTest: Increase performance threshold from 10ms to 20ms for slower CI runners
- LoginHistoryCleanupWorkerTest: Fix boundary test using 364 days instead of 365
to account for timing difference between test setup and worker execution
Overall: 7422 tests, 0 failures, suite runs in 57.4s
This commit implements the unified checks architecture that consolidates
SNMP monitoring with HTTP/TCP/DNS checks under a single "check" abstraction,
enabling consistent graphing, alerting, and management across all check types.
Database Changes:
- Add source_type and source_id to checks table for tracking auto-discovered
vs manually created checks
- Add value field to check_results for storing numeric sensor readings
- Maintain backward compatibility with existing check_results data
New SNMP Executors:
- SnmpSensorExecutor: Poll sensor OIDs and return formatted values with
status determination (OK/WARNING/CRITICAL based on limits)
- SnmpInterfaceExecutor: Poll interface stats (bandwidth, packets, errors)
- SnmpProcessorExecutor: Poll CPU/processor usage
- SnmpStorageExecutor: Poll disk/memory usage with percentage calculations
Check Execution Worker:
- CheckExecutorWorker: Unified Oban worker that dispatches to appropriate
executor based on check_type (snmp_sensor, snmp_interface, http, tcp, etc.)
- Self-schedules next execution with distributed polling offsets
- Records results in check_results TimescaleDB hypertable
- Updates check state (OK/WARNING/CRITICAL/UNKNOWN)
Discovery Integration:
- Auto-creates checks during SNMP discovery for sensors, interfaces,
processors, and storage
- Links checks to source entities via source_id for data lookup
- Enables/disables checks based on discovery results
UI Enhancements:
- Checks tab on device detail page with grouped display
- FormComponent for adding manual HTTP/TCP/DNS checks
- Empty state with "Run Discovery" prompt
- Check status badges and last checked times
Graphing:
- Update GraphLive to accept check_id parameter
- Query check_results table for time-series data
- Support all check types (SNMP, HTTP response times, etc.)
Testing:
- Comprehensive test suite for SnmpSensorExecutor (5 tests)
- Test suite for CheckExecutorWorker (7 tests)
- Test coverage for discovery check creation (6 tests)
- Remove deprecated monitoring_test.exs testing old API
Bug Fixes:
- Fix SNMP executors reading credentials from correct Device schema fields
(device.snmp_version instead of device.snmp_device.version)
- Update agent channel test to query MonitoringCheck table directly
Code Quality:
- Extract add_snmp_credentials helper to reduce cyclomatic complexity
- Use map-based lookups for sensor formatting and check type grouping
- Apply pattern matching in dispatcher to reduce complexity
- All credo checks passing with no issues
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Enhanced AgentChannel logging to track device IDs across job list updates.
Added tests verifying server correctly excludes deleted devices from job lists.
Root cause: Bug is in Go agent (appends jobs without clearing), not Phoenix server.
- use 64-bit HC counters (ifHCInOctets/ifHCOutOctets) instead of 32-bit
counters that wrap every ~34s on gigabit links
- fix negative integer parsing in replay adapter so sensor scale values
like -3 produce correct divisors (1000) instead of falling through to 1
- replace walking entire Cisco enterprise tree with targeted subtrees to
avoid timeouts; add Juniper, HP/H3C, Fortinet, Cambium vendor OIDs
- add missing discovery OIDs: ENTITY-STATE-MIB, HOST-RESOURCES-MIB
tables, and UCD-SNMP-MIB Linux CPU stats fallback
newly discovered devices sat idle with no metric data until the
next scheduling cycle. now the channel sends a poll job (and ping
job if monitoring enabled) right after discovery succeeds, using
send(self(), ...) so discovery DB writes commit first.
also reduces k8s replicas from 3 to 2.
