towerops/docs/TEST_COVERAGE.md

TowerOps Test Coverage Report

Generated: 2026-03-13 Test Run: 90 doctests, 55 properties, 8,134 tests | 4 failures, 38 skipped, 231 excluded Duration: 158.2s (40.5s async, 117.6s sync)

Summary

Metric	Count
Total test files	547
Total tests (incl. doctests/properties)	8,279
Passing	8,237
Failures	4
Skipped	38
Excluded	231
Source modules (excl. vendor profiles, migrations, proto)	~230
Web modules (controllers, LiveViews, plugs, etc.)	~140
Modules with NO test file	~100

Current Failures (4)

1. ToweropsWeb.AgentLive.IndexTest — Unique constraint violation on agents.slug during setup (test data collision)
2. ToweropsWeb.AgentLive.IndexTest — Same slug collision in cloud poller creation setup
3. ToweropsWeb.AgentLive.IndexTest — Same root cause, different describe block
4. ToweropsWeb.GraphQL.Resolvers.TimeSeriesTest — Deadlock on create_check_result (async test hitting ShareRowExclusiveLock)

Both are infrastructure/setup issues, not logic bugs. The agent slug collision likely needs a unique factory counter; the deadlock needs async: false or transaction isolation.

---

Test Infrastructure & Patterns

Case Templates

• ToweropsWeb.ConnCase — HTTP/LiveView tests. Imports Phoenix.ConnTest, Plug.Conn, verified routes. Has register_and_log_in_user/1 and sudo variant.
• Towerops.DataCase — Context/schema tests. Imports Ecto, Ecto.Changeset, Ecto.Query, Towerops.Repo. SQL Sandbox isolation.
• ToweropsWeb.LiveViewTestHelpers — Shared helpers for LiveView tests.

Fixtures (Factory Pattern)

All in test/support/fixtures/:

• accounts_fixtures.ex — Users, tokens, TOTP
• agents_fixtures.ex — Agents, assignments, tokens
• billing_fixtures.ex — Stripe data
• devices_fixtures.ex — Devices, firmware
• gaiia_fixtures.ex — GAIIA accounts, subscribers
• integrations_fixtures.ex — Integration configs
• jobs_fixtures.ex — Oban jobs
• maintenance_fixtures.ex — Maintenance windows
• on_call_fixtures.ex — Schedules, escalation policies
• organizations_fixtures.ex — Orgs, memberships, invitations
• snmp_fixtures.ex — SNMP devices, interfaces

Test Types

• Async tests: ~442 files use async: true
• Property tests: 55 (using StreamData)
• Doctests: 90
• Integration tests: test/integration/ (SNMP, discovery parity)
• SnmpKit tests: Dedicated test/snmpkit/ directory with ~45 test files

Mocking

• Mox — Used for behaviour-based mocks (SNMP, ping, HTTP)
• ping_stub.ex — Stub for ping behaviour
• snmp_kit_mock.ex — SNMP client mock

---

Coverage by Domain

✅ Well-Covered Domains

Domain	Source Files	Test Files	Notes
Accounts	13	14+	Comprehensive auth, TOTP, sessions, HIBP, browser sessions
Agents	5	8+	Assignments, tokens, stats, release checker, cloud poller
Alerts	2	2	Alert schema + context
API Tokens	2	2	Token CRUD
Billing	5	4	Stripe client, webhook processor, notifier (missing: stripe_webhook_event schema)
Devices	14	10+	Core CRUD, firmware, backups, differ, version comparator
GAIIA	14	13	Excellent: sync, reconciliation, subscriber matching, impact analysis
Organizations	6	7+	Policies, invitations, subscriptions, property tests
Preseem	14	14	Full coverage: sync, baselines, insights, fleet intelligence
Profiles	7	5	YAML profiles, device/sensor OIDs
SNMP Core	30+	25+	Discovery, polling, sanitizer, sensor change, MIB parsing
SNMP Vendors	180+	180+	Auto-generated, 1:1 coverage
Workers	30	22	Most critical workers covered
Web: Plugs	13	11	Auth, rate limiting, security headers, brute force
Web: Controllers	25+	25+	API v1, admin, auth flow controllers
Web: LiveViews	40+	35+	Most major pages tested

⚠️ Gaps — Modules Without Test Files

🔴 Critical (Needs Tests NOW)

These modules handle auth, billing, alert routing, or data mutations with no dedicated tests:

Module	Why Critical
Towerops.Monitoring (context)	Core monitoring orchestration — check creation, status evaluation
Towerops.Monitoring.Executor	Dispatches monitoring checks to executors
Towerops.Monitoring.Executors.DnsExecutor	DNS monitoring — untested execution path
Towerops.Monitoring.Executors.HttpExecutor	HTTP monitoring — untested execution path
Towerops.Monitoring.Executors.TcpExecutor	TCP monitoring — untested execution path
Towerops.Monitoring.Executors.SnmpInterfaceExecutor	SNMP interface monitoring
Towerops.Monitoring.Executors.SnmpProcessorExecutor	SNMP CPU monitoring
Towerops.Monitoring.Executors.SnmpStorageExecutor	SNMP storage monitoring
Towerops.OnCall.Notifier	Sends on-call notifications — critical alert path
Towerops.OnCall.Incident	Incident lifecycle — create, ack, resolve
Towerops.OnCall.EscalationPolicy	Escalation rule evaluation
Towerops.OnCall.Notification	Notification delivery schema
Towerops.PagerDuty.Notifier	External PagerDuty alert dispatch
Towerops.Security.CloudflareClient	Cloudflare IP banning — security mutation
Towerops.Security.IpBlock / IpWhitelist	IP access control schemas
Towerops.Billing.StripeWebhookEvent	Stripe event persistence schema
Towerops.Accounts.UserRecoveryCode	Account recovery — security-critical
Towerops.Accounts.UserToken	Token schema (tested indirectly via Accounts context but no direct schema test)

🟠 High Priority

Module	Why
Towerops.Maintenance (context)	Maintenance window CRUD and scheduling logic
Towerops.ConfigChanges (context)	Config change tracking and correlation
Towerops.ConfigChanges.Correlator	Correlates config changes to alerts
Towerops.Topology.Lldp	LLDP neighbor discovery logic
Towerops.Topology.Identifier	Device identification heuristics
Towerops.Netbox.Sync	External NetBox sync — data import
Towerops.Sonar.Client / Sync	Sonar billing integration
Towerops.Weather.Client	Weather API integration
Towerops.Workers.EscalationCheckWorker	Periodic escalation evaluation
Towerops.Workers.CheckWorker	Monitoring check scheduling
Towerops.Workers.MikrotikBackupWorker	MikroTik backup orchestration
Towerops.Workers.CloudflareBanWorker	Security: auto-ban via Cloudflare
ToweropsWeb.Controllers.Api.V1.PagerdutyWebhookController	Inbound PagerDuty webhooks
ToweropsWeb.Controllers.Api.V1.ActivityController	Activity feed API
ToweropsWeb.Controllers.Api.V1.CheckResultsController	Check results API
ToweropsWeb.Controllers.InvitationController	Org invitation acceptance
ToweropsWeb.Controllers.UserConfirmationController	Email confirmation flow
ToweropsWeb.Controllers.UserRegistrationController	User signup flow
ToweropsWeb.Live.OnboardingLive	New user onboarding
ToweropsWeb.Live.Org.GaiiaMappingLive	GAIIA device mapping UI
ToweropsWeb.Live.Org.GaiiaReconciliationLive	GAIIA reconciliation UI

🟡 Medium Priority

Module	Why
Towerops.Geocoding	Address-to-coordinates (read-only helper)
Towerops.GeoIp.Block / Location	GeoIP lookup schemas
Towerops.Vault	Encryption key management
Towerops.RateLimit	Rate limiting logic
Towerops.Devices.Event / BackupRequest	Event/request schemas
Towerops.Devices.MikrotikBackup	Backup schema
Towerops.OnCall.Schedule / Layer / Override	On-call schedule schemas
Towerops.Organizations.Membership	Membership schema
Towerops.SNMP.Interface / Sensor / Device schemas	Data schemas
Towerops.Workers.*SyncWorker (sonar, splynx, visp, weather, netbox)	Sync orchestration
ToweropsWeb.Live.CapacityLive	Capacity planning page
ToweropsWeb.Live.ConfigTimelineLive	Config change timeline
ToweropsWeb.Live.ActivityFeedLive	Activity feed page
ToweropsWeb.Live.MikrotikBackupLive.Compare	Backup diff viewer
ToweropsWeb.GraphQL.Context	GraphQL context setup

🟢 Low Priority

Module	Notes
Towerops.EctoTypes.EncryptedBinary	Simple Ecto type (encrypted_map is tested)
Towerops.Settings.ApplicationSetting	Simple schema
Towerops.Mailer	Swoosh wrapper
Towerops.Release	Mix release tasks
Towerops.Repo	Ecto repo module
Towerops.SNMP.SnmpBehaviour	Behaviour definition
Towerops.SNMP.PollerBehaviour	Behaviour definition
Towerops.Monitoring.PingBehaviour	Behaviour definition
Towerops.Profiles.MibCache / ProfileWatcher	GenServer infrastructure
Various SNMP data schemas (readings, stats)	Simple structs

---

Top 10 Recommended Tests to Write

1. test/towerops/monitoring_test.exs — Monitoring Context

Impact: 🔴 Critical Cover: create_check/2, update_check/2, delete_check/2, list_checks_for_device/1, check result creation and status evaluation. This is the core monitoring orchestration layer with zero test coverage.

2. test/towerops/monitoring/executor_test.exs — Check Executor Dispatch

Impact: 🔴 Critical Cover: Executor dispatch logic, timeout handling, error wrapping. Test with mocked executors that the right executor is called for each check type (HTTP, DNS, TCP, SNMP).

3. test/towerops/on_call/notifier_test.exs — On-Call Notification Delivery

Impact: 🔴 Critical Cover: Notification channel selection (email, SMS, push), retry logic, escalation triggering, de-duplication. Mock external delivery and verify correct notification flow.

4. test/towerops/on_call/incident_test.exs — Incident Lifecycle

Impact: 🔴 Critical Cover: Incident creation from alert, acknowledgment, resolution, re-escalation on timeout. Verify state machine transitions and escalation policy evaluation.

5. test/towerops/monitoring/executors/http_executor_test.exs — HTTP Check Execution

Impact: 🔴 Critical Cover: HTTP status checks, response time measurement, SSL verification, timeout handling, redirect following, body content matching. Mock HTTP client.

6. test/towerops/config_changes_test.exs — Config Change Tracking

Impact: 🟠 High Cover: Change event creation, correlation with alerts/devices, timeline querying, diff computation. Verify correlator links config changes to relevant alerts.

7. test/towerops/maintenance_test.exs — Maintenance Window Context

Impact: 🟠 High Cover: Window creation/scheduling, overlap detection, active window queries, alert suppression during maintenance. Verify is_in_maintenance?/2 logic.

8. test/towerops_web/controllers/api/v1/pagerduty_webhook_controller_test.exs — PagerDuty Webhooks

Impact: 🟠 High Cover: Webhook signature verification, event type handling (trigger, acknowledge, resolve), idempotency, error responses for invalid payloads.

9. test/towerops/security/cloudflare_client_test.exs — Cloudflare IP Management

Impact: 🟠 High Cover: IP block creation/deletion via Cloudflare API, error handling, rate limiting. Mock Cloudflare API. Verify the ban worker integrates correctly.

10. test/towerops_web/controllers/user_registration_controller_test.exs — User Registration

Impact: 🟠 High Cover: Successful registration, duplicate email handling, password validation, email confirmation trigger, registration with invitation token. Critical auth flow.

---

Test Health Notes

• Async ratio is excellent — ~81% of test files use async: true
• Property testing is in use — 55 property tests via StreamData (found in schema validation, Ecto type tests)
• Fixture coverage is comprehensive — 14 fixture modules covering all major domains
• Vendor profile tests are auto-generated — 180+ vendor tests likely from mix gen_vendor_tests
• The 4 current failures are infrastructure issues, not logic bugs:
  • Agent slug uniqueness collision in test factories → fix: add System.unique_integer() to slug generation
  • Deadlock in async time series test → fix: set async: false or restructure DB access
• 231 excluded tests — likely tagged :integration or :external for CI isolation
• 38 skipped tests — worth auditing to see if any are permanently broken