graham/towerops
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
towerops/lib/towerops
History
Graham McIntire 997bc46cbf fix: dns and ping checks not executing in production (#71) 
## Summary

- **Ping checks never scheduled**: `ensure_default_ping_check` created check records but never called `schedule_check`, so auto-created ping checks never executed
- **Skipped checks permanently orphaned**: `CheckExecutorWorker` only rescheduled checks that actually executed — skipped checks (agent-assigned, SNMP disabled) broke the scheduling chain permanently
- **No recovery mechanism**: `JobHealthCheckWorker` only recovered `DeviceMonitorWorker` jobs, not `CheckExecutorWorker` jobs — any lost check job was gone forever

## Changes

- `monitoring.ex`: Call `schedule_check` after creating ping check in `ensure_default_ping_check`
- `check_executor_worker.ex`: Reschedule skipped-but-enabled checks so conditions are re-evaluated next cycle
- `job_health_check_worker.ex`: Recover orphaned `CheckExecutorWorker` jobs for enabled checks (runs every 10 min)
- `devices.ex`: Wrap `ensure_default_ping_check` in try/rescue for deadlock resilience

## Test plan

- [x] New tests: ping check scheduling on creation, no duplicate scheduling on idempotent call
- [x] Updated tests: skipped checks now assert rescheduling instead of orphaning
- [x] New tests: health check worker recovers orphaned check executor jobs, skips disabled checks
- [x] All 177 targeted tests pass
- [x] Compile clean with warnings-as-errors
- [x] Pre-commit hooks pass (credo, format)

Reviewed-on: graham/towerops-web#71
2026-03-18 08:43:37 -05:00
..
accounts add 100% test coverage for Towerops.Accounts 2026-03-11 10:37:03 -05:00
admin feat: add Admin.update_global_pricing with Stripe integration 2026-03-06 13:55:12 -06:00
agent wire up service checks (HTTP/TCP/DNS) to remote agents (#50) 2026-03-16 17:25:02 -05:00
agents perf: disable Req retry for faster error tests 2026-03-10 16:19:31 -05:00
alerts fix: make sidebar sticky with header and resolve all credo issues (#32) 2026-03-15 18:19:09 -05:00
api_tokens refactor: use API token auth for profile imports instead of session cookies 2026-01-18 09:30:21 -06:00
billing Add GeoIP import HTTP tests 2026-03-10 16:02:49 -05:00
capacity feat: add backhaul capacity analysis foundation 2026-03-06 17:54:32 -06:00
cn_maestro fix: make sidebar sticky with header and resolve all credo issues (#32) 2026-03-15 18:19:09 -05:00
config_changes refactor: fix all credo strict issues, format all code, fix broken routes 2026-02-13 19:34:40 -06:00
contexts help update 2026-02-04 17:56:48 -06:00
devices perf: LiveView event batching & PubSub optimization (TOW-26) 2026-03-15 14:08:27 -05:00
ecto_types add encrypted map ecto type for integration credentials 2026-02-12 16:52:05 -06:00
gaiia fix: humanize raw error strings in integration sync modules 2026-03-12 13:53:28 -05:00
geoip remove Mix.env and support ip database import 2026-01-28 13:08:06 -06:00
integrations paperclip/TOW-21-uisp-integration-phase1 (#31) 2026-03-15 17:57:36 -05:00
job_monitoring more tests and fixes 2026-02-07 11:50:18 -06:00
maintenance Add maintenance windows and weather overlay (backend) 2026-02-16 10:01:19 -06:00
mobile_sessions security: implement comprehensive security audit fixes 2026-03-05 13:08:10 -06:00
monitoring add ssl/tls certificate expiration check type (#64) 2026-03-17 16:14:22 -05:00
netbox fix: humanize raw error strings in integration sync modules 2026-03-12 13:53:28 -05:00
on_call feat: add on-call schedules and escalation policies 2026-03-11 12:32:54 -05:00
organizations Add alert routing setting (builtin/pagerduty/both) 2026-03-13 15:26:25 -05:00
pagerduty fix: make sidebar sticky with header and resolve all credo issues (#32) 2026-03-15 18:19:09 -05:00
preseem fix: humanize raw error strings in integration sync modules 2026-03-12 13:53:28 -05:00
profiles fix: handle FileSystem :ignore in ProfileWatcher for CI 2026-03-07 16:02:32 -06:00
proto add ssl/tls certificate expiration check type (#64) 2026-03-17 16:14:22 -05:00
reports fix: make sidebar sticky with header and resolve all credo issues (#32) 2026-03-15 18:19:09 -05:00
security Add GeoIP import HTTP tests 2026-03-10 16:02:49 -05:00
settings feat: add decimal type support to ApplicationSetting 2026-03-06 13:34:37 -06:00
sites feat: add geographic location to sites with geocoding, address fields, and Leaflet.js network map 2026-02-14 13:07:14 -06:00
snmp perf: LiveView event batching & PubSub optimization (TOW-26) 2026-03-15 14:08:27 -05:00
sonar fix: humanize raw error strings in integration sync modules 2026-03-12 13:53:28 -05:00
splynx fix: humanize raw error strings in integration sync modules 2026-03-12 13:53:28 -05:00
status_pages fix: make sidebar sticky with header and resolve all credo issues (#32) 2026-03-15 18:19:09 -05:00
topology Improve topology discovery matching 2026-03-10 15:44:46 -05:00
uisp fix: make sidebar sticky with header and resolve all credo issues (#32) 2026-03-15 18:19:09 -05:00
visp fix: humanize raw error strings in integration sync modules 2026-03-12 13:53:28 -05:00
weather Add GeoIP import HTTP tests 2026-03-10 16:02:49 -05:00
workers fix: dns and ping checks not executing in production (#71) 2026-03-18 08:43:37 -05:00
accounts.ex fix: NimbleTOTP base32 decoding and complete e2e test setup 2026-03-06 16:42:41 -06:00
activity_feed.ex fix: convert binary UUID to text in activity feed sync query 2026-03-06 17:33:34 -06:00
admin.ex feat: add Admin.update_global_pricing with Stripe integration 2026-03-06 13:55:12 -06:00
agents.ex fix: auto-dismiss agent offline insights when agent reconnects 2026-03-04 16:25:08 -06:00
alerts.ex add real-time GraphQL API with time-series queries and subscriptions 2026-03-12 10:24:39 -05:00
api_tokens.ex fix my-data page 2026-01-30 16:47:11 -06:00
application.ex fix: remove duplicate StormDetector child spec 2026-03-15 16:05:30 -05:00
billing.ex fix: infer device role during discovery and handle string price values 2026-03-10 13:25:51 -05:00
capacity.ex feat: add capacity insight worker, UI, and reporting views 2026-03-07 14:42:40 -06:00
config_changes.ex refactor: fix all credo strict issues, format all code, fix broken routes 2026-02-13 19:34:40 -06:00
dashboard.ex perf: optimize dashboard device count queries (partial N+1 fix) 2026-03-05 13:15:55 -06:00
devices.ex fix: dns and ping checks not executing in production (#71) 2026-03-18 08:43:37 -05:00
error_tracker_ignorer.ex Ignore pod shutdown errors in ErrorTracker 2026-02-17 10:59:51 -06:00
gaiia.ex improve Gaiia integration UX 2026-03-12 13:04:17 -05:00
geocoding.ex Add GeoIP import HTTP tests 2026-03-10 16:02:49 -05:00
geoip.ex remove Mix.env and support ip database import 2026-01-28 13:08:06 -06:00
honeybadger_filter.ex fix: suppress benign shutdown errors during K8s pod rollouts 2026-02-12 12:16:40 -06:00
http.ex Add GeoIP import HTTP tests 2026-03-10 16:02:49 -05:00
integrations.ex Store billing subscriber/MRR data on integrations for dashboard 2026-02-15 16:51:33 -06:00
job_monitoring.ex more tests and fixes 2026-02-07 11:50:18 -06:00
log_filter.ex Add comprehensive Dialyzer type specifications 2026-01-17 10:52:02 -06:00
logger_filters.ex fix: suppress benign shutdown errors during K8s pod rollouts 2026-02-12 12:16:40 -06:00
mailer.ex moduledocs 2026-01-17 15:08:49 -06:00
maintenance.ex fix: remove duplicate StormDetector child spec 2026-03-15 16:05:30 -05:00
mobile_sessions.ex Security fixes: mask credentials in logs/API, fix cookie/CSP/LIKE injection/webhooks 2026-02-15 09:09:04 -06:00
monitoring.ex fix: dns and ping checks not executing in production (#71) 2026-03-18 08:43:37 -05:00
numeric.ex perf: disable Req retry for faster error tests 2026-03-10 16:19:31 -05:00
on_call.ex fix: input validation, SSRF, API hardening, and cookie security 2026-03-14 14:48:59 -05:00
organizations.ex perf: LiveView event batching & PubSub optimization (TOW-26) 2026-03-15 14:08:27 -05:00
preseem.ex feat: add FluxCD image automation, fix Preseem duplicate AP crash 2026-02-14 10:30:12 -06:00
profiles.ex add universoal import 2026-01-18 16:59:34 -06:00
query_helpers.ex fix: input validation, SSRF, API hardening, and cookie security 2026-03-14 14:48:59 -05:00
rate_limit.ex update hammer 2026-02-08 10:08:31 -06:00
redis_health_check.ex fix: remove to_charlist conversion for Redix host option 2026-01-24 17:49:57 -06:00
release.ex fix dialyzer things 2026-01-03 12:56:40 -06:00
repo.ex moduledocs 2026-01-17 15:08:49 -06:00
reports.ex paperclip/TOW-21-uisp-integration-phase1 (#31) 2026-03-15 17:57:36 -05:00
result.ex more tests 2026-02-03 13:20:33 -06:00
rf_links.ex fix: make sidebar sticky with header and resolve all credo issues (#32) 2026-03-15 18:19:09 -05:00
search.ex refactor: fix all credo strict issues, format all code, fix broken routes 2026-02-13 19:34:40 -06:00
settings.ex remote agent/cloud improvements 2026-01-24 09:16:41 -06:00
sites.ex Prevent Oban polling/monitoring job stacking per device 2026-02-16 16:37:48 -06:00
snmp.ex fix: show ARP error samples in log message text instead of metadata 2026-03-10 15:12:09 -05:00
status_pages.ex paperclip/TOW-21-uisp-integration-phase1 (#31) 2026-03-15 17:57:36 -05:00
topology.ex fix: make sidebar sticky with header and resolve all credo issues (#32) 2026-03-15 18:19:09 -05:00
trace.ex feat: add site search to trace 2026-03-06 15:37:23 -06:00
url_validator.ex test: improve test coverage and code quality 2026-03-15 14:49:55 -05:00
vault.ex add mikrotik device handling 2026-02-01 16:42:39 -06:00
weather.ex Prevent Oban polling/monitoring job stacking per device 2026-02-16 16:37:48 -06:00