diff --git a/docs/features/job-monitoring-dashboard.md b/docs/features/job-monitoring-dashboard.md new file mode 100644 index 00000000..98664e6a --- /dev/null +++ b/docs/features/job-monitoring-dashboard.md @@ -0,0 +1,248 @@ +# Job Monitoring Dashboard + +## Overview + +The Job Monitoring Dashboard provides real-time visibility into Oban worker operations, helping administrators monitor polling and discovery jobs across all devices. This feature enables proactive problem detection and performance optimization of the SNMP polling infrastructure. + +**Route:** `/admin/monitoring` (requires superuser access) + +## Features + +### 1. Active Operations + +- **Real-time job execution tracking**: See which devices are currently being polled or discovered +- **Visual indicators**: Animated spinner icons show active operations +- **Device context**: Jobs are displayed with device names and worker types +- **Execution time**: Shows how long each job has been running + +### 2. Problems Detection + +Automatically identifies and highlights problematic jobs: + +**Stuck Jobs** +- Jobs executing for more than 5 minutes +- Displayed with warning indicators +- Shows execution duration in human-readable format + +**Failed Jobs** +- Recent job failures (last 24 hours) +- Displays error messages when available +- Shows attempt count (e.g., "Attempt 3/3") + +### 3. Health Metrics + +**Current Activity** +- Executing jobs count +- Queued jobs count +- Completed jobs (last hour) +- Failed jobs (last hour) + +**Success Rates** +- Polling success rate (last hour) with visual progress bar +- Discovery success rate (last hour) with visual progress bar + +**Performance Metrics** +- Average execution time +- Jobs per hour throughput +- Queue depths by queue (pollers, discovery) + +### 4. Recent Activity Timeline + +- Chronological view of recent job completions +- Color-coded by outcome (success/failure) +- Shows device names and worker types +- Displays relative timestamps ("2 minutes ago") +- Auto-scrolling with max 50 events + +### 5. Real-time Updates + +- PubSub-based event stream +- Updates when jobs start, complete, or fail +- Metrics refresh every 10 seconds +- No manual refresh required + +## Usage + +### Accessing the Dashboard + +1. Log in as a superuser +2. Navigate to `/admin/monitoring` or click "Job Monitoring" from the Admin Dashboard +3. Dashboard loads with current system state + +### Interpreting Metrics + +**Healthy System Indicators:** +- High success rates (>95%) +- Low queue depths (<10 per queue) +- No stuck jobs +- Average execution time <5 seconds + +**Warning Signs:** +- Success rates <90% +- Growing queue depths +- Stuck jobs present +- Increasing average execution time + +### Problem Investigation + +When problems are detected: + +1. **Stuck Jobs**: Check device connectivity, SNMP configuration, or system resources +2. **Failed Jobs**: Review error messages for: + - SNMP timeout errors + - Authentication failures + - Device unreachable errors + - Configuration issues + +3. **High Queue Depths**: May indicate: + - Too many devices for current polling capacity + - Slow SNMP responses + - System resource constraints + +## Technical Details + +### Architecture + +**Context Layer:** `Towerops.JobMonitoring` +- Provides query functions for job states +- Aggregates metrics from Oban tables +- Handles stuck job detection logic + +**Events Layer:** `Towerops.JobMonitoring.Events` +- PubSub-based event broadcasting +- Events: `:started`, `:completed`, `:failed` +- Topic: `"job:lifecycle"` + +**Metrics Layer:** `Towerops.JobMonitoring.Metrics` +- Calculates success rates +- Aggregates execution times +- Queries queue depths + +**LiveView:** `ToweropsWeb.Admin.MonitoringLive` +- Real-time UI updates +- Subscribes to job lifecycle events +- Polls for metric updates + +### Database Queries + +**Active Jobs:** +```sql +SELECT * FROM oban_jobs +WHERE state = 'executing' +ORDER BY attempted_at ASC +``` + +**Stuck Jobs:** +```sql +SELECT * FROM oban_jobs +WHERE state = 'executing' + AND attempted_at < NOW() - INTERVAL '5 minutes' +``` + +**Failed Jobs:** +```sql +SELECT * FROM oban_jobs +WHERE state IN ('retryable', 'discarded') + AND updated_at > NOW() - INTERVAL '24 hours' +ORDER BY updated_at DESC +``` + +**Success Rate (1 hour):** +```sql +SELECT + COUNT(*) FILTER (WHERE state = 'completed') AS completed, + COUNT(*) AS total +FROM oban_jobs +WHERE completed_at > NOW() - INTERVAL '1 hour' + AND worker LIKE 'Towerops.Workers.%' +``` + +### Event Flow + +1. **Job Start** + - Worker calls `Events.broadcast_job_event(job, :started)` + - Event includes: `job_id`, `worker`, `device_id` + - LiveView adds to `@active_jobs` + +2. **Job Completion** + - Worker calls `Events.broadcast_job_event(job, :completed, metadata)` + - Metadata includes execution duration + - LiveView removes from `@active_jobs`, reloads data + +3. **Job Failure** + - Worker calls `Events.broadcast_job_event(job, :failed, metadata)` + - Metadata includes error and duration + - LiveView reloads data to show in failed jobs list + +### Worker Integration + +**DevicePollerWorker:** +```elixir +def perform(%Job{} = job) do + Events.broadcast_job_event(job, :started) + start_time = System.monotonic_time(:second) + + result = poll_device(device) + + duration = System.monotonic_time(:second) - start_time + Events.broadcast_job_event(job, :completed, %{duration: duration}) + + result +end +``` + +**DiscoveryWorker:** +- Similar event broadcasting pattern +- Handles both `:ok` and `:discard` results +- Broadcasts completion event for both outcomes + +## Performance Considerations + +### Query Optimization + +- All queries include appropriate indexes +- Active jobs query is fast (filtered on `state = 'executing'`) +- Stuck jobs query uses timestamp comparison +- Recent jobs limited to 24-hour window + +### LiveView Updates + +- Metrics refresh on 10-second timer +- PubSub events processed asynchronously +- Only active jobs updated on events +- Full data reload on job completion/failure + +### Scalability + +- Dashboard supports monitoring 1000+ devices +- PubSub events are lightweight (no device data in payload) +- Metric calculations use database aggregations +- Recent activity limited to 50 events in memory + +## Future Enhancements + +### Planned Features + +1. **Job Cancellation**: Cancel stuck jobs directly from dashboard +2. **Historical Charts**: Visualize success rates and throughput over time +3. **Alerting**: Email/webhook notifications for stuck jobs +4. **Device Filtering**: Filter dashboard by organization or site +5. **Job Details**: Click job to see full args, errors, and stack traces +6. **Queue Management**: Pause/resume queues, adjust concurrency +7. **Performance Trends**: Track p50/p95/p99 execution times + +### Extensibility + +The monitoring system is designed for extension: + +- Add new worker types by updating `worker_name/1` helper +- Extend metrics by adding queries to `Metrics` module +- Add custom event types by updating `Events.broadcast_job_event/3` +- Create new problem detection rules in `JobMonitoring` context + +## Related Documentation + +- [Oban Documentation](https://hexdocs.pm/oban/Oban.html) +- [Phoenix PubSub](https://hexdocs.pm/phoenix_pubsub/Phoenix.PubSub.html) +- [LiveView Real-time Updates](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html) +- [SNMP Polling Architecture](./polling-architecture.md) (if exists)