docs: add job monitoring dashboard feature documentation
Comprehensive documentation covering architecture, usage, technical details, and future enhancements for the job monitoring dashboard. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
137d7c83ec
commit
9ed26eaad1
1 changed files with 248 additions and 0 deletions
248
docs/features/job-monitoring-dashboard.md
Normal file
248
docs/features/job-monitoring-dashboard.md
Normal file
|
|
@ -0,0 +1,248 @@
|
|||
# Job Monitoring Dashboard
|
||||
|
||||
## Overview
|
||||
|
||||
The Job Monitoring Dashboard provides real-time visibility into Oban worker operations, helping administrators monitor polling and discovery jobs across all devices. This feature enables proactive problem detection and performance optimization of the SNMP polling infrastructure.
|
||||
|
||||
**Route:** `/admin/monitoring` (requires superuser access)
|
||||
|
||||
## Features
|
||||
|
||||
### 1. Active Operations
|
||||
|
||||
- **Real-time job execution tracking**: See which devices are currently being polled or discovered
|
||||
- **Visual indicators**: Animated spinner icons show active operations
|
||||
- **Device context**: Jobs are displayed with device names and worker types
|
||||
- **Execution time**: Shows how long each job has been running
|
||||
|
||||
### 2. Problems Detection
|
||||
|
||||
Automatically identifies and highlights problematic jobs:
|
||||
|
||||
**Stuck Jobs**
|
||||
- Jobs executing for more than 5 minutes
|
||||
- Displayed with warning indicators
|
||||
- Shows execution duration in human-readable format
|
||||
|
||||
**Failed Jobs**
|
||||
- Recent job failures (last 24 hours)
|
||||
- Displays error messages when available
|
||||
- Shows attempt count (e.g., "Attempt 3/3")
|
||||
|
||||
### 3. Health Metrics
|
||||
|
||||
**Current Activity**
|
||||
- Executing jobs count
|
||||
- Queued jobs count
|
||||
- Completed jobs (last hour)
|
||||
- Failed jobs (last hour)
|
||||
|
||||
**Success Rates**
|
||||
- Polling success rate (last hour) with visual progress bar
|
||||
- Discovery success rate (last hour) with visual progress bar
|
||||
|
||||
**Performance Metrics**
|
||||
- Average execution time
|
||||
- Jobs per hour throughput
|
||||
- Queue depths by queue (pollers, discovery)
|
||||
|
||||
### 4. Recent Activity Timeline
|
||||
|
||||
- Chronological view of recent job completions
|
||||
- Color-coded by outcome (success/failure)
|
||||
- Shows device names and worker types
|
||||
- Displays relative timestamps ("2 minutes ago")
|
||||
- Auto-scrolling with max 50 events
|
||||
|
||||
### 5. Real-time Updates
|
||||
|
||||
- PubSub-based event stream
|
||||
- Updates when jobs start, complete, or fail
|
||||
- Metrics refresh every 10 seconds
|
||||
- No manual refresh required
|
||||
|
||||
## Usage
|
||||
|
||||
### Accessing the Dashboard
|
||||
|
||||
1. Log in as a superuser
|
||||
2. Navigate to `/admin/monitoring` or click "Job Monitoring" from the Admin Dashboard
|
||||
3. Dashboard loads with current system state
|
||||
|
||||
### Interpreting Metrics
|
||||
|
||||
**Healthy System Indicators:**
|
||||
- High success rates (>95%)
|
||||
- Low queue depths (<10 per queue)
|
||||
- No stuck jobs
|
||||
- Average execution time <5 seconds
|
||||
|
||||
**Warning Signs:**
|
||||
- Success rates <90%
|
||||
- Growing queue depths
|
||||
- Stuck jobs present
|
||||
- Increasing average execution time
|
||||
|
||||
### Problem Investigation
|
||||
|
||||
When problems are detected:
|
||||
|
||||
1. **Stuck Jobs**: Check device connectivity, SNMP configuration, or system resources
|
||||
2. **Failed Jobs**: Review error messages for:
|
||||
- SNMP timeout errors
|
||||
- Authentication failures
|
||||
- Device unreachable errors
|
||||
- Configuration issues
|
||||
|
||||
3. **High Queue Depths**: May indicate:
|
||||
- Too many devices for current polling capacity
|
||||
- Slow SNMP responses
|
||||
- System resource constraints
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Architecture
|
||||
|
||||
**Context Layer:** `Towerops.JobMonitoring`
|
||||
- Provides query functions for job states
|
||||
- Aggregates metrics from Oban tables
|
||||
- Handles stuck job detection logic
|
||||
|
||||
**Events Layer:** `Towerops.JobMonitoring.Events`
|
||||
- PubSub-based event broadcasting
|
||||
- Events: `:started`, `:completed`, `:failed`
|
||||
- Topic: `"job:lifecycle"`
|
||||
|
||||
**Metrics Layer:** `Towerops.JobMonitoring.Metrics`
|
||||
- Calculates success rates
|
||||
- Aggregates execution times
|
||||
- Queries queue depths
|
||||
|
||||
**LiveView:** `ToweropsWeb.Admin.MonitoringLive`
|
||||
- Real-time UI updates
|
||||
- Subscribes to job lifecycle events
|
||||
- Polls for metric updates
|
||||
|
||||
### Database Queries
|
||||
|
||||
**Active Jobs:**
|
||||
```sql
|
||||
SELECT * FROM oban_jobs
|
||||
WHERE state = 'executing'
|
||||
ORDER BY attempted_at ASC
|
||||
```
|
||||
|
||||
**Stuck Jobs:**
|
||||
```sql
|
||||
SELECT * FROM oban_jobs
|
||||
WHERE state = 'executing'
|
||||
AND attempted_at < NOW() - INTERVAL '5 minutes'
|
||||
```
|
||||
|
||||
**Failed Jobs:**
|
||||
```sql
|
||||
SELECT * FROM oban_jobs
|
||||
WHERE state IN ('retryable', 'discarded')
|
||||
AND updated_at > NOW() - INTERVAL '24 hours'
|
||||
ORDER BY updated_at DESC
|
||||
```
|
||||
|
||||
**Success Rate (1 hour):**
|
||||
```sql
|
||||
SELECT
|
||||
COUNT(*) FILTER (WHERE state = 'completed') AS completed,
|
||||
COUNT(*) AS total
|
||||
FROM oban_jobs
|
||||
WHERE completed_at > NOW() - INTERVAL '1 hour'
|
||||
AND worker LIKE 'Towerops.Workers.%'
|
||||
```
|
||||
|
||||
### Event Flow
|
||||
|
||||
1. **Job Start**
|
||||
- Worker calls `Events.broadcast_job_event(job, :started)`
|
||||
- Event includes: `job_id`, `worker`, `device_id`
|
||||
- LiveView adds to `@active_jobs`
|
||||
|
||||
2. **Job Completion**
|
||||
- Worker calls `Events.broadcast_job_event(job, :completed, metadata)`
|
||||
- Metadata includes execution duration
|
||||
- LiveView removes from `@active_jobs`, reloads data
|
||||
|
||||
3. **Job Failure**
|
||||
- Worker calls `Events.broadcast_job_event(job, :failed, metadata)`
|
||||
- Metadata includes error and duration
|
||||
- LiveView reloads data to show in failed jobs list
|
||||
|
||||
### Worker Integration
|
||||
|
||||
**DevicePollerWorker:**
|
||||
```elixir
|
||||
def perform(%Job{} = job) do
|
||||
Events.broadcast_job_event(job, :started)
|
||||
start_time = System.monotonic_time(:second)
|
||||
|
||||
result = poll_device(device)
|
||||
|
||||
duration = System.monotonic_time(:second) - start_time
|
||||
Events.broadcast_job_event(job, :completed, %{duration: duration})
|
||||
|
||||
result
|
||||
end
|
||||
```
|
||||
|
||||
**DiscoveryWorker:**
|
||||
- Similar event broadcasting pattern
|
||||
- Handles both `:ok` and `:discard` results
|
||||
- Broadcasts completion event for both outcomes
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Query Optimization
|
||||
|
||||
- All queries include appropriate indexes
|
||||
- Active jobs query is fast (filtered on `state = 'executing'`)
|
||||
- Stuck jobs query uses timestamp comparison
|
||||
- Recent jobs limited to 24-hour window
|
||||
|
||||
### LiveView Updates
|
||||
|
||||
- Metrics refresh on 10-second timer
|
||||
- PubSub events processed asynchronously
|
||||
- Only active jobs updated on events
|
||||
- Full data reload on job completion/failure
|
||||
|
||||
### Scalability
|
||||
|
||||
- Dashboard supports monitoring 1000+ devices
|
||||
- PubSub events are lightweight (no device data in payload)
|
||||
- Metric calculations use database aggregations
|
||||
- Recent activity limited to 50 events in memory
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Features
|
||||
|
||||
1. **Job Cancellation**: Cancel stuck jobs directly from dashboard
|
||||
2. **Historical Charts**: Visualize success rates and throughput over time
|
||||
3. **Alerting**: Email/webhook notifications for stuck jobs
|
||||
4. **Device Filtering**: Filter dashboard by organization or site
|
||||
5. **Job Details**: Click job to see full args, errors, and stack traces
|
||||
6. **Queue Management**: Pause/resume queues, adjust concurrency
|
||||
7. **Performance Trends**: Track p50/p95/p99 execution times
|
||||
|
||||
### Extensibility
|
||||
|
||||
The monitoring system is designed for extension:
|
||||
|
||||
- Add new worker types by updating `worker_name/1` helper
|
||||
- Extend metrics by adding queries to `Metrics` module
|
||||
- Add custom event types by updating `Events.broadcast_job_event/3`
|
||||
- Create new problem detection rules in `JobMonitoring` context
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Oban Documentation](https://hexdocs.pm/oban/Oban.html)
|
||||
- [Phoenix PubSub](https://hexdocs.pm/phoenix_pubsub/Phoenix.PubSub.html)
|
||||
- [LiveView Real-time Updates](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html)
|
||||
- [SNMP Polling Architecture](./polling-architecture.md) (if exists)
|
||||
Loading…
Add table
Reference in a new issue