# Job Monitoring Dashboard

## Overview

The Job Monitoring Dashboard provides real-time visibility into Oban worker operations, helping administrators monitor polling and discovery jobs across all devices. This feature enables proactive problem detection and performance optimization of the SNMP polling infrastructure.

**Route:** `/admin/monitoring` (requires superuser access)

## Features

### 1. Active Operations

- **Real-time job execution tracking**: See which devices are currently being polled or discovered
- **Visual indicators**: Animated spinner icons show active operations
- **Device context**: Jobs are displayed with device names and worker types
- **Execution time**: Shows how long each job has been running

### 2. Problems Detection

Automatically identifies and highlights problematic jobs:

**Stuck Jobs**
- Jobs executing for more than 5 minutes
- Displayed with warning indicators
- Shows execution duration in human-readable format

**Failed Jobs**
- Recent job failures (last 24 hours)
- Displays error messages when available
- Shows attempt count (e.g., "Attempt 3/3")

### 3. Health Metrics

**Current Activity**
- Executing jobs count
- Queued jobs count
- Completed jobs (last hour)
- Failed jobs (last hour)

**Success Rates**
- Polling success rate (last hour) with visual progress bar
- Discovery success rate (last hour) with visual progress bar

**Performance Metrics**
- Average execution time
- Jobs per hour throughput
- Queue depths by queue (pollers, discovery)

### 4. Recent Activity Timeline

- Chronological view of recent job completions
- Color-coded by outcome (success/failure)
- Shows device names and worker types
- Displays relative timestamps ("2 minutes ago")
- Auto-scrolling with max 50 events

### 5. Real-time Updates

- PubSub-based event stream
- Updates when jobs start, complete, or fail
- Metrics refresh every 10 seconds
- No manual refresh required

## Usage

### Accessing the Dashboard

1. Log in as a superuser
2. Navigate to `/admin/monitoring` or click "Job Monitoring" from the Admin Dashboard
3. Dashboard loads with current system state

### Interpreting Metrics

**Healthy System Indicators:**
- High success rates (>95%)
- Low queue depths (<10 per queue)
- No stuck jobs
- Average execution time <5 seconds

**Warning Signs:**
- Success rates <90%
- Growing queue depths
- Stuck jobs present
- Increasing average execution time

### Problem Investigation

When problems are detected:

1. **Stuck Jobs**: Check device connectivity, SNMP configuration, or system resources
2. **Failed Jobs**: Review error messages for:
   - SNMP timeout errors
   - Authentication failures
   - Device unreachable errors
   - Configuration issues

3. **High Queue Depths**: May indicate:
   - Too many devices for current polling capacity
   - Slow SNMP responses
   - System resource constraints

## Technical Details

### Architecture

**Context Layer:** `Towerops.JobMonitoring`
- Provides query functions for job states
- Aggregates metrics from Oban tables
- Handles stuck job detection logic

**Events Layer:** `Towerops.JobMonitoring.Events`
- PubSub-based event broadcasting
- Events: `:started`, `:completed`, `:failed`
- Topic: `"job:lifecycle"`

**Metrics Layer:** `Towerops.JobMonitoring.Metrics`
- Calculates success rates
- Aggregates execution times
- Queries queue depths

**LiveView:** `ToweropsWeb.Admin.MonitoringLive`
- Real-time UI updates
- Subscribes to job lifecycle events
- Polls for metric updates

### Database Queries

**Active Jobs:**
```sql
SELECT * FROM oban_jobs
WHERE state = 'executing'
ORDER BY attempted_at ASC
```

**Stuck Jobs:**
```sql
SELECT * FROM oban_jobs
WHERE state = 'executing'
  AND attempted_at < NOW() - INTERVAL '5 minutes'
```

**Failed Jobs:**
```sql
SELECT * FROM oban_jobs
WHERE state IN ('retryable', 'discarded')
  AND updated_at > NOW() - INTERVAL '24 hours'
ORDER BY updated_at DESC
```

**Success Rate (1 hour):**
```sql
SELECT
  COUNT(*) FILTER (WHERE state = 'completed') AS completed,
  COUNT(*) AS total
FROM oban_jobs
WHERE completed_at > NOW() - INTERVAL '1 hour'
  AND worker LIKE 'Towerops.Workers.%'
```

### Event Flow

1. **Job Start**
   - Worker calls `Events.broadcast_job_event(job, :started)`
   - Event includes: `job_id`, `worker`, `device_id`
   - LiveView adds to `@active_jobs`

2. **Job Completion**
   - Worker calls `Events.broadcast_job_event(job, :completed, metadata)`
   - Metadata includes execution duration
   - LiveView removes from `@active_jobs`, reloads data

3. **Job Failure**
   - Worker calls `Events.broadcast_job_event(job, :failed, metadata)`
   - Metadata includes error and duration
   - LiveView reloads data to show in failed jobs list

### Worker Integration

**DevicePollerWorker:**
```elixir
def perform(%Job{} = job) do
  Events.broadcast_job_event(job, :started)
  start_time = System.monotonic_time(:second)

  result = poll_device(device)

  duration = System.monotonic_time(:second) - start_time
  Events.broadcast_job_event(job, :completed, %{duration: duration})

  result
end
```

**DiscoveryWorker:**
- Similar event broadcasting pattern
- Handles both `:ok` and `:discard` results
- Broadcasts completion event for both outcomes

## Performance Considerations

### Query Optimization

- All queries include appropriate indexes
- Active jobs query is fast (filtered on `state = 'executing'`)
- Stuck jobs query uses timestamp comparison
- Recent jobs limited to 24-hour window

### LiveView Updates

- Metrics refresh on 10-second timer
- PubSub events processed asynchronously
- Only active jobs updated on events
- Full data reload on job completion/failure

### Scalability

- Dashboard supports monitoring 1000+ devices
- PubSub events are lightweight (no device data in payload)
- Metric calculations use database aggregations
- Recent activity limited to 50 events in memory

## Future Enhancements

### Planned Features

1. **Job Cancellation**: Cancel stuck jobs directly from dashboard
2. **Historical Charts**: Visualize success rates and throughput over time
3. **Alerting**: Email/webhook notifications for stuck jobs
4. **Device Filtering**: Filter dashboard by organization or site
5. **Job Details**: Click job to see full args, errors, and stack traces
6. **Queue Management**: Pause/resume queues, adjust concurrency
7. **Performance Trends**: Track p50/p95/p99 execution times

### Extensibility

The monitoring system is designed for extension:

- Add new worker types by updating `worker_name/1` helper
- Extend metrics by adding queries to `Metrics` module
- Add custom event types by updating `Events.broadcast_job_event/3`
- Create new problem detection rules in `JobMonitoring` context

## Related Documentation

- [Oban Documentation](https://hexdocs.pm/oban/Oban.html)
- [Phoenix PubSub](https://hexdocs.pm/phoenix_pubsub/Phoenix.PubSub.html)
- [LiveView Real-time Updates](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html)
- [SNMP Polling Architecture](./polling-architecture.md) (if exists)
