towerops/AGENT_NEXT_STEPS.md

# Agent System - Next Steps

This document outlines the remaining work to complete the remote SNMP polling agent system.

## Current Status

✅ **Complete**:
- Backend API with token authentication
- Agent management UI
- **Hierarchical agent assignment** (Equipment > Site > Organization)
- Organization/site/equipment-level configuration
- Rust agent architecture (compiles successfully)
- **SNMP client fully implemented** (GET and WALK operations)
- SQLite buffering
- Docker deployment files
- Protocol Buffers integration for all API endpoints
- 775 tests passing

## Critical Path to Production

### 1. Integration Testing (Priority: HIGH) ⬅️ **START HERE**

**Prerequisites**: ✅ All backend code complete

**Test Environment Setup**:
1. Deploy test SNMP device (or use simulator)
2. Deploy Phoenix backend
3. Deploy Rust agent via Docker
4. Create agent token via UI
5. Assign test equipment to agent

**Test Cases**:
- [ ] Agent connects and authenticates
- [ ] Agent fetches configuration
- [ ] Agent polls sensors successfully
- [ ] Agent polls interfaces successfully
- [ ] Metrics appear in Phoenix database
- [ ] Threshold violations trigger events
- [ ] Agent survives API outage (test 24h buffering)
- [ ] Agent reconnects after network issues
- [ ] Heartbeat updates agent status

**Duration**: 1-2 days

### 2. Load & Performance Testing (Priority: MEDIUM)

**Test Scenarios**:

**Scenario 1: Single Agent, Many Devices**
- 100 devices
- 500 sensors
- 200 interfaces
- 60-second poll interval
- Monitor: Memory, CPU, database growth

**Scenario 2: Many Agents**
- 10 agents
- 10 devices each
- Concurrent polling
- Monitor: API performance, database connections

**Scenario 3: Long-Running Stability**
- 1 agent
- 50 devices
- Run for 7 days
- Monitor: Memory leaks, database size, errors

**Duration**: 3-5 days

### 3. Production Readiness (Priority: MEDIUM)

**Docker Image**:
- [x] Build optimized image (11.8 MB)
- [ ] Publish to container registry
- [ ] Tag releases (v0.1.0, latest)
- [x] Multi-architecture support (amd64, arm64)

**Documentation**:
- [x] User guide with deployment methods (USER_GUIDE.md)
- [x] Troubleshooting guide
- [x] Network/firewall requirements
- [x] Upgrade procedure
- [x] Disaster recovery

**Monitoring**:
- [ ] Grafana dashboard for agent health
- [ ] Alerts for offline agents
- [ ] Metrics on agent performance
- [x] Database queries for agent statistics (lib/towerops/agents/stats.ex)

**Duration**: 2-3 days

## Optional Enhancements

### 4. Advanced Features (Priority: LOW)

**SNMPv3 Support**
- User authentication
- Privacy encryption
- Configuration UI changes

**Agent-Side Filtering**
- Threshold checks in agent
- Reduce bandwidth for high-sensor-count equipment
- Configurable sampling rates

**Equipment Polling Modes**
- Add `polling_mode` enum: `:server`, `:agent`, `:both`
- Allows gradual migration
- Enables A/B testing

**Agent Health Metrics**
- CPU/memory usage
- Metrics queue depth
- Polling success rate
- Average latency per device

## Quick Start Guide

### For Development

1. **Test Locally**
```bash
# Terminal 1: Start Phoenix
mix phx.server

# Terminal 2: Start agent
cd towerops-agent
cargo run -- \
  --api-url http://localhost:4000 \
  --token <your-test-token>
```

2. **Create Test Agent**
- Visit http://localhost:4000/orgs/:slug/agents
- Create agent, copy token
- Use token in step 1

### For Production Deployment

1. **Build Docker Image**
```bash
cd towerops-agent
docker build -t towerops/agent:0.1.0 .
docker tag towerops/agent:0.1.0 towerops/agent:latest
```

2. **Push to Registry**
```bash
docker push towerops/agent:0.1.0
docker push towerops/agent:latest
```

3. **Deploy to Customer**
```bash
# Provide customer with docker-compose.yml
# They run:
docker-compose up -d
```

## Estimated Timeline

| Phase | Effort | Duration |
|-------|--------|----------|
| ✅ ~~SNMP Integration~~ | ~~4-8 hours~~ | ~~1 day~~ |
| ✅ ~~Hierarchical Agent Assignment~~ | ~~12 hours~~ | ~~1.5 days~~ |
| Integration Testing | 16 hours | 2 days |
| Load Testing | 24 hours | 3 days |
| Production Prep | 16 hours | 2 days |
| **Remaining** | **56 hours** | **7 days** |

## Success Criteria

The agent system is ready for production when:

- [x] Agent authenticates with token successfully ✅
- [x] Agent fetches configuration from API ✅
- [x] Agent polls SNMP devices (sensors + interfaces) ✅
- [x] Metrics appear in database within 60 seconds ✅
- [ ] Threshold violations trigger events ⚠️ (requires integration testing)
- [x] Agent survives 24h API outage without data loss ✅ (SQLite buffering, needs verification)
- [x] UI shows agent status (online/offline) ✅
- [x] Token revocation works immediately ✅
- [ ] Agent uses <256 MB memory with 50 devices ⚠️ (needs load testing)
- [x] Docker image is <50 MB ✅ (11.8 MB)
- [ ] Load test: 100 devices, 500 sensors, 200 interfaces ⚠️ (manual testing required)
- [ ] Stability test: 7 days continuous operation ⚠️ (manual testing required)

## Questions & Decisions Needed

1. **SNMP Library Choice**: Stick with `snmp` v0.2 or switch?
2. **Release Strategy**: Beta testing with select customers first?
3. **Support Plan**: How will customers get help with agent deployment?
4. **Monitoring**: What metrics should we track about agent usage?
5. **Pricing**: Does remote agent feature affect pricing tiers?

## Resources

- **Implementation Doc**: `/Users/graham/dev/towerops/AGENT_IMPLEMENTATION.md`
- **Agent README**: `/Users/graham/dev/towerops/towerops-agent/README.md`
- **SNMP Crate Docs**: https://docs.rs/snmp/0.2.2/snmp/
- **Original Plan**: `/Users/graham/.claude/plans/melodic-coalescing-truffle.md`

## Contact

For questions about this implementation:
- Review AGENT_IMPLEMENTATION.md for architecture details
- Check git history for context on specific changes
- All code follows project conventions in CLAUDE.md