Commit graph

288 commits

Author SHA1 Message Date
ba1e8933e4
Add hierarchical SNMP configuration with org/site/equipment fallback
Implements a hierarchical SNMP community configuration system where settings
can be defined at the organization level (global default), overridden at the
site level, or overridden at the equipment level.

Changes:
- Migration: Add snmp_version and snmp_community to organizations and sites
- Organization schema: Add SNMP fields with default version "2c"
- Site schema: Add SNMP fields that override organization defaults
- Equipment context: Add get_snmp_config/1 with hierarchical fallback logic
- Discovery: Use Equipment.get_snmp_config instead of equipment.snmp_community
- PollerWorker: Use Equipment.get_snmp_config instead of equipment.snmp_community

The fallback order is:
1. Equipment-level (highest priority)
2. Site-level
3. Organization-level (default)

All 785 tests passing.
2026-01-17 11:47:17 -06:00
de002f9903
Cap storage/memory usage sensors at 100% to handle device firmware bugs
Fixes an issue where storage usage was reporting values over 200,000% due to
incorrect SNMP values from the device. This can occur when:
- Device firmware has bugs in hrStorageUsed/hrStorageSize reporting
- SNMP allocation units differ between OIDs
- Values are read during device reconfiguration

Changes:
- PollerWorker: Cap percentage sensors at 100%, log warning with raw values
- Mikrotik profile: Cap storage usage at 100% during discovery
- NetSnmp profile: Cap memory usage at 100% during discovery

The raw percentage is logged before capping to help diagnose device issues.
All percentage sensors now have consistent validation across polling and discovery.
2026-01-17 11:36:37 -06:00
79dabd840b
wording 2026-01-17 11:21:18 -06:00
07b0ddb67d
Set DEPLOY_TIMESTAMP from GitLab CI instead of pod metadata
- Update GitLab CI deploy job to set DEPLOY_TIMESTAMP when deploying
- Use 'kubectl set env' to update all pods with same timestamp
- Change deployment.yaml to use static placeholder value
- Update documentation to explain GitLab CI timestamp approach

This ensures all pods show the same deployment time, regardless of:
- Which pod handles the request (with replicas=2)
- When individual pods were created or restarted
- Pod restarts from crashes or rolling updates

The timestamp now represents when the deployment was initiated by GitLab CI,
not when individual pods were created.
2026-01-17 11:17:07 -06:00
9fcf19b304
Use Kubernetes pod creation timestamp for deploy time in footer
- Read DEPLOY_TIMESTAMP environment variable for actual deploy time
- Fall back to compile time in development when env var not set
- Add Kubernetes Downward API to inject metadata.creationTimestamp
- Update k8s/deployment.yaml to set DEPLOY_TIMESTAMP env var
- Document deployment timestamp mechanism in k8s/README.md

This ensures the footer shows when the pod was deployed to the cluster,
not when the Docker image was built in CI/CD.
2026-01-17 11:14:15 -06:00
ae16acf016
Add build timestamp to footer with shared time formatting helpers
- Add compile-time @build_timestamp to Application module
- Create ToweropsWeb.TimeHelpers module for shared time formatting
- Update footer to display 'Last deployed Xm ago · YYYY-MM-DD HH:MM:SS UTC'
- Refactor AgentLive.Helpers to delegate to shared TimeHelpers
- Consolidate duplicate time formatting logic into single module
- Timestamp updates automatically on each build/deploy
2026-01-17 11:10:50 -06:00
654fcb541c
Add footer with dynamic copyright year to all layouts
- Add footer component to app, authenticated, and admin layouts
- Update marketing layout footer to use dynamic year
- Use flexbox layout to push footer to bottom of viewport
- Footer displays 'Copyright © <year> Towerops'
- Year updates automatically using Date.utc_today().year
2026-01-17 11:05:39 -06:00
b53a53b199
Add comprehensive Dialyzer type specifications
- Add @type definitions to all schema modules:
  - User, UserCredential, Membership, Invitation
  - Organization, Site, AgentAssignment
  - InterfaceStat, SensorReading, Alert
- Fix all compilation warnings with stronger Elixir types:
  - Remove unused require Logger in log_filter.ex
  - Remove unused parse_float(nil) clause
  - Add pin operators (^) for variables in binary pattern matches
- Fix Dialyzer errors (25 → 0):
  - Remove unreachable pattern match clauses
  - Fix unmatched return values with _ = prefix
  - Update @spec for deliver_alert_notification/1
- Properly handle all Phoenix.PubSub.subscribe and Task.start return values
- Explicitly ignore if statement return values where needed

All files now pass mix compile --warnings-as-errors and mix dialyzer.
2026-01-17 10:52:02 -06:00
02c163986e
Fix 21 Dialyzer errors by adding schema types and fixing unmatched returns
Schema types added:
- AgentToken.t/0
- Equipment.t/0
- Device.t/0
- Interface.t/0
- Sensor.t/0

Fixed unmatched returns in AgentChannel:
- Agents.update_agent_token_heartbeat/3 (2 instances)
- process_snmp_result/2

Reduced Dialyzer errors from 48 to 27.
2026-01-17 10:40:47 -06:00
e1b9976219
Add Dialyzer typespecs to WebSocket agent code and Agents context
- Add @spec to AgentSocket callbacks (connect/3, id/1)
- Add @spec to AgentChannel callbacks (join/3, handle_info/2, handle_in/3)
- Add @spec to AgentChannel private function build_jobs_for_agent/1
- Add @spec to Agents.verify_agent_token/1
- Add @spec to Agents.update_agent_token_heartbeat/3
- Add @spec to Agents.list_agent_polling_targets/1

These specs provide strict type checking for the critical WebSocket
agent communication code implemented in this session.
2026-01-17 10:37:27 -06:00
bdde71462a
Fix memory/CPU chart fill: explicitly disable area fill for non-traffic charts
- Set fill: false for CPU, memory, and storage charts
- Prevents Chart.js from auto-filling when backgroundColor is set
- Only traffic charts should have filled area (fill: 'origin')
- Fixes visual bug where 20% memory usage appeared as 100% fill
2026-01-17 10:28:52 -06:00
5b62910bae
Remove unused module attributes and alias in Ubiquiti profile
- Remove unused Discovery alias
- Remove unused @radio_tx_capacity and @radio_rx_capacity attributes
- Remove unused @radio_uptime_pct, @radio_firmware, @radio_status
- Remove unused @radio_remote_mac and @radio_remote_ip attributes
- Fixes all compile warnings
2026-01-17 10:20:19 -06:00
305b22dc65
Fix organization_id access: use equipment.site.organization_id 2026-01-16 19:21:58 -06:00
6b7a3ba36e
Fix binary protobuf encoding for JSON serializer
- Use base64-encoded binary in JSON payload instead of {:binary, data} tuples
- V1.JSONSerializer cannot encode tuples, needs plain JSON objects
- Agent already sends/expects base64-encoded binary in 'binary' key
2026-01-16 18:36:24 -06:00
9b88a32f33
Move token authentication from URL to Phoenix channel join payload 2026-01-16 18:22:57 -06:00
538f582fec
Update agent setup instructions with Watchtower auto-updates
- Change default agent image to gmcintire/towerops-agent:main
- Add Watchtower service to docker-compose instructions
- Configure 15-minute update check interval for development
- Add DOCKER_API_VERSION=1.44 to prevent API compatibility errors
- Remove deprecated docker-compose version field
2026-01-16 17:38:30 -06:00
eb91798729
agent improvements and agent config 2026-01-16 17:26:51 -06:00
a7ed057976
neighbor improvements 2026-01-16 16:25:49 -06:00
dcd4588304
fix neighbors 2026-01-16 16:09:15 -06:00
6627235981
add tests and add snmp neighbor discovery 2026-01-16 13:44:29 -06:00
3539e36f12
updates 2026-01-16 13:15:59 -06:00
6908661bd1
Fix mobile API organizations endpoint
Fixed bug where list_user_organizations was being called with user struct
instead of user_id, causing 400 Bad Request errors.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-15 16:54:28 -06:00
7068ab2466
Convert User Settings from controller to LiveView
- Create UserSettingsLive to replace UserSettingsController
- Convert email/password forms to LiveView with proper form handling
- Add mobile session management (toggle alerts, revoke devices)
- Add require_sudo_mode on_mount hook to UserAuth
- Create dedicated live_session for sudo mode routes in router
- Keep UserSettingsController for email confirmation via token only
- Add MobileSessions.get_session/1 function for test support
- Update tests to match LiveView behavior (password update redirects to login)
- Remove old controller tests for email/password updates (now in LiveView)
2026-01-15 16:29:50 -06:00
1786198c11
Fix datetime truncation for mobile session schemas
All datetime fields in mobile session schemas must truncate to :second
because the database uses :utc_datetime type (without microseconds).

Fixed DateTime.utc_now() calls in:
- QRLoginToken.complete_changeset/2
- QRLoginToken.put_expiration/1
- MobileSession.touch_changeset/1
- MobileSession.put_timestamps/1
2026-01-15 16:17:05 -06:00
0ea2addc91
Fix Bandit configuration and user settings HTML test
Bandit 1.10.1 does not support read_timeout/write_timeout in http_1_options
or stream_idle_timeout in http_2_options. These invalid options caused the
application to crash on startup with 'Unsupported key(s) in http_1_options'.

Moved read_timeout to thousand_island_options where it belongs, and removed
the unsupported write_timeout and stream_idle_timeout options.

Also fixed UserSettingsHTMLTest to include the missing mobile_sessions assign
that is required by the edit.html.heex template.
2026-01-15 16:06:01 -06:00
c7f02eac24
add api for mobile login 2026-01-15 15:36:14 -06:00
0a410ecf81
Show state transitions in equipment event logs
Update interface operational status event messages to show both the
previous state and new state (e.g., 'Interface eth0 changed from DOWN
to UP' instead of just 'Interface eth0 is now up'). This provides
better context in the event logs.

Other event types already showed transitions:
- Admin status changes
- Speed changes
- MAC address changes
- Sensor threshold events
- Sensor spikes/drops
2026-01-15 13:35:50 -06:00
85abdedf8b
Update table borders to use lighter grey instead of black
Replace black ring border with lighter zinc-200/zinc-700 border on all
tables for better visual consistency across the application. Also update
tbody dividers to match the new border styling.
2026-01-15 13:32:09 -06:00
ab6ecbdfdc
Convert JavaScript to TypeScript with proper type definitions
- Add comprehensive type definitions for vendor libraries (Chart.js, topbar)
- Add TypeScript types for LiveView hooks and WebAuthn interfaces
- Convert all JS files to TypeScript with proper typing:
  - app.js → app.ts
  - webauthn.js → webauthn.ts
  - webauthn_utils.js → webauthn_utils.ts
  - passkey_settings.js → passkey_settings.ts
  - passkey_login.js → passkey_login.ts
  - passkey_discoverable.js → passkey_discoverable.ts
- Update tsconfig.json with modern ES2022 and strict type checking
- Update esbuild config to use app.ts as entry point
- All assets compile successfully with esbuild's native TypeScript support
- No node_modules required, following Phoenix/Elixir conventions
2026-01-15 13:29:49 -06:00
2bacf6337c
Fix equipment charts to display full 24-hour time range
Set explicit min/max bounds on chart x-axis to ensure all graphs
consistently show 24 hours of data, even when data points are sparse.

Also fix AlertNotifierTest race condition by disabling async execution.
2026-01-15 13:12:29 -06:00
82413df58b
Improve monitor script to eliminate screen flicker
Instead of clearing the entire screen every 5 seconds, the script now:
- Prints the header once and saves cursor position
- Fetches all API data before updating display (no blank period)
- Restores cursor position and updates only the content area
- Hides cursor during display for cleaner look
- Shows cursor on exit via cleanup trap

Uses tput commands:
- sc/rc: Save/restore cursor position
- ed: Clear from cursor to end of screen
- civis/cnorm: Hide/show cursor
2026-01-15 13:04:41 -06:00
4fe583b220
cleanup 2026-01-15 13:01:29 -06:00
480f4dba7f
Add script to apply Pangolin security rules
Helper script to upload security rules to Pangolin instance.
Requires PANGOLIN_KEY environment variable.
2026-01-15 12:28:35 -06:00
5e9032314b
Add comprehensive Pangolin security rules
Created 30+ security rules to protect against common web attacks:

Protection categories:
- SQL injection (UNION, OR/AND, comments)
- XSS (script tags, event handlers)
- Path traversal (directory traversal, absolute paths)
- Common exploits (WordPress, PHPMyAdmin, admin panels)
- Malicious bots (scanners, scrapers, empty user agents)
- Shell injection (commands, ShellShock)
- File upload exploits (PHP, double extensions)
- Information disclosure (.git, .env, backups, configs)
- Protocol attacks (HTTP/0.9, TRACE, TRACK)
- Known CVEs (Log4Shell, SSRF, XXE)

Features:
- Allowlist for legitimate traffic (health checks, agents, monitoring)
- Detailed documentation with examples
- Testing and troubleshooting guides
- Performance impact analysis
- Compliance mapping (OWASP, PCI DSS, SOC 2)

Documentation: docs/PANGOLIN_SECURITY.md
Configuration: pangolin-security-rules.toml
2026-01-15 12:25:21 -06:00
0a6f30299b
Filter out Bandit HTTP/0.9 errors from port scanners
Added log filter to suppress noisy errors from:
- HTTP/0.9 requests (obsolete protocol from 1991)
- Invalid HTTP version errors from port scanners and bots
- Malformed HTTP requests from automated tools

These errors are expected on public-facing servers and don't indicate
application problems. The filter uses Elixir's logger handler filters
to suppress them at the logging layer.

Resolves sporadic 'Bandit.HTTPError: Invalid HTTP version: {0, 9}' errors
in production logs.
2026-01-15 10:00:53 -06:00
c9e4e8d105
Show pipeline creation time as relative 'X ago' format
- Added format_time_ago() function
- Handles both GNU date (Linux) and BSD date (macOS)
- Properly parses ISO 8601 timestamps with UTC timezone
- Shows: 30s ago, 5m ago, 2h ago, 3d ago
- Changed 'Created' field from raw timestamp to human-readable format
2026-01-15 09:36:00 -06:00
97b1e7bb9e
Fix agent project ID in monitoring script
Updated from 63766041 to 77595235 (correct ID for towerops/towerops-agent)
2026-01-15 09:33:53 -06:00
58433ec072
Enable Docker socket mount by default in agent setup
- Removed comment from Docker socket volume mount
- Auto-updates now work out of the box when users copy config
- Socket mount: /var/run/docker.sock:/var/run/docker.sock
2026-01-15 09:29:58 -06:00
a98e1823f3
Fix project ID and duration formatting
- Updated main app project ID: 77257437 → 77618300 (towerops/towerops)
- Fixed duration formatting to handle decimal seconds from GitLab API
- Script now fetches correct current pipelines instead of old deleted project
2026-01-15 09:10:24 -06:00
de3c282825
Fix debug info display - show after clear command
- Moved debug fields (Local HEAD, Created, Commit) into monitoring loop
- Removed duplicate output that was immediately cleared
- Added warning when local commit doesn't match pipeline commit
- Now debug info stays visible throughout monitoring
2026-01-15 09:07:23 -06:00
37160fc33f
Improve pipeline detection and debugging
- Show local HEAD commit for comparison
- Add explicit sort order (id DESC) to get most recent pipeline
- Display pipeline created time and commit SHA
- Remove duplicate pipeline info output
2026-01-15 09:04:19 -06:00
d4e3846e69
Add debug output to pipeline monitoring
- Show pipeline creation time and commit SHA
- Show job count and job IDs
- Help diagnose incorrect job/status issues
2026-01-15 09:03:07 -06:00
8f434f18bc
Fix main app project path to towerops/towerops
Changed from graham/towerops to towerops/towerops
2026-01-15 09:01:28 -06:00
b33d6a5c25
Fix job URLs to use project path instead of numeric ID
Failed job URLs now correctly use graham/towerops instead of 77257437
2026-01-15 09:00:46 -06:00
6be6ff34f9
Fix GitLab API calls - use numeric project IDs
- Changed from URL-encoded paths to numeric project IDs
- Added -L flag to curl to follow redirects
- Improved error handling with HTTP status codes
- Shows API errors with response body
2026-01-15 08:59:48 -06:00
b036716eb7
Fix environment variable name in monitoring script
Changed GITLAB_TOKEN to GITLAB_ACCESS_TOKEN throughout
2026-01-15 08:58:27 -06:00
c1d944a612
Add CI/CD monitoring script
Features:
- Auto-detects project from git remote
- Shows real-time pipeline progress
- Color-coded job statuses
- Links to failed job logs
- For main app: monitors Kubernetes rollout
- Auto-refreshes every 5 seconds

Usage:
  export GITLAB_TOKEN=<token>
  ./scripts/monitor-deploy.sh [app|agent]

Or just run from repo directory for auto-detection
2026-01-15 08:52:40 -06:00
b70ddfe166
timestamp improvement 2026-01-15 08:03:30 -06:00
bb5766f1a6
Optimize list_agent_polling_targets to use SQL filtering
Rewrote list_agent_polling_targets/1 to filter equipment using SQL WHERE clause
instead of loading all SNMP-enabled equipment into memory and filtering in Elixir.

Previous implementation:
- Loaded ALL SNMP-enabled equipment with preloads (potentially 10,000+ records)
- Filtered in application code with equipment_matches_agent?/2 helper
- Called get_equipment_assignment/1 for each piece of equipment (N+1 queries)
- Memory-intensive and slow at scale

New implementation:
- Single SQL query with LEFT JOIN and compound WHERE clause
- Evaluates hierarchical assignment (equipment → site → organization) in SQL
- Only loads equipment assigned to the specific agent
- Scales efficiently to 10,000+ equipment

Performance improvement:
- With 10,000 equipment, 1000 assigned to agent:
  - Before: Load 10,000 records + 10,000 queries = massive memory + slow
  - After: Load 1,000 records in 1 query = 10x memory reduction + instant

Removed unused helper functions:
- equipment_matches_agent?/2
- site_matches_agent?/2

All 40 agent tests passing.
2026-01-15 07:33:52 -06:00
723f064315
Add batch loading for sensor/interface stats to eliminate N+1 queries
Added two new functions to Snmp context:
- get_latest_sensor_readings_batch/1 - Loads latest readings for multiple sensors in one query
- get_latest_interface_stats_batch/1 - Loads latest stats for multiple interfaces in one query

Both use PostgreSQL DISTINCT ON to efficiently get only the latest record per ID.

Updated EquipmentLive.Show.load_snmp_data/1 to use batch functions:
- Equipment with 20 sensors: 20 queries → 1 query (20x reduction)
- Equipment with 10 interfaces: 10 queries → 1 query (10x reduction)
- Total improvement: 30 queries → 2 queries for typical equipment page load

This significantly improves page load performance for equipment with many sensors/interfaces.
2026-01-15 07:31:57 -06:00