diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 173daa95..3498c83d 120000 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1 +1 @@ -/nix/store/zx636v118rzzqzwafgrw2aybh7l0szrl-pre-commit-config.json \ No newline at end of file +/nix/store/4xlkmdd947h5qlhj2rmbyavq08grkz27-pre-commit-config.json \ No newline at end of file diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 00000000..1fb0e8f9 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,1012 @@ +# TowerOps Architecture + +> Network infrastructure monitoring platform for ISPs/WISPs built with Phoenix LiveView, Oban, and TimescaleDB. + +--- + +## 1. Project Structure + +``` +towerops/ +├── assets/ # Frontend: JS, CSS, Tailwind +├── config/ # Runtime & env config (dev/test/prod/runtime) +├── c_src/ # C NIFs (SNMP native interop) +├── docs/ # Documentation +├── e2e/ # End-to-end tests (Playwright) +├── flux/ # FluxCD / GitOps manifests +├── k8s/ # Kubernetes manifests +├── lib/ +│ ├── mix/ # Custom Mix tasks +│ ├── snmpkit/ # SNMP toolkit (native extension) +│ ├── towerops/ # Core business logic (contexts, schemas, workers) +│ └── towerops_web/ # Web layer (LiveView, controllers, channels, GraphQL) +├── mibs/ # SNMP MIB files +├── priv/ +│ ├── repo/migrations/ # Ecto migrations +│ ├── static/ # Static assets +│ └── gettext/ # i18n translations +├── rel/ # Release configuration +├── scripts/ # Operational scripts +├── test/ # Test suite +├── towerops/ # Agent-related files +└── vendor/ # Vendored dependencies +``` + +### Key `lib/towerops/` Subdirectories + +| Directory | Purpose | +|-----------|---------| +| `accounts/` | User, auth tokens, TOTP, recovery codes, sessions, consent | +| `admin/` | Superuser audit logs | +| `agents/` | Remote polling agent tokens & assignments | +| `alerts/` | Alert schema and queries | +| `api_tokens/` | API token management (hashed bearer tokens) | +| `billing/` | Stripe integration, usage metering | +| `capacity/` | Interface throughput & utilization calculations | +| `config_changes/` | MikroTik config change event tracking | +| `contexts/` | Shared context helpers | +| `devices/` | Device schema, backups, firmware, events | +| `ecto_types/` | Custom Ecto types (IP address, MAC address, SNMP OID) | +| `gaiia/` | Gaiia billing/CRM integration (accounts, subscriptions, impact analysis) | +| `geoip/` | GeoIP lookup (blocks, locations) | +| `integrations/` | Integration schema (multi-provider) | +| `job_monitoring/` | Oban job health events | +| `maintenance/` | Maintenance windows | +| `mobile_sessions/` | Mobile app sessions & QR login | +| `monitoring/` | Check definitions & check results | +| `netbox/` | NetBox DCIM sync | +| `on_call/` | Schedules, escalation policies, incidents, notifications | +| `organizations/` | Org, membership, invitations, policies | +| `pagerduty/` | PagerDuty client & notifier | +| `preseem/` | Preseem QoE integration (APs, baselines, fleet intelligence) | +| `profiles/` | SNMP device profiles (OID mappings) | +| `proto/` | Protobuf definitions (agent communication) | +| `security/` | Brute-force protection, IP blocks/whitelists | +| `settings/` | Global application settings (key-value store) | +| `sites/` | Site schema & tree structure | +| `snmp/` | SNMP device data (interfaces, sensors, readings, wireless, topology) | +| `sonar/` | Sonar billing integration | +| `splynx/` | Splynx billing integration | +| `topology/` | Network topology discovery (LLDP, CDP, ARP, MAC) | +| `visp/` | VISP billing integration | +| `weather/` | Weather observations & alerts per site | +| `workers/` | All Oban background workers | + +### Key `lib/towerops_web/` Subdirectories + +| Directory | Purpose | +|-----------|---------| +| `channels/` | Phoenix Channels (agent WebSocket) | +| `components/` | Reusable UI components (breadcrumbs, tables, forms) | +| `controllers/` | Traditional controllers (auth, webhooks, health, API) | +| `graphql/` | Absinthe GraphQL schema, types, resolvers, subscriptions | +| `helpers/` | View helpers (time formatting, status display) | +| `live/` | All LiveView pages | +| `plugs/` | Plug middleware (auth, rate limiting, security headers) | + +--- + +## 2. Data Model + +All primary keys are UUIDs (`:binary_id`). Multi-tenancy is organization-scoped. + +### Core Entities + +``` +┌──────────────┐ ┌──────────────────┐ ┌──────────────┐ +│ User │────<│ Membership │>────│ Organization │ +│ │ │ (role-based) │ │ │ +│ email │ │ role: owner| │ │ name, slug │ +│ hashed_pass │ │ admin|executive │ │ subscription │ +│ totp_secret │ │ |technician| │ │ snmp_config │ +│ is_superuser │ │ member|viewer │ │ mikrotik_cfg │ +│ timezone │ │ is_default │ │ stripe_* │ +│ default_org │ └──────────────────┘ │ alert_routing│ +└──────────────┘ └──────┬───────┘ + │ + ┌────────────────┬──────┴──────┬───────────────┐ + │ │ │ │ + ┌─────┴─────┐ ┌──────┴──────┐ ┌───┴────┐ ┌──────┴──────┐ + │ Site │ │ Device │ │ Alert │ │ Integration │ + │ │ │ │ │ │ │ │ + │ name │ │ name │ │ type │ │ provider │ + │ lat/lng │ │ ip_address │ │ sev │ │ credentials │ + │ address │ │ status │ │ msg │ │ sync_status │ + │ snmp_cfg │ │ snmp_cfg │ │ device │ │ (encrypted) │ + │ parent_id │ │ mikrotik_cfg│ │ check │ └─────────────┘ + │ agent_tok │ │ site_id │ └────────┘ + └───────────┘ │ agent_assign│ + │ check_intv │ + └─────────────┘ +``` + +### On-Call / Incident Management + +``` +Schedule ──has_many──> Layer ──has_many──> LayerMember ──belongs_to──> User + │ + └──has_many──> Override (user, start_at, end_at) + +EscalationPolicy ──has_many──> EscalationRule ──has_many──> EscalationTarget + │ │ │ + │ │ position, delay_minutes ├── schedule_id + │ └──────────────────────── └── user_id + │ +Incident ──belongs_to──> Alert + │ EscalationPolicy + │ Organization + └──has_many──> Notification ──belongs_to──> User +``` + +### SNMP Data Model + +``` +Device (towerops) ──1:1──> Snmp.Device + │ + ├── Snmp.Interface ──> InterfaceStat (time-series) + ├── Snmp.Sensor ──> SensorReading (time-series) + ├── Snmp.StateSensor + ├── Snmp.Processor ──> ProcessorReading + ├── Snmp.Storage ──> StorageReading + ├── Snmp.WirelessClient ──> WirelessClientReading + ├── Snmp.Neighbor (LLDP/CDP) + ├── Snmp.ArpEntry + ├── Snmp.Vlan + └── Snmp.PhysicalEntity +``` + +### Billing / CRM Schemas + +- **Gaiia**: `Account`, `BillingSubscription`, `NetworkSite`, `InventoryItem`, `DeviceSubscriberLink` +- **Preseem**: `AccessPoint`, `SubscriberMetric`, `FleetProfile`, `DeviceBaseline`, `Insight`, `SyncLog` +- **Billing**: `StripeWebhookEvent` + +### Other Schemas + +| Schema | Table | Purpose | +|--------|-------|---------| +| `AgentToken` | `agent_tokens` | Remote polling agent credentials | +| `AgentAssignment` | `agent_assignments` | Device → agent mapping | +| `ApiToken` | `api_tokens` | REST/GraphQL API bearer tokens | +| `BrowserSession` | `browser_sessions` | Active login sessions | +| `LoginAttempt` | `login_attempts` | Auth attempt audit log | +| `UserConsent` | `user_consents` | Privacy/terms consent records | +| `PolicyVersion` | `policy_versions` | Privacy policy versions | +| `UserTotpDevice` | `user_totp_devices` | Multi-device TOTP | +| `UserRecoveryCode` | `user_recovery_codes` | MFA recovery codes | +| `MaintenanceWindow` | `maintenance_windows` | Scheduled maintenance | +| `ConfigChangeEvent` | `config_change_events` | MikroTik config diffs | +| `DeviceFirmwareHistory` | `device_firmware_histories` | Firmware change tracking | +| `FirmwareRelease` | `firmware_releases` | Known firmware versions | +| `BackupRequest` | `backup_requests` | MikroTik backup queue | +| `MikrotikBackup` | `mikrotik_backups` | Stored config backups | +| `Check` | `checks` | Monitoring check definitions | +| `CheckResult` | `check_results` | Check execution results | +| `MonitoringCheck` | `monitoring_checks` | Check configuration | +| `DeviceLink` | `device_links` | Discovered topology links | +| `DeviceLinkEvidence` | `device_link_evidence` | Link confidence scores | +| `DeviceNeighbor` | `device_neighbors` | Raw neighbor data | +| `ApplicationSetting` | `application_settings` | Global config KV store | +| `AuditLog` | `audit_logs` | Admin action audit trail | +| `IpBlock` | `ip_blocks` | Blocked IPs | +| `IpWhitelist` | `ip_whitelists` | Whitelisted IPs | +| `MobileSession` | `mobile_sessions` | Mobile app sessions | +| `QrLoginToken` | `qr_login_tokens` | QR-based mobile auth | +| `Weather.Observation` | `weather_observations` | Site weather data | +| `Weather.Alert` | `weather_alerts` | Active weather alerts | +| `GeoIp.Block` / `Location` | `geoip_*` | IP geolocation | + +--- + +## 3. Context Modules + +### `Towerops.Accounts` +User management, authentication, TOTP/MFA, password management, email changes, session tracking. + +```elixir +# User lookup +def get_user_by_email(email) +def get_user_by_email_and_password(email, password) +def get_user(id) +def get_user!(id) + +# Registration +def register_user(attrs) +def register_user_with_organization(attrs) +def change_user_registration(user, attrs \\ %{}) + +# TOTP / MFA +def generate_totp_secret() +def generate_totp_uri(user, secret) +def generate_totp_qr_code(user, secret) +def verify_totp(secret, code) +def enable_totp(user, secret) +def totp_enabled?(user) +def verify_user_totp(user, code) +def list_user_totp_devices(user_id) +def create_totp_device(user_id, device_name) +def verify_user_totp_any_device(user, code) +def delete_totp_device(device_id, user_id) +def rename_totp_device(device_id, user_id, new_name) +def generate_recovery_codes(user_id) +def verify_recovery_code(user_id, code) +def count_unused_recovery_codes(user_id) +def verify_user_mfa(user, code) + +# Profile & Settings +def change_user_profile(user, attrs \\ %{}) +def update_user_profile(user, attrs) +def sudo_mode?(user, minutes \\ -10) +def grant_sudo_mode(user) + +# Email & Password +def change_user_email(user, attrs \\ %{}, opts \\ []) +def update_user_email(user, token) +def change_user_password(user, attrs \\ %{}, opts \\ []) + +# Sessions (+ more token functions not listed) +``` + +### `Towerops.Organizations` +Multi-tenant org management, memberships, invitations, SNMP/agent propagation. + +```elixir +def list_user_organizations(user_id) +def user_has_access?(user_id, organization_id) +def get_organization!(id) +def get_organization_by_slug!(slug) +def create_organization(attrs, user_id, opts \\ []) +def update_organization(organization, attrs) +def delete_organization(organization) +def change_organization(organization, attrs \\ %{}) + +# Memberships +def get_membership(organization_id, user_id) +def list_organization_memberships(organization_id) +def list_organization_members(organization_id) +def remove_member(organization_id, user_id) +def update_member_role(organization_id, user_id, new_role) +def create_membership(attrs) +def get_default_membership(user_id) +def set_default_organization(user_id, organization_id) +def list_organization_notification_recipients(organization_id) + +# Invitations +def list_pending_invitations(organization_id) +def create_invitation(attrs) +def get_invitation_by_token(token) +def accept_invitation(invitation, user_id) +def delete_invitation(invitation) + +# Config propagation +def apply_snmp_config_to_all_equipment(organization_id) +def apply_agent_to_all_equipment(organization_id) +def clear_all_site_assignments(organization_id) + +# Billing lookups +def get_organization_by_stripe_customer_id(stripe_customer_id) +def list_organizations_with_active_subscriptions() +``` + +### `Towerops.Devices` +Device CRUD, SNMP/MikroTik config resolution (cascading: org → site → device), monitoring. + +```elixir +def list_site_devices(site_id) +def list_organization_devices(organization_id, filters \\ %{}) +def count_organization_devices(organization_id) +def search_devices(organization_id, query) +def get_device_status_counts(organization_id) +def list_monitored_devices() +def list_snmp_enabled_devices() +def list_mikrotik_devices_with_api() +def get_device(id) +def get_device!(id) +def get_device_with_details(id) +def get_device_by_ip(ip_address, site_id, exclude_id \\ nil) +def get_snmp_config(device) # Resolves cascading config +def get_mikrotik_config(device) # Resolves cascading config +def get_snmpv3_config(device) # Resolves cascading config +def resolve_agent_token_id(device) # Device → site → org → global fallback +def propagate_site_snmpv3_change(site_id, attrs) +def propagate_organization_mikrotik_change(organization_id, attrs) +# ... CRUD, reordering, etc. +``` + +### `Towerops.Sites` +Site CRUD, hierarchical tree structure, config propagation. + +```elixir +def list_organization_sites(organization_id) +def list_root_sites(organization_id) +def list_child_sites(parent_site_id) +def build_site_tree(organization_id) +def list_sites_with_coordinates() +def search_sites(organization_id, query) +def get_site!(id) +def create_site(attrs) +def update_site(site, attrs) +def delete_site(site) +def apply_snmp_config_to_all_equipment(site_id) +def apply_agent_to_all_equipment(site_id) +def reorder_site(site_id, new_position) +``` + +### `Towerops.Alerts` +Alert lifecycle: creation, acknowledgement, resolution, Gaiia impact analysis. + +```elixir +def create_alert(attrs) +def list_devices_alerts(device_id, limit \\ 100) +def list_organization_active_alerts(organization_id) +def list_organization_alerts(organization_id, filters) +def count_active_alerts(organization_id) +def get_alert!(id) +def get_alert(id) +def acknowledge_alert(alert, user_id) +def resolve_alert(alert) +def resolve_alert_silent(alert) # No PubSub broadcast +def acknowledge_alert_silent(alert) # No PubSub broadcast +def has_active_alert?(device_id, alert_type) +def get_active_alert(device_id, alert_type) +def has_active_check_alert?(check_id) +def resolve_check_alerts(check_id) +def store_gaiia_impact(alert, impact_data) +``` + +### `Towerops.Integrations` +Generic integration CRUD for all billing/CRM providers. + +```elixir +def list_integrations(organization_id) +def get_integration(organization_id, provider) +def create_integration(organization_id, attrs) +def update_integration(integration, attrs) +def update_sync_status(integration, status, message \\ nil) +def delete_integration(integration) +def get_billing_summary(organization_id) +def list_enabled_integrations(provider) +def update_billing_totals(integration, subscriber_count, total_mrr) +``` + +**Valid providers:** `preseem`, `gaiia`, `pagerduty`, `netbox`, `sonar`, `splynx`, `visp` + +### `Towerops.OnCall` +On-call schedules, escalation policies, incidents. + +```elixir +# Schedules +def list_schedules(organization_id) +def get_schedule!(id) / get_schedule(id) +def create_schedule(attrs) / update_schedule(schedule, attrs) / delete_schedule(schedule) +def create_layer(attrs) / update_layer(layer, attrs) / delete_layer(layer) +def add_layer_member(attrs) / remove_layer_member(member) +def create_override(attrs) / delete_override(override) +def who_is_on_call(schedule_id, datetime \\ DateTime.utc_now()) + +# Escalation Policies +def list_escalation_policies(organization_id) +def get_escalation_policy!(id) / get_escalation_policy(id) +def create_escalation_policy(attrs) / update_escalation_policy(policy, attrs) / delete_escalation_policy(policy) +def create_escalation_rule(attrs) / update_escalation_rule(rule, attrs) / delete_escalation_rule(rule) +def create_escalation_target(attrs) / delete_escalation_target(target) +``` + +### `Towerops.Billing` +Stripe billing integration: checkout, portal, usage metering, subscription management. + +```elixir +def default_free_devices() +def default_price_per_device() +def create_checkout_session(organization, opts \\ []) +def create_portal_session(organization, return_url) +def billable_device_count(organization) +def effective_free_device_count(org) +def effective_price_per_device(org) +def estimated_monthly_cost(organization) +def sync_usage_to_stripe(organization) +def update_subscription_from_stripe(organization, stripe_subscription) +def update_payment_method_status(organization, status) +def migrate_all_subscriptions_to_price(new_price_id) +``` + +### `Towerops.Agents` +Remote polling agent lifecycle: tokens, heartbeat, assignments, mass updates, debug mode. + +```elixir +def create_agent_token(organization_id, name) +def create_cloud_poller(name) +def list_organization_agent_tokens(organization_id) +def list_cloud_pollers() +def verify_agent_token(token) +def update_agent_token_heartbeat(agent_token_id, ip_address, metadata \\ nil) +def update_agent_token(agent_token, attrs) +def restart_agent(agent_token_id) +def update_agent(agent_token_id, url, checksum) +def toggle_agent_debug(agent_token, enabled, user) +def revoke_agent_token(id) +def delete_agent_token(id) +def assign_device_to_agent(agent_token_id, device_id) +def unassign_device(device_id) +def list_agent_devices(agent_token_id) +def list_agent_polling_targets(agent_token_id) +def get_effective_agent_token(device) +def broadcast_mass_update() +``` + +### `Towerops.Monitoring` +Check definitions, execution, result tracking, state machine (soft/hard states). + +```elixir +def list_checks(organization_id, opts \\ []) +def get_check(id) / get_check!(id) +def create_check(attrs) / update_check(check, attrs) / delete_check(check) +def schedule_check(check) +def create_check_result(attrs) +def get_latest_check_result(check_id) +def get_check_results(check_id, opts \\ []) +def get_check_graph_data(check_id, from_time, to_time) +def update_check_state(check, status, output) # Soft/hard state machine +def stop_device_checks(device_id) +def disable_device_checks(device_id) +def get_device_health_summary(device_ids) +def get_device_latest_response_times(device_ids) +``` + +### `Towerops.Snmp` +SNMP device discovery, interface/sensor management, readings. + +```elixir +def test_connection(ip, community, version, port \\ 161) +def discover_device(device) +def discover_all_for_org(org_id) +def create_checks_from_discovery(device, snmp_device) +def get_device(device_id) / get_device_with_associations(device_id) +def list_interfaces(device_id) / list_monitored_interfaces(device_id) +def list_sensors(device_id) / list_sensors_by_type(device_id) +def get_sensor_readings(sensor_id, opts \\ []) +def get_interface_stats(interface_id, opts \\ []) +def set_manual_capacity(interface_id, capacity_bps) +# ... many more reading/stat functions +``` + +### `Towerops.Gaiia` +Gaiia CRM/billing data: accounts, subscriptions, network sites, inventory, impact analysis. + +```elixir +def list_accounts(organization_id) +def upsert_account(organization_id, attrs) +def count_active_subscribers(organization_id) +def sum_active_mrr(organization_id) +def list_network_sites(organization_id) +def upsert_network_site(organization_id, attrs) +def list_inventory_items(organization_id) +def list_billing_subscriptions(organization_id) +def get_device_subscriber_counts(device_ids) +def suggest_site_matches(organization_id, network_site) +def suggest_device_matches(organization_id, inventory_item) +``` + +Sub-modules: `Gaiia.Client`, `Gaiia.Webhooks`, `Gaiia.ImpactAnalysis`, `Gaiia.Reconciliation`, `Gaiia.SubscriberMatching`, `Gaiia.Actions` + +### `Towerops.Preseem` +Preseem QoE integration: access points, subscriber metrics, fleet intelligence. + +```elixir +def get_access_point_for_device(device_id) +def list_access_points(organization_id) +def list_unmatched_access_points(organization_id) +def list_matched_access_points(organization_id) +def list_subscriber_metrics(access_point_id, opts \\ []) +def link_access_point(access_point_id, device_id) +def unlink_access_point(access_point_id) +def get_site_qoe_summary(site_id) +``` + +Sub-modules: `Preseem.Client`, `Preseem.Sync`, `Preseem.Baseline`, `Preseem.DeviceMatcher`, `Preseem.FleetIntelligence`, `Preseem.Insights` + +### Other Context Modules + +| Module | Purpose | +|--------|---------| +| `Towerops.Maintenance` | Maintenance window CRUD, device/site in-maintenance checks | +| `Towerops.Topology` | Network topology discovery (LLDP/CDP/ARP/MAC), link inference | +| `Towerops.Dashboard` | Dashboard summary, health scores, impact analysis | +| `Towerops.Search` | Cross-entity full-text search (devices, sites, accounts, alerts) | +| `Towerops.ActivityFeed` | Aggregated activity feed (config changes, alerts, syncs, events) | +| `Towerops.ApiTokens` | API token CRUD, hashed token verification | +| `Towerops.Capacity` | Interface throughput & utilization calculations | +| `Towerops.ConfigChanges` | MikroTik config change diff tracking | +| `Towerops.MobileSessions` | Mobile app sessions, QR login flow | +| `Towerops.Weather` | Weather observations & alerts per site | +| `Towerops.Admin` | Superuser admin: user/org listing, audit logs, billing overrides | +| `Towerops.Profiles` | SNMP device profile matching & OID management | +| `Towerops.Settings` | Global application settings (key-value store) | +| `Towerops.Geocoding` | Address → lat/lng geocoding | +| `Towerops.GeoIp` | IP → location lookup | +| `Towerops.Http` | Shared HTTP client helpers | +| `Towerops.Trace` | Subscriber trace functionality | +| `Towerops.RateLimit` | Rate limiting logic | +| `Towerops.Numeric` | Numeric formatting utilities | + +--- + +## 4. LiveView Pages + +### Main Application (requires auth + org) + +| Route | LiveView | Purpose | +|-------|----------|---------| +| `/dashboard` | `DashboardLive` | Org dashboard with health score, alert summary, site overview | +| `/alerts` | `AlertLive.Index` | Alert list with filtering (active/ack/resolved), real-time updates | +| `/agents` | `AgentLive.Index` | Agent list with health status | +| `/agents/:id` | `AgentLive.Show` | Agent detail: assigned devices, heartbeat, metadata | +| `/agents/:id/edit` | `AgentLive.Edit` | Edit agent settings | +| `/devices` | `DeviceLive.Index` | Device list with status, filtering, real-time updates | +| `/devices/new` | `DeviceLive.Form` | Create device form (with SNMP credential test) | +| `/devices/:id` | `DeviceLive.Show` | Device detail: status, interfaces, sensors, alerts, backups | +| `/devices/:id/edit` | `DeviceLive.Form` | Edit device form | +| `/devices/:id/graph/:sensor_type` | `GraphLive.Show` | Time-series graphs for device metrics | +| `/devices/:device_id/backups/compare` | `MikrotikBackupLive.Compare` | Config backup diff viewer | +| `/devices/:device_id/config-timeline` | `ConfigTimelineLive` | Config change timeline | +| `/sites` | `SiteLive.Index` | Site list with device counts | +| `/sites/new` | `SiteLive.Form` | Create site | +| `/sites/:id` | `SiteLive.Show` | Site detail: devices, capacity, weather | +| `/sites/:id/edit` | `SiteLive.Form` | Edit site | +| `/insights` | `InsightsLive.Index` | AI-generated operational insights | +| `/trace` | `TraceLive.Index` | Subscriber trace tool | +| `/activity` | `ActivityFeedLive` | Activity feed (config changes, alerts, syncs) | +| `/schedules` | `ScheduleLive.Index` | On-call schedule list | +| `/schedules/new` | `ScheduleLive.Form` | Create schedule | +| `/schedules/:id` | `ScheduleLive.Show` | Schedule detail with calendar view | +| `/schedules/:id/edit` | `ScheduleLive.Form` | Edit schedule | +| `/schedules/escalation-policies` | `EscalationPolicyLive.Index` | Escalation policy list | +| `/schedules/escalation-policies/new` | `EscalationPolicyLive.Form` | Create escalation policy | +| `/schedules/escalation-policies/:id` | `EscalationPolicyLive.Show` | Policy detail with rules | +| `/schedules/escalation-policies/:id/edit` | `EscalationPolicyLive.Form` | Edit policy | +| `/maintenance` | `MaintenanceLive.Index` | Maintenance window list | +| `/maintenance/new` | `MaintenanceLive.Form` | Create maintenance window | +| `/maintenance/:id` | `MaintenanceLive.Show` | Window detail | +| `/maintenance/:id/edit` | `MaintenanceLive.Form` | Edit window | +| `/changelog` | `ChangelogLive` | Product changelog | +| `/network-map` | `NetworkMapLive` | Network topology visualization | +| `/sites-map` | `MapLive.Index` | Geographic site map | + +### Organization Settings (requires auth + specific org slug) + +| Route | LiveView | Purpose | +|-------|----------|---------| +| `/orgs/:org_slug/settings` | `Org.SettingsLive` | Org settings, SNMP defaults, billing | +| `/orgs/:org_slug/settings/integrations` | `Org.IntegrationsLive` | Integration management | +| `/orgs/:org_slug/settings/integrations/preseem/devices` | `Org.PreseemDevicesLive` | Preseem AP ↔ device matching | +| `/orgs/:org_slug/settings/integrations/preseem/insights` | `Org.PreseemInsightsLive` | Preseem fleet insights | +| `/orgs/:org_slug/settings/integrations/gaiia/mapping` | `Org.GaiiaMappingLive` | Gaiia site/device mapping | +| `/orgs/:org_slug/settings/integrations/gaiia/reconciliation` | `Org.GaiiaReconciliationLive` | Gaiia data reconciliation | + +### Account / User Pages + +| Route | LiveView | Purpose | +|-------|----------|---------| +| `/orgs` | `OrgLive.Index` | Organization picker | +| `/orgs/new` | `OrgLive.New` | Create organization | +| `/users/settings` | `UserSettingsLive` | Profile, email, password, API tokens, sessions, TOTP | +| `/users/my-data` | `AccountLive.MyData` | GDPR data export | +| `/account/activity` | `AccountLive.Activity` | User login history | +| `/account/totp-enrollment` | `AccountLive.TotpEnrollment` | TOTP setup wizard | +| `/mobile/qr-login` | `MobileQRLive` | QR code for mobile app login | +| `/users/register` | `UserRegistrationLive` | Registration form | +| `/users/reset-password/:token` | `UserResetPasswordLive` | Password reset | +| `/help` | `HelpLive.Index` | Help / documentation page | + +### Admin Pages (superuser only) + +| Route | LiveView | Purpose | +|-------|----------|---------| +| `/admin/` | `Admin.DashboardLive` | Admin overview | +| `/admin/users` | `Admin.UserLive.Index` | All users management | +| `/admin/organizations` | `Admin.OrgLive.Index` | All orgs management | +| `/admin/audit` | `Admin.AuditLive.Index` | Audit log viewer | +| `/admin/monitoring` | `Admin.MonitoringLive` | System monitoring | +| `/admin/agents` | `Admin.AgentLive.Index` | All agents health | +| `/admin/security` | `Admin.SecurityLive.Index` | IP blocks/whitelists | + +--- + +## 5. PubSub Topics + +### Alert Topics + +| Topic | Events | Subscribers | +|-------|--------|-------------| +| `alerts:org:{org_id}:new` | New alert created | `AlertLive.Index`, `DashboardLive`, `ActivityFeedLive` | +| `alerts:org:{org_id}:resolved` | Alert resolved | `AlertLive.Index`, `DashboardLive`, `ActivityFeedLive` | +| `organization:{org_id}:alerts` | Alert status updates | `UserAuth.subscribe_to_status_updates` (nav badge) | + +### Device Topics + +| Topic | Events | Subscribers | +|-------|--------|-------------| +| `device:{device_id}` | Device status changes, poll results, sensor data | `DeviceLive.Show`, `GraphLive.Show` | +| `device:events` | All device events (cross-device) | `Devices.EventLogger` | +| `device_events:org:{org_id}` | Org-scoped device events | `ActivityFeedLive` | +| `devices:org:{org_id}` | Device created/updated/deleted | `DeviceLive.Index`, `ActivityFeedLive` | +| `wireless_clients:device:{device_id}` | Wireless client updates | `DeviceLive.Show` | + +### Agent Topics + +| Topic | Events | Subscribers | +|-------|--------|-------------| +| `agents:health` | Agent heartbeat, stale detection | `AgentLive.Index`, `DeviceLive.Show`, `Admin.AgentLive` | +| `admin:agents` | Admin-level agent events | `Admin.AgentLive.Index` | +| `agent:{agent_id}:assignments` | Assignment changes | `AgentChannel` | +| `agent:{agent_id}:discovery` | Discovery commands | `AgentChannel` | +| `agent:{agent_id}:backup` | Backup commands | `AgentChannel` | +| `agent:{agent_id}:credential_test` | Credential test requests | `AgentChannel` | +| `agent:{agent_id}:live_poll` | Live poll requests | `AgentChannel` | +| `agent:{agent_id}:lifecycle` | Restart/update commands | `AgentChannel` | +| `agent:{agent_id}:latency_probe` | Latency probe requests | `AgentChannel` | + +### Other Topics + +| Topic | Events | Subscribers | +|-------|--------|-------------| +| `config:org:{org_id}` | Config change events | `ActivityFeedLive` | +| `sync:org:{org_id}` | Integration sync events | `ActivityFeedLive` | +| `topology:{org_id}` | Topology link changes | `NetworkMapLive` | +| `credential_test:{test_id}` | SNMP credential test results | `DeviceLive.Form` | +| `security:blocks` | IP block changes | `Admin.SecurityLive` | +| `security:whitelist` | IP whitelist changes | `Admin.SecurityLive` | +| `job_monitoring` | Oban job health events | `Admin.MonitoringLive` | + +--- + +## 6. Workers (Oban) + +### Queue: `pollers` + +| Worker | Purpose | +|--------|---------| +| `DevicePollerWorker` | SNMP polling: interfaces, sensors, storage, processors, wireless, ARP, topology. The main data collection worker. | + +### Queue: `monitors` + +| Worker | Purpose | +|--------|---------| +| `DeviceMonitorWorker` | Device up/down monitoring via ICMP ping, triggers alerts on status change | + +### Queue: `discovery` + +| Worker | Purpose | +|--------|---------| +| `DiscoveryWorker` | SNMP device discovery: sysinfo, interfaces, sensors, profiles | + +### Queue: `checks` + +| Worker | Purpose | +|--------|---------| +| `CheckWorker` | Schedules and dispatches monitoring check execution | + +### Queue: `check_executors` + +| Worker | Purpose | +|--------|---------| +| `CheckExecutorWorker` | Executes individual monitoring checks (HTTP, ping, etc.) | + +### Queue: `notifications` + +| Worker | Purpose | +|--------|---------| +| `AlertNotificationWorker` | Sends alert notifications (email, PagerDuty, on-call) | +| `WelcomeEmailWorker` | Sends welcome email after registration | +| `EscalationCheckWorker` | Checks for escalation timeouts and advances incidents | + +### Queue: `weather` + +| Worker | Purpose | +|--------|---------| +| `WeatherSyncWorker` | Fetches weather data for sites with coordinates | + +### Queue: `maintenance` + +| Worker | Purpose | +|--------|---------| +| `BillingSyncWorker` | Syncs device count usage to Stripe | +| `GaiiaSyncWorker` | Syncs data from Gaiia billing/CRM | +| `GaiiaInsightWorker` | Generates Gaiia-based insights | +| `PreseemSyncWorker` | Syncs data from Preseem | +| `PreseemBaselineWorker` | Calculates Preseem performance baselines | +| `NetboxSyncWorker` | Syncs data from NetBox DCIM | +| `SonarSyncWorker` | Syncs data from Sonar billing | +| `SplynxSyncWorker` | Syncs data from Splynx billing | +| `VispSyncWorker` | Syncs data from VISP billing | +| `MikrotikBackupWorker` | Fetches MikroTik config backups | +| `BackupSummaryWorker` | Generates backup comparison summaries | +| `BackupTimeoutWorker` | Cleans up timed-out backup requests | +| `FirmwareVersionFetcherWorker` | Fetches latest firmware versions from vendors | +| `StaleAgentWorker` | Detects agents that stopped heartbeating | +| `AgentLatencyEvaluator` | Evaluates agent-to-device latency data | +| `CloudLatencyProbeWorker` | Probes cloud poller latency | +| `SessionCleanupWorker` | Cleans up expired browser/mobile sessions | +| `LoginHistoryCleanupWorker` | Purges old login attempt records | +| `ExpiredBanCleanupWorker` | Removes expired IP bans | +| `StaleViolationCleanupWorker` | Cleans up stale rate limit violations | +| `CloudflareBanWorker` | Syncs IP bans to Cloudflare WAF | +| `JobHealthCheckWorker` | Monitors Oban job queue health | +| `DeviceHealthInsightWorker` | Generates device health insights | +| `CapacityInsightWorker` | Generates capacity utilization insights | +| `SystemInsightWorker` | Generates system-level insights | +| `WirelessInsightWorker` | Generates wireless performance insights | + +### Additional Worker + +| Worker | Queue | Purpose | +|--------|-------|---------| +| `Snmp.NeighborCleanupWorker` | (in snmp module) | Cleans up stale SNMP neighbor records | + +--- + +## 7. Integrations Architecture + +### Overview + +Integrations are stored as `Integration` records with encrypted credentials (`Cloak.Ecto`). Each provider has: +- A **client module** (`Client`) for API communication +- A **sync module** (`Sync`) for data synchronization +- An **Oban worker** for periodic sync + +### Billing/CRM Providers + +| Provider | Modules | Data Synced | +|----------|---------|-------------| +| **Gaiia** | `Gaiia.Client`, `Gaiia.Sync`, `Gaiia.Webhooks`, `Gaiia.ImpactAnalysis`, `Gaiia.Reconciliation`, `Gaiia.SubscriberMatching`, `Gaiia.Actions` | Accounts, billing subscriptions, network sites, inventory items, device-subscriber links | +| **Preseem** | `Preseem.Client`, `Preseem.Sync`, `Preseem.Baseline`, `Preseem.DeviceMatcher`, `Preseem.FleetIntelligence`, `Preseem.Insights` | Access points, subscriber metrics, fleet profiles, device baselines | +| **Sonar** | `Sonar.Client`, `Sonar.Sync` | Subscriber/billing data | +| **Splynx** | `Splynx.Client`, `Splynx.Sync` | Subscriber/billing data | +| **VISP** | `Visp.Client`, `Visp.Sync` | Subscriber/billing data | + +### Infrastructure Providers + +| Provider | Modules | Purpose | +|----------|---------|---------| +| **NetBox** | `Netbox.Client`, `Netbox.Sync` | DCIM inventory sync (devices, sites) | +| **PagerDuty** | `PagerDuty.Client`, `PagerDuty.Notifier` | Alert routing & incident escalation | + +### Stripe Billing + +Stripe integration is handled separately from the integration framework: +- `Towerops.Billing` — checkout sessions, portal sessions, usage metering +- `BillingSyncWorker` — periodic usage sync +- `StripeWebhookController` — receives Stripe webhook events +- `StripeWebhookEvent` schema — idempotent webhook processing +- Usage-based billing: free device tier, then per-device pricing +- Organization fields: `stripe_customer_id`, `stripe_subscription_id`, `subscription_status` + +### Gaiia Specifics + +Gaiia has the deepest integration: +- **Webhook receiver** at `/api/v1/webhooks/gaiia/:organization_id` +- **Impact analysis** — when a device goes down, calculates affected subscribers & MRR +- **Reconciliation** — UI for matching Gaiia network sites to TowerOps sites +- **Device-subscriber links** — maps which subscribers are served by which device + +### Alert Routing + +Organizations configure `alert_routing`: +- `"builtin"` — uses on-call schedules + escalation policies +- `"pagerduty"` — routes to PagerDuty +- `"both"` — routes to both + +--- + +## 8. Auth Flow + +### Authentication Stack + +``` +Request → Browser Pipeline → fetch_session → fetch_current_scope_for_user + │ + ├─ Check session token + ├─ Check remember_me cookie + ├─ Build Scope (user + org + tz) + └─ Handle impersonation +``` + +### `Towerops.Accounts.Scope` + +The `Scope` struct flows through the entire request lifecycle: + +```elixir +%Scope{ + user: %User{}, # Current user (or impersonated user) + superuser: %User{} | nil, # Actual superuser if impersonating + impersonating?: boolean(), + organization: %Organization{} | nil, + timezone: "America/Chicago", + time_format: "24h" +} +``` + +### Session Lifecycle + +1. **Login** — `UserSessionController.create` validates email/password +2. **TOTP verification** — If TOTP enabled, redirect to `/users/log-in/totp` +3. **Session creation** — Token stored in session + optional remember_me cookie +4. **Organization loading** — Default org set from membership, or from URL slug +5. **Session renewal** — Tokens reissued after configurable interval +6. **Sudo mode** — Sensitive operations require recent password re-entry (`last_sudo_at`) + +### LiveView `on_mount` Hooks + +```elixir +:load_cookie_consent # Load GDPR consent state +:require_authenticated_user # Redirect to login if no session +:require_totp_enrollment # Redirect to TOTP setup if required +:load_current_organization # Load org from URL slug param +:load_default_organization # Load user's default org +:subscribe_to_status_updates # Subscribe to org alert PubSub +:require_superuser # Admin-only pages +:require_sudo_mode # Require recent password entry +:handle_policy_consent # Check privacy/terms acceptance +:redirect_if_user_is_authenticated # Login page redirect +:require_organization_owner # Owner-only operations +``` + +### API Authentication + +- **REST API** (`/api/v1/*`) — Bearer token via `ToweropsWeb.Plugs.ApiAuth`, verified against hashed `ApiToken` records +- **GraphQL** (`/api/graphql`) — Same bearer token auth + `GraphQL.Context` plug +- **Mobile API** (`/api/v1/mobile/*`) — Session token via `ToweropsWeb.Plugs.MobileAuth` +- **Webhooks** (`/api/v1/webhooks/*`) — Shared secret via `ToweropsWeb.Plugs.WebhookAuth` +- **Agent WebSocket** — Token-based auth in `AgentSocket`/`AgentChannel` + +### Organization Switching + +- `POST /orgs/switch` — changes the current organization in session +- User must have a membership in the target organization +- Default org stored on `Membership.is_default` flag +- URL-based: `/orgs/:org_slug/settings/*` uses slug from URL + +### Security Features + +- **Rate limiting** — Per-route rate limits (auth, API, admin) via `RateLimit` plug +- **Brute-force protection** — `Security.BruteForce` with IP blocking + Cloudflare WAF sync +- **TOTP/MFA** — Required enrollment, multi-device support, recovery codes +- **Sudo mode** — Time-limited elevated access for sensitive operations +- **IP whitelisting/blocking** — `IpBlock`, `IpWhitelist` schemas +- **Session tracking** — `BrowserSession` records with device/location info +- **Impersonation** — Superuser can impersonate users, tracked in `Scope` + +--- + +## 9. Key Patterns + +### Cascading Configuration + +SNMP and MikroTik credentials cascade: **Organization → Site → Device**. Each level can override or inherit: + +```elixir +# In Devices context +def resolve_agent_token_id(device) do + get_direct_agent(device) # Device-level assignment + || get_site_agent(device) # Site default agent + || get_org_default_agent(device) # Organization default + || get_global_default_agent() # Global cloud poller fallback +end +``` + +### AccessControl Helpers + +`ToweropsWeb.Live.Helpers.AccessControl` provides organization-scoped access verification: + +```elixir +verify_device_access(device_id, organization_id) # → {:ok, device} | {:error, :not_found | :unauthorized} +verify_site_access(site_id, organization_id) +verify_alert_access(alert_id, organization_id) +``` + +### Membership Roles + +Ordered by privilege: `owner > admin > executive > technician > member > viewer` + +- **owner** — Full access, can delete organization +- **admin** — Full access except deleting org +- **executive** — Read-only with financial visibility (MRR, revenue) +- **technician** — Field ops: devices, sites, alerts, no financials +- **viewer** — Read-only, no financials + +### Breadcrumbs + +`ToweropsWeb.Components.Breadcrumbs` provides a DaisyUI-based breadcrumb component: + +```heex +<.breadcrumb items={[ + %{label: "Sites", path: ~p"/sites"}, + %{label: @site.name, path: ~p"/sites/#{@site.id}"}, + %{label: "Edit"} +]} /> +``` + +### PubSub Pattern + +Contexts broadcast changes; LiveViews subscribe on mount: + +```elixir +# In context (e.g., Alerts) +Phoenix.PubSub.broadcast(Towerops.PubSub, "alerts:org:#{org_id}:new", {:new_alert, alert}) + +# In LiveView mount +Phoenix.PubSub.subscribe(Towerops.PubSub, "alerts:org:#{organization.id}:new") + +# In LiveView handle_info +def handle_info({:new_alert, alert}, socket), do: ... +``` + +### Encrypted Fields + +Sensitive data (SNMP passwords, API credentials, integration secrets) uses `Cloak.Ecto` encryption: + +```elixir +field :snmpv3_auth_password, Towerops.Encrypted.Binary +field :credentials, Towerops.Encrypted.Map +``` + +### Multi-Tenancy + +All queries are scoped by `organization_id`. The org context flows through: +1. `Scope` struct in conn/socket assigns +2. URL slug (`/orgs/:org_slug/...`) +3. Default organization from user membership + +### Form Pattern + +LiveViews use Phoenix changesets with `to_form/1`: + +```elixir +# In mount/handle_params +changeset = Context.change_entity(entity, %{}) +socket = assign(socket, form: to_form(changeset)) + +# In handle_event("save", ...) +case Context.update_entity(entity, params) do + {:ok, entity} -> {:noreply, push_navigate(socket, to: ...)} + {:error, changeset} -> {:noreply, assign(socket, form: to_form(changeset))} +end +``` + +### Flash Messages + +Standard Phoenix flash with custom styling: + +```elixir +socket |> put_flash(:info, "Device created successfully") |> push_navigate(to: ...) +socket |> put_flash(:error, "Failed to save device") +``` + +### Agent Communication + +Agents connect via WebSocket (`AgentChannel`): +1. Agent authenticates with token via `AgentSocket` +2. Channel subscribes to agent-specific PubSub topics +3. Server pushes commands: poll, discover, backup, restart, update +4. Agent sends results: poll data, discovery results, backup content +5. Protobuf used for efficient data serialization (`lib/towerops/proto/`) + +### GraphQL API + +Absinthe-based GraphQL at `/api/graphql`: +- Schema: `ToweropsWeb.GraphQL.Schema` +- Types mirror core entities (devices, sites, alerts, agents, schedules, etc.) +- Complexity analysis enabled (max 500 complexity, max 10 depth) +- Subscriptions via `ToweropsWeb.GraphQL.Subscriptions` +- API token authentication via `GraphQL.Context` plug + +### Health Checks + +- `GET /health` — Kubernetes probe endpoint (no auth) +- `DeviceMonitorWorker` — ICMP ping monitoring +- `JobHealthCheckWorker` — Oban queue health +- `AgentLatencyEvaluator` — Agent responsiveness + +--- + +*Generated from source code analysis. Last updated: 2026-03-13.* diff --git a/docs/COMPETITIVE_ANALYSIS.md b/docs/COMPETITIVE_ANALYSIS.md new file mode 100644 index 00000000..f5f855a6 --- /dev/null +++ b/docs/COMPETITIVE_ANALYSIS.md @@ -0,0 +1,524 @@ +# TowerOps Competitive Analysis + +**Last Updated**: 2026-03-13 +**Purpose**: Map the ISP/WISP network monitoring and operations tool landscape, identify gaps, and position TowerOps for differentiation. + +--- + +## Table of Contents + +1. [LibreNMS](#1-librenms) +2. [PRTG Network Monitor](#2-prtg-network-monitor) +3. [Auvik](#3-auvik) +4. [The Dude (MikroTik)](#4-the-dude-mikrotik) +5. [Preseem](#5-preseem) +6. [Sonar](#6-sonar) +7. [Splynx](#7-splynx) +8. [Oxidized / RANCID](#8-oxidized--rancid) +9. [Grafana + Prometheus](#9-grafana--prometheus) +10. [Competitive Matrix](#competitive-matrix) +11. [TowerOps Positioning](#towerops-positioning) + +--- + +## 1. LibreNMS + +**Website**: librenms.org +**License**: GPL v3 (open source) + +### Target Market +ISPs, WISPs, enterprises, hosting providers, universities — anyone with network infrastructure and the technical staff to self-host. + +### Pricing Model +Free and open source. No licensing costs. Community support via Discord and forums. Professional support available through third-party consultants. + +### Key Strengths +- **Massive device support**: Auto-discovery via CDP, FDP, LLDP, OSPF, BGP, SNMP, and ARP. 786+ OS profiles. +- **Billing module**: Built-in bandwidth billing system for ports based on usage or transfer — rare in an open-source monitoring tool. +- **Distributed polling**: Horizontal scaling with poller groups. Handles large networks (10k+ devices). +- **Alerting flexibility**: Alerts via email, IRC, Slack, PagerDuty, Telegram, and many more transports. +- **API access**: Full REST API for automation and integration. +- **Community**: Large, active community. Active Discord. Regular contributions. +- **Zero cost**: No per-device, per-sensor, or per-subscriber fees. + +### Key Weaknesses / Gaps +- **UX is dated**: Web UI is functional but looks like it was designed in 2015. Not responsive or modern. Navigation is confusing for new users. +- **RRDtool limitations**: Default time-series backend (RRDtool) has fixed retention periods and is not SQL-queryable. InfluxDB support exists but is less mature. +- **No multi-tenancy**: Single-organization design. No proper tenant isolation for MSPs or multi-client environments. +- **Complex setup**: Requires Linux sysadmin skills. Installation, upgrades, and maintenance are manual. No SaaS option. +- **No agent architecture**: Polling is centralized or distributed-pollers, but no lightweight edge agents for remote sites behind NAT/firewalls. +- **No subscriber awareness**: Monitors devices and interfaces, not subscribers. No concept of "this customer is having a bad experience." +- **Config management**: No built-in config backup or change tracking. Requires Oxidized or RANCID alongside it. +- **Mobile app**: Community-built mobile app exists (by Wavedirect) but not officially maintained. + +### What TowerOps Can Learn / Differentiate On +- **Learn**: LibreNMS's device profile coverage (786 OS types) is the gold standard. TowerOps has matched this — maintain parity. +- **Learn**: Bandwidth billing module is a unique feature that ISPs value. TowerOps integrates with billing platforms (Sonar, Splynx) instead — a different but arguably better approach. +- **Differentiate**: Modern LiveView UI vs. LibreNMS's aging PHP interface. This is a major selling point. +- **Differentiate**: Multi-tenancy from day one. LibreNMS can't serve MSPs or multi-org setups cleanly. +- **Differentiate**: TimescaleDB > RRDtool. SQL-queryable, better compression, better retention policies. +- **Differentiate**: Agent-based distributed polling for sites behind NAT. LibreNMS requires VPN tunnels or direct access. +- **Differentiate**: Subscriber-aware monitoring. LibreNMS thinks in devices; TowerOps can think in customers. + +--- + +## 2. PRTG Network Monitor + +**Website**: paessler.com/prtg +**Vendor**: Paessler AG (German company) + +### Target Market +Mid-market enterprise IT departments, MSPs, some ISPs. Generalist network monitoring — not ISP/WISP-specific. + +### Pricing Model +Subscription-based by sensor count: +- **PRTG 500** (50 devices): $179/month +- **PRTG 1000** (100 devices): $325/month +- **PRTG 2500** (250 devices): $675/month +- **PRTG 5000** (500 devices): $1,183/month +- **PRTG 10000** (1000 devices): $1,492/month +- **Enterprise**: Custom pricing for 10k+ sensors +- Free tier: 100 sensors (~10 devices) + +### Key Strengths +- **Ease of use**: Highly polished UI. Web interface, desktop app, and mobile apps (iOS/Android). Best-in-class for non-technical users. +- **Sensor-based architecture**: Granular monitoring — each metric is a "sensor." Flexible but can get expensive quickly. +- **Broad protocol support**: SNMP, WMI, SSH, HTTP, NetFlow, sFlow, jFlow, packet sniffing, and more. +- **Maps & dashboards**: Built-in map designer with real-time status. Good for NOC displays. +- **Notifications**: Email, push notifications, HTTP requests, SMS, Slack, Teams, etc. +- **Windows-native**: Runs on Windows Server. Familiar for Windows-shop IT teams. + +### Key Weaknesses / Gaps +- **Expensive at scale**: An ISP with 500 devices needs PRTG 5000 ($1,183/mo) and will burn through sensors quickly. Interface monitoring alone eats 5-10 sensors per device. +- **Windows-only server**: Core server runs only on Windows. Odd choice for network monitoring in a Linux-heavy ISP world. +- **Not ISP-aware**: No subscriber mapping, no billing integration, no WISP-specific features. It's a generic IT monitoring tool. +- **No config management**: No config backup or diff tracking. +- **Sensor sprawl**: The sensor model means costs escalate unpredictably. Hard to budget for growing networks. +- **No distributed agents**: Remote probes exist but are Windows-based. No lightweight Linux agent for tower sites. +- **Limited automation**: Basic auto-discovery but no sophisticated topology mapping or network automation. + +### What TowerOps Can Learn / Differentiate On +- **Learn**: PRTG's UX polish is excellent. Maps, dashboards, mobile apps — they invest heavily in usability. TowerOps should aim for this level of polish. +- **Learn**: The sensor model, while expensive, gives users granular control. Consider offering similar granularity with better pricing. +- **Differentiate**: Per-device or flat pricing vs. per-sensor. ISPs hate unpredictable costs. +- **Differentiate**: Linux-native, cloud-native. No Windows dependency. +- **Differentiate**: ISP/WISP-specific features that PRTG will never build (subscriber awareness, billing integration, tower management). + +--- + +## 3. Auvik + +**Website**: auvik.com +**Type**: Cloud-based SaaS + +### Target Market +MSPs (Managed Service Providers) primarily. Some mid-market IT departments. Not ISP/WISP-focused. + +### Pricing Model +Quote-based (not publicly listed). Reports suggest: +- Per-device pricing, tiered by network size +- Estimated $3-8/device/month depending on volume +- Multi-tenant by design (MSP model) +- No free tier; 14-day free trial + +### Key Strengths +- **Cloud-first**: No server to manage. Deploy a lightweight collector and go. Full visibility in under an hour. +- **Auto-discovery & mapping**: Best-in-class automated network topology mapping. Live, auto-updating maps. +- **64+ pre-configured alerts**: Works out of the box with sensible defaults. Low time-to-value. +- **Configuration backup**: Automatic config backup with version history and diff comparison. Built-in (no Oxidized needed). +- **700+ vendor support**: Broad multi-vendor device support out of the box. +- **NetFlow/traffic analysis**: Deep traffic visibility including encrypted traffic analysis. +- **MSP integrations**: ConnectWise, Autotask, ServiceNow, IT Glue, PagerDuty, Slack, Teams, etc. +- **Multi-tenant**: Built for MSPs managing many client networks. Proper tenant isolation. +- **Modern UX**: Clean, intuitive interface. Junior techs can use it immediately. + +### Key Weaknesses / Gaps +- **Not ISP-focused**: Designed for corporate LANs, not carrier networks. No subscriber awareness, no WISP features, no tower management. +- **Expensive for ISPs**: Per-device pricing adds up fast for ISPs with hundreds/thousands of devices. An ISP with 1000 devices could be paying $3,000-8,000/month. +- **No SNMP depth**: Good for L2/L3 basics but lacks deep SNMP sensor discovery (temperature, power, optical levels) that ISPs need for tower equipment. +- **No billing integration**: No connection to ISP billing platforms (Sonar, Splynx, VISP). +- **Cloud-only**: No self-hosted option. Data sovereignty concerns for some operators. +- **Limited customization**: Pre-built dashboards are good but custom metrics/sensors are limited compared to PRTG or LibreNMS. +- **No QoE**: No subscriber quality-of-experience monitoring. + +### What TowerOps Can Learn / Differentiate On +- **Learn**: Auvik's time-to-value is exceptional. Deploy collector → see network in minutes. TowerOps's agent model should aim for similar simplicity. +- **Learn**: Auto-updating topology maps are a killer feature. TowerOps already has LLDP topology discovery — push this further. +- **Learn**: Built-in config backup eliminates the need for a separate Oxidized/RANCID install. TowerOps should include this. +- **Differentiate**: ISP/WISP domain expertise vs. Auvik's MSP/corporate focus. +- **Differentiate**: Deep SNMP sensor discovery (optical, RF, environmental) vs. Auvik's shallow device monitoring. +- **Differentiate**: Billing platform integrations (Sonar, Splynx, VISP, Gaiia) that Auvik will never build. + +--- + +## 4. The Dude (MikroTik) + +**Website**: mikrotik.com (under Tools/Legacy) +**Vendor**: MikroTik + +### Target Market +Small-to-medium WISPs and networks running MikroTik equipment. Hobbyists. Budget-conscious operators. + +### Pricing Model +**Free**. Included with MikroTik RouterOS. Server can run on any RouterOS device or as a Windows/Wine application. + +### Key Strengths +- **Free**: Zero cost. Hard to compete with free for budget-conscious WISPs. +- **Deep MikroTik integration**: Direct RouterOS API access. Can monitor and configure MikroTik devices natively without SNMP overhead. +- **Auto-discovery**: Subnet scanning with service detection. Draws network maps automatically. +- **Lightweight**: Runs on a MikroTik router itself (CHR or hardware). No separate server needed. +- **Network maps**: Visual topology with automatic layout. Real-time status overlay. +- **Multi-protocol**: SNMP, ICMP, DNS, TCP services monitoring. +- **Widely used in WISP community**: De facto standard for MikroTik-heavy WISPs. + +### Key Weaknesses / Gaps +- **Legacy/neglected product**: MikroTik lists it under "Legacy Tools." Development has stagnated. The client is a Windows app (or runs under Wine). No web UI. +- **MikroTik-centric**: Works great for MikroTik gear but limited for multi-vendor environments (Ubiquiti, Cambium, Cisco, etc.). +- **No web interface**: Desktop client only. No remote access without VPN/RDP. No mobile app. +- **Limited alerting**: Basic notifications but no sophisticated alert routing, escalation, or integration with modern tools (PagerDuty, Slack, etc.). +- **No time-series storage**: Stores current state but lacks historical trending, capacity planning, or long-term data retention. +- **No API**: No REST API or integration capabilities. +- **No multi-tenancy**: Single-user design. +- **Scaling issues**: Performance degrades with large networks (500+ devices). Not designed for ISP-scale operations. +- **No config backup**: No configuration management features. +- **No subscriber awareness**: Device-centric only. + +### What TowerOps Can Learn / Differentiate On +- **Learn**: The Dude's auto-discovery and visual mapping are beloved by WISPs. Simple, immediate, visual. TowerOps topology should feel this intuitive. +- **Learn**: Running on the router itself (zero infrastructure) is appealing for tiny WISPs. TowerOps's agent-on-router approach can replicate this for larger operators. +- **Differentiate**: Everything. The Dude is a legacy tool with no active development. TowerOps is the modern replacement. +- **Differentiate**: Web UI, multi-vendor, multi-tenant, alerting, API, historical data, subscriber awareness — all things The Dude lacks. +- **Key opportunity**: Many WISPs have outgrown The Dude but haven't found a replacement that "gets" their world. TowerOps is that replacement. + +--- + +## 5. Preseem + +**Website**: preseem.com +**Type**: Cloud-based SaaS with on-prem appliance for traffic analysis + +### Target Market +Regional ISPs and WISPs. Purpose-built for fixed wireless and fiber operators. + +### Pricing Model +- **Per-subscriber**: Starting at $0.40/subscriber/month +- **Minimum**: $300/month +- **Volume discounts** available +- **Month-to-month**: No long-term contracts required +- **Hardware**: Optional pre-loaded appliances or BYO Linux server +- 30-day free trial + +### Key Strengths +- **QoE-focused**: Quality of Experience monitoring is their core differentiator. Measures what subscribers actually experience, not just device uptime. +- **Traffic shaping**: Adaptive traffic management that reduces congestion and improves subscriber QoE. This is unique — most monitoring tools only observe. +- **Multi-vendor, multi-access**: Supports fixed wireless (Cambium, Ubiquiti, MikroTik, Baicells, etc.) and fiber. Unified view across technologies. +- **Billing integrations**: Connects to Sonar, Splynx, Powercode, VISP, BillMax, Azotel, and more. Subscriber-aware from day one. +- **Churn reduction**: Proven results — customers report 25-30% churn reduction. Direct revenue impact. +- **Industry recognition**: 5x WISPA Service of the Year winner. Deeply embedded in the WISP community. +- **Cloud analytics**: Centralized view across all vendors and network elements. No need to log into multiple systems. +- **Scales to 100+ Gbps**: Handles large traffic volumes. + +### Key Weaknesses / Gaps +- **Not a full monitoring platform**: Preseem monitors traffic and QoE but doesn't replace a full NMS. No SNMP polling, no device inventory, no ping monitoring, no alerting for device failures. +- **Requires inline deployment**: The appliance sits inline or uses port mirroring. Adds a potential point of failure and complexity. +- **No config management**: No config backup or change tracking. +- **No topology mapping**: No network topology discovery or visualization. +- **No on-call/incident management**: No paging, escalation, or incident workflows. +- **Pricing adds up**: At $0.40/subscriber, a 5,000-subscriber ISP pays $2,000/month just for QoE. On top of their NMS, billing platform, etc. +- **Complementary, not complete**: ISPs still need LibreNMS/PRTG/etc. alongside Preseem. More tools = more cost and complexity. + +### What TowerOps Can Learn / Differentiate On +- **Learn**: Subscriber-aware monitoring is incredibly valuable. Preseem proved the market wants this. TowerOps should integrate subscriber context from billing platforms. +- **Learn**: QoE metrics (latency, jitter, packet loss per subscriber) matter more than device uptime to ISP operators. Build this into TowerOps. +- **Learn**: Preseem's billing integrations are a model. They connect to every major ISP billing platform — TowerOps already integrates with Sonar, Splynx, VISP, and Gaiia. +- **Differentiate**: TowerOps can be the unified platform that includes SNMP monitoring + subscriber awareness + alerting + topology — eliminating the need for Preseem as a separate tool. +- **Complement or compete**: In the near term, TowerOps + Preseem could work together. Long term, TowerOps could build QoE features and reduce the need for Preseem. + +--- + +## 6. Sonar + +**Website**: sonar.software +**Type**: Cloud-based SaaS + +### Target Market +ISPs of all types: cable, fiber, wireless, VoIP, MDU, enterprise. Focused on billing and operations, not monitoring. + +### Pricing Model +Per-subscriber tiered pricing with $500/month minimum (with contract): +- **1-5,000 subscribers**: $1.25/subscriber/month +- **5,001-10,000**: $1.00/subscriber/month +- **10,001-50,000**: $0.75/subscriber/month +- **50,000+**: Enterprise/custom pricing + +### Key Strengths +- **Comprehensive ISP operations**: Billing, CRM, ticketing, inventory, field service scheduling, provisioning — all in one platform. +- **Financial tools**: 25+ pre-built financial reports. Tax management across jurisdictions. Payment processing. +- **Field service**: Job scheduling, dispatch, GPS tracking, work orders, checklists. Mobile app for field techs. +- **Network visibility**: Basic monitoring, fault detection, device provisioning, performance tracking. +- **DataConnect**: Snowflake data warehouse integration for advanced analytics. +- **Customer portal**: Self-service portal for subscribers. +- **US/Canada support**: Real humans, no outsourcing, Mon-Fri 8am-6pm CT. + +### Key Weaknesses / Gaps +- **Monitoring is basic**: Sonar's network monitoring is rudimentary compared to a dedicated NMS. Basic ICMP/SNMP, no deep sensor discovery, no QoE. +- **Not a replacement for an NMS**: ISPs using Sonar still need LibreNMS, PRTG, or similar for real network monitoring. +- **Expensive at scale**: A 5,000-subscriber ISP pays $6,250/month. A 10,000-subscriber ISP pays ~$7,500/month. This is before monitoring, QoE, or other tools. +- **Cloud-only**: No self-hosted option. +- **No topology/mapping**: No network topology visualization. +- **No config management**: No device config backup or change tracking. + +### What TowerOps Can Learn / Differentiate On +- **Learn**: Sonar owns the "ISP operations platform" positioning. TowerOps should position as the monitoring/network operations complement, not a billing competitor. +- **Learn**: Sonar's integration ecosystem is a model. Being the platform that other tools integrate with is powerful. +- **Differentiate**: TowerOps is the deep network monitoring that Sonar can't do. Sonar does billing; TowerOps does monitoring. They're complementary. +- **Integration opportunity**: TowerOps's existing Sonar integration should be a first-class feature. Pull subscriber data from Sonar → correlate with network health → give NOC teams subscriber context. +- **Differentiate**: TowerOps can surface network problems that impact billing (e.g., "these 50 subscribers have been experiencing degraded service for 3 days — expect churn"). + +--- + +## 7. Splynx + +**Website**: splynx.com +**Type**: Cloud-hosted or self-hosted SaaS + +### Target Market +Small-to-medium ISPs worldwide. Strong presence in Africa, Eastern Europe, Southeast Asia, and Latin America. 1,000+ ISP customers globally. + +### Pricing Model +Per-subscriber, subscription-based (prices in USD): +- Scales with active subscriber count (intervals of 100, entry level up to 400 subscribers) +- Includes license + maintenance fee (hosting, support, backups) +- Cloud or on-premise deployment options +- Add-ons: ACS (TR-069) from $100/month, Whalebone security from $80/month +- All features included in base license (no feature tiers) + +### Key Strengths +- **All-in-one ISP platform**: Billing, RADIUS, TR-069 (ACS), IPAM, CRM, ticketing, field service, inventory — comprehensive. +- **Networking features**: Built-in RADIUS server, bandwidth management, QoE, IPAM, network monitoring, config backups, change control. +- **Self-hosted option**: Unlike Sonar (cloud-only), Splynx can be self-hosted. Important for operators in regions with data sovereignty requirements. +- **Customer portal & mobile app**: White-labeled mobile app for subscriber self-service. +- **Field tech mobile app**: Dedicated app for technicians with scheduling, work orders, maps. +- **Global reach**: 1,000+ ISPs worldwide. Strong in developing markets where cost matters. +- **Fast support**: 85% of tickets resolved in under 60 minutes. +- **TR-069 / ACS**: CPE device provisioning and management. Increasingly important for fiber ISPs. + +### Key Weaknesses / Gaps +- **Monitoring is secondary**: Network monitoring exists but is not the primary focus. Basic compared to dedicated NMS tools. +- **Not deep SNMP**: Doesn't match LibreNMS or TowerOps for SNMP sensor discovery, vendor-specific MIBs, or environmental monitoring. +- **Smaller than Sonar in US market**: Sonar dominates the US ISP billing space. Splynx is stronger internationally. +- **No topology mapping**: No automated network topology discovery or visualization. +- **No QoE**: No subscriber quality-of-experience monitoring or traffic shaping. +- **UI less polished than Sonar**: Functional but not as modern as Sonar's interface. + +### What TowerOps Can Learn / Differentiate On +- **Learn**: Splynx's all-in-one approach (billing + networking + CRM) appeals to small ISPs who don't want 5 separate tools. TowerOps should integrate deeply enough that it feels like part of the stack. +- **Learn**: Self-hosted option matters for international markets. TowerOps's agent architecture and potential self-hosted option could appeal to the same market. +- **Differentiate**: TowerOps provides the deep monitoring that Splynx's built-in monitoring can't match. Position as the "upgrade your monitoring" path for Splynx users. +- **Integration opportunity**: TowerOps's existing Splynx integration should pull subscriber data, service plans, and customer context to enrich network monitoring. + +--- + +## 8. Oxidized / RANCID + +**Website**: github.com/ytti/oxidized (Oxidized), shrubbery.net/rancid (RANCID) +**License**: Open source + +### Target Market +Any network operator needing config backup. ISPs, enterprises, MSPs. Technical users comfortable with CLI tools. + +### Pricing Model +**Free** (open source). Self-hosted only. + +### Key Strengths +- **Oxidized**: Modern RANCID replacement. Ruby-based. Supports 130+ OS types. Git-backed config storage with blame/diff. REST API. Hooks for notifications (Slack, email). Syslog integration to trigger config fetch on change events. +- **RANCID**: Battle-tested (20+ years). CVS/SVN-backed. Widely deployed. Simple and reliable. +- **Config versioning**: Git integration means full history, diff, blame. Know who changed what and when. +- **Lightweight**: Single-purpose tools that do one thing well. +- **Event-driven**: Oxidized can react to syslog events (config change detected → immediately fetch new config). + +### Key Weaknesses / Gaps +- **Config backup only**: No monitoring, no alerting, no topology, no dashboards. Single-purpose. +- **Oxidized maintenance concerns**: GitHub repo has a "Maintainer Wanted" notice. Bus factor risk. +- **No web UI**: CLI/API only. No visual diff viewer out of the box (though tools like oxidized-web exist). +- **Manual setup**: Requires configuration, cron jobs, and integration work. +- **No compliance features**: No automated compliance checking against config templates. +- **RANCID is legacy**: CVS-based, Perl scripts, showing its age. Most operators have migrated to Oxidized. + +### What TowerOps Can Learn / Differentiate On +- **Learn**: Config backup with git-backed versioning and diff is table stakes for ISPs. TowerOps already has config change tracking for MikroTik — expand this to all supported vendors. +- **Learn**: Event-driven config fetch (syslog trigger → immediate backup) is smart. TowerOps agents could do this natively. +- **Learn**: "Git blame" for config changes (who changed what line) is incredibly valuable for troubleshooting and compliance. +- **Differentiate**: TowerOps can build config backup into the platform, eliminating a separate Oxidized install. One less tool to maintain. +- **Differentiate**: Config compliance checking (compare running config against a template, flag drift) — something neither Oxidized nor RANCID does. + +--- + +## 9. Grafana + Prometheus + +**Website**: grafana.com, prometheus.io +**License**: Open source (Grafana AGPL, Prometheus Apache 2.0) + +### Target Market +DevOps teams, SREs, cloud-native organizations, and technically sophisticated ISPs who want full control and customization. + +### Pricing Model +**Free** (self-hosted open source). Grafana Cloud offers managed hosting: +- Free tier: 10k metrics, 50GB logs +- Pro: ~$29/month per active series +- Enterprise: Custom pricing +- Most ISPs self-host to avoid costs. + +### Key Strengths +- **Visualization**: Grafana is the gold standard for dashboards. Flexible, beautiful, and infinitely customizable. +- **Prometheus data model**: Labels-based metrics with powerful PromQL query language. Excellent for high-cardinality data. +- **Ecosystem**: Thousands of exporters (SNMP Exporter, Node Exporter, Blackbox Exporter, etc.). Alertmanager for alerting. Loki for logs. +- **Cloud-native**: Designed for containerized, microservices architectures. Kubernetes-native. +- **Scalability**: Horizontal scaling with Thanos, Cortex, or Mimir for long-term storage. +- **Community**: Massive community. Extensive documentation. Frequent releases. +- **Flexibility**: Can monitor anything. Network, servers, applications, custom metrics. + +### Key Weaknesses / Gaps +- **DIY assembly required**: Not a product — it's a toolkit. ISPs must assemble SNMP Exporter + Prometheus + Alertmanager + Grafana + dashboards. Significant engineering effort. +- **No auto-discovery**: Prometheus uses static config or service discovery (Consul, DNS, file). No SNMP/LLDP auto-discovery of network devices. +- **SNMP support is bolted on**: snmp_exporter works but requires SNMP MIB compilation into a generator config. Painful compared to LibreNMS's plug-and-play. +- **No network awareness**: No topology mapping, no device inventory, no interface table management. You're staring at metrics, not a network. +- **Alert fatigue**: Alertmanager is powerful but complex to configure. No built-in alert correlation or deduplication intelligence. +- **No subscriber awareness**: Purely infrastructure metrics. No concept of customers or service impact. +- **No config management**: No config backup or change tracking. +- **Maintenance burden**: Self-hosted Prometheus requires capacity planning, storage management, retention tuning. It's a full-time job at scale. +- **Dashboard sprawl**: Without discipline, you end up with 200 dashboards and no one knows which to look at. + +### When ISPs Choose Grafana+Prometheus Over a Product +- They have DevOps/SRE staff who already know the stack +- They want custom dashboards that no product can provide +- They're already using it for server/application monitoring and want to extend to network +- They're philosophically opposed to vendor lock-in +- They have unique metrics or integrations that products don't support +- Cost-sensitive and willing to trade engineering time for license fees + +### What TowerOps Can Learn / Differentiate On +- **Learn**: Grafana's dashboard flexibility is unmatched. TowerOps should offer embeddable Grafana-style dashboards or a dashboard builder that approaches this flexibility. +- **Learn**: The Prometheus ecosystem's breadth is incredible. TowerOps should be able to export metrics in Prometheus format for users who want Grafana alongside TowerOps. +- **Differentiate**: TowerOps is a product, not a toolkit. Auto-discovery, device inventory, subscriber awareness, alerting, topology — all built-in. No assembly required. +- **Differentiate**: SNMP-first design. TowerOps's 786 OS profiles and vendor-specific MIB support is years of work that Prometheus users would have to replicate. +- **Differentiate**: ISP domain knowledge. Grafana doesn't know what a tower site is or what subscriber churn means. TowerOps does. +- **Coexistence strategy**: Offer Prometheus-compatible metric export. Let users who love Grafana use it alongside TowerOps. Don't force them to choose. + +--- + +## Competitive Matrix + +| Feature | TowerOps | LibreNMS | PRTG | Auvik | The Dude | Preseem | Sonar | Splynx | Oxidized | Grafana+Prom | +|---------|----------|----------|------|-------|----------|---------|-------|--------|----------|-------------| +| **SNMP Monitoring** | ✅ Deep | ✅ Deep | ✅ Deep | ⚠️ Basic | ⚠️ Basic | ❌ | ⚠️ Basic | ⚠️ Basic | ❌ | ⚠️ Bolt-on | +| **Device Auto-Discovery** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | +| **Topology Mapping** | ✅ LLDP | ✅ | ❌ | ✅ Best | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | +| **Multi-Tenancy** | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | +| **Subscriber Awareness** | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ Best | ✅ | ✅ | ❌ | ❌ | +| **QoE Monitoring** | 🔜 | ❌ | ❌ | ❌ | ❌ | ✅ Best | ❌ | ⚠️ | ❌ | ❌ | +| **Config Backup** | ✅ MikroTik | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ Best | ❌ | +| **Billing Integration** | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ Native | ✅ Native | ❌ | ❌ | +| **Alerting/On-Call** | ✅ | ✅ | ✅ | ✅ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ✅ | +| **Web UI** | ✅ Modern | ⚠️ Dated | ✅ | ✅ | ❌ Desktop | ✅ | ✅ | ✅ | ❌ CLI | ✅ | +| **Mobile App** | ✅ | ⚠️ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | +| **API** | ✅ REST+GraphQL | ✅ REST | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ REST | ✅ | +| **Agent/Distributed** | ✅ | ⚠️ Pollers | ⚠️ Probes | ✅ Collector | ❌ | ✅ Appliance | ❌ | ❌ | ❌ | ✅ Exporters | +| **Open Source** | ❌ | ✅ | ❌ | ❌ | ✅ Free | ❌ | ❌ | ❌ | ✅ | ✅ | +| **ISP/WISP Focus** | ✅ | ⚠️ | ❌ | ❌ | ⚠️ MikroTik | ✅ | ✅ | ✅ | ❌ | ❌ | + +--- + +## TowerOps Positioning + +### Where TowerOps Fits + +TowerOps occupies a unique position in the landscape: **the only purpose-built network monitoring platform designed specifically for ISPs and WISPs, with modern architecture and subscriber awareness.** + +The current landscape forces ISPs to choose between: +1. **Generic monitoring tools** (LibreNMS, PRTG, Grafana) that don't understand ISP operations +2. **ISP billing platforms** (Sonar, Splynx) with rudimentary monitoring bolted on +3. **QoE-only tools** (Preseem) that complement but don't replace an NMS +4. **Legacy WISP tools** (The Dude) that can't scale + +TowerOps eliminates the "duct-tape stack" of LibreNMS + Oxidized + Preseem + Grafana that most ISPs cobble together. + +### Target Customer Profile + +**Primary**: Regional ISPs and WISPs with 500-50,000 subscribers +- Running mixed equipment: MikroTik, Ubiquiti, Cambium, Cisco, Juniper +- Currently using LibreNMS or The Dude and outgrowing them +- Using Sonar or Splynx for billing and wanting monitoring that integrates +- Frustrated with managing 3-5 separate tools for network operations +- Small NOC team (1-5 people) that needs efficiency, not complexity + +**Secondary**: MSPs managing ISP/WISP clients +- Need multi-tenant monitoring with ISP-specific features +- Currently using Auvik but hitting its limits for carrier networks + +### Unique Value Propositions + +1. **ISP-native, not ISP-adapted**: Built from day one for ISP/WISP operations. Not a generic monitoring tool with ISP features bolted on. Sites, towers, sectors, subscribers — first-class concepts. + +2. **Unified platform**: SNMP monitoring + topology mapping + config management + alerting + on-call + subscriber awareness in one product. Replaces LibreNMS + Oxidized + PagerDuty + custom Grafana dashboards. + +3. **Modern architecture**: Phoenix LiveView for real-time UI, TimescaleDB for time-series (SQL-queryable, not RRDtool), distributed agent architecture for sites behind NAT/firewalls. + +4. **Subscriber-aware monitoring**: Pull subscriber data from Sonar/Splynx/VISP/Gaiia and correlate with network health. "These 47 subscribers on Tower-12 Sector-3 are experiencing degraded service" — not just "interface utilization is high." + +5. **Agent-based distributed monitoring**: Lightweight agents deployed at tower sites, headends, or POPs. Monitor behind NAT, through firewalls, with local data collection and central aggregation. No VPN tunnels or direct SNMP access needed. + +6. **Deep SNMP + fast polling**: 786 OS profiles with vendor-specific sensor discovery. 60-second default polling (vs LibreNMS's 5-minute). Environmental monitoring (temperature, power, optical levels) that Auvik and billing platforms can't do. + +7. **Billing platform integrations**: First-class integrations with Sonar, Splynx, VISP, and Gaiia. Not just API calls — bidirectional sync that enriches both platforms. + +### Features Competitors Lack + +| Feature | Why It Matters | Who's Missing It | +|---------|---------------|-----------------| +| **Site/tower hierarchy** | ISPs think in sites, not flat device lists | LibreNMS, PRTG, Auvik, Grafana | +| **Agent-based polling behind NAT** | Tower sites often lack direct connectivity | LibreNMS, PRTG, The Dude | +| **Subscriber ↔ device correlation** | "Which customers are affected by this outage?" | LibreNMS, PRTG, Auvik, Grafana | +| **Built-in on-call/escalation** | No separate PagerDuty/Opsgenie needed | LibreNMS, Preseem, Sonar, Splynx | +| **Config backup + deep monitoring** | Eliminates Oxidized as separate tool | LibreNMS, PRTG, Preseem | +| **Multi-tenant with ISP features** | MSPs managing WISPs need both | LibreNMS, The Dude, Preseem | +| **60-second polling** | Faster fault detection, better capacity data | LibreNMS (5min default) | +| **TimescaleDB time-series** | SQL-queryable historical data, continuous aggregates | LibreNMS (RRDtool), The Dude (none) | +| **Capacity planning** | Interface utilization trends and forecasting | The Dude, Preseem, billing platforms | + +### Competitive Positioning Summary + +``` + ISP/WISP Specific + ↑ + | + Preseem | TowerOps ★ + (QoE only) | (Full NMS + ISP features) + | + Sonar/Splynx | + (Billing + basic | + monitoring) | + | + Generic ←──────────────┼──────────────→ Deep Monitoring + | + The Dude | LibreNMS + (Legacy) | (Open source NMS) + | + Auvik | PRTG + (MSP focus) | (Enterprise IT) + | + Grafana+Prometheus + (DIY toolkit) + | + Generic IT +``` + +### Go-To-Market Priorities + +1. **Migrate The Dude users**: Largest addressable audience of frustrated WISP operators. Low switching cost (The Dude is free but limited). Marketing message: "You've outgrown The Dude." + +2. **Replace LibreNMS for ISPs**: Technical operators who've invested in LibreNMS but want modern UX, subscriber awareness, and less maintenance. Message: "Everything LibreNMS does, plus everything it doesn't." + +3. **Complement Sonar/Splynx**: Don't compete with billing platforms. Be the monitoring layer they recommend. Message: "The monitoring platform that actually talks to your billing system." + +4. **Displace the duct-tape stack**: ISPs running LibreNMS + Oxidized + Grafana + PagerDuty + Preseem. Message: "One platform instead of five." diff --git a/docs/CONVENTIONS.md b/docs/CONVENTIONS.md new file mode 100644 index 00000000..dd1dcb65 --- /dev/null +++ b/docs/CONVENTIONS.md @@ -0,0 +1,227 @@ +# TowerOps Coding Conventions + +Project-specific patterns and conventions for the TowerOps Phoenix LiveView application. + +## File Organization + +### LiveView Modules + +LiveView pages live under `lib/towerops_web/live/` organized by domain: + +``` +lib/towerops_web/live/ + schedule_live/ + index.ex # List page module + index.html.heex # List page template + form.ex # Create/edit form module + form.html.heex # Form template + show.ex # Detail page module + show.html.heex # Detail page template +``` + +- **`.ex`** files contain the LiveView module (mount, handle_params, handle_event, etc.) +- **`.html.heex`** files contain the corresponding HEEx template +- Module naming: `ToweropsWeb.ScheduleLive.Index`, `ToweropsWeb.ScheduleLive.Form`, etc. + +### Test Files + +Tests mirror the `lib/` structure under `test/`: + +``` +test/ + towerops/ # Context/business logic tests + towerops_web/ # Web layer tests + snmpkit/ # SNMP toolkit tests + integration/ # Integration tests (SNMP, discovery) + support/ # Test helpers and fixtures + fixtures/ # Factory/fixture modules + mix/ # Mix task tests + test_helper.exs +``` + +## Navigation & Page Structure + +### `active_page` Assign + +Every LiveView sets an `active_page` assign in `mount/3` to highlight the correct nav item: + +```elixir +def mount(_params, _session, socket) do + {:ok, assign(socket, :active_page, "schedules")} +end +``` + +Valid values: `"dashboard"`, `"devices"`, `"sites"`, `"network-map"`, `"alerts"`, `"maintenance"`, `"schedules"`, `"insights"`, `"trace"`, `"activity"`, `"settings"`, `"help"` + +The layout component (`Layouts.authenticated`) checks `@active_page` to apply the `active` class to nav links. + +### Breadcrumbs + +Use the `ToweropsWeb.Components.Breadcrumbs` component (`.breadcrumb`) for hierarchical navigation: + +```heex +<.breadcrumb items={[ + %{label: "Dashboard", navigate: ~p"/dashboard"}, + %{label: "Devices", navigate: ~p"/devices"}, + %{label: "My Device"} +]} /> +``` + +- Each item is a map with `:label` (required) and `:navigate` (optional) +- Omit `:navigate` on the last item (current page) +- The first item automatically gets a home icon + +## Forms + +### Standard Form Pattern + +Forms use `phx-change="validate"` for real-time validation and `phx-submit="save"` for submission: + +```heex +<.simple_form for={@form} phx-change="validate" phx-submit="save"> + <.input field={@form[:name]} label={t("Name")} /> + <:actions> + <.button>{t("Save")} + + +``` + +The LiveView handles both events: + +```elixir +def handle_event("validate", %{"schedule" => params}, socket) do + changeset = Schedule.changeset(socket.assigns.schedule, params) + {:noreply, assign(socket, :form, to_form(changeset, action: :validate))} +end + +def handle_event("save", %{"schedule" => params}, socket) do + # ... create or update logic +end +``` + +## Flash Messages + +Use `put_flash/3` with `:info` for success and `:error` for failures. Always wrap messages in `t()`: + +```elixir +socket +|> put_flash(:info, t("Schedule created successfully.")) +|> push_navigate(to: ~p"/schedules") +``` + +```elixir +socket +|> put_flash(:error, t("Could not save schedule.")) +``` + +## Translations (i18n) + +All user-facing strings must use the `t()` macro (or domain-specific variants). Defined in `ToweropsWeb.GettextHelpers`: + +| Macro | Domain | Example | +|-------|--------|---------| +| `t/1` | Common UI | `t("Save")`, `t("Cancel")` | +| `t_auth/1` | Authentication | `t_auth("Log in")` | +| `t_equipment/1` | Equipment | `t_equipment("Add Device")` | +| `t_admin/1` | Admin | `t_admin("Impersonate User")` | +| `t_email/1` | Emails | `t_email("Welcome!")` | + +Interpolation: `t("Welcome, %{name}!", name: user.name)` + +Error messages use the standard `gettext/1` function. + +## `current_scope` Assign + +The `@current_scope` assign is available in all authenticated views. It's a struct containing: + +- **`.user`** — the current `User` struct (email, id, `is_superuser`, etc.) +- **`.organization`** — the current `Organization` (name, slug, settings) +- **`.timezone`** — the user's timezone string (falls back to `"UTC"`) +- **`.impersonating?`** — boolean, true when a superuser is impersonating another user + +Access in templates: + +```heex +<%= @current_scope.user.email %> +<%= @current_scope.organization.name %> +``` + +Access in LiveView: + +```elixir +organization = socket.assigns.current_scope.organization +user = socket.assigns.current_scope.user +``` + +The layout displays an impersonation banner when `@current_scope.impersonating?` is true. + +## CSS & Styling + +### Tailwind CSS + +All styling uses Tailwind utility classes. The project uses **DaisyUI** as a component library on top of Tailwind. + +### Dark Mode + +Dark mode is supported via the `dark:` prefix: + +```heex +
+``` + +Always provide dark mode variants for backgrounds, text colors, and borders. + +### Icons + +Icons use Heroicons via the `.icon` component with `hero-*` names: + +```heex +<.icon name="hero-plus" /> +<.icon name="hero-pencil-square" class="size-5" /> +<.icon name="hero-check" class="size-4 text-green-300" /> +<.icon name="hero-home-mini" class="h-4 w-4" /> +``` + +Use the mini variant (`hero-*-mini`) for small inline icons and the default (outline) for larger UI elements. + +## LiveView Conventions + +### Module Boilerplate + +```elixir +defmodule ToweropsWeb.ScheduleLive.Index do + @moduledoc false + use ToweropsWeb, :live_view + + alias Towerops.OnCall + + @impl true + def mount(_params, _session, socket) do + {:ok, assign(socket, :active_page, "schedules")} + end + + @impl true + def handle_params(params, _url, socket) do + # ... + end +end +``` + +- Always use `@impl true` for callback annotations +- Set `active_page` in `mount/3` +- Use `handle_params/3` for loading data based on URL params / live_action + +### Scoped Routes + +Routes are scoped under the organization slug: + +``` +/orgs/:org_slug/devices +/orgs/:org_slug/schedules/:id +``` + +Use `~p` sigil for path generation with `@current_scope`: + +```elixir +~p"/orgs/#{@current_scope.organization.slug}/devices" +``` diff --git a/docs/SECURITY_AUDIT_2026_03_14.md b/docs/SECURITY_AUDIT_2026_03_14.md new file mode 100644 index 00000000..5da9641e --- /dev/null +++ b/docs/SECURITY_AUDIT_2026_03_14.md @@ -0,0 +1,272 @@ +# Security Audit Report — TowerOps Web Application + +**Date:** 2026-03-14 +**Scope:** Authentication, authorization, and session security +**Auditor:** Automated code review (Skippy) + +--- + +## Summary + +The TowerOps application demonstrates a well-structured security posture overall. Authentication uses Argon2 password hashing, TOTP-based MFA, session token management with expiry, and CSRF protection. Organization-scoped data access is consistently enforced through `ScopedResource` and `AccessControl` helpers. + +The findings below are verified issues found in the source code. + +--- + +## Findings + +### 1. API Device Create Allows `organization_id` Override + +**Severity:** HIGH +**File:** `lib/towerops_web/controllers/api/v1/devices_controller.ex`, lines 245–249 +**Issue:** The `default_organization_id/2` function only sets `organization_id` when the param is `nil` or empty. If a client provides a *different* `organization_id` in the request body, it is passed through to `Devices.create_device/2` unchanged. This allows an API token holder for Org A to create devices in Org B. + +```elixir +defp default_organization_id(device_params, organization_id) do + case Map.get(device_params, "organization_id") do + nil -> Map.put(device_params, "organization_id", organization_id) + "" -> Map.put(device_params, "organization_id", organization_id) + _provided_id -> device_params # ← attacker-controlled org_id accepted + end +end +``` + +**Suggested Fix:** Always force the authenticated organization_id: +```elixir +defp default_organization_id(device_params, organization_id) do + Map.put(device_params, "organization_id", organization_id) +end +``` + +--- + +### 2. Brute Force Protection Disabled in Production + +**Severity:** HIGH +**File:** `lib/towerops_web/plugs/brute_force_protection.ex`, line 38 +**Issue:** The plug's `call/2` function unconditionally returns `conn` without any processing. The comment says "TEMPORARILY DISABLED." While Cloudflare provides edge-level protection, if Cloudflare is bypassed or misconfigured, the application has no server-side brute force protection beyond rate limiting. + +```elixir +def call(conn, _opts) do + # TEMPORARILY DISABLED - brute force protection bypassed + conn +end +``` + +**Suggested Fix:** Re-enable the brute force protection or remove the plug entirely to avoid a false sense of security. At minimum, document the Cloudflare dependency as a security requirement. + +--- + +### 3. Admin API MIB/GeoIP Routes Use Application-Level Superuser Check, Not Pipeline + +**Severity:** MEDIUM +**File:** `lib/towerops_web/router.ex`, lines 168–176 +**Files affected:** `lib/towerops_web/controllers/api/v1/mib_controller.ex`, `lib/towerops_web/controllers/api/v1/geoip_controller.ex` +**Issue:** The admin API scope (`/admin/api`) uses only the `:api_v1` pipeline (Bearer token auth) without the `:require_superuser` pipeline plug. Superuser checks are done inside each controller action via `require_superuser/1`. This is defense-in-depth weakness — if a developer adds a new action and forgets the check, it would be accessible to any API token holder. + +```elixir +# Router - no superuser pipeline plug +scope "/admin/api", V1 do + pipe_through :api_v1 + # ... +end +``` + +The controllers do implement their own `require_superuser` check, which works, but the `halt()` return from within `with :ok <- require_superuser(conn)` is not propagated correctly because `halt()` is called inside the private function but `with` still proceeds. Let me verify... + +Actually, reviewing more carefully: the `require_superuser` function calls `halt()` on the conn and returns the halted conn. The `with :ok <-` pattern means if it doesn't return `:ok`, the `with` block falls through. But the `require_superuser/1` function calls `json` + `halt` and returns the conn, not `:ok`. So the `with` clause correctly handles this — if `require_superuser` returns the halted conn instead of `:ok`, the `with` doesn't match and the function returns without executing the body. **However**, the halted conn is not actually returned to the caller — it's discarded. The controller action returns normally without the halted conn. + +**Wait** — re-reading the code: the `with :ok <- require_superuser(conn)` pattern means if `require_superuser` returns anything other than `:ok`, the `with` block uses its else clause or falls through. Since there's no explicit else, the non-matching return value (the halted conn) is returned as the function's return value. Phoenix will see that the conn is halted and stop processing. This is correct. + +**Revised assessment:** The controller-level checks work correctly. The concern remains that this is a defense-in-depth gap — a pipeline-level check would be more robust. + +**Suggested Fix:** Add a `require_superuser_api` plug to the admin API pipeline: +```elixir +scope "/admin/api", V1 do + pipe_through [:api_v1, :require_superuser_api] +end +``` + +--- + +### 4. Honeybadger API Key Hardcoded in Source + +**Severity:** MEDIUM +**File:** `config/config.exs`, line 29 +**Issue:** The Honeybadger API key (`hbp_xe5xMnpLZ2XJsXQJujoEkgCmiqCfwa0uYA3Y`) is hardcoded in version-controlled source. While Honeybadger is an error-tracking service (not end-user-facing), a leaked API key could allow attackers to send fake error reports, pollute error data, or potentially access error data containing sensitive information (stack traces, request params). + +```elixir +config :honeybadger, + api_key: "hbp_xe5xMnpLZ2XJsXQJujoEkgCmiqCfwa0uYA3Y", +``` + +**Suggested Fix:** Move to an environment variable: +```elixir +config :honeybadger, + api_key: System.get_env("HONEYBADGER_API_KEY"), +``` + +--- + +### 5. Org Settings LiveView Missing Role-Based Authorization on Sensitive Operations + +**Severity:** MEDIUM +**File:** `lib/towerops_web/live/org/settings_live.ex`, lines 103–225 +**Issue:** The `handle_event` callbacks for `"save"` (update org settings), `"send_invitation"`, `"remove_member"`, `"change_role"`, `"apply_snmp_to_all"`, and `"apply_agent_to_all"` do not check the current user's role/permissions. Any authenticated user who is a member of the organization can access the settings page (it's in the `require_authenticated_user_and_organization` live_session) and perform these operations, regardless of whether they are an owner, admin, or technician. + +The `Permissions` module exists but is not imported or used in this LiveView. Only the backup delete event in `DeviceLive.Show` uses `owner?()`. + +**Suggested Fix:** Add role checks to sensitive operations: +```elixir +def handle_event("save", %{"organization" => org_params}, socket) do + import ToweropsWeb.Permissions + unless admin?(socket), do: {:noreply, put_flash(socket, :error, "Permission denied")} + # ... existing logic +end + +def handle_event("send_invitation", params, socket) do + import ToweropsWeb.Permissions + unless admin?(socket), do: {:noreply, put_flash(socket, :error, "Permission denied")} + # ... existing logic +end +``` + +--- + +### 6. Session Cookie Missing `secure: true` Flag + +**Severity:** MEDIUM +**File:** `lib/towerops_web/endpoint.ex`, lines 9–14 +**Issue:** The session cookie options don't set `secure: true`. While the comment in `prod.exs` says "SSL/TLS is handled by Cloudflared proxy," the cookie `secure` flag is independent of TLS termination — it tells the browser to only send the cookie over HTTPS connections. Without it, if a user somehow accesses the app over HTTP (e.g., direct IP access bypassing Cloudflare), the session cookie would be transmitted in cleartext. + +```elixir +@session_options [ + store: :cookie, + key: "_towerops_key", + signing_salt: "hrDZxLhd", + encryption_salt: "vK3p8mNx", + same_site: "Lax" + # missing: secure: true +] +``` + +**Suggested Fix:** Add `secure: true` for production: +```elixir +@session_options [ + store: :cookie, + key: "_towerops_key", + signing_salt: "hrDZxLhd", + encryption_salt: "vK3p8mNx", + same_site: "Lax", + secure: Application.compile_env(:towerops, :env) == :prod +] +``` + +Or configure via `config/prod.exs`. + +--- + +### 7. Remember-Me Cookie Missing `http_only: true` Flag + +**Severity:** LOW +**File:** `lib/towerops_web/user_auth.ex`, lines 19–23 +**Issue:** The `@remember_me_options` for the remember-me cookie don't explicitly set `http_only: true`. While Phoenix's `put_resp_cookie` defaults to `http_only: true`, and `secure` defaults to the conn's scheme, it's best practice to be explicit. The `secure` flag is also not set, same concern as finding #6. + +```elixir +@remember_me_options [ + sign: true, + max_age: @max_cookie_age_in_days * 24 * 60 * 60, + same_site: "Lax" + # missing explicit: http_only: true, secure: true +] +``` + +**Suggested Fix:** Be explicit: +```elixir +@remember_me_options [ + sign: true, + max_age: @max_cookie_age_in_days * 24 * 60 * 60, + same_site: "Lax", + http_only: true, + secure: true +] +``` + +--- + +### 8. No `force_ssl` Configured for Production + +**Severity:** LOW +**File:** `config/prod.exs`, line 41 +**Issue:** The comment acknowledges that SSL is handled by Cloudflared proxy, so `force_ssl` is not configured. However, without `force_ssl`, the application won't set HSTS headers. HSTS ensures browsers always use HTTPS even if a user types `http://`. This is a defense-in-depth measure. + +**Suggested Fix:** Enable `force_ssl` with HSTS even behind Cloudflare: +```elixir +config :towerops, ToweropsWeb.Endpoint, + force_ssl: [hsts: true, rewrite_on: [:x_forwarded_proto]] +``` + +The `rewrite_on: [:x_forwarded_proto]` ensures it works correctly behind a proxy. + +--- + +### 9. Alert Show Leaks Existence of Alerts Across Organizations via Error Type + +**Severity:** LOW +**File:** `lib/towerops_web/controllers/api/v1/alerts_controller.ex`, lines 32–44 +**Issue:** The `show` action uses `Alerts.get_alert!(id)` which raises `Ecto.NoResultsError` for non-existent alerts, but returns a 404 with "Alert not found" for alerts that exist in other organizations. An attacker can distinguish between "alert doesn't exist" and "alert exists but belongs to another org" by observing whether the error is caught by the rescue clause or the if-else branch. In practice, the response is the same 404 message, but the code path differs (potential timing difference). + +This is a very minor information leak. The same pattern exists in `acknowledge` and `resolve`. + +**Suggested Fix:** Use a non-raising `get_alert/1` function that returns nil: +```elixir +def show(conn, %{"id" => id}) do + organization_id = conn.assigns.current_organization_id + + case Alerts.get_alert(id) do + nil -> + conn |> put_status(:not_found) |> json(%{error: "Alert not found"}) + alert -> + if alert.device && alert.device.organization_id == organization_id do + json(conn, %{data: format_alert(alert)}) + else + conn |> put_status(:not_found) |> json(%{error: "Alert not found"}) + end + end +end +``` + +--- + +## Positive Observations + +These are **not findings** — they are security strengths worth noting: + +1. **Password hashing:** Uses Argon2 with timing-safe comparison and `no_user_verify()` to prevent user enumeration. +2. **TOTP implementation:** Properly separates TOTP from recovery codes for sudo mode. Recovery codes cannot bypass sudo verification. +3. **Session management:** Tokens expire after 14 days, sessions are reissued after 7 days, and there's a 30-minute inactivity timeout. +4. **CSRF protection:** `protect_from_forgery` plug in browser pipeline, session renewal on login to prevent fixation. +5. **Organization scoping:** `ScopedResource` pattern consistently enforces org-level isolation in API controllers. +6. **LiveView auth:** All live_sessions use `on_mount` hooks for authentication/authorization checks. +7. **Impersonation audit trail:** Impersonation start/stop creates audit log entries with IP addresses. +8. **API token hashing:** Tokens are SHA-256 hashed before storage; raw tokens shown only once. +9. **Rate limiting:** Auth endpoints limited to 10 req/min, API to 1000 req/min per IP. +10. **GraphQL hardening:** Introspection disabled in production, complexity and depth limits enforced. +11. **Webhook signature verification:** Uses `Plug.Crypto.secure_compare` for timing-safe comparison. +12. **Content Security Policy:** Properly configured with `frame-ancestors 'none'` to prevent clickjacking. +13. **MIB upload path traversal protection:** Vendor names and filenames validated; archive contents checked for path traversal. +14. **Mobile API authorization:** Each endpoint verifies the user has access to the requested organization. + +--- + +## Risk Summary + +| Severity | Count | Key Issues | +|----------|-------|-----------| +| CRITICAL | 0 | — | +| HIGH | 2 | API org_id override, brute force protection disabled | +| MEDIUM | 3 | Admin API pipeline gap, hardcoded API key, missing role checks in org settings | +| LOW | 3 | Cookie flags, no HSTS, minor info leak | + +**Overall Assessment:** No critical vulnerabilities found. The two HIGH findings should be addressed promptly — the API `organization_id` override is the most actionable. The brute force protection gap is mitigated by Cloudflare but represents a defense-in-depth weakness. diff --git a/docs/SECURITY_AUDIT_API.md b/docs/SECURITY_AUDIT_API.md new file mode 100644 index 00000000..604d5c09 --- /dev/null +++ b/docs/SECURITY_AUDIT_API.md @@ -0,0 +1,404 @@ +# Security Audit: API & External Integrations + +**Date:** 2026-03-14 +**Scope:** REST API, GraphQL API, webhook handlers, integration clients, background workers +**Auditor:** Automated code review + +--- + +## Summary + +| Severity | Count | +|----------|-------| +| High | 2 | +| Medium | 5 | +| Low | 3 | + +Overall the codebase demonstrates strong security practices: webhook signature verification uses `Plug.Crypto.secure_compare`, API tokens are hashed with SHA-256, credentials are encrypted at rest (Cloak AES-256-GCM), rate limiting is applied to auth and API endpoints, and GraphQL has complexity/depth limits. The findings below are areas for improvement. + +--- + +## HIGH Severity + +### H1: SSRF via User-Controlled Integration URLs (Sonar, Splynx) + +**Files:** +- `lib/towerops/sonar/client.ex` — lines 141-142 +- `lib/towerops/splynx/client.ex` — lines 13-15 (via `authenticate/3`), lines 95-96 (via `get/4`) +- `lib/towerops/sonar/sync.ex` — line 21 +- `lib/towerops/splynx/sync.ex` — line 20 + +**Description:** +Sonar and Splynx integrations accept a user-supplied `instance_url` stored in `integration.credentials["instance_url"]`. This URL is directly concatenated into HTTP requests without any validation: + +```elixir +# sonar/client.ex:142 +url = String.trim_trailing(instance_url, "/") <> "/api/graphql" + +# splynx/client.ex — authenticate/3 and get/4 +url = String.trim_trailing(instance_url, "/") <> "/api/2.0/admin/auth/tokens" +``` + +An attacker with org-admin access could set `instance_url` to `http://169.254.169.254` (cloud metadata), `http://localhost:5432`, or any internal service, causing the server to make requests to internal network resources. The integration sync workers then execute these requests periodically in the background. + +**Impact:** Server-Side Request Forgery (SSRF). Could access cloud metadata endpoints (AWS/GCP instance credentials), internal services, or scan internal networks. + +**Suggested Fix:** +Add URL validation in `Towerops.Integrations` when creating/updating integrations: + +```elixir +defp validate_instance_url(changeset) do + validate_change(changeset, :credentials, fn :credentials, creds -> + case creds["instance_url"] do + nil -> [] + url -> + uri = URI.parse(url) + cond do + uri.scheme not in ["https", "http"] -> + [credentials: "instance_url must use http or https"] + is_private_ip?(uri.host) -> + [credentials: "instance_url cannot point to a private IP address"] + uri.host in ["localhost", "127.0.0.1", "::1", "0.0.0.0"] -> + [credentials: "instance_url cannot point to localhost"] + true -> [] + end + end + end) +end +``` + +Also consider resolving the hostname and checking the IP before making requests. + +--- + +### H2: Admin API Routes Missing Rate Limiting + +**File:** `lib/towerops_web/router.ex` — lines 172-180 + +**Description:** +The admin API scope (`/admin/api`) only pipes through `:api_v1` but does **not** include `:rate_limit_api` or any rate limiting pipeline: + +```elixir +scope "/admin/api", V1 do + pipe_through :api_v1 # <-- no rate_limit_api + + get "/mibs", MibController, :index + post "/mibs", MibController, :upload + delete "/mibs/:vendor", MibController, :delete + post "/geoip/import", GeoipController, :import_database +end +``` + +While these endpoints do check for superuser status in the controller, a compromised superuser API token could be used to flood the server with MIB uploads or GeoIP imports without any rate limiting, potentially causing DoS. + +**Impact:** Denial of service via resource exhaustion (disk space from MIB uploads, database from GeoIP imports). + +**Suggested Fix:** +Add rate limiting to the admin API pipeline: + +```elixir +scope "/admin/api", V1 do + pipe_through [:api_v1, :rate_limit_api] # or a stricter :admin_rate_limit + ... +end +``` + +--- + +## MEDIUM Severity + +### M1: GraphQL Complexity/Depth Limits Disabled in Non-Production Environments + +**File:** `lib/towerops_web/graphql/context.ex` — lines 15-20 + +**Description:** +The GraphQL context plug conditionally applies complexity analysis only in production: + +```elixir +opts = + if Application.get_env(:towerops, :env) == :prod do + [context: context, analyze_complexity: true, max_complexity: 500] + else + [context: context] + end +``` + +Note: The router (`router.ex:139-141`) does set `analyze_complexity: true, max_complexity: 500, max_depth: 10` at the Absinthe.Plug level. However, the Context plug **overwrites** these options via `Absinthe.Plug.put_options/2`, which replaces the entire options map. In non-production environments, this effectively removes the complexity and depth limits set in the router. + +**Impact:** In staging/dev environments (which may be publicly accessible), deeply nested or highly complex queries could cause CPU/memory exhaustion. + +**Suggested Fix:** +Apply complexity limits in all environments: + +```elixir +def call(conn, _opts) do + context = build_context(conn) + Absinthe.Plug.put_options(conn, + context: context, + analyze_complexity: true, + max_complexity: 500 + ) +end +``` + +--- + +### M2: GraphQL Introspection Block is Bypassable + +**File:** `lib/towerops_web/plugs/graphql_introspection.ex` — lines 18-24 + +**Description:** +The introspection check uses simple string matching: + +```elixir +String.contains?(query, "__schema") or String.contains?(query, "__type") +``` + +This can be bypassed using: +1. **GraphQL aliases:** `query { myAlias: __schema { types { name } } }` — Actually still contains `__schema`, so this specific bypass won't work. +2. **Batch requests:** If the `query` parameter is sent as part of a JSON array (batch queries), `conn.body_params` won't match the `%{"query" => query}` pattern since it would be a list, not a map. +3. **The `operationName` + `variables` pattern:** The check only examines the `query` field, but if the GraphQL request uses persisted queries or extensions, the query text may not be in `body_params["query"]`. + +More critically, the check only runs on `body_params["query"]`. If the query is sent via GET request with query parameters (which Absinthe supports by default), the body params would be empty and the check is bypassed entirely. + +**Impact:** Schema exposure in production, revealing the full API surface to attackers. + +**Suggested Fix:** +Also check query params for GET requests, and consider using Absinthe's built-in introspection control: + +```elixir +# In schema.ex - more reliable approach +def context(ctx) do + if Application.get_env(:towerops, :env) == :prod do + Map.put(ctx, :introspection, false) + else + ctx + end +end +``` + +Or disable introspection at the Absinthe level by adding a phase or middleware. + +--- + +### M3: Webhook Endpoints (Gaiia, PagerDuty) Lack Dedicated Rate Limiting + +**File:** `lib/towerops_web/router.ex` — lines 163-169 + +**Description:** +The Gaiia and PagerDuty webhook routes share the general `:rate_limit_api` (1000 req/min per IP): + +```elixir +scope "/api/v1/webhooks", V1 do + pipe_through [:api, :rate_limit_api] + + post "/gaiia/:organization_id", GaiiaWebhookController, :create + post "/pagerduty/:organization_id", PagerdutyWebhookController, :create +end +``` + +These endpoints are unauthenticated (signature verification happens in the controller after the request is processed). At 1000 requests/minute, an attacker can: +- Force the server to perform 1000 database lookups per minute (to find the integration and its webhook secret) +- Force 1000 HMAC computations per minute +- The `organization_id` in the URL can be enumerated to discover valid organizations + +**Impact:** Organization enumeration and moderate DoS potential through resource consumption. + +**Suggested Fix:** +Apply stricter rate limiting (e.g., 60 req/min) to webhook endpoints: + +```elixir +pipeline :rate_limit_webhook do + plug RateLimit, type: :webhook # Add a :webhook type with lower limits +end +``` + +--- + +### M4: Stripe Webhook Raw Body May Be Empty + +**File:** `lib/towerops_web/controllers/api/v1/stripe_webhook_controller.ex` — line 14 + +**Description:** +The Stripe webhook controller reads the raw body with a fallback to empty string: + +```elixir +raw_body = conn.private[:raw_body] || "" +``` + +If `raw_body` is empty (e.g., due to a middleware issue or the body being consumed), `verify_signature/2` would still attempt verification against an empty payload. The `parse_signature_header/1` would crash on `nil` input if the `stripe-signature` header is also missing (since `List.first([])` returns `nil`, and `String.split(nil, ",")` raises). + +**Impact:** Unhandled exception could leak stack traces in non-production error responses, and the empty body fallback could mask misconfiguration issues. + +**Suggested Fix:** +Add explicit nil/empty checks: + +```elixir +def create(conn, _params) do + with signature when is_binary(signature) <- get_req_header(conn, "stripe-signature") |> List.first(), + raw_body when is_binary(raw_body) and raw_body != "" <- conn.private[:raw_body] do + # ... process + else + nil -> conn |> put_status(400) |> json(%{error: "Missing signature header"}) + "" -> conn |> put_status(400) |> json(%{error: "Empty request body"}) + end +end +``` + +--- + +### M5: `to_atom_keys` in IntegrationsController Could Crash on Invalid Keys + +**File:** `lib/towerops_web/controllers/api/v1/integrations_controller.ex` — lines 89-93 + +**Description:** +The `to_atom_keys/1` helper converts user-supplied string keys to atoms using `String.to_existing_atom/1`: + +```elixir +defp to_atom_keys(map) when is_map(map) do + Map.new(map, fn {k, v} -> {String.to_existing_atom(k), v} end) +rescue + ArgumentError -> map +end +``` + +While `String.to_existing_atom/1` is safe (won't create new atoms), the `rescue` clause catches `ArgumentError` and returns the original map with string keys. This means: +1. If ANY key is not an existing atom, ALL keys remain as strings (including valid ones) +2. The changeset then receives a mix of expected atom keys and unexpected string keys, potentially silently dropping valid fields + +This is a correctness bug more than a security issue, but it could lead to integration updates silently failing to apply credential changes. + +**Suggested Fix:** +Convert keys individually, falling back per-key: + +```elixir +defp to_atom_keys(map) when is_map(map) do + Map.new(map, fn {k, v} -> + key = try do + String.to_existing_atom(k) + rescue + ArgumentError -> k + end + {key, v} + end) +end +``` + +--- + +## LOW Severity + +### L1: API Token Verification Uses Database Lookup Instead of Constant-Time Comparison + +**File:** `lib/towerops/api_tokens.ex` — lines 79-97 + +**Description:** +API token verification hashes the input token and does a database lookup: + +```elixir +def verify_token(raw_token) do + token_hash = hash_token(raw_token) + case Repo.get_by(ApiToken, token_hash: token_hash) do + nil -> {:error, :invalid_token} + token -> ... + end +end +``` + +This is the standard and correct approach for hashed tokens — database lookup timing is dominated by query execution time, making timing attacks impractical. The hash comparison happens in PostgreSQL which uses constant-time comparison for indexed lookups. + +However, the SHA-256 hash is not salted. While this doesn't enable practical attacks (tokens are 32 random bytes with ~256 bits of entropy), salted hashing (e.g., bcrypt) would provide defense-in-depth if the database were compromised. + +**Impact:** Theoretical — if the database is breached, unsalted SHA-256 hashes of tokens are exposed. Given the high entropy of the tokens, brute-forcing is infeasible. + +**No immediate action required.** Note for future consideration. + +--- + +### L2: Mobile Session Token Lookup Timing + +**File:** `lib/towerops/mobile_sessions.ex` — lines 38-47 + +**Description:** +Mobile session tokens are hashed and looked up via database query: + +```elixir +def get_session_by_token(token) when is_binary(token) do + hashed = MobileSession.hash_token(token) + MobileSession + |> where([s], s.token == ^hashed) + |> where([s], s.expires_at > ^now) + |> Repo.one() +end +``` + +Same pattern as API tokens — hash-then-lookup. This is secure for the same reasons as L1. + +**No action required.** + +--- + +### L3: Gaiia Webhook Signature Mismatch Logging Leaks Partial Signatures + +**File:** `lib/towerops_web/controllers/api/v1/gaiia_webhook_controller.ex` — lines 96-103 + +**Description:** +On signature mismatch, the controller logs partial signature values: + +```elixir +Logger.warning( + "Gaiia webhook signature mismatch — " <> + "ts=#{timestamp} body_len=#{byte_size(raw_body)} " <> + "secret_len=#{byte_size(secret)} " <> + "expected=#{String.slice(expected, 0, 12)}… " <> + "received=#{String.slice(v1_signature, 0, 12)}…" +) +``` + +This logs: +- The length of the webhook secret (`secret_len`) +- The first 12 characters of the expected HMAC signature + +While the HMAC output is not the secret itself, logging the secret length reveals information that could help an attacker. The first 12 hex chars of the expected signature provide limited information (6 bytes of a 32-byte HMAC). + +**Impact:** Minor information leakage in logs. Not directly exploitable but violates principle of least information. + +**Suggested Fix:** +Remove `secret_len` and `expected` from the log message: + +```elixir +Logger.warning( + "Gaiia webhook signature mismatch — " <> + "ts=#{timestamp} body_len=#{byte_size(raw_body)}" +) +``` + +--- + +## Positive Findings (No Issues) + +These areas were reviewed and found to be properly secured: + +1. **Webhook signature verification:** All three webhook handlers (Stripe, Gaiia, PagerDuty) use `Plug.Crypto.secure_compare/2` for constant-time signature comparison. ✅ + +2. **Agent webhook auth:** Uses `Plug.Crypto.secure_compare/2` against a shared secret. ✅ + +3. **Rate limiting:** Applied to auth endpoints (10/min), API endpoints (1000/min), and admin browser routes (100/min). ✅ + +4. **GraphQL depth/complexity limits:** Router configures `max_depth: 10` and `max_complexity: 500` with `analyze_complexity: true`. ✅ (But see M1 about context plug overwriting in non-prod.) + +5. **Credentials at rest:** Integration credentials use `Encrypted.Map` (Cloak AES-256-GCM). ✅ + +6. **CSRF protection:** Browser pipeline includes `:protect_from_forgery`. ✅ + +7. **Security headers:** CSP, frame-ancestors, and secure browser headers configured. ✅ + +8. **MIB upload path traversal protection:** Vendor names validated with regex, extracted archive paths validated against extraction directory. ✅ + +9. **Oban worker error handling:** Workers use `max_attempts` configuration and return `:ok` on non-retryable errors, `{:error, reason}` on retryable errors. No infinite retry loops found. ✅ + +10. **Organization scoping:** API controllers consistently scope queries to `conn.assigns.current_organization_id`. ✅ + +11. **Integration client error handling:** All HTTP clients (VISP, Sonar, Splynx, Gaiia, Preseem) handle error responses gracefully with pattern matching and don't leak credentials in error messages. ✅ + +12. **Token generation:** Uses `:crypto.strong_rand_bytes(32)` for API tokens — cryptographically secure. ✅ diff --git a/docs/SECURITY_AUDIT_INPUT_VALIDATION.md b/docs/SECURITY_AUDIT_INPUT_VALIDATION.md new file mode 100644 index 00000000..6a14f29e --- /dev/null +++ b/docs/SECURITY_AUDIT_INPUT_VALIDATION.md @@ -0,0 +1,350 @@ +# Security Audit: Input Validation, Injection & Data Handling + +**Date:** 2026-03-14 +**Auditor:** Automated Security Review (Skippy) +**Scope:** `lib/towerops/`, `lib/towerops_web/live/`, `lib/towerops_web/controllers/`, `lib/towerops_web/components/` + +--- + +## Summary + +| Severity | Count | +|----------|-------| +| 🔴 High | 3 | +| 🟠 Medium | 5 | +| 🟡 Low | 4 | + +--- + +## 🔴 HIGH Severity + +### H1: IDOR — API Device Creation Allows Cross-Organization Assignment + +**File:** `lib/towerops_web/controllers/api/v1/devices_controller.ex`, lines 245–250 +**File:** `lib/towerops/devices/device.ex`, lines 133–140 (`organization_id` in cast list) + +**Issue:** The `default_organization_id/2` function only sets `organization_id` when it's `nil` or `""`. If the caller provides a different `organization_id`, it passes through unmodified. Since `organization_id` is in the Device changeset's `cast` list, an authenticated API user can create devices in other organizations by supplying a foreign `organization_id` in the request body. + +```elixir +# devices_controller.ex:245-250 +defp default_organization_id(device_params, organization_id) do + case Map.get(device_params, "organization_id") do + nil -> Map.put(device_params, "organization_id", organization_id) + "" -> Map.put(device_params, "organization_id", organization_id) + _provided_id -> device_params # ← User-supplied org_id passes through! + end +end +``` + +**Impact:** An authenticated user with a valid API token can create devices (and trigger SNMP discovery) in any organization they know the ID of. + +**Fix:** Always force the authenticated organization's ID: +```elixir +defp default_organization_id(device_params, organization_id) do + Map.put(device_params, "organization_id", organization_id) +end +``` + +--- + +### H2: IDOR — Agent Token Deletion Without Organization Scoping + +**File:** `lib/towerops_web/live/agent_live/index.ex`, line 163 +**File:** `lib/towerops/agents.ex`, line 451 + +**Issue:** The `"delete_agent"` event handler passes the user-supplied `id` directly to `Agents.delete_agent_token(id)` without verifying the agent token belongs to the current user's organization. `delete_agent_token/1` fetches by raw ID with `Repo.get!`, no org check. + +```elixir +# agent_live/index.ex:163 +def handle_event("delete_agent", %{"id" => id}, socket) do + case Agents.delete_agent_token(id) do # ← No org scope check! +``` + +**Impact:** A user could delete agent tokens belonging to other organizations by crafting a WebSocket event with a known agent token ID. + +**Fix:** Verify organization ownership before deletion: +```elixir +def handle_event("delete_agent", %{"id" => id}, socket) do + organization = socket.assigns.current_scope.organization + case Agents.get_organization_agent_token(organization.id, id) do + nil -> {:noreply, put_flash(socket, :error, "Agent not found")} + agent_token -> + case Agents.delete_agent_token(agent_token.id) do + # ... + end + end +end +``` + +--- + +### H3: IDOR — Schedule/Escalation Policy Access Without Organization Check + +**Files:** +- `lib/towerops_web/live/schedule_live/show.ex`, line 19 +- `lib/towerops_web/live/schedule_live/form.ex`, line 27 +- `lib/towerops_web/live/escalation_policy_live/show.ex`, line 10 +- `lib/towerops_web/live/escalation_policy_live/form.ex`, line 27 + +**Issue:** These LiveViews fetch schedules and escalation policies by raw ID (`OnCall.get_schedule!(id)`, `OnCall.get_escalation_policy!(id)`) without verifying the resource belongs to the user's current organization. Any authenticated user who knows a schedule/policy UUID can view, edit, and delete it. + +```elixir +# schedule_live/show.ex:19 +schedule = OnCall.get_schedule!(id) # ← No org check +# escalation_policy_live/show.ex:10 +policy = OnCall.get_escalation_policy!(id) # ← No org check +``` + +**Impact:** Cross-organization read/write/delete access to on-call schedules and escalation policies. + +**Fix:** Add organization scoping to the queries: +```elixir +def get_schedule!(id, organization_id) do + Schedule + |> where(id: ^id, organization_id: ^organization_id) + |> Repo.one!() + |> Repo.preload(overrides: :user, layers: [members: :user]) +end +``` + +Then use `OnCall.get_schedule!(id, organization.id)` in the LiveViews. + +--- + +## 🟠 MEDIUM Severity + +### M1: LIKE Wildcard Injection in Multiple Search Functions + +**Files:** +- `lib/towerops/devices.ex`, line 148 (`search_devices/2`) +- `lib/towerops/sites.ex`, line 40 (`search_sites/2`) +- `lib/towerops/gaiia.ex`, lines 122, 133, 208 (`search_inventory_items`, `search_network_sites`, `suggest_site_matches`) + +**Issue:** These functions build ILIKE patterns as `"%#{query}%"` without escaping `%` and `_` wildcards. A user can submit `%` as a search query to match all records, or use `_` for single-character wildcards. + +Compare with `lib/towerops/search.ex` (lines 89–93) and `lib/towerops/trace.ex` (lines 456–461) which correctly call `sanitize`/`sanitize_like`. + +```elixir +# devices.ex:148 — NO sanitization +search_term = "%#{query}%" + +# search.ex:89 — CORRECT +defp sanitize(query) do + query + |> String.replace("%", "\\%") + |> String.replace("_", "\\_") +end +``` + +**Impact:** Information disclosure — users can craft patterns to enumerate all records or perform wildcard-based probing. Low direct risk since queries are org-scoped, but violates defense-in-depth. + +**Fix:** Add `sanitize_like/1` to all search functions that build ILIKE patterns, or extract a shared helper. + +--- + +### M2: `Jason.decode!` on Untrusted Input (Crash on Malformed JSON) + +**File:** `lib/towerops_web/live/device_live/index.ex`, line 196 + +**Issue:** `Jason.decode!(params["identifier"])` will raise a `Jason.DecodeError` if the client sends malformed JSON. While Phoenix LiveView catches this and the process restarts, it causes a brief disruption for the user and generates noisy error logs. + +```elixir +def handle_event("add_discovered_device", params, socket) do + identifier = Jason.decode!(params["identifier"]) # ← Crashes on bad input +``` + +**Impact:** Denial of service for the user's LiveView session. The process restarts, but data in assigns is lost. + +**Fix:** Use `Jason.decode/1` with error handling: +```elixir +case Jason.decode(params["identifier"]) do + {:ok, identifier} -> # proceed + {:error, _} -> {:noreply, put_flash(socket, :error, "Invalid identifier")} +end +``` + +--- + +### M3: Role Change Without Caller Permission Check + +**File:** `lib/towerops_web/live/org/settings_live.ex`, lines 208–223 + +**Issue:** The `"change_role"` event handler calls `Organizations.update_member_role/3` without checking whether the **current user** has permission to change roles. Any authenticated member of the organization can change anyone else's role (except the owner's). A `member` or `viewer` role user could promote themselves to `admin`. + +The `update_member_role/3` function only prevents changing the owner's role, but doesn't verify the caller's role: + +```elixir +def update_member_role(organization_id, user_id, new_role) do + case get_membership(organization_id, user_id) do + %Membership{role: :owner} -> {:error, :cannot_change_owner_role} + %Membership{} = membership -> update_membership(membership, %{role: new_role}) + # ← No check that the CALLER is an owner/admin + end +end +``` + +**Impact:** Privilege escalation — any org member can promote themselves to `admin` role. + +**Fix:** Check that the current user is an owner or admin before allowing role changes: +```elixir +def handle_event("change_role", %{"user-id" => user_id, "role" => role}, socket) do + organization = socket.assigns.organization + current_user = socket.assigns.current_scope.user + + # Verify caller has permission + caller_membership = Organizations.get_membership(organization.id, current_user.id) + unless caller_membership.role in [:owner, :admin] do + {:noreply, put_flash(socket, :error, "You don't have permission to change roles")} + else + # proceed with role change + end +end +``` + +--- + +### M4: SSRF via HTTP Monitoring Checks + +**File:** `lib/towerops/monitoring/executors/http_executor.ex` +**File:** `lib/towerops_web/live/check_live/form_component.ex`, lines 282–303 + +**Issue:** Users can create HTTP monitoring checks with arbitrary URLs. The executor makes server-side HTTP requests to these URLs without restricting the target. This could be used to probe internal services (metadata endpoints, localhost services, internal APIs). + +```elixir +# http_executor.ex — No URL validation/restriction +opts = [ + method: String.to_atom(method), + url: Map.fetch!(config, "url"), # ← Any URL, including internal +``` + +**Impact:** Server-side request forgery (SSRF). Can probe internal network, cloud metadata services (169.254.169.254), and localhost services. + +**Fix:** Validate URLs against an allowlist or block internal/private IP ranges before making the request. At minimum, block requests to `127.0.0.0/8`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `169.254.169.254`, and `::1`. + +--- + +### M5: `inspect()` Leaking Internal Details in API Error Responses + +**Files:** +- `lib/towerops_web/controllers/api/v1/mib_controller.ex`, line 93 +- `lib/towerops_web/controllers/api/v1/agent_release_webhook_controller.ex`, line 26 + +**Issue:** These endpoints return `inspect(reason)` in JSON error responses, which can leak internal Elixir data structures, module names, and system paths to API clients. + +```elixir +# mib_controller.ex:93 +|> json(%{error: inspect(reason)}) # ← Leaks internal error details + +# agent_release_webhook_controller.ex:26 +|> json(%{status: "error", error: inspect(reason)}) +``` + +**Impact:** Information disclosure — internal error details aid attackers in understanding the application structure. + +**Fix:** Return generic error messages to clients and log the detailed error server-side: +```elixir +Logger.error("MIB listing failed: #{inspect(reason)}") +json(conn, %{error: "Internal server error"}) +``` + +--- + +## 🟡 LOW Severity + +### L1: `String.to_integer/1` on URL Params Without Validation + +**Files:** +- `lib/towerops_web/live/device_live/index.ex`, line 67: `String.to_integer()` +- `lib/towerops_web/live/device_live/show.ex`, line 76: `String.to_integer()` + +**Issue:** `String.to_integer/1` raises `ArgumentError` on non-numeric input. If a user crafts a URL with `?page=abc`, the LiveView process crashes. + +**Impact:** Minor DoS — LiveView process restarts. Low severity because LiveView handles this gracefully with reconnection. + +**Fix:** Use `Integer.parse/1` with a default fallback: +```elixir +page = case Integer.parse(Map.get(params, "page", "1")) do + {n, ""} when n > 0 -> n + _ -> 1 +end +``` + +--- + +### L2: Unverified TOTP Device Deletion Race + +**File:** `lib/towerops_web/live/user_settings_live/totp_manager.ex`, line 57 + +**Issue:** `Repo.get!(UserTotpDevice, device_id)` fetches the TOTP device by ID from `socket.assigns.new_device_id` without verifying it still belongs to the current user. While `new_device_id` is set server-side, a race condition exists if the user opens the verification flow in multiple tabs. + +**Impact:** Minimal — the device ID comes from server-side assigns, not user input. Theoretical only. + +**Fix:** Scope the query: `Repo.get_by!(UserTotpDevice, id: device_id, user_id: user.id)` + +--- + +### L3: `raw()` Usage in Registration Template with `t_auth()` Translations + +**File:** `lib/towerops_web/controllers/user_registration_html/new.html.heex`, lines 164, 187 + +**Issue:** The `raw()` call wraps the output of `t_auth()` translation, which includes interpolated values. If translation strings are ever sourced from user-editable content (CMS, admin panel), this becomes an XSS vector. Currently safe because translations are compile-time `.po` files. + +```heex +raw(~s(#{t_auth("Privacy Policy")})) +``` + +**Impact:** Currently no risk (compile-time translations). Becomes XSS if translation source changes. + +**Fix:** Use HEEx components instead of `raw()` for link construction: +```heex +<%= t_auth("Privacy Policy") %> +``` + +--- + +### L4: HTTP Executor `String.to_atom/1` on User Input + +**File:** `lib/towerops/monitoring/executors/http_executor.ex`, line 57 + +**Issue:** `String.to_atom(method)` converts the HTTP method from config to an atom. While the method comes from check config (not direct user request params), atoms are not garbage collected. Repeated creation of unique method strings could leak atoms, though in practice the valid HTTP methods are a small fixed set. + +```elixir +method: String.to_atom(method), # ← Atom table pollution risk +``` + +**Impact:** Theoretical atom table exhaustion. Low because check config is validated and limited. + +**Fix:** Use `String.to_existing_atom/1` or a whitelist: +```elixir +defp method_to_atom("get"), do: :get +defp method_to_atom("post"), do: :post +defp method_to_atom("put"), do: :put +defp method_to_atom("delete"), do: :delete +defp method_to_atom("head"), do: :head +defp method_to_atom("patch"), do: :patch +defp method_to_atom(_), do: :get +``` + +--- + +## ✅ Positive Findings (Things Done Well) + +1. **AccessControl module** (`lib/towerops_web/live/helpers/access_control.ex`) — Devices, sites, and alerts are properly scoped to organization. Good pattern. + +2. **LIKE sanitization** in `Search`, `Trace`, and `Snmp` modules — Properly escapes `%` and `_` wildcards. + +3. **MIB Controller path traversal protection** — Vendor names are validated against `^[a-zA-Z0-9_-]+$`, extracted archive paths are validated against the extraction directory, and filenames use `Path.basename()`. + +4. **Stripe webhook signature verification** — Uses `Plug.Crypto.secure_compare/2` for timing-safe comparison and validates timestamp tolerance. + +5. **Error JSON handler** — `ErrorJSON` returns generic 500 messages without leaking internals. + +6. **CSRF protection** — Phoenix's built-in CSRF tokens are active for all forms. + +7. **HEEx auto-escaping** — All templates use `{@variable}` syntax which auto-escapes by default. No XSS from dynamic user content in templates. + +8. **Ecto parameterized queries** — All `fragment()` calls use `^variable` bindings, preventing SQL injection. + +9. **Encrypted sensitive fields** — SNMP passwords and MikroTik credentials use `Encrypted.Binary` type. + +10. **Backup deletion** — Properly verifies device organization access via `AccessControl.verify_device_access/2` before allowing deletion. diff --git a/docs/TEST_COVERAGE.md b/docs/TEST_COVERAGE.md new file mode 100644 index 00000000..fae145a3 --- /dev/null +++ b/docs/TEST_COVERAGE.md @@ -0,0 +1,234 @@ +# TowerOps Test Coverage Report + +**Generated:** 2026-03-13 +**Test Run:** 90 doctests, 55 properties, 8,134 tests | 4 failures, 38 skipped, 231 excluded +**Duration:** 158.2s (40.5s async, 117.6s sync) + +## Summary + +| Metric | Count | +|---|---| +| Total test files | 547 | +| Total tests (incl. doctests/properties) | 8,279 | +| Passing | 8,237 | +| Failures | 4 | +| Skipped | 38 | +| Excluded | 231 | +| Source modules (excl. vendor profiles, migrations, proto) | ~230 | +| Web modules (controllers, LiveViews, plugs, etc.) | ~140 | +| Modules with NO test file | ~100 | + +## Current Failures (4) + +1. **`ToweropsWeb.AgentLive.IndexTest`** — Unique constraint violation on `agents.slug` during setup (test data collision) +2. **`ToweropsWeb.AgentLive.IndexTest`** — Same slug collision in cloud poller creation setup +3. **`ToweropsWeb.AgentLive.IndexTest`** — Same root cause, different describe block +4. **`ToweropsWeb.GraphQL.Resolvers.TimeSeriesTest`** — Deadlock on `create_check_result` (async test hitting `ShareRowExclusiveLock`) + +> Both are infrastructure/setup issues, not logic bugs. The agent slug collision likely needs a unique factory counter; the deadlock needs `async: false` or transaction isolation. + +--- + +## Test Infrastructure & Patterns + +### Case Templates +- **`ToweropsWeb.ConnCase`** — HTTP/LiveView tests. Imports `Phoenix.ConnTest`, `Plug.Conn`, verified routes. Has `register_and_log_in_user/1` and sudo variant. +- **`Towerops.DataCase`** — Context/schema tests. Imports `Ecto`, `Ecto.Changeset`, `Ecto.Query`, `Towerops.Repo`. SQL Sandbox isolation. +- **`ToweropsWeb.LiveViewTestHelpers`** — Shared helpers for LiveView tests. + +### Fixtures (Factory Pattern) +All in `test/support/fixtures/`: +- `accounts_fixtures.ex` — Users, tokens, TOTP +- `agents_fixtures.ex` — Agents, assignments, tokens +- `billing_fixtures.ex` — Stripe data +- `devices_fixtures.ex` — Devices, firmware +- `gaiia_fixtures.ex` — GAIIA accounts, subscribers +- `integrations_fixtures.ex` — Integration configs +- `jobs_fixtures.ex` — Oban jobs +- `maintenance_fixtures.ex` — Maintenance windows +- `on_call_fixtures.ex` — Schedules, escalation policies +- `organizations_fixtures.ex` — Orgs, memberships, invitations +- `snmp_fixtures.ex` — SNMP devices, interfaces + +### Test Types +- **Async tests:** ~442 files use `async: true` +- **Property tests:** 55 (using `StreamData`) +- **Doctests:** 90 +- **Integration tests:** `test/integration/` (SNMP, discovery parity) +- **SnmpKit tests:** Dedicated `test/snmpkit/` directory with ~45 test files + +### Mocking +- **`Mox`** — Used for behaviour-based mocks (SNMP, ping, HTTP) +- **`ping_stub.ex`** — Stub for ping behaviour +- **`snmp_kit_mock.ex`** — SNMP client mock + +--- + +## Coverage by Domain + +### ✅ Well-Covered Domains + +| Domain | Source Files | Test Files | Notes | +|---|---|---|---| +| **Accounts** | 13 | 14+ | Comprehensive auth, TOTP, sessions, HIBP, browser sessions | +| **Agents** | 5 | 8+ | Assignments, tokens, stats, release checker, cloud poller | +| **Alerts** | 2 | 2 | Alert schema + context | +| **API Tokens** | 2 | 2 | Token CRUD | +| **Billing** | 5 | 4 | Stripe client, webhook processor, notifier (missing: `stripe_webhook_event` schema) | +| **Devices** | 14 | 10+ | Core CRUD, firmware, backups, differ, version comparator | +| **GAIIA** | 14 | 13 | Excellent: sync, reconciliation, subscriber matching, impact analysis | +| **Organizations** | 6 | 7+ | Policies, invitations, subscriptions, property tests | +| **Preseem** | 14 | 14 | Full coverage: sync, baselines, insights, fleet intelligence | +| **Profiles** | 7 | 5 | YAML profiles, device/sensor OIDs | +| **SNMP Core** | 30+ | 25+ | Discovery, polling, sanitizer, sensor change, MIB parsing | +| **SNMP Vendors** | 180+ | 180+ | Auto-generated, 1:1 coverage | +| **Workers** | 30 | 22 | Most critical workers covered | +| **Web: Plugs** | 13 | 11 | Auth, rate limiting, security headers, brute force | +| **Web: Controllers** | 25+ | 25+ | API v1, admin, auth flow controllers | +| **Web: LiveViews** | 40+ | 35+ | Most major pages tested | + +### ⚠️ Gaps — Modules Without Test Files + +#### 🔴 Critical (Needs Tests NOW) + +These modules handle auth, billing, alert routing, or data mutations with no dedicated tests: + +| Module | Why Critical | +|---|---| +| `Towerops.Monitoring` (context) | Core monitoring orchestration — check creation, status evaluation | +| `Towerops.Monitoring.Executor` | Dispatches monitoring checks to executors | +| `Towerops.Monitoring.Executors.DnsExecutor` | DNS monitoring — untested execution path | +| `Towerops.Monitoring.Executors.HttpExecutor` | HTTP monitoring — untested execution path | +| `Towerops.Monitoring.Executors.TcpExecutor` | TCP monitoring — untested execution path | +| `Towerops.Monitoring.Executors.SnmpInterfaceExecutor` | SNMP interface monitoring | +| `Towerops.Monitoring.Executors.SnmpProcessorExecutor` | SNMP CPU monitoring | +| `Towerops.Monitoring.Executors.SnmpStorageExecutor` | SNMP storage monitoring | +| `Towerops.OnCall.Notifier` | Sends on-call notifications — critical alert path | +| `Towerops.OnCall.Incident` | Incident lifecycle — create, ack, resolve | +| `Towerops.OnCall.EscalationPolicy` | Escalation rule evaluation | +| `Towerops.OnCall.Notification` | Notification delivery schema | +| `Towerops.PagerDuty.Notifier` | External PagerDuty alert dispatch | +| `Towerops.Security.CloudflareClient` | Cloudflare IP banning — security mutation | +| `Towerops.Security.IpBlock` / `IpWhitelist` | IP access control schemas | +| `Towerops.Billing.StripeWebhookEvent` | Stripe event persistence schema | +| `Towerops.Accounts.UserRecoveryCode` | Account recovery — security-critical | +| `Towerops.Accounts.UserToken` | Token schema (tested indirectly via Accounts context but no direct schema test) | + +#### 🟠 High Priority + +| Module | Why | +|---|---| +| `Towerops.Maintenance` (context) | Maintenance window CRUD and scheduling logic | +| `Towerops.ConfigChanges` (context) | Config change tracking and correlation | +| `Towerops.ConfigChanges.Correlator` | Correlates config changes to alerts | +| `Towerops.Topology.Lldp` | LLDP neighbor discovery logic | +| `Towerops.Topology.Identifier` | Device identification heuristics | +| `Towerops.Netbox.Sync` | External NetBox sync — data import | +| `Towerops.Sonar.Client` / `Sync` | Sonar billing integration | +| `Towerops.Weather.Client` | Weather API integration | +| `Towerops.Workers.EscalationCheckWorker` | Periodic escalation evaluation | +| `Towerops.Workers.CheckWorker` | Monitoring check scheduling | +| `Towerops.Workers.MikrotikBackupWorker` | MikroTik backup orchestration | +| `Towerops.Workers.CloudflareBanWorker` | Security: auto-ban via Cloudflare | +| `ToweropsWeb.Controllers.Api.V1.PagerdutyWebhookController` | Inbound PagerDuty webhooks | +| `ToweropsWeb.Controllers.Api.V1.ActivityController` | Activity feed API | +| `ToweropsWeb.Controllers.Api.V1.CheckResultsController` | Check results API | +| `ToweropsWeb.Controllers.InvitationController` | Org invitation acceptance | +| `ToweropsWeb.Controllers.UserConfirmationController` | Email confirmation flow | +| `ToweropsWeb.Controllers.UserRegistrationController` | User signup flow | +| `ToweropsWeb.Live.OnboardingLive` | New user onboarding | +| `ToweropsWeb.Live.Org.GaiiaMappingLive` | GAIIA device mapping UI | +| `ToweropsWeb.Live.Org.GaiiaReconciliationLive` | GAIIA reconciliation UI | + +#### 🟡 Medium Priority + +| Module | Why | +|---|---| +| `Towerops.Geocoding` | Address-to-coordinates (read-only helper) | +| `Towerops.GeoIp.Block` / `Location` | GeoIP lookup schemas | +| `Towerops.Vault` | Encryption key management | +| `Towerops.RateLimit` | Rate limiting logic | +| `Towerops.Devices.Event` / `BackupRequest` | Event/request schemas | +| `Towerops.Devices.MikrotikBackup` | Backup schema | +| `Towerops.OnCall.Schedule` / `Layer` / `Override` | On-call schedule schemas | +| `Towerops.Organizations.Membership` | Membership schema | +| `Towerops.SNMP.Interface` / `Sensor` / `Device` schemas | Data schemas | +| `Towerops.Workers.*SyncWorker` (sonar, splynx, visp, weather, netbox) | Sync orchestration | +| `ToweropsWeb.Live.CapacityLive` | Capacity planning page | +| `ToweropsWeb.Live.ConfigTimelineLive` | Config change timeline | +| `ToweropsWeb.Live.ActivityFeedLive` | Activity feed page | +| `ToweropsWeb.Live.MikrotikBackupLive.Compare` | Backup diff viewer | +| `ToweropsWeb.GraphQL.Context` | GraphQL context setup | + +#### 🟢 Low Priority + +| Module | Notes | +|---|---| +| `Towerops.EctoTypes.EncryptedBinary` | Simple Ecto type (encrypted_map is tested) | +| `Towerops.Settings.ApplicationSetting` | Simple schema | +| `Towerops.Mailer` | Swoosh wrapper | +| `Towerops.Release` | Mix release tasks | +| `Towerops.Repo` | Ecto repo module | +| `Towerops.SNMP.SnmpBehaviour` | Behaviour definition | +| `Towerops.SNMP.PollerBehaviour` | Behaviour definition | +| `Towerops.Monitoring.PingBehaviour` | Behaviour definition | +| `Towerops.Profiles.MibCache` / `ProfileWatcher` | GenServer infrastructure | +| Various SNMP data schemas (readings, stats) | Simple structs | + +--- + +## Top 10 Recommended Tests to Write + +### 1. `test/towerops/monitoring_test.exs` — Monitoring Context +**Impact: 🔴 Critical** +Cover: `create_check/2`, `update_check/2`, `delete_check/2`, `list_checks_for_device/1`, check result creation and status evaluation. This is the core monitoring orchestration layer with zero test coverage. + +### 2. `test/towerops/monitoring/executor_test.exs` — Check Executor Dispatch +**Impact: 🔴 Critical** +Cover: Executor dispatch logic, timeout handling, error wrapping. Test with mocked executors that the right executor is called for each check type (HTTP, DNS, TCP, SNMP). + +### 3. `test/towerops/on_call/notifier_test.exs` — On-Call Notification Delivery +**Impact: 🔴 Critical** +Cover: Notification channel selection (email, SMS, push), retry logic, escalation triggering, de-duplication. Mock external delivery and verify correct notification flow. + +### 4. `test/towerops/on_call/incident_test.exs` — Incident Lifecycle +**Impact: 🔴 Critical** +Cover: Incident creation from alert, acknowledgment, resolution, re-escalation on timeout. Verify state machine transitions and escalation policy evaluation. + +### 5. `test/towerops/monitoring/executors/http_executor_test.exs` — HTTP Check Execution +**Impact: 🔴 Critical** +Cover: HTTP status checks, response time measurement, SSL verification, timeout handling, redirect following, body content matching. Mock HTTP client. + +### 6. `test/towerops/config_changes_test.exs` — Config Change Tracking +**Impact: 🟠 High** +Cover: Change event creation, correlation with alerts/devices, timeline querying, diff computation. Verify correlator links config changes to relevant alerts. + +### 7. `test/towerops/maintenance_test.exs` — Maintenance Window Context +**Impact: 🟠 High** +Cover: Window creation/scheduling, overlap detection, active window queries, alert suppression during maintenance. Verify `is_in_maintenance?/2` logic. + +### 8. `test/towerops_web/controllers/api/v1/pagerduty_webhook_controller_test.exs` — PagerDuty Webhooks +**Impact: 🟠 High** +Cover: Webhook signature verification, event type handling (trigger, acknowledge, resolve), idempotency, error responses for invalid payloads. + +### 9. `test/towerops/security/cloudflare_client_test.exs` — Cloudflare IP Management +**Impact: 🟠 High** +Cover: IP block creation/deletion via Cloudflare API, error handling, rate limiting. Mock Cloudflare API. Verify the ban worker integrates correctly. + +### 10. `test/towerops_web/controllers/user_registration_controller_test.exs` — User Registration +**Impact: 🟠 High** +Cover: Successful registration, duplicate email handling, password validation, email confirmation trigger, registration with invitation token. Critical auth flow. + +--- + +## Test Health Notes + +- **Async ratio is excellent** — ~81% of test files use `async: true` +- **Property testing is in use** — 55 property tests via StreamData (found in schema validation, Ecto type tests) +- **Fixture coverage is comprehensive** — 14 fixture modules covering all major domains +- **Vendor profile tests are auto-generated** — 180+ vendor tests likely from `mix gen_vendor_tests` +- **The 4 current failures are infrastructure issues**, not logic bugs: + - Agent slug uniqueness collision in test factories → fix: add `System.unique_integer()` to slug generation + - Deadlock in async time series test → fix: set `async: false` or restructure DB access +- **231 excluded tests** — likely tagged `:integration` or `:external` for CI isolation +- **38 skipped tests** — worth auditing to see if any are permanently broken diff --git a/docs/references/elixir-best-practices.md b/docs/references/elixir-best-practices.md new file mode 100644 index 00000000..72484261 --- /dev/null +++ b/docs/references/elixir-best-practices.md @@ -0,0 +1,2149 @@ +# Elixir Best Practices Reference + +A comprehensive guide covering OTP patterns, Phoenix LiveView, Ecto, testing, performance, code organization, and deployment. + +--- + +## Table of Contents + +1. [OTP Patterns](#1-otp-patterns) +2. [Phoenix LiveView](#2-phoenix-liveview) +3. [Ecto Best Practices](#3-ecto-best-practices) +4. [Testing](#4-testing) +5. [Performance](#5-performance) +6. [Code Organization](#6-code-organization) +7. [Deployment](#7-deployment) + +--- + +## 1. OTP Patterns + +### GenServer Best Practices + +#### Keep GenServer callbacks thin + +The GenServer module should be a thin wrapper. Push business logic into pure functions in separate modules. + +```elixir +# ✅ Good: Thin GenServer, logic in pure module +defmodule MyApp.RateLimiter do + use GenServer + + def start_link(opts) do + name = Keyword.fetch!(opts, :name) + config = Keyword.fetch!(opts, :config) + GenServer.start_link(__MODULE__, config, name: name) + end + + def check(server, key), do: GenServer.call(server, {:check, key}) + + @impl true + def init(config) do + {:ok, RateLimiter.State.new(config)} + end + + @impl true + def handle_call({:check, key}, _from, state) do + {result, new_state} = RateLimiter.Logic.check(state, key) + {:reply, result, new_state} + end +end + +defmodule MyApp.RateLimiter.Logic do + # Pure functions — easy to test without starting a process + def check(state, key) do + now = System.monotonic_time(:millisecond) + # ... pure logic returning {result, new_state} + end +end +``` + +```elixir +# ❌ Bad: Fat GenServer with all logic inline +defmodule MyApp.RateLimiter do + use GenServer + + @impl true + def handle_call({:check, key}, _from, state) do + now = System.monotonic_time(:millisecond) + bucket = Map.get(state.buckets, key, %{tokens: state.max, last: now}) + elapsed = now - bucket.last + refilled = min(bucket.tokens + elapsed * state.rate, state.max) + # ... 40 more lines of logic mixed with GenServer plumbing + end +end +``` + +#### Always define a client API + +```elixir +# ✅ Good: Client API hides GenServer details +defmodule MyApp.Cache do + use GenServer + + # --- Client API --- + def get(server \\ __MODULE__, key) do + GenServer.call(server, {:get, key}) + end + + def put(server \\ __MODULE__, key, value, ttl \\ :infinity) do + GenServer.call(server, {:put, key, value, ttl}) + end + + # --- Server Callbacks --- + @impl true + def handle_call({:get, key}, _from, state) do + {:reply, Map.get(state, key), state} + end +end + +# Callers just do: +MyApp.Cache.get(:user_123) +``` + +#### Use `handle_continue` for post-init work + +```elixir +defmodule MyApp.DataLoader do + use GenServer + + @impl true + def init(config) do + # Don't block the supervisor — return immediately, do heavy work in continue + {:ok, %{config: config, data: nil}, {:continue, :load_data}} + end + + @impl true + def handle_continue(:load_data, state) do + data = expensive_load(state.config) + {:noreply, %{state | data: data}} + end +end +``` + +#### Timeouts and `:hibernate` + +```elixir +# For processes that are mostly idle, hibernate to reclaim memory +@impl true +def handle_call(:get_status, _from, state) do + {:reply, state.status, state, :hibernate} +end + +# Use Process.send_after for periodic work instead of :timer +@impl true +def init(state) do + schedule_cleanup() + {:ok, state} +end + +@impl true +def handle_info(:cleanup, state) do + new_state = do_cleanup(state) + schedule_cleanup() + {:noreply, new_state} +end + +defp schedule_cleanup do + Process.send_after(self(), :cleanup, :timer.minutes(5)) +end +``` + +#### Reply later with `handle_call` + manual reply + +```elixir +@impl true +def handle_call(:expensive_op, from, state) do + # Don't block the GenServer — spawn the work + Task.start(fn -> + result = do_expensive_thing() + GenServer.reply(from, result) + end) + + {:noreply, state} +end +``` + +### Supervisor Trees + +#### Use granular supervision strategies + +```elixir +defmodule MyApp.Application do + use Application + + @impl true + def start(_type, _args) do + children = [ + # Infrastructure first + MyApp.Repo, + {Phoenix.PubSub, name: MyApp.PubSub}, + + # Application-specific supervisors + MyApp.WorkerSupervisor, + MyApp.SchedulerSupervisor, + + # Web layer last (depends on everything above) + MyAppWeb.Endpoint + ] + + opts = [strategy: :one_for_one, name: MyApp.Supervisor] + Supervisor.start_link(children, opts) + end +end +``` + +#### Choose the right strategy + +```elixir +# :one_for_one — children are independent +# Most common. Use when children don't depend on each other. +Supervisor.start_link(children, strategy: :one_for_one) + +# :one_for_all — if one dies, restart all +# Use when children are tightly coupled (e.g., producer + consumer). +Supervisor.start_link(children, strategy: :one_for_all) + +# :rest_for_one — if one dies, restart it and all children started after it +# Use when later children depend on earlier ones. +Supervisor.start_link([ + MyApp.Database, # if this crashes... + MyApp.Cache, # ...restart this + MyApp.API # ...and this +], strategy: :rest_for_one) +``` + +#### DynamicSupervisor for runtime-spawned children + +```elixir +defmodule MyApp.SessionSupervisor do + use DynamicSupervisor + + def start_link(init_arg) do + DynamicSupervisor.start_link(__MODULE__, init_arg, name: __MODULE__) + end + + @impl true + def init(_init_arg) do + DynamicSupervisor.init(strategy: :one_for_one, max_children: 1000) + end + + def start_session(session_id, opts) do + spec = {MyApp.Session, [id: session_id] ++ opts} + DynamicSupervisor.start_child(__MODULE__, spec) + end + + def stop_session(pid) do + DynamicSupervisor.terminate_child(__MODULE__, pid) + end +end +``` + +### When to Use Agent vs GenServer vs Task + +| Use Case | Choice | Why | +|----------|--------|-----| +| Simple state wrapper | `Agent` | No custom messages needed | +| State + complex message handling | `GenServer` | Need `handle_info`, timeouts, multi-step logic | +| Fire-and-forget work | `Task.start` | Don't need result | +| Async work, need result | `Task.async` + `Task.await` | Short-lived, supervised via caller | +| Background work under supervisor | `Task.Supervisor.start_child` | Crash isolation, can be `async_nolink` | +| Long-running background job | `GenServer` | Needs lifecycle management, restart | +| Pool of concurrent workers | `Task.Supervisor` | Bounded concurrency with `max_children` | + +```elixir +# Agent: dead simple state. Use sparingly — often a GenServer is clearer. +{:ok, counter} = Agent.start_link(fn -> 0 end, name: MyApp.Counter) +Agent.update(MyApp.Counter, &(&1 + 1)) +Agent.get(MyApp.Counter, & &1) #=> 1 + +# Task: fire-and-forget +Task.start(fn -> send_welcome_email(user) end) + +# Task: async/await (short-lived, caller-linked) +task = Task.async(fn -> fetch_external_data(url) end) +result = Task.await(task, 5_000) + +# Task.Supervisor: supervised tasks +{:ok, result} = Task.Supervisor.async_nolink(MyApp.TaskSup, fn -> + risky_external_call() +end) |> Task.yield(10_000) || {:error, :timeout} +``` + +**Anti-pattern: Using Agent for complex state management** + +```elixir +# ❌ Bad: Agent doing too much +Agent.update(pid, fn state -> + # Complex multi-step logic that should be in a GenServer + state + |> validate_transition() + |> apply_side_effects() # Side effects inside Agent.update = bad + |> update_counters() +end) + +# ✅ Good: Use GenServer when logic is non-trivial +``` + +### Process Linking vs Monitoring + +```elixir +# Link: bidirectional. If either process crashes, the other gets an EXIT signal. +# Use for: tightly coupled processes that should die together. +Process.link(pid) + +# Monitor: unidirectional. Observer gets a :DOWN message, monitored process is unaffected. +# Use for: watching a process you don't own / shouldn't crash with. +ref = Process.monitor(pid) + +receive do + {:DOWN, ^ref, :process, ^pid, reason} -> + Logger.warning("Process #{inspect(pid)} went down: #{inspect(reason)}") +end + +# In GenServer, handle :DOWN in handle_info: +@impl true +def handle_info({:DOWN, _ref, :process, pid, _reason}, state) do + {:noreply, remove_worker(state, pid)} +end +``` + +**Rule of thumb:** +- **Link** = "we live and die together" (parent-child, co-dependent) +- **Monitor** = "I want to know when you die" (observer pattern) + +### Registry Patterns + +```elixir +# Start registry in supervision tree +children = [ + {Registry, keys: :unique, name: MyApp.Registry}, + # ... +] + +# Register a GenServer via name +defmodule MyApp.Session do + use GenServer + + def start_link(session_id) do + GenServer.start_link(__MODULE__, session_id, + name: via_tuple(session_id) + ) + end + + def get_state(session_id) do + GenServer.call(via_tuple(session_id), :get_state) + end + + defp via_tuple(session_id) do + {:via, Registry, {MyApp.Registry, {:session, session_id}}} + end +end + +# Lookup +case Registry.lookup(MyApp.Registry, {:session, "abc123"}) do + [{pid, _value}] -> {:ok, pid} + [] -> {:error, :not_found} +end + +# Dispatch to all registered processes (use :duplicate keys) +{Registry, keys: :duplicate, name: MyApp.PubSubRegistry} + +Registry.dispatch(MyApp.PubSubRegistry, :event_type, fn entries -> + for {pid, _} <- entries, do: send(pid, {:event, payload}) +end) +``` + +#### Partitioned Registry for high concurrency + +```elixir +{Registry, keys: :unique, name: MyApp.Registry, partitions: System.schedulers_online()} +``` + +--- + +## 2. Phoenix LiveView + +### Assigns Optimization + +#### `temporary_assigns` — free memory after render + +```elixir +# ✅ Good: Large lists that only need to render once +def mount(_params, _session, socket) do + {:ok, + assign(socket, :messages, list_messages()), + temporary_assigns: [messages: []]} +end + +# After the first render, socket.assigns.messages is reset to [] +# The client retains the DOM. New messages are appended via handle_event/handle_info. +``` + +#### Streams — the modern replacement for temporary_assigns + +Streams (LiveView 0.19+) are the preferred way to handle large collections. They track items by ID and send only diffs. + +```elixir +def mount(_params, _session, socket) do + {:ok, + socket + |> stream(:messages, list_messages()) + |> assign(:page, 1)} +end + +# Adding items +def handle_info({:new_message, message}, socket) do + {:noreply, stream_insert(socket, :messages, message)} +end + +# Removing items +def handle_event("delete", %{"id" => id}, socket) do + message = Messages.get!(id) + Messages.delete(message) + {:noreply, stream_delete(socket, :messages, message)} +end + +# Bulk reset +def handle_event("refresh", _, socket) do + {:noreply, stream(socket, :messages, list_messages(), reset: true)} +end +``` + +In the template: + +```heex +
+
+

<%= message.body %>

+
+
+``` + +**Anti-pattern: Storing huge lists in assigns without streaming** + +```elixir +# ❌ Bad: 10,000 items sitting in process memory permanently +def mount(_, _, socket) do + {:ok, assign(socket, :items, Repo.all(Item))} # All in memory, all diffed every render +end + +# ✅ Good: Use streams +def mount(_, _, socket) do + {:ok, stream(socket, :items, Repo.all(Item))} +end +``` + +### `handle_params` vs `mount` + +```elixir +# mount/3 — called ONCE when the LiveView process starts. +# Use for: one-time setup, subscriptions, session-derived state. + +# handle_params/3 — called on EVERY URL change (including initial mount). +# Use for: loading data based on URL params, pagination, filtering. + +defmodule MyAppWeb.ProductLive.Index do + use MyAppWeb, :live_view + + # Called once + @impl true + def mount(_params, _session, socket) do + if connected?(socket) do + Phoenix.PubSub.subscribe(MyApp.PubSub, "products") + end + + {:ok, socket} + end + + # Called on every navigation (including initial) + @impl true + def handle_params(params, _uri, socket) do + page = String.to_integer(params["page"] || "1") + category = params["category"] + + {:noreply, + socket + |> assign(:page, page) + |> assign(:category, category) + |> stream(:products, list_products(page, category), reset: true)} + end +end +``` + +**Anti-pattern: Loading URL-dependent data in `mount`** + +```elixir +# ❌ Bad: Data loaded in mount won't refresh on live_patch navigation +def mount(%{"id" => id}, _session, socket) do + product = Products.get!(id) + {:ok, assign(socket, :product, product)} +end + +# The user live_patches to /products/456 — mount is NOT called again, +# so the old product stays. Use handle_params instead. +``` + +### `live_session` Boundaries + +```elixir +# live_session groups LiveViews that share authentication/layout. +# Navigating between different live_sessions triggers a full page load (dead render + reconnect). +# Within the same live_session, live_patch/live_redirect keeps the WebSocket alive. + +scope "/", MyAppWeb do + pipe_through [:browser] + + live_session :public, + on_mount: [{MyAppWeb.InitAssigns, :default}] do + live "/", HomeLive.Index + live "/about", AboutLive + end + + live_session :authenticated, + on_mount: [{MyAppWeb.UserAuth, :ensure_authenticated}] do + live "/dashboard", DashboardLive + live "/settings", SettingsLive + live "/projects/:id", ProjectLive.Show + end + + live_session :admin, + on_mount: [{MyAppWeb.UserAuth, :ensure_admin}] do + live "/admin", AdminLive.Index + live "/admin/users", AdminLive.Users + end +end +``` + +```elixir +# on_mount hook for authentication +defmodule MyAppWeb.UserAuth do + import Phoenix.LiveView + import Phoenix.Component + + def on_mount(:ensure_authenticated, _params, %{"user_token" => token}, socket) do + case Accounts.get_user_by_session_token(token) do + nil -> {:halt, redirect(socket, to: "/login")} + user -> {:cont, assign(socket, :current_user, user)} + end + end + + def on_mount(:ensure_authenticated, _params, _session, socket) do + {:halt, redirect(socket, to: "/login")} + end +end +``` + +### JS Hooks + Server Coordination + +```elixir +# In the template +
+ +# phx-hook value must match the hook name in app.js +``` + +```javascript +// assets/js/hooks/chart.js +const Chart = { + mounted() { + this.chart = new ChartLib(this.el, { + data: JSON.parse(this.el.dataset.points) + }); + + // Receive events from server + this.handleEvent("update-chart", ({ points }) => { + this.chart.update(points); + }); + + // Send events to server + this.el.addEventListener("click", (e) => { + this.pushEvent("chart-clicked", { x: e.offsetX, y: e.offsetY }); + }); + }, + + updated() { + // Called when the element's attributes change from server + const points = JSON.parse(this.el.dataset.points); + this.chart.update(points); + }, + + destroyed() { + this.chart.destroy(); + } +}; + +export default Chart; +``` + +```javascript +// assets/js/app.js +import Chart from "./hooks/chart"; +import InfiniteScroll from "./hooks/infinite_scroll"; + +let Hooks = { Chart, InfiniteScroll }; +let liveSocket = new LiveSocket("/live", Socket, { + params: { _csrf_token: csrfToken }, + hooks: Hooks +}); +``` + +```elixir +# Server-side: push events to the hook +def handle_info({:data_updated, points}, socket) do + {:noreply, push_event(socket, "update-chart", %{points: points})} +end + +# Handle events from the hook +def handle_event("chart-clicked", %{"x" => x, "y" => y}, socket) do + # Process click coordinates + {:noreply, socket} +end +``` + +### `phx-update` Strategies + +```heex +<%!-- Default: replace entire container contents --%> +
+ <%= for n <- @notifications do %> +
<%= n.text %>
+ <% end %> +
+ +<%!-- Stream: managed by stream_insert/stream_delete --%> +
+
+ <%= msg.body %> +
+
+ +<%!-- Append: new children appended, existing preserved (legacy, prefer streams) --%> +
+ <%= for entry <- @log_entries do %> +
<%= entry.text %>
+ <% end %> +
+ +<%!-- Ignore: server never updates this container (for JS-managed DOM) --%> +
+
JS manages this subtree
+
+``` + +### Dead Views vs Live Views + +```elixir +# Dead view: traditional request/response. Use for: +# - Static pages (about, terms, privacy) +# - SEO-critical content that doesn't need interactivity +# - Simple forms that don't need real-time validation + +# In router: +get "/terms", PageController, :terms + +# Live view: persistent WebSocket. Use for: +# - Real-time updates (dashboards, notifications) +# - Complex interactive forms +# - Collaborative features +# - Any UI that benefits from server push + +# Rule of thumb: if it doesn't need a WebSocket, a dead view is simpler and cheaper. +``` + +--- + +## 3. Ecto Best Practices + +### Composable Queries + +```elixir +defmodule MyApp.Devices.Query do + import Ecto.Query + + def base, do: from(d in MyApp.Devices.Device, as: :device) + + def by_site(query, site_id) do + where(query, [device: d], d.site_id == ^site_id) + end + + def by_status(query, status) do + where(query, [device: d], d.status == ^status) + end + + def with_interfaces(query) do + preload(query, [:interfaces]) + end + + def search(query, nil), do: query + def search(query, ""), do: query + def search(query, term) do + term = "%#{term}%" + where(query, [device: d], ilike(d.hostname, ^term) or ilike(d.ip_address, ^term)) + end + + def paginate(query, page, per_page \\ 25) do + offset = (page - 1) * per_page + + query + |> limit(^per_page) + |> offset(^offset) + end + + def order_by_hostname(query) do + order_by(query, [device: d], asc: d.hostname) + end +end + +# Usage — compose like pipes: +import MyApp.Devices.Query + +base() +|> by_site(site_id) +|> by_status(:active) +|> search(params["q"]) +|> order_by_hostname() +|> paginate(page) +|> with_interfaces() +|> Repo.all() +``` + +### Multi-Tenancy Patterns + +#### Option A: Foreign Key (simpler, recommended for most apps) + +```elixir +# Every query scopes by tenant_id +defmodule MyApp.Devices do + def list_devices(%Tenant{id: tenant_id}) do + Device + |> where(tenant_id: ^tenant_id) + |> Repo.all() + end +end + +# Enforce at the changeset level too +def changeset(device, attrs, tenant) do + device + |> cast(attrs, [:hostname, :ip_address]) + |> put_assoc(:tenant, tenant) + |> unique_constraint([:hostname, :tenant_id]) +end +``` + +#### Option B: Postgres Schema Prefix (stronger isolation) + +```elixir +# Each tenant gets a separate Postgres schema +defmodule MyApp.Repo do + def put_tenant_prefix(tenant_slug) do + # Call this at the start of each request + prefix = "tenant_#{tenant_slug}" + put_dynamic_repo({__MODULE__, prefix}) + prefix + end +end + +# In queries +def list_devices(tenant_prefix) do + Device + |> Repo.all(prefix: tenant_prefix) +end + +# Migrations must run per-tenant +def migrate_tenant(tenant_prefix) do + Ecto.Migrator.run(Repo, :up, all: true, prefix: tenant_prefix) +end +``` + +**Trade-offs:** +- Foreign key: simpler ops, cross-tenant queries easy, must remember to scope every query +- Prefix: stronger isolation, harder cross-tenant queries, migration complexity + +### Changesets + +#### Embedded schemas for complex nested data + +```elixir +defmodule MyApp.Devices.SNMPConfig do + use Ecto.Schema + import Ecto.Changeset + + # Not a database table — embedded in a JSONB column + embedded_schema do + field :community, :string + field :version, Ecto.Enum, values: [:v1, :v2c, :v3] + field :port, :integer, default: 161 + field :timeout_ms, :integer, default: 5000 + end + + def changeset(config, attrs) do + config + |> cast(attrs, [:community, :version, :port, :timeout_ms]) + |> validate_required([:community, :version]) + |> validate_number(:port, greater_than: 0, less_than: 65536) + |> validate_number(:timeout_ms, greater_than: 0) + end +end + +defmodule MyApp.Devices.Device do + use Ecto.Schema + import Ecto.Changeset + + schema "devices" do + field :hostname, :string + embeds_one :snmp_config, MyApp.Devices.SNMPConfig, on_replace: :update + timestamps() + end + + def changeset(device, attrs) do + device + |> cast(attrs, [:hostname]) + |> cast_embed(:snmp_config, required: true) + |> validate_required([:hostname]) + end +end +``` + +#### `cast_assoc` vs `put_assoc` + +```elixir +# cast_assoc: user-facing forms where you cast nested params +# Expects params like %{"addresses" => [%{"street" => "123 Main"}]} +def changeset(user, attrs) do + user + |> cast(attrs, [:name]) + |> cast_assoc(:addresses, with: &Address.changeset/2) +end + +# put_assoc: programmatic. You already have the struct/changeset. +def add_address(user, address) do + user + |> change() + |> put_assoc(:addresses, [address | user.addresses]) +end +``` + +### `Repo.transaction` Patterns + +#### Multi for composable transactions + +```elixir +alias Ecto.Multi + +def create_team_with_owner(team_attrs, user) do + Multi.new() + |> Multi.insert(:team, Team.changeset(%Team{}, team_attrs)) + |> Multi.insert(:membership, fn %{team: team} -> + Membership.changeset(%Membership{}, %{ + team_id: team.id, + user_id: user.id, + role: :owner + }) + end) + |> Multi.run(:audit, fn repo, %{team: team} -> + repo.insert(AuditLog.entry(:team_created, team, user)) + end) + |> Repo.transaction() + |> case do + {:ok, %{team: team, membership: membership}} -> + {:ok, team} + + {:error, :team, changeset, _changes} -> + {:error, changeset} + + {:error, :membership, changeset, _changes} -> + {:error, changeset} + end +end +``` + +#### Manual transactions with early returns + +```elixir +Repo.transaction(fn -> + with {:ok, user} <- create_user(attrs), + {:ok, _profile} <- create_profile(user, profile_attrs), + :ok <- send_welcome_email(user) do + user + else + {:error, reason} -> Repo.rollback(reason) + end +end) +``` + +### Preload Strategies (Avoid N+1) + +```elixir +# ❌ Bad: N+1 queries +devices = Repo.all(Device) +Enum.map(devices, fn d -> + interfaces = Repo.all(from i in Interface, where: i.device_id == ^d.id) + %{d | interfaces: interfaces} +end) + +# ✅ Good: Preload in the query (single JOIN) +Repo.all( + from d in Device, + join: i in assoc(d, :interfaces), + preload: [interfaces: i] +) + +# ✅ Good: Separate preload query (two queries total, often faster than JOIN for large datasets) +Device +|> Repo.all() +|> Repo.preload(:interfaces) + +# ✅ Good: Nested preloads +Repo.all( + from d in Device, + preload: [interfaces: :ip_addresses, site: :organization] +) + +# ✅ Good: Custom preload query +active_interfaces_query = from i in Interface, where: i.status == :up + +Repo.all( + from d in Device, + preload: [interfaces: ^active_interfaces_query] +) +``` + +### Upserts + +```elixir +# Insert or update on conflict +def upsert_device(attrs) do + %Device{} + |> Device.changeset(attrs) + |> Repo.insert( + on_conflict: {:replace, [:hostname, :ip_address, :updated_at]}, + conflict_target: :serial_number, + returning: true + ) +end + +# Bulk upsert +def upsert_metrics(metrics) do + now = NaiveDateTime.utc_now() |> NaiveDateTime.truncate(:second) + + entries = Enum.map(metrics, fn m -> + %{ + device_id: m.device_id, + metric_name: m.name, + value: m.value, + inserted_at: now, + updated_at: now + } + end) + + Repo.insert_all(Metric, entries, + on_conflict: {:replace, [:value, :updated_at]}, + conflict_target: [:device_id, :metric_name], + returning: true + ) +end +``` + +### Advisory Locks + +```elixir +# Use PostgreSQL advisory locks for distributed locking +defmodule MyApp.Lock do + @doc """ + Runs `fun` only if the advisory lock is acquired. + Lock is automatically released when the transaction commits. + """ + def with_lock(lock_key, fun) when is_integer(lock_key) do + Repo.transaction(fn -> + case Repo.query("SELECT pg_try_advisory_xact_lock($1)", [lock_key]) do + {:ok, %{rows: [[true]]}} -> + fun.() + + {:ok, %{rows: [[false]]}} -> + Repo.rollback(:lock_not_acquired) + end + end) + end + + # Helper to convert string keys to integers + def key(name) when is_binary(name) do + :erlang.phash2(name) + end +end + +# Usage +case MyApp.Lock.with_lock(Lock.key("sync:site:#{site_id}"), fn -> + perform_sync(site_id) +end) do + {:ok, result} -> {:ok, result} + {:error, :lock_not_acquired} -> {:ok, :already_running} +end +``` + +--- + +## 4. Testing + +### ExUnit Best Practices + +#### Describe blocks and clear naming + +```elixir +defmodule MyApp.DevicesTest do + use MyApp.DataCase, async: true + + alias MyApp.Devices + + describe "create_device/1" do + test "creates device with valid attributes" do + attrs = %{hostname: "switch-01", ip_address: "10.0.0.1"} + assert {:ok, device} = Devices.create_device(attrs) + assert device.hostname == "switch-01" + end + + test "returns error changeset with invalid attributes" do + assert {:error, %Ecto.Changeset{}} = Devices.create_device(%{}) + end + + test "enforces unique hostname per site" do + site = insert_site() + attrs = %{hostname: "switch-01", site_id: site.id} + {:ok, _} = Devices.create_device(attrs) + assert {:error, changeset} = Devices.create_device(attrs) + assert "has already been taken" in errors_on(changeset).hostname + end + end +end +``` + +#### Setup blocks + +```elixir +describe "device operations" do + setup do + site = insert_site() + device = insert_device(site: site) + %{site: site, device: device} + end + + test "updates device hostname", %{device: device} do + assert {:ok, updated} = Devices.update_device(device, %{hostname: "new-name"}) + assert updated.hostname == "new-name" + end +end + +# Named setup for reuse +setup :create_user + +defp create_user(_context) do + %{user: insert_user()} +end +``` + +### Property-Based Testing with StreamData + +```elixir +defmodule MyApp.ParserTest do + use ExUnit.Case, async: true + use ExUnitProperties + + property "parsing and formatting are inverse operations" do + check all ip <- ip_address_generator() do + assert ip == ip |> MyApp.IP.format() |> MyApp.IP.parse!() + end + end + + property "all valid SNMP communities pass validation" do + check all community <- string(:alphanumeric, min_length: 1, max_length: 32) do + changeset = SNMPConfig.changeset(%SNMPConfig{}, %{ + community: community, + version: :v2c + }) + assert changeset.valid? + end + end + + property "rate limiter never allows more than max requests" do + check all requests <- list_of(constant(:request), min_length: 1, max_length: 100), + max <- integer(1..10) do + state = RateLimiter.Logic.new(max_requests: max, window_ms: 1000) + + {allowed, _state} = + Enum.reduce(requests, {0, state}, fn _, {count, s} -> + case RateLimiter.Logic.check(s, "key") do + {:ok, new_s} -> {count + 1, new_s} + {:error, new_s} -> {count, new_s} + end + end) + + assert allowed <= max + end + end + + # Custom generators + defp ip_address_generator do + gen all a <- integer(0..255), + b <- integer(0..255), + c <- integer(0..255), + d <- integer(0..255) do + {a, b, c, d} + end + end +end +``` + +### Mox for Behaviours + +```elixir +# 1. Define a behaviour +defmodule MyApp.SNMP.Client do + @callback get(String.t(), String.t(), keyword()) :: {:ok, term()} | {:error, term()} + @callback walk(String.t(), String.t(), keyword()) :: {:ok, [term()]} | {:error, term()} +end + +# 2. Real implementation +defmodule MyApp.SNMP.NetSNMP do + @behaviour MyApp.SNMP.Client + + @impl true + def get(host, oid, opts \\ []) do + # Real SNMP call + end +end + +# 3. Define mock in test/support/mocks.ex +Mox.defmock(MyApp.SNMP.MockClient, for: MyApp.SNMP.Client) + +# 4. Configure in config/test.exs +config :my_app, :snmp_client, MyApp.SNMP.MockClient + +# 5. Use in application code +defmodule MyApp.Poller do + @snmp_client Application.compile_env(:my_app, :snmp_client) + + def poll_device(device) do + @snmp_client.get(device.ip_address, @sys_descr_oid) + end +end + +# 6. Test with expectations +defmodule MyApp.PollerTest do + use MyApp.DataCase, async: true + + import Mox + + setup :verify_on_exit! + + test "polls device and stores result" do + device = insert_device(ip_address: "10.0.0.1") + + expect(MyApp.SNMP.MockClient, :get, fn "10.0.0.1", _oid, _opts -> + {:ok, "Cisco IOS"} + end) + + assert {:ok, result} = MyApp.Poller.poll_device(device) + assert result.sys_descr == "Cisco IOS" + end + + test "handles SNMP timeout" do + device = insert_device() + + expect(MyApp.SNMP.MockClient, :get, fn _, _, _ -> + {:error, :timeout} + end) + + assert {:error, :timeout} = MyApp.Poller.poll_device(device) + end +end +``` + +### Async Test Isolation + +```elixir +# DataCase uses Ecto SQL Sandbox for async isolation +defmodule MyApp.DataCase do + use ExUnit.CaseTemplate + + using do + quote do + alias MyApp.Repo + import Ecto + import Ecto.Changeset + import Ecto.Query + import MyApp.DataCase + end + end + + setup tags do + MyApp.DataCase.setup_sandbox(tags) + :ok + end + + def setup_sandbox(tags) do + pid = Ecto.Adapters.SQL.Sandbox.start_owner!(MyApp.Repo, + shared: !tags[:async] + ) + on_exit(fn -> Ecto.Adapters.SQL.Sandbox.stop_owner(pid) end) + end +end + +# Mark test modules as async when they don't share state +defmodule MyApp.AccountsTest do + use MyApp.DataCase, async: true # Runs in parallel +end +``` + +### Factory Patterns (Without ExMachina) + +```elixir +# Simple factory module — no library needed +defmodule MyApp.Factory do + alias MyApp.Repo + + def build(:user) do + %MyApp.Accounts.User{ + email: "user-#{System.unique_integer([:positive])}@example.com", + name: "Test User", + hashed_password: Bcrypt.hash_pwd_salt("password123") + } + end + + def build(:device) do + %MyApp.Devices.Device{ + hostname: "device-#{System.unique_integer([:positive])}", + ip_address: "10.0.0.#{:rand.uniform(254)}", + status: :active + } + end + + def build(:site) do + %MyApp.Sites.Site{ + name: "Site #{System.unique_integer([:positive])}", + slug: "site-#{System.unique_integer([:positive])}" + } + end + + # Override fields + def build(factory, attrs) do + factory + |> build() + |> struct!(attrs) + end + + # Insert into DB + def insert!(factory, attrs \\ []) do + factory + |> build(attrs) + |> Repo.insert!() + end + + # Convenience helpers for tests + def insert_user(attrs \\ []), do: insert!(:user, attrs) + def insert_device(attrs \\ []), do: insert!(:device, attrs) + def insert_site(attrs \\ []), do: insert!(:site, attrs) +end + +# Import in DataCase: +# import MyApp.Factory + +# Usage in tests: +test "deactivates device" do + device = insert_device(status: :active) + assert {:ok, updated} = Devices.deactivate(device) + assert updated.status == :inactive +end +``` + +#### Trait-like patterns + +```elixir +def build(:device, :with_interfaces) do + device = build(:device) + interfaces = for i <- 1..4, do: build(:interface, device_id: device.id, index: i) + %{device | interfaces: interfaces} +end + +def insert_device_with_interfaces(attrs \\ []) do + device = insert_device(attrs) + for i <- 1..4 do + insert!(:interface, device_id: device.id, index: i) + end + Repo.preload(device, :interfaces) +end +``` + +### ConnCase vs DataCase + +```elixir +# DataCase: tests that need the database but NOT HTTP/LiveView +# - Context module tests (Accounts.create_user/1) +# - Business logic tests +# - Query tests + +# ConnCase: tests that need HTTP connection or LiveView +# - Controller tests +# - LiveView tests +# - Plug tests +# - API endpoint tests + +defmodule MyAppWeb.DeviceLiveTest do + use MyAppWeb.ConnCase, async: true + + import Phoenix.LiveViewTest + + test "lists devices", %{conn: conn} do + device = insert_device(hostname: "switch-01") + {:ok, view, html} = live(conn, ~p"/devices") + assert html =~ "switch-01" + end + + test "creates device via form", %{conn: conn} do + {:ok, view, _} = live(conn, ~p"/devices/new") + + view + |> form("#device-form", device: %{hostname: "new-switch"}) + |> render_submit() + + assert_redirect(view, ~p"/devices") + end +end +``` + +--- + +## 5. Performance + +### ETS for Caching + +```elixir +defmodule MyApp.Cache do + use GenServer + + @table __MODULE__ + @default_ttl :timer.minutes(5) + + def start_link(_opts) do + GenServer.start_link(__MODULE__, [], name: __MODULE__) + end + + @impl true + def init(_) do + table = :ets.new(@table, [ + :set, + :public, # Any process can read (fast, no bottleneck) + :named_table, + read_concurrency: true + ]) + + schedule_cleanup() + {:ok, %{table: table}} + end + + # Client API — direct ETS access, no GenServer bottleneck for reads + def get(key) do + case :ets.lookup(@table, key) do + [{^key, value, expires_at}] -> + if System.monotonic_time(:millisecond) < expires_at do + {:ok, value} + else + :ets.delete(@table, key) + :miss + end + + [] -> + :miss + end + end + + def put(key, value, ttl \\ @default_ttl) do + expires_at = System.monotonic_time(:millisecond) + ttl + :ets.insert(@table, {key, value, expires_at}) + :ok + end + + def delete(key), do: :ets.delete(@table, key) + + # Fetch-or-compute pattern + def fetch(key, ttl \\ @default_ttl, fun) do + case get(key) do + {:ok, value} -> value + :miss -> + value = fun.() + put(key, value, ttl) + value + end + end + + # Periodic cleanup of expired entries + @impl true + def handle_info(:cleanup, state) do + now = System.monotonic_time(:millisecond) + + :ets.foldl(fn {key, _value, expires_at}, acc -> + if now >= expires_at, do: :ets.delete(@table, key) + acc + end, nil, @table) + + schedule_cleanup() + {:noreply, state} + end + + defp schedule_cleanup, do: Process.send_after(self(), :cleanup, :timer.minutes(1)) +end + +# Usage +MyApp.Cache.fetch("device:#{device_id}:interfaces", :timer.minutes(10), fn -> + Repo.preload(device, :interfaces).interfaces +end) +``` + +### `:persistent_term` for Config + +```elixir +# persistent_term is optimized for reads (zero-copy), expensive for writes. +# Perfect for: config, feature flags, compiled schemas. + +defmodule MyApp.Config do + def load! do + # Called once at startup or on config reload + :persistent_term.put({__MODULE__, :features}, %{ + new_dashboard: true, + beta_api: false + }) + + :persistent_term.put({__MODULE__, :limits}, %{ + max_devices_per_site: 10_000, + poll_interval_ms: 30_000 + }) + end + + # Lightning-fast reads — no message passing, no copying + def feature_enabled?(flag) do + features = :persistent_term.get({__MODULE__, :features}) + Map.get(features, flag, false) + end + + def get_limit(key) do + limits = :persistent_term.get({__MODULE__, :limits}) + Map.fetch!(limits, key) + end +end + +# ❌ Don't: Write to persistent_term frequently +# Every write triggers a global GC of all copies. Use ETS for frequently changing data. +``` + +### Telemetry Instrumentation + +```elixir +# Emit telemetry events from your application +defmodule MyApp.Devices do + def poll_device(device) do + start_time = System.monotonic_time() + + result = do_poll(device) + + :telemetry.execute( + [:my_app, :device, :poll], + %{duration: System.monotonic_time() - start_time}, + %{device_id: device.id, site_id: device.site_id, result: elem(result, 0)} + ) + + result + end + + # Or use :telemetry.span/3 for automatic start/stop/exception events + def sync_site(site) do + :telemetry.span([:my_app, :site, :sync], %{site_id: site.id}, fn -> + result = do_sync(site) + {result, %{device_count: length(result.devices)}} + end) + end +end + +# Attach handlers in application startup +defmodule MyApp.Telemetry do + def setup do + :telemetry.attach_many("my-app-metrics", [ + [:my_app, :device, :poll], + [:my_app, :site, :sync, :stop], + [:my_app, :site, :sync, :exception], + [:phoenix, :endpoint, :stop], + [:my_app, :repo, :query] + ], &handle_event/4, nil) + end + + def handle_event([:my_app, :device, :poll], measurements, metadata, _config) do + duration_ms = System.convert_time_unit(measurements.duration, :native, :millisecond) + + if duration_ms > 5_000 do + Logger.warning("Slow poll for device #{metadata.device_id}: #{duration_ms}ms") + end + + # Push to metrics system (Prometheus, StatsD, etc.) + :telemetry.execute([:prometheus, :histogram, :observe], %{ + name: "device_poll_duration_ms", + value: duration_ms, + labels: %{result: metadata.result} + }) + end +end +``` + +### GenStage / Broadway for Backpressure + +```elixir +# Broadway: higher-level, batteries-included pipeline +defmodule MyApp.SNMPPollPipeline do + use Broadway + + def start_link(_opts) do + Broadway.start_link(__MODULE__, + name: __MODULE__, + producer: [ + module: {MyApp.PollProducer, []}, + transformer: {__MODULE__, :transform, []}, + concurrency: 1 + ], + processors: [ + default: [concurrency: 50, max_demand: 10] + ], + batchers: [ + default: [batch_size: 100, batch_timeout: 1_000, concurrency: 5] + ] + ) + end + + def transform(event, _opts) do + %Broadway.Message{ + data: event, + acknowledger: {__MODULE__, :ack_id, :ack_data} + } + end + + @impl true + def handle_message(_, %Broadway.Message{data: device} = message, _) do + case poll_device(device) do + {:ok, result} -> + Broadway.Message.put_data(message, result) + + {:error, reason} -> + Broadway.Message.failed(message, reason) + end + end + + @impl true + def handle_batch(_, messages, _, _) do + # Bulk insert results + results = Enum.map(messages, & &1.data) + MyApp.Metrics.bulk_insert(results) + messages + end + + @impl true + def handle_failed(messages, _context) do + Enum.each(messages, fn msg -> + Logger.error("Poll failed: #{inspect(msg.data)} — #{inspect(msg.status)}") + end) + messages + end +end +``` + +### Connection Pooling with Finch + +```elixir +# In application supervisor +children = [ + {Finch, name: MyApp.Finch, + pools: %{ + "https://api.example.com" => [size: 25, count: 2], + :default => [size: 10] + } + } +] + +# Usage +def get_device_info(device_id) do + Finch.build(:get, "https://api.example.com/devices/#{device_id}", + [{"authorization", "Bearer #{token()}"}] + ) + |> Finch.request(MyApp.Finch) + |> case do + {:ok, %{status: 200, body: body}} -> {:ok, Jason.decode!(body)} + {:ok, %{status: status}} -> {:error, {:http_error, status}} + {:error, reason} -> {:error, reason} + end +end + +# Streaming large responses +Finch.build(:get, url) +|> Finch.stream(MyApp.Finch, acc, fn + {:status, status}, acc -> %{acc | status: status} + {:headers, headers}, acc -> %{acc | headers: headers} + {:data, chunk}, acc -> %{acc | body: acc.body <> chunk} +end) +``` + +### Binary Optimization + +```elixir +# Binaries > 64 bytes are reference-counted (shared, not copied). +# Sub-binaries point into the original — beware keeping references to large binaries. + +# ❌ Bad: Keeping a small slice alive holds the entire large binary in memory +def extract_header(<>) do + header # This is a sub-binary — the entire original stays in memory +end + +# ✅ Good: Copy when you need only a small part of a large binary +def extract_header(<>) do + :binary.copy(header) # New, independent 4-byte binary +end + +# Pattern matching on binaries is very efficient +def parse_tlv(<>) do + {type, value, rest} +end + +# Use iodata/iolists to avoid concatenation +# ❌ Bad: Repeated concatenation creates intermediate binaries +def build_response(items) do + Enum.reduce(items, "", fn item, acc -> + acc <> "
  • " <> item.name <> "
    • " + end) +end + +# ✅ Good: iolists — zero-copy, sent directly to IO +def build_response(items) do + Enum.map(items, fn item -> + ["
  • ", item.name, "
    • "] + end) +end +# Pass to IO.iodata_to_binary/1 only if you truly need a flat binary +``` + +--- + +## 6. Code Organization + +### Context Boundaries (DDD-lite) + +Phoenix contexts are your primary code organization tool. Think of them as bounded contexts. + +``` +lib/my_app/ +├── accounts/ # User management +│ ├── accounts.ex # Public API (the context module) +│ ├── user.ex # Schema +│ ├── user_token.ex # Schema +│ └── user_notifier.ex +├── devices/ # Device management +│ ├── devices.ex # Public API +│ ├── device.ex # Schema +│ ├── interface.ex # Schema +│ └── query.ex # Composable queries +├── monitoring/ # Polling & metrics +│ ├── monitoring.ex +│ ├── poller.ex +│ ├── metric.ex +│ └── alert_rule.ex +└── sites/ + ├── sites.ex + └── site.ex +``` + +#### The context module is the public API + +```elixir +defmodule MyApp.Devices do + @moduledoc """ + The Devices context — manages network devices and their interfaces. + """ + alias MyApp.Repo + alias MyApp.Devices.{Device, Interface, Query} + + # --- Public API --- + + def list_devices(site_id, opts \\ []) do + Query.base() + |> Query.by_site(site_id) + |> Query.maybe_search(opts[:search]) + |> Query.paginate(opts[:page] || 1, opts[:per_page] || 25) + |> Repo.all() + end + + def get_device!(id), do: Repo.get!(Device, id) + + def create_device(attrs) do + %Device{} + |> Device.changeset(attrs) + |> Repo.insert() + end + + # Cross-context: accept IDs, not structs from other contexts + def assign_to_site(device_id, site_id) do + device_id + |> get_device!() + |> Device.changeset(%{site_id: site_id}) + |> Repo.update() + end +end +``` + +**Anti-pattern: Reaching into another context's schemas** + +```elixir +# ❌ Bad: Controller directly queries another context's schema +def index(conn, params) do + devices = Repo.all( + from d in MyApp.Devices.Device, + join: s in MyApp.Sites.Site, on: d.site_id == s.id, + where: s.organization_id == ^conn.assigns.org_id + ) +end + +# ✅ Good: Go through context APIs +def index(conn, params) do + devices = MyApp.Devices.list_devices_for_org(conn.assigns.org_id, params) +end +``` + +### When to Split Contexts + +**Split when:** +- Two areas of the codebase change for different reasons +- You're tempted to build circular dependencies between schemas +- A context module exceeds ~300-400 lines +- Different areas need different data access patterns (e.g., one is read-heavy, another write-heavy) + +**Don't split when:** +- It would force every operation to cross context boundaries +- The schemas are tightly coupled and always change together +- You'd just be creating pass-through functions + +### Service Modules vs Context Functions + +```elixir +# Keep simple CRUD in the context module: +defmodule MyApp.Devices do + def create_device(attrs), do: ... + def update_device(device, attrs), do: ... + def delete_device(device), do: ... +end + +# Extract complex operations into dedicated service modules: +defmodule MyApp.Devices.Discovery do + @moduledoc "Handles SNMP-based device auto-discovery." + + def discover(subnet, opts \\ []) do + # 50+ lines of complex discovery logic + end +end + +defmodule MyApp.Devices.BulkImport do + @moduledoc "Imports devices from CSV/spreadsheet." + + def import_csv(path, site_id) do + # Complex parsing, validation, deduplication + end +end + +# The context still exposes them if needed: +defmodule MyApp.Devices do + defdelegate discover(subnet, opts \\ []), to: MyApp.Devices.Discovery + defdelegate import_csv(path, site_id), to: MyApp.Devices.BulkImport +end +``` + +### Avoiding God Modules + +Signs of a god module: +- 500+ lines +- Mix of CRUD, business logic, notifications, reporting +- Functions that don't share state or purpose + +```elixir +# ❌ Bad: God module +defmodule MyApp.Devices do + def create_device(attrs), do: ... + def update_device(device, attrs), do: ... + def poll_device(device), do: ... # Should be in Monitoring + def generate_report(site_id), do: ... # Should be in Reports + def send_alert(device, message), do: ... # Should be in Notifications + def calculate_uptime(device_id), do: ... # Should be in Monitoring/Analytics + def export_to_csv(devices), do: ... # Should be in Exports + # ... 800 more lines +end + +# ✅ Good: Split by responsibility +defmodule MyApp.Devices do # CRUD + device-specific logic +defmodule MyApp.Monitoring do # Polling, metrics, uptime +defmodule MyApp.Notifications do # Alerts, emails +defmodule MyApp.Reports do # Report generation, exports +``` + +--- + +## 7. Deployment + +### Releases with `mix release` + +```elixir +# mix.exs +def project do + [ + app: :my_app, + version: "1.0.0", + elixir: "~> 1.17", + releases: [ + my_app: [ + include_executables_for: [:unix], + steps: [:assemble, :tar], + cookie: "my-secure-cookie-#{Mix.env()}" + ] + ] + ] +end +``` + +```bash +# Build a release +MIX_ENV=prod mix release + +# Run it +_build/prod/rel/my_app/bin/my_app start + +# Remote console +_build/prod/rel/my_app/bin/my_app remote +``` + +### Runtime Config + +```elixir +# config/runtime.exs — evaluated at boot time, not compile time +import Config + +config :my_app, MyApp.Repo, + url: System.get_env("DATABASE_URL") || raise("DATABASE_URL not set"), + pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10"), + ssl: System.get_env("DATABASE_SSL") == "true", + ssl_opts: [verify: :verify_none] + +config :my_app, MyAppWeb.Endpoint, + url: [host: System.get_env("PHX_HOST") || "localhost", port: 443, scheme: "https"], + http: [ + ip: {0, 0, 0, 0}, + port: String.to_integer(System.get_env("PORT") || "4000") + ], + secret_key_base: System.get_env("SECRET_KEY_BASE") || raise("SECRET_KEY_BASE not set") + +# ❌ Bad: Using Application.get_env at compile time for runtime values +# config/config.exs +config :my_app, api_key: System.get_env("API_KEY") # Baked in at compile time! + +# ✅ Good: Use runtime.exs or Application.compile_env only for truly static config +``` + +### Migrations in Production + +```elixir +# rel/overlays/bin/migrate — migration script for releases +defmodule MyApp.Release do + @moduledoc "Release tasks (run without Mix)." + + @app :my_app + + def migrate do + load_app() + + for repo <- repos() do + {:ok, _, _} = + Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :up, all: true)) + end + end + + def rollback(repo, version) do + load_app() + {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :down, to: version)) + end + + defp repos do + Application.fetch_env!(@app, :ecto_repos) + end + + defp load_app do + Application.ensure_all_started(:ssl) + Application.load(@app) + end +end + +# Run from release: +# bin/my_app eval "MyApp.Release.migrate()" +``` + +#### Safe migration practices + +```elixir +# ✅ Safe: Add columns as nullable first, backfill, then add constraint +def change do + alter table(:devices) do + add :firmware_version, :string # nullable, no default + end +end + +# Later migration: +def change do + alter table(:devices) do + modify :firmware_version, :string, null: false, default: "unknown" + end +end + +# ✅ Safe: Create index concurrently (no table lock) +@disable_ddl_transaction true +@disable_migration_lock true + +def change do + create index(:devices, [:site_id, :status], concurrently: true) +end + +# ❌ Dangerous: Renaming columns (breaks running code during deploy) +# Instead: add new column, backfill, deploy code using new column, drop old column +``` + +### Health Checks + +```elixir +defmodule MyAppWeb.HealthController do + use MyAppWeb, :controller + + def check(conn, _params) do + checks = %{ + database: check_database(), + migrations: check_migrations(), + memory: check_memory() + } + + status = if Enum.all?(Map.values(checks), &(&1 == :ok)), do: 200, else: 503 + + json(conn, %{ + status: if(status == 200, do: "healthy", else: "unhealthy"), + checks: checks, + version: Application.spec(:my_app, :vsn) |> to_string(), + node: Node.self() + }) + end + + # Liveness: "is the process alive?" (for k8s liveness probe) + def liveness(conn, _params) do + send_resp(conn, 200, "ok") + end + + # Readiness: "can it serve traffic?" (for k8s readiness probe / load balancer) + def readiness(conn, _params) do + if ready?() do + send_resp(conn, 200, "ready") + else + send_resp(conn, 503, "not ready") + end + end + + defp check_database do + case Ecto.Adapters.SQL.query(MyApp.Repo, "SELECT 1", []) do + {:ok, _} -> :ok + {:error, _} -> :error + end + end + + defp check_migrations do + # Ensure no pending migrations + case Ecto.Migrator.migrations(MyApp.Repo) do + migrations when is_list(migrations) -> + if Enum.any?(migrations, fn {status, _, _} -> status == :down end), + do: :pending, + else: :ok + + _ -> + :error + end + end + + defp check_memory do + mem = :erlang.memory(:total) + # Alert if using more than 1GB + if mem > 1_073_741_824, do: :warning, else: :ok + end + + defp ready? do + # Check if all required services are up + check_database() == :ok + end +end + +# Router +scope "/health", MyAppWeb do + pipe_through :api + get "/", HealthController, :check + get "/live", HealthController, :liveness + get "/ready", HealthController, :readiness +end +``` + +### Graceful Shutdown + +```elixir +# BEAM handles SIGTERM gracefully by default in releases. +# Customize shutdown behavior: + +defmodule MyApp.Application do + use Application + + @impl true + def start(_type, _args) do + children = [ + MyApp.Repo, + {Phoenix.PubSub, name: MyApp.PubSub}, + # Workers that need graceful shutdown + {MyApp.Poller, shutdown: :timer.seconds(30)}, + MyAppWeb.Endpoint + ] + + opts = [strategy: :one_for_one, name: MyApp.Supervisor] + Supervisor.start_link(children, opts) + end + + @impl true + def prep_stop(_state) do + # Called before supervision tree shuts down + # Deregister from load balancer, stop accepting new work + Logger.info("Application shutting down, draining connections...") + MyAppWeb.Endpoint.config_change([], []) + end + + @impl true + def stop(_state) do + Logger.info("Application stopped") + end +end + +# In GenServer, handle terminate for cleanup +defmodule MyApp.Poller do + use GenServer + + @impl true + def terminate(reason, state) do + Logger.info("Poller terminating: #{inspect(reason)}") + # Flush pending results, close connections, etc. + flush_pending(state) + :ok + end +end + +# vm.args.eex — configure shutdown timeout +# In rel/vm.args.eex: +# +sbwt none +# -shutdown_time 30000 +``` + +### Rolling Deploys + +```elixir +# Dockerfile for releases +# syntax=docker/dockerfile:1 +FROM hexpm/elixir:1.17.3-erlang-27.1-debian-bookworm-20240904 AS build + +RUN apt-get update && apt-get install -y git build-essential + +WORKDIR /app +ENV MIX_ENV=prod + +COPY mix.exs mix.lock ./ +RUN mix deps.get --only $MIX_ENV +RUN mix deps.compile + +COPY config/config.exs config/prod.exs config/ +COPY lib lib +COPY priv priv +COPY assets assets + +RUN mix assets.deploy +RUN mix compile +COPY config/runtime.exs config/ +RUN mix release + +# --- Runtime --- +FROM debian:bookworm-slim + +RUN apt-get update && apt-get install -y libssl3 libncurses6 locales \ + && rm -rf /var/lib/apt/lists/* + +RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen +ENV LANG=en_US.UTF-8 + +WORKDIR /app +COPY --from=build /app/_build/prod/rel/my_app ./ + +ENV PHX_SERVER=true +CMD ["bin/my_app", "eval", "MyApp.Release.migrate()"] # Run migrations +CMD ["bin/my_app", "start"] +``` + +#### Blue-green / rolling deploy checklist + +1. **Database migrations first** — run `migrate` before deploying new code +2. **Backward-compatible migrations only** — new code must work with old schema during transition +3. **Health check gates** — don't route traffic until `/health/ready` returns 200 +4. **Graceful drain** — send SIGTERM, wait for in-flight requests to complete (30s timeout) +5. **Connection draining** — WebSocket/LiveView connections reconnect automatically +6. **Feature flags** — for risky changes, deploy code behind a flag, enable after verification + +```yaml +# Example k8s deployment spec +apiVersion: apps/v1 +kind: Deployment +spec: + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 + template: + spec: + terminationGracePeriodSeconds: 30 + containers: + - name: app + livenessProbe: + httpGet: + path: /health/live + port: 4000 + initialDelaySeconds: 5 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /health/ready + port: 4000 + initialDelaySeconds: 10 + periodSeconds: 5 +``` + +--- + +## Quick Reference: Common Anti-Patterns + +| Anti-Pattern | Better Approach | +|---|---| +| Fat GenServer callbacks | Extract logic into pure-function modules | +| `Application.get_env` in module body | `Application.compile_env` or `runtime.exs` | +| Loading URL-dependent data in `mount` | Use `handle_params` | +| Storing large lists in assigns | Use `stream/3` | +| N+1 Ecto queries in loops | Preload or batch queries | +| God context module (500+ lines) | Split by responsibility | +| `Agent` for complex state | Use `GenServer` | +| Renaming DB columns in one migration | Add → backfill → migrate code → drop | +| `System.get_env` in `config.exs` | Move to `runtime.exs` | +| `String.concat` in loops | Use iolists | +| Frequent `:persistent_term` writes | Use ETS for mutable cache | +| Blocking work in `GenServer.init` | Use `handle_continue` | +| `Process.link` for observing | Use `Process.monitor` | +| `Repo.all` then filter in Elixir | Filter in the query | diff --git a/docs/references/gleam-best-practices.md b/docs/references/gleam-best-practices.md new file mode 100644 index 00000000..af659f21 --- /dev/null +++ b/docs/references/gleam-best-practices.md @@ -0,0 +1,1725 @@ +# Gleam Language Reference & Best Practices + +> Practical reference for building production Gleam applications on the BEAM. +> Covers language fundamentals, OTP, web development, and real-world patterns. + +--- + +## Table of Contents + +1. [Language Fundamentals](#1-language-fundamentals) +2. [OTP from Gleam](#2-otp-from-gleam) +3. [Web Development](#3-web-development) +4. [Package Ecosystem](#4-package-ecosystem) +5. [Interop (Erlang, Elixir, JavaScript)](#5-interop) +6. [Testing](#6-testing) +7. [Project Structure](#7-project-structure) +8. [Real-World Patterns](#8-real-world-patterns) + +--- + +## 1. Language Fundamentals + +### The Type System + +Gleam is statically typed with full type inference. No `null`, no exceptions, no implicit conversions. + +#### Result Type + +The core error handling primitive. Every fallible operation returns `Result(value, error)`. + +```gleam +import gleam/result +import gleam/int + +pub fn parse_age(input: String) -> Result(Int, String) { + case int.parse(input) { + Ok(age) if age >= 0 && age <= 150 -> Ok(age) + Ok(_) -> Error("Age out of range") + Error(_) -> Error("Not a number") + } +} + +pub fn main() { + // Chain results with result.try (like flatMap/bind) + let output = + "25" + |> parse_age + |> result.map(fn(age) { age + 1 }) + + // Pattern match on results + case output { + Ok(age) -> echo age + Error(msg) -> echo msg + } +} +``` + +#### Option (gleam/option) + +Gleam has no `null`. Optional values use `Option(a)` which is `Some(a) | None`. + +```gleam +import gleam/option.{type Option, None, Some} + +pub type User { + User(name: String, email: Option(String)) +} + +pub fn display_email(user: User) -> String { + case user.email { + Some(email) -> email + None -> "No email provided" + } +} + +// option module has helpers +pub fn get_email_or_default(user: User) -> String { + user.email + |> option.unwrap("no-reply@example.com") +} +``` + +#### Custom Types (Algebraic Data Types) + +Custom types are the backbone of Gleam data modeling. They're tagged unions (sum types) with named fields (product types). + +```gleam +// Simple enum +pub type Color { + Red + Green + Blue +} + +// Record with fields +pub type User { + User(id: Int, name: String, role: Role) +} + +// Sum type (tagged union) +pub type Role { + Admin + Moderator(permissions: List(String)) + Member +} + +// Generic types +pub type Validated(a) { + Valid(value: a) + Invalid(errors: List(String)) +} + +// Opaque types - hide internals from other modules +pub opaque type Email { + Email(String) +} + +pub fn new_email(value: String) -> Result(Email, String) { + case value { + _ if value == "" -> Error("Email cannot be empty") + _ -> Ok(Email(value)) + } +} + +pub fn to_string(email: Email) -> String { + let Email(value) = email + value +} +``` + +#### Record Updates + +```gleam +pub type Config { + Config(host: String, port: Int, debug: Bool) +} + +pub fn enable_debug(config: Config) -> Config { + Config(..config, debug: True) +} +``` + +### Pattern Matching + +Pattern matching is pervasive in Gleam — it's the primary flow control mechanism. + +```gleam +import gleam/int +import gleam/list + +// Match on custom types +pub fn describe_role(role: Role) -> String { + case role { + Admin -> "Administrator" + Moderator(perms) -> "Mod with " <> int.to_string(list.length(perms)) <> " permissions" + Member -> "Regular member" + } +} + +// Multiple subjects +pub fn classify(x: Int, y: Int) -> String { + case x, y { + 0, 0 -> "Origin" + 0, _ -> "Y axis" + _, 0 -> "X axis" + _, _ -> "Somewhere else" + } +} + +// Guards +pub fn categorize_age(age: Int) -> String { + case age { + a if a < 0 -> "Invalid" + a if a < 13 -> "Child" + a if a < 18 -> "Teenager" + a if a < 65 -> "Adult" + _ -> "Senior" + } +} + +// String prefix matching +pub fn parse_command(input: String) -> Result(String, String) { + case input { + "GET " <> path -> Ok(path) + "POST " <> path -> Ok(path) + _ -> Error("Unknown command") + } +} + +// List patterns +pub fn first_two(items: List(a)) -> String { + case items { + [] -> "empty" + [_] -> "one item" + [_, _, ..] -> "two or more" + } +} + +// Nested patterns +pub fn get_admin_name(user: User) -> Result(String, Nil) { + case user { + User(name:, role: Admin, ..) -> Ok(name) + _ -> Error(Nil) + } +} + +// As patterns (bind a name to the whole matched value) +pub fn check(user: User) -> User { + case user { + User(role: Admin, ..) as admin -> { + echo "Admin found" + admin + } + other -> other + } +} +``` + +### Use Expressions + +`use` is syntactic sugar for callback functions. It converts the rest of the function body into an anonymous function passed as the last argument. + +```gleam +import gleam/result +import gleam/list + +// WITHOUT use - nested callbacks +pub fn without_use(input: String) -> Result(String, String) { + result.try(parse_name(input), fn(name) { + result.try(validate_name(name), fn(valid_name) { + result.map(format_greeting(valid_name), fn(greeting) { + greeting + }) + }) + }) +} + +// WITH use - flat and readable +pub fn with_use(input: String) -> Result(String, String) { + use name <- result.try(parse_name(input)) + use valid_name <- result.try(validate_name(name)) + use greeting <- result.map(format_greeting(valid_name)) + greeting +} + +// use with list.map +pub fn double_all(numbers: List(Int)) -> List(Int) { + use n <- list.map(numbers) + n * 2 +} + +// use with list.filter +pub fn only_positive(numbers: List(Int)) -> List(Int) { + use n <- list.filter(numbers) + n > 0 +} + +// use for middleware (very common in web handlers) +pub fn handle(req: Request) -> Response { + use <- wisp.log_request(req) + use <- wisp.serve_static(req, under: "/static", from: "/public") + use req <- middleware(req) + route(req) +} +``` + +**How `use` works mechanically:** + +```gleam +// This: +use x <- some_function(arg1, arg2) +do_something(x) + +// Desugars to: +some_function(arg1, arg2, fn(x) { + do_something(x) +}) +``` + +### Pipelines + +The `|>` operator passes the left-hand value as the first argument to the right-hand function. + +```gleam +import gleam/string +import gleam/list +import gleam/int + +pub fn format_names(names: List(String)) -> String { + names + |> list.map(string.trim) + |> list.filter(fn(s) { s != "" }) + |> list.sort(string.compare) + |> list.map(string.capitalise) + |> string.join(", ") +} + +// Pipe to a different argument position with function capture +pub fn example() { + "world" + |> string.append("hello ", _) // Becomes string.append("hello ", "world") +} + +// Debug in the middle of a pipeline +pub fn debug_pipeline(x: Int) -> Int { + x + |> int.multiply(2) + |> echo // prints the intermediate value, passes it through + |> int.add(1) +} +``` + +### Labelled Arguments + +```gleam +pub fn create_user( + name name: String, + email email: String, + role role: Role, +) -> User { + User(id: 0, name:, role:) +} + +// Call with labels (order doesn't matter for labelled args) +pub fn example() { + create_user(role: Admin, name: "Alice", email: "alice@example.com") +} + +// Shorthand: when variable name matches label +pub fn example2() { + let name = "Bob" + let email = "bob@example.com" + let role = Member + create_user(name:, email:, role:) +} +``` + +### Generics + +```gleam +// Generic function +pub fn first(pair: #(a, b)) -> a { + pair.0 +} + +// Generic custom type +pub type Stack(a) { + Stack(items: List(a)) +} + +pub fn push(stack: Stack(a), item: a) -> Stack(a) { + Stack(items: [item, ..stack.items]) +} + +pub fn pop(stack: Stack(a)) -> Result(#(a, Stack(a)), Nil) { + case stack.items { + [] -> Error(Nil) + [top, ..rest] -> Ok(#(top, Stack(items: rest))) + } +} + +// Constrained generics don't exist in Gleam — +// use higher-order functions instead: +pub fn map_pair(pair: #(a, a), f: fn(a) -> b) -> #(b, b) { + #(f(pair.0), f(pair.1)) +} +``` + +--- + +## 2. OTP from Gleam + +Gleam runs on the BEAM and has first-class OTP support via `gleam_erlang` and `gleam_otp`. + +``` +gleam add gleam_otp gleam_erlang +``` + +### Processes and Subjects + +A `Subject(msg)` is a type-safe process mailbox reference. It's Gleam's answer to untyped Erlang PIDs. + +```gleam +import gleam/erlang/process.{type Subject} + +pub fn spawn_logger() -> Subject(String) { + // process.new_subject() creates a subject for receiving messages + let subject = process.new_subject() + + // Spawn a process + process.start(linked: True, running: fn() { + logger_loop(subject) + }) + + subject +} + +fn logger_loop(subject: Subject(String)) -> Nil { + // Receive a message (blocks until one arrives) + let msg = process.receive(subject, within: 5000) + case msg { + Ok(text) -> { + echo text + logger_loop(subject) + } + Error(Nil) -> { + echo "Logger timed out, stopping" + Nil + } + } +} + +pub fn main() { + let logger = spawn_logger() + process.send(logger, "Hello from main!") + process.send(logger, "Another message") + process.sleep(100) +} +``` + +### Actors (gleam_otp) + +Actors are the recommended abstraction — they handle OTP system messages automatically. + +```gleam +import gleam/erlang/process.{type Subject} +import gleam/otp/actor + +// Define your message type +pub type CounterMsg { + Increment(by: Int) + Decrement(by: Int) + GetCount(reply_to: Subject(Int)) + Reset +} + +// State is just a plain type +pub type CounterState { + CounterState(count: Int, name: String) +} + +// Start the actor +pub fn start(name: String) -> Result(Subject(CounterMsg), actor.StartError) { + let init_state = CounterState(count: 0, name:) + + actor.new(init_state) + |> actor.on_message(handle_message) + |> actor.start +} + +// Handle messages — return actor.Next to continue or actor.Stop to shut down +fn handle_message( + state: CounterState, + message: CounterMsg, +) -> actor.Next(CounterState, CounterMsg) { + case message { + Increment(by:) -> { + let new_state = CounterState(..state, count: state.count + by) + actor.continue(new_state) + } + + Decrement(by:) -> { + let new_state = CounterState(..state, count: state.count - by) + actor.continue(new_state) + } + + GetCount(reply_to:) -> { + actor.send(reply_to, state.count) + actor.continue(state) + } + + Reset -> { + actor.continue(CounterState(..state, count: 0)) + } + } +} + +// Client API — hide the message protocol behind functions +pub fn increment(counter: Subject(CounterMsg), by amount: Int) -> Nil { + actor.send(counter, Increment(by: amount)) +} + +pub fn get_count(counter: Subject(CounterMsg)) -> Int { + // actor.call sends a message and waits for a reply + actor.call(counter, GetCount, within: 1000) +} + +pub fn main() { + let assert Ok(counter) = start("my_counter") + + increment(counter, by: 5) + increment(counter, by: 3) + actor.send(counter, Decrement(by: 2)) + + let count = get_count(counter) + echo count // 6 +} +``` + +### Supervisors + +```gleam +import gleam/otp/static_supervisor + +pub fn start_app() { + static_supervisor.new(static_supervisor.OneForOne) + |> static_supervisor.add(static_supervisor.worker_child( + id: "counter_1", + run: fn(_) { start("counter_1") }, + )) + |> static_supervisor.add(static_supervisor.worker_child( + id: "counter_2", + run: fn(_) { start("counter_2") }, + )) + |> static_supervisor.start +} +``` + +### Selectors (Listening to Multiple Subjects) + +```gleam +import gleam/erlang/process.{type Selector, type Subject} + +pub type Event { + UserMessage(String) + SystemAlert(String) + Tick +} + +pub fn listen( + user_sub: Subject(String), + system_sub: Subject(String), + tick_sub: Subject(Nil), +) -> Event { + // Build a selector that maps different subjects to a unified Event type + let selector = + process.new_selector() + |> process.selecting(user_sub, UserMessage) + |> process.selecting(system_sub, SystemAlert) + |> process.selecting(tick_sub, fn(_) { Tick }) + + // Wait for any of them + let assert Ok(event) = process.select(selector, within: 5000) + event +} +``` + +--- + +## 3. Web Development + +### Wisp Framework + +Wisp is the standard web framework for Gleam. It wraps `gleam_http` types and provides middleware. + +``` +gleam add wisp mist gleam_http gleam_erlang +``` + +#### Basic App Structure + +```gleam +// src/app.gleam — entry point +import gleam/erlang/process +import mist +import wisp +import wisp/wisp_mist +import app/router +import app/web.{type Context, Context} + +pub fn main() { + wisp.configure_logger() + + let secret_key_base = wisp.random_string(64) + let ctx = Context( + secret_key_base:, + db: Nil, // your DB connection goes here + ) + + let assert Ok(_) = + wisp_mist.handler(router.handle_request(_, ctx), secret_key_base) + |> mist.new + |> mist.port(8000) + |> mist.start_http + + process.sleep_forever() +} +``` + +#### Context Type + +```gleam +// src/app/web.gleam +pub type Context { + Context(secret_key_base: String, db: connection) +} +``` + +#### Routing + +```gleam +// src/app/router.gleam +import wisp.{type Request, type Response} +import app/web.{type Context} +import app/handlers/user_handler +import app/handlers/health_handler + +pub fn handle_request(req: Request, ctx: Context) -> Response { + // Apply global middleware + use req <- middleware(req, ctx) + + // Route based on method + path segments + case wisp.path_segments(req) { + // GET / + [] -> wisp.ok() |> wisp.string_body("Welcome!") + + // GET /health + ["health"] -> health_handler.index(req) + + // /users/... + ["users", ..rest] -> user_routes(req, ctx, rest) + + // 404 + _ -> wisp.not_found() + } +} + +fn user_routes(req: Request, ctx: Context, path: List(String)) -> Response { + case req.method, path { + // GET /users + http.Get, [] -> user_handler.list(req, ctx) + + // POST /users + http.Post, [] -> user_handler.create(req, ctx) + + // GET /users/:id + http.Get, [id] -> user_handler.show(req, ctx, id) + + // PUT /users/:id + http.Put, [id] -> user_handler.update(req, ctx, id) + + // DELETE /users/:id + http.Delete, [id] -> user_handler.delete(req, ctx, id) + + // Method not allowed + _, _ -> wisp.method_not_allowed([http.Get, http.Post, http.Put, http.Delete]) + } +} + +fn middleware(req: Request, ctx: Context, handler: fn(Request) -> Response) -> Response { + let req = wisp.method_override(req) + use <- wisp.log_request(req) + use <- wisp.rescue_crashes + use req <- wisp.handle_head(req) + use <- wisp.serve_static(req, under: "/static", from: static_directory()) + + handler(req) +} + +fn static_directory() -> String { + let assert Ok(priv) = wisp.priv_directory("app") + priv <> "/static" +} +``` + +#### Request Handling + +```gleam +// src/app/handlers/user_handler.gleam +import gleam/http +import gleam/json +import gleam/dynamic/decode +import wisp.{type Request, type Response} +import app/web.{type Context} + +pub fn create(req: Request, ctx: Context) -> Response { + // Read JSON body + use json_body <- wisp.require_json(req) + + // Decode the JSON + let decoder = + decode.into({ + use name <- decode.parameter + use email <- decode.parameter + #(name, email) + }) + |> decode.field("name", decode.string) + |> decode.field("email", decode.string) + + case decode.run(json_body, decoder) { + Ok(#(name, email)) -> { + // Do something with the data... + let response_json = + json.object([ + #("name", json.string(name)), + #("email", json.string(email)), + #("id", json.int(1)), + ]) + |> json.to_string_tree + + wisp.json_response(response_json, 201) + } + Error(_) -> wisp.unprocessable_entity() + } +} + +pub fn list(_req: Request, _ctx: Context) -> Response { + let body = + json.array([ + json.object([#("id", json.int(1)), #("name", json.string("Alice"))]), + json.object([#("id", json.int(2)), #("name", json.string("Bob"))]), + ], of: fn(x) { x }) + |> json.to_string_tree + + wisp.json_response(body, 200) +} +``` + +### Lustre (Frontend Framework) + +Lustre is Gleam's Elm-inspired frontend framework. Model-Update-View architecture. + +``` +gleam add lustre +gleam add --dev lustre_dev_tools +``` + +#### Counter App + +```gleam +import gleam/int +import lustre +import lustre/attribute +import lustre/element.{text} +import lustre/element/html.{button, div, h1, p} +import lustre/event + +// MODEL +pub type Model { + Model(count: Int, name: String) +} + +fn init(_flags) -> Model { + Model(count: 0, name: "Counter") +} + +// UPDATE +pub type Msg { + Increment + Decrement + Reset + SetName(String) +} + +fn update(model: Model, msg: Msg) -> Model { + case msg { + Increment -> Model(..model, count: model.count + 1) + Decrement -> Model(..model, count: model.count - 1) + Reset -> Model(..model, count: 0) + SetName(name) -> Model(..model, name:) + } +} + +// VIEW +fn view(model: Model) -> element.Element(Msg) { + div([], [ + h1([], [text(model.name)]), + p([], [text("Count: " <> int.to_string(model.count))]), + button([event.on_click(Increment)], [text("+")]), + button([event.on_click(Decrement)], [text("-")]), + button([event.on_click(Reset)], [text("Reset")]), + html.input([ + attribute.value(model.name), + event.on_input(SetName), + attribute.placeholder("Counter name"), + ]), + ]) +} + +// MAIN +pub fn main() { + let app = lustre.simple(init, update, view) + let assert Ok(_) = lustre.start(app, "#app", Nil) + Nil +} +``` + +#### Lustre with Effects + +For apps that need side effects (HTTP requests, timers, etc.), use `lustre.application` instead of `lustre.simple`. + +```gleam +import lustre +import lustre/effect.{type Effect} + +pub fn main() { + let app = lustre.application(init, update, view) + let assert Ok(_) = lustre.start(app, "#app", Nil) + Nil +} + +fn init(_flags) -> #(Model, Effect(Msg)) { + #(Model(loading: True, data: None), fetch_data()) +} + +fn update(model: Model, msg: Msg) -> #(Model, Effect(Msg)) { + case msg { + DataLoaded(data) -> #(Model(loading: False, data: Some(data)), effect.none()) + FetchFailed -> #(Model(loading: False, data: None), effect.none()) + Refresh -> #(Model(..model, loading: True), fetch_data()) + } +} + +fn fetch_data() -> Effect(Msg) { + // Use lustre_http or similar packages for HTTP effects + effect.none() // placeholder +} +``` + +#### Lustre Server Components + +Lustre components can run on the server and push updates to connected clients, similar to Phoenix LiveView. + +```gleam +import lustre +import lustre/server_component + +// Create a server component +pub fn app() { + lustre.component(init, update, view, on_attribute_change()) +} + +fn on_attribute_change() -> Dict(String, Decoder(Msg)) { + dict.new() +} +``` + +### Mist HTTP Server + +Mist is the HTTP server that Wisp runs on top of. You can use it directly for lower-level control. + +```gleam +import gleam/bytes_tree +import gleam/http/response +import gleam/erlang/process +import mist + +pub fn main() { + let assert Ok(_) = + fn(_req) { + response.new(200) + |> response.set_body(mist.Bytes( + bytes_tree.from_string("Hello from Mist!"), + )) + } + |> mist.new + |> mist.port(3000) + |> mist.start_http + + process.sleep_forever() +} +``` + +--- + +## 4. Package Ecosystem + +### Key Packages + +| Package | Purpose | Install | +|---------|---------|---------| +| `gleam_stdlib` | Standard library (lists, strings, result, option, etc.) | Included by default | +| `gleam_json` | JSON encoding/decoding | `gleam add gleam_json` | +| `gleam_http` | HTTP types (Request, Response, Method) | `gleam add gleam_http` | +| `gleam_crypto` | Hashing, HMAC, secure random | `gleam add gleam_crypto` | +| `gleam_erlang` | Erlang interop, process, atoms | `gleam add gleam_erlang` | +| `gleam_otp` | OTP actors, supervisors | `gleam add gleam_otp` | +| `wisp` | Web framework | `gleam add wisp` | +| `mist` | HTTP server | `gleam add mist` | +| `lustre` | Frontend framework | `gleam add lustre` | +| `sqlight` | SQLite bindings | `gleam add sqlight` | +| `gleam_pgo` | PostgreSQL client (PGO) | `gleam add gleam_pgo` | +| `cake` | SQL query builder | `gleam add cake` | +| `envoy` | Environment variables | `gleam add envoy` | +| `argv` | CLI argument parsing | `gleam add argv` | +| `tom` | TOML parser | `gleam add tom` | +| `simplifile` | File system operations | `gleam add simplifile` | +| `gleeunit` | Test runner | `gleam add --dev gleeunit` | + +### gleam_json + +```gleam +import gleam/json +import gleam/dynamic/decode + +// Encoding +pub fn encode_user(user: User) -> String { + json.object([ + #("id", json.int(user.id)), + #("name", json.string(user.name)), + #("role", encode_role(user.role)), + ]) + |> json.to_string +} + +fn encode_role(role: Role) -> json.Json { + case role { + Admin -> json.string("admin") + Moderator(perms) -> json.object([ + #("type", json.string("moderator")), + #("permissions", json.array(perms, json.string)), + ]) + Member -> json.string("member") + } +} + +// Decoding +pub fn decode_user(data: String) -> Result(User, json.DecodeError) { + let user_decoder = + decode.into({ + use id <- decode.parameter + use name <- decode.parameter + use role <- decode.parameter + User(id:, name:, role:) + }) + |> decode.field("id", decode.int) + |> decode.field("name", decode.string) + |> decode.field("role", role_decoder()) + + json.parse(data, user_decoder) +} + +fn role_decoder() -> decode.Decoder(Role) { + decode.one_of(decode.string |> decode.then(fn(s) { + case s { + "admin" -> decode.into(Admin) + "member" -> decode.into(Member) + _ -> decode.fail("Role") + } + }), [ + // Or decode the object form + decode.into({ + use perms <- decode.parameter + Moderator(permissions: perms) + }) + |> decode.field("permissions", decode.list(decode.string)), + ]) +} +``` + +### sqlight (SQLite) + +```gleam +import sqlight + +pub fn main() { + use conn <- sqlight.with_connection("mydb.sqlite") + + // Create table + let assert Ok(_) = + sqlight.exec( + "CREATE TABLE IF NOT EXISTS users ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL, + email TEXT NOT NULL UNIQUE + )", + conn, + ) + + // Insert + let assert Ok(_) = + sqlight.query( + "INSERT INTO users (name, email) VALUES (?, ?)", + conn, + [sqlight.text("Alice"), sqlight.text("alice@example.com")], + dynamic.dynamic, + ) + + // Query + let assert Ok(rows) = + sqlight.query( + "SELECT id, name, email FROM users WHERE name = ?", + conn, + [sqlight.text("Alice")], + decode.into({ + use id <- decode.parameter + use name <- decode.parameter + use email <- decode.parameter + #(id, name, email) + }) + |> decode.field(0, decode.int) + |> decode.field(1, decode.string) + |> decode.field(2, decode.string), + ) + + echo rows +} +``` + +### Publishing to Hex + +```toml +# gleam.toml +name = "my_package" +version = "1.0.0" +description = "A useful Gleam package" +licences = ["Apache-2.0"] + +[repository] +type = "github" +user = "myuser" +repo = "my_package" +``` + +```bash +# Build and publish +gleam publish + +# Retire a version (if you published something broken) +gleam hex retire my_package 1.0.0 security "Use 1.0.1 instead" +``` + +--- + +## 5. Interop + +### Calling Erlang from Gleam (FFI) + +Create an Erlang file alongside your Gleam module: + +```gleam +// src/app/crypto_utils.gleam + +// Declare external Erlang functions +@external(erlang, "crypto", "hash") +fn erlang_hash(algorithm: atom, data: BitArray) -> BitArray + +// Wrap it in a Gleam-friendly API +import gleam/erlang/atom + +pub fn sha256(data: BitArray) -> BitArray { + let assert Ok(algo) = atom.from_string("sha256") + erlang_hash(algo, data) +} +``` + +For custom Erlang code, create a `.erl` file in `src/`: + +```erlang +%% src/my_ffi.erl +-module(my_ffi). +-export([system_time/0, format_timestamp/1]). + +system_time() -> + erlang:system_time(millisecond). + +format_timestamp(Ms) -> + calendar:system_time_to_rfc3339(Ms div 1000, [{unit, second}]). +``` + +```gleam +// src/app/time.gleam + +@external(erlang, "my_ffi", "system_time") +pub fn system_time_ms() -> Int + +@external(erlang, "my_ffi", "format_timestamp") +pub fn format_timestamp(ms: Int) -> String +``` + +### Calling Elixir from Gleam + +Elixir modules are just Erlang modules with an `Elixir.` prefix: + +```gleam +// Call Elixir's Jason library +@external(erlang, "Elixir.Jason", "encode!") +fn jason_encode(term: dynamic.Dynamic) -> String + +// Call Elixir's Enum module +@external(erlang, "Elixir.Enum", "shuffle") +pub fn shuffle(list: List(a)) -> List(a) +``` + +**Important:** To use Elixir dependencies, add them to your `gleam.toml`: + +```toml +[dependencies] +jason = ">= 1.4.0 and < 2.0.0" # Elixir/Erlang hex packages work +``` + +### JavaScript Target FFI + +For the JavaScript target, create `.mjs` files: + +```javascript +// src/app/dom_ffi.mjs +export function get_element(id) { + const el = document.getElementById(id); + if (el) return new Ok(el); + return new Error(undefined); +} + +export function set_inner_html(element, html) { + element.innerHTML = html; + return undefined; +} +``` + +```gleam +// src/app/dom.gleam + +@external(javascript, "./dom_ffi.mjs", "get_element") +pub fn get_element(id: String) -> Result(Dynamic, Nil) + +@external(javascript, "./dom_ffi.mjs", "set_inner_html") +pub fn set_inner_html(element: Dynamic, html: String) -> Nil +``` + +### Target-Conditional Code + +```gleam +// Provide different implementations per target +@external(erlang, "my_ffi", "monotonic_time") +@external(javascript, "./time_ffi.mjs", "monotonic_time") +pub fn monotonic_time() -> Int +``` + +### Erlang Target vs JavaScript Target + +| Feature | Erlang (BEAM) | JavaScript | +|---------|---------------|------------| +| Concurrency | Full OTP (actors, supervisors) | Single-threaded (async) | +| Int size | Arbitrary precision | 64-bit float | +| FFI target | `.erl` files | `.mjs` files | +| Use cases | Servers, distributed systems | Browser apps, serverless | +| OTP support | ✅ Full | ❌ Not available | +| Package compat | Erlang + Elixir hex packages | npm packages via FFI | +| Build output | BEAM bytecode | JavaScript modules | + +--- + +## 6. Testing + +### gleeunit + +Gleam's standard test runner. Any public function in `test/` ending in `_test` is run as a test. + +```gleam +// test/app_test.gleam +import gleeunit +import gleeunit/should +import app/user + +pub fn main() { + gleeunit.main() +} + +pub fn parse_age_valid_test() { + user.parse_age("25") + |> should.be_ok + |> should.equal(25) +} + +pub fn parse_age_negative_test() { + user.parse_age("-1") + |> should.be_error + |> should.equal("Age out of range") +} + +pub fn parse_age_not_number_test() { + user.parse_age("abc") + |> should.be_error + |> should.equal("Not a number") +} +``` + +```bash +gleam test +``` + +### Custom Assertions + +```gleam +// test/test_helpers.gleam +import gleam/string + +pub fn should_contain(haystack: String, needle: String) -> String { + case string.contains(haystack, needle) { + True -> haystack + False -> + panic as { + "Expected \"" <> haystack <> "\" to contain \"" <> needle <> "\"" + } + } +} + +pub fn should_be_between(value: Int, low: Int, high: Int) -> Int { + case value >= low && value <= high { + True -> value + False -> + panic as "Expected value to be between bounds" + } +} +``` + +```gleam +// test/string_test.gleam +import test_helpers.{should_contain} + +pub fn greeting_test() { + build_greeting("Alice") + |> should_contain("Alice") + |> should_contain("Hello") +} +``` + +### Testing Actors + +```gleam +import gleam/erlang/process + +pub fn counter_increment_test() { + let assert Ok(counter) = counter.start("test") + + counter.increment(counter, by: 5) + counter.increment(counter, by: 3) + + counter.get_count(counter) + |> should.equal(8) +} +``` + +### Testing Wisp Handlers + +```gleam +import wisp/testing + +pub fn health_check_test() { + let req = testing.get("/health", []) + let resp = router.handle_request(req, test_context()) + + resp.status + |> should.equal(200) +} + +pub fn create_user_test() { + let body = "{\"name\": \"Alice\", \"email\": \"alice@test.com\"}" + let req = testing.post_json("/users", [], body) + let resp = router.handle_request(req, test_context()) + + resp.status + |> should.equal(201) +} + +fn test_context() -> Context { + Context(secret_key_base: "test-secret", db: Nil) +} +``` + +--- + +## 7. Project Structure + +### gleam.toml Configuration + +```toml +name = "my_app" +version = "1.0.0" +target = "erlang" # or "javascript" +description = "My Gleam application" +licences = ["Apache-2.0"] + +# OTP application config (for BEAM deployment) +[erlang] +application_start_module = "my_app/application" +extra_applications = ["crypto", "ssl"] # Erlang apps to start + +[repository] +type = "github" +user = "myuser" +repo = "my_app" + +[dependencies] +gleam_stdlib = ">= 0.34.0 and < 2.0.0" +gleam_erlang = ">= 0.25.0 and < 2.0.0" +gleam_otp = ">= 0.10.0 and < 2.0.0" +gleam_http = ">= 3.6.0 and < 4.0.0" +gleam_json = ">= 1.0.0 and < 3.0.0" +wisp = ">= 1.0.0 and < 3.0.0" +mist = ">= 1.0.0 and < 4.0.0" + +[dev-dependencies] +gleeunit = ">= 1.0.0 and < 2.0.0" +``` + +### Module Organization + +``` +my_app/ +├── gleam.toml +├── manifest.toml # Lock file (commit this) +├── src/ +│ ├── my_app.gleam # Entry point (pub fn main) +│ ├── my_app/ +│ │ ├── application.gleam # OTP application (supervisor tree) +│ │ ├── router.gleam # HTTP routing +│ │ ├── web.gleam # Context type, shared web helpers +│ │ ├── internal.gleam # Internal helpers (not public API) +│ │ ├── handlers/ +│ │ │ ├── user_handler.gleam +│ │ │ └── health_handler.gleam +│ │ ├── models/ +│ │ │ ├── user.gleam # User type + encoding/decoding +│ │ │ └── session.gleam +│ │ ├── services/ +│ │ │ ├── auth.gleam # Business logic +│ │ │ └── email.gleam +│ │ └── db/ +│ │ ├── repo.gleam # Database connection + queries +│ │ └── migrations.gleam +│ └── my_ffi.erl # Erlang FFI (if needed) +├── test/ +│ ├── my_app_test.gleam +│ └── my_app/ +│ ├── handlers/ +│ │ └── user_handler_test.gleam +│ └── models/ +│ └── user_test.gleam +└── priv/ + └── static/ # Static files served by wisp + ├── css/ + └── js/ +``` + +### Gleam vs Elixir: When to Use Each + +| Dimension | Gleam | Elixir | +|-----------|-------|--------| +| **Type safety** | ✅ Full static types, no runtime type errors | ❌ Dynamic typing, runtime crashes possible | +| **Ecosystem maturity** | 🟡 Growing rapidly, some gaps | ✅ Mature, battle-tested (Phoenix, Ecto, LiveView) | +| **OTP support** | 🟡 Subset (type-safe actors, supervisors) | ✅ Full OTP (GenServer, GenStage, DynamicSupervisor) | +| **Metaprogramming** | ❌ No macros (by design) | ✅ Powerful macros | +| **Learning curve** | ✅ Small language, few concepts | 🟡 More features to learn | +| **Refactoring** | ✅ Compiler catches everything | 🟡 Tests + dialyzer help, but gaps exist | +| **Web (backend)** | 🟡 Wisp is solid but young | ✅ Phoenix is world-class | +| **Web (frontend)** | ✅ Lustre (Elm-like, universal components) | 🟡 LiveView (server-dependent) | +| **Database** | 🟡 sqlight, gleam_pgo, cake (basic) | ✅ Ecto (migrations, schemas, changesets) | +| **Interop** | ✅ Calls Erlang & Elixir directly | ✅ Calls Erlang directly | +| **JS target** | ✅ Compiles to JavaScript | ❌ BEAM only | + +**Use Gleam when:** +- Type safety is a priority +- You want compile-time guarantees +- Building API services where Wisp's simplicity is sufficient +- Frontend + backend in one language (JavaScript target) +- Starting a new service that doesn't need Ecto/Phoenix ecosystem + +**Use Elixir when:** +- You need the full Phoenix/LiveView stack +- Ecto's database abstractions are important +- You need macros or DSLs +- The library you need only exists in Elixir +- You're extending an existing Elixir codebase + +**Use both:** +- Gleam and Elixir interop seamlessly on the BEAM +- Write type-safe core logic in Gleam, use Elixir for Phoenix/Ecto +- Gradually introduce Gleam into an Elixir project + +--- + +## 8. Real-World Patterns + +### Error Handling with Result Chains + +```gleam +import gleam/result + +pub type AppError { + NotFound(resource: String) + Unauthorized + ValidationError(field: String, message: String) + DatabaseError(detail: String) +} + +// Chain operations that can fail +pub fn update_user_email( + db: Connection, + user_id: Int, + new_email: String, +) -> Result(User, AppError) { + use user <- result.try(find_user(db, user_id)) + use validated_email <- result.try(validate_email(new_email)) + use updated <- result.try(save_user(db, User(..user, email: validated_email))) + Ok(updated) +} + +// Map errors between types +pub fn find_user(db: Connection, id: Int) -> Result(User, AppError) { + db_query(db, id) + |> result.map_error(fn(_) { NotFound("user") }) +} + +// Recover from errors +pub fn get_user_or_guest(db: Connection, id: Int) -> User { + find_user(db, id) + |> result.unwrap(guest_user()) +} + +// Convert AppError to HTTP responses +pub fn error_to_response(error: AppError) -> Response { + case error { + NotFound(resource) -> + wisp.not_found() + |> wisp.string_body(resource <> " not found") + Unauthorized -> + wisp.response(401) + |> wisp.string_body("Unauthorized") + ValidationError(field, message) -> + wisp.unprocessable_entity() + |> wisp.string_body(field <> ": " <> message) + DatabaseError(_) -> + wisp.internal_server_error() + } +} +``` + +### Configuration + +```gleam +import envoy +import gleam/int +import gleam/result + +pub type Config { + Config( + port: Int, + host: String, + database_url: String, + secret_key: String, + log_level: LogLevel, + ) +} + +pub type LogLevel { + Debug + Info + Warn + Error +} + +pub fn load() -> Result(Config, String) { + use port <- result.try( + envoy.get("PORT") + |> result.then(int.parse) + |> result.replace_error("Invalid PORT"), + ) + use host <- result.try( + envoy.get("HOST") + |> result.replace_error("Missing HOST"), + ) + use database_url <- result.try( + envoy.get("DATABASE_URL") + |> result.replace_error("Missing DATABASE_URL"), + ) + use secret_key <- result.try( + envoy.get("SECRET_KEY") + |> result.replace_error("Missing SECRET_KEY"), + ) + let log_level = + envoy.get("LOG_LEVEL") + |> result.unwrap("info") + |> parse_log_level + + Ok(Config(port:, host:, database_url:, secret_key:, log_level:)) +} + +fn parse_log_level(s: String) -> LogLevel { + case s { + "debug" -> Debug + "warn" -> Warn + "error" -> Error + _ -> Info + } +} +``` + +### Logging + +```gleam +import gleam/io +import gleam/erlang + +// Use wisp's built-in logging (wraps Erlang's logger) +import wisp + +pub fn main() { + wisp.configure_logger() + wisp.log_info("Application starting") + wisp.log_warning("Something might be wrong") + wisp.log_error("Something went wrong!") +} + +// Or use Erlang's logger directly via FFI +@external(erlang, "logger", "info") +pub fn log_info(message: String) -> Nil +``` + +### JSON API Design Pattern + +```gleam +import gleam/json +import gleam/dynamic/decode +import gleam/http +import wisp.{type Request, type Response} + +// Consistent API response wrapper +pub fn json_success(data: json.Json, status: Int) -> Response { + json.object([ + #("ok", json.bool(True)), + #("data", data), + ]) + |> json.to_string_tree + |> wisp.json_response(status) +} + +pub fn json_error(message: String, status: Int) -> Response { + json.object([ + #("ok", json.bool(False)), + #("error", json.string(message)), + ]) + |> json.to_string_tree + |> wisp.json_response(status) +} + +// Full CRUD handler example +pub fn handle(req: Request, ctx: Context, id: String) -> Response { + case req.method { + http.Get -> show(ctx, id) + http.Put -> { + use body <- wisp.require_json(req) + update(ctx, id, body) + } + http.Delete -> delete(ctx, id) + _ -> wisp.method_not_allowed([http.Get, http.Put, http.Delete]) + } +} + +fn show(ctx: Context, id: String) -> Response { + case find_item(ctx.db, id) { + Ok(item) -> json_success(encode_item(item), 200) + Error(NotFound(_)) -> json_error("Not found", 404) + Error(_) -> json_error("Internal error", 500) + } +} + +fn update(ctx: Context, id: String, body: Dynamic) -> Response { + let decoder = + decode.into({ + use name <- decode.parameter + use value <- decode.parameter + #(name, value) + }) + |> decode.field("name", decode.string) + |> decode.field("value", decode.int) + + case decode.run(body, decoder) { + Ok(#(name, value)) -> { + case save_item(ctx.db, id, name, value) { + Ok(item) -> json_success(encode_item(item), 200) + Error(e) -> error_to_response(e) + } + } + Error(_) -> json_error("Invalid request body", 422) + } +} +``` + +### Database Access Pattern + +```gleam +import gleam/pgo +import gleam/dynamic/decode + +pub type Repo { + Repo(db: pgo.Connection) +} + +pub fn connect(database_url: String) -> Repo { + let config = + pgo.url_config(database_url) + |> pgo.pool_size(10) + + let db = pgo.connect(config) + Repo(db:) +} + +pub fn find_user_by_email( + repo: Repo, + email: String, +) -> Result(User, AppError) { + let query = "SELECT id, name, email, role FROM users WHERE email = $1" + + let user_decoder = + decode.into({ + use id <- decode.parameter + use name <- decode.parameter + use email <- decode.parameter + use role <- decode.parameter + User(id:, name:, email:, role: parse_role(role)) + }) + |> decode.field(0, decode.int) + |> decode.field(1, decode.string) + |> decode.field(2, decode.string) + |> decode.field(3, decode.string) + + case pgo.execute(query, repo.db, [pgo.text(email)], user_decoder) { + Ok(pgo.Returned(_, [user])) -> Ok(user) + Ok(pgo.Returned(_, [])) -> Error(NotFound("user")) + Error(e) -> Error(DatabaseError(string.inspect(e))) + } +} + +pub fn create_user( + repo: Repo, + name: String, + email: String, + role: String, +) -> Result(User, AppError) { + let query = + "INSERT INTO users (name, email, role) VALUES ($1, $2, $3) + RETURNING id, name, email, role" + + let user_decoder = + decode.into({ + use id <- decode.parameter + use name <- decode.parameter + use email <- decode.parameter + use role <- decode.parameter + User(id:, name:, email:, role: parse_role(role)) + }) + |> decode.field(0, decode.int) + |> decode.field(1, decode.string) + |> decode.field(2, decode.string) + |> decode.field(3, decode.string) + + case pgo.execute(query, repo.db, [pgo.text(name), pgo.text(email), pgo.text(role)], user_decoder) { + Ok(pgo.Returned(_, [user])) -> Ok(user) + Error(e) -> Error(DatabaseError(string.inspect(e))) + } +} + +fn parse_role(s: String) -> Role { + case s { + "admin" -> Admin + "moderator" -> Moderator(permissions: []) + _ -> Member + } +} +``` + +--- + +## Quick Reference + +### Common Imports + +```gleam +import gleam/io // println, debug +import gleam/int // Int operations +import gleam/float // Float operations +import gleam/string // String operations +import gleam/list // List operations +import gleam/dict.{type Dict} // Dict (hash map) +import gleam/option.{type Option, Some, None} +import gleam/result // Result helpers +import gleam/json // JSON encode/decode +import gleam/dynamic/decode // Dynamic value decoding +import gleam/http.{Get, Post, Put, Delete} +import gleam/erlang/process.{type Subject} +import gleam/otp/actor // OTP actors +``` + +### Common Operations Cheat Sheet + +```gleam +// String interpolation (there isn't any — use <>) +"Hello " <> name <> "!" + +// Convert to string +int.to_string(42) +float.to_string(3.14) + +// Parse +int.parse("42") // Ok(42) +float.parse("3.14") // Ok(3.14) + +// List operations +list.map(items, fn(x) { x + 1 }) +list.filter(items, fn(x) { x > 0 }) +list.fold(items, 0, fn(acc, x) { acc + x }) +list.find(items, fn(x) { x.id == target_id }) + +// Dict operations +dict.new() +dict.insert(d, "key", "value") +dict.get(d, "key") // Result(value, Nil) +dict.from_list([#("a", 1), #("b", 2)]) + +// Tuples +let pair = #("hello", 42) +pair.0 // "hello" +pair.1 // 42 + +// Debug print any value +echo some_value + +// Assert (crash if pattern doesn't match — use sparingly) +let assert Ok(value) = might_fail() +``` + +--- + +*Last updated: 2026-03-13. Based on Gleam ~1.x, gleam_otp 1.2.0, wisp 2.2.1, lustre 5.6.0.* diff --git a/docs/references/golang-best-practices.md b/docs/references/golang-best-practices.md new file mode 100644 index 00000000..4c5fb964 --- /dev/null +++ b/docs/references/golang-best-practices.md @@ -0,0 +1,1883 @@ +# Go (Golang) Best Practices Reference + +> Comprehensive reference for idiomatic, production-grade Go. Concrete examples throughout. + +--- + +## Table of Contents + +1. [Idiomatic Go](#1-idiomatic-go) +2. [Concurrency](#2-concurrency) +3. [Project Structure](#3-project-structure) +4. [Networking](#4-networking) +5. [Testing](#5-testing) +6. [Performance](#6-performance) +7. [Observability](#7-observability) +8. [Build & Deploy](#8-build--deploy) + +--- + +## 1. Idiomatic Go + +### Error Handling + +#### Wrapping errors with `%w` + +Always wrap errors with context using `fmt.Errorf` and `%w` so callers can unwrap: + +```go +func readConfig(path string) (*Config, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("readConfig %s: %w", path, err) + } + var cfg Config + if err := json.Unmarshal(data, &cfg); err != nil { + return nil, fmt.Errorf("readConfig unmarshal: %w", err) + } + return &cfg, nil +} + +// Caller can inspect: +if errors.Is(err, os.ErrNotExist) { + // handle missing file +} +``` + +#### Sentinel errors + +Define package-level sentinel errors for conditions callers need to check: + +```go +package repo + +import "errors" + +var ( + ErrNotFound = errors.New("not found") + ErrConflict = errors.New("conflict") + ErrUnauthorized = errors.New("unauthorized") +) + +func (r *Repo) GetDevice(id string) (*Device, error) { + d, ok := r.devices[id] + if !ok { + return nil, fmt.Errorf("device %s: %w", id, ErrNotFound) + } + return d, nil +} +``` + +#### Custom error types + +Use when you need structured error data beyond a message: + +```go +type ValidationError struct { + Field string + Message string +} + +func (e *ValidationError) Error() string { + return fmt.Sprintf("validation: %s — %s", e.Field, e.Message) +} + +// Check with errors.As: +var ve *ValidationError +if errors.As(err, &ve) { + log.Printf("bad field: %s", ve.Field) +} +``` + +#### Don't use `errors.Is` / `errors.As` for every error + +Only sentinel/typed errors need programmatic inspection. Most errors just bubble up with wrapping — that's fine. Don't over-engineer. + +### Interface Design + +#### Small interfaces + +The best Go interfaces have 1–3 methods. The stdlib sets the standard: + +```go +type Reader interface { + Read(p []byte) (n int, err error) +} + +type Writer interface { + Write(p []byte) (n int, err error) +} +``` + +Define your own small interfaces close to where they're consumed: + +```go +// In the handler package, not the repo package +type DeviceFinder interface { + FindDevice(ctx context.Context, id string) (*model.Device, error) +} + +type handler struct { + finder DeviceFinder +} +``` + +#### Accept interfaces, return structs + +```go +// Good — accepts interface, returns concrete type +func NewService(repo DeviceRepo, logger *slog.Logger) *Service { + return &Service{repo: repo, logger: logger} +} + +// Bad — returning an interface hides the concrete type unnecessarily +func NewService(repo DeviceRepo) ServiceInterface { ... } +``` + +This lets callers benefit from the full concrete type while keeping your function testable via the interface parameter. + +#### Don't define interfaces prematurely + +> "The bigger the interface, the weaker the abstraction." — Rob Pike + +Define interfaces when you have ≥2 implementations OR need to mock in tests. Not before. + +### Embedding + +Use struct embedding for composition, not inheritance: + +```go +type BaseModel struct { + ID string `json:"id"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` +} + +type Device struct { + BaseModel + Name string `json:"name"` + IP string `json:"ip"` + SNMPPort int `json:"snmp_port"` +} + +// Device now has .ID, .CreatedAt, etc. promoted +d := Device{Name: "switch-01"} +d.ID = "dev_123" +``` + +Embed interfaces to signal partial implementation or to compose behaviors: + +```go +type ReadCloser struct { + io.Reader + io.Closer +} +``` + +**Watch out**: Embedding promotes all methods, including ones you don't want. If the embedded type has a `String()` method, your type now satisfies `fmt.Stringer` — maybe unintentionally. + +### Functional Options Pattern + +For constructors with many optional parameters: + +```go +type Server struct { + addr string + readTimeout time.Duration + writeTimeout time.Duration + maxConns int + logger *slog.Logger +} + +type Option func(*Server) + +func WithReadTimeout(d time.Duration) Option { + return func(s *Server) { s.readTimeout = d } +} + +func WithWriteTimeout(d time.Duration) Option { + return func(s *Server) { s.writeTimeout = d } +} + +func WithMaxConns(n int) Option { + return func(s *Server) { s.maxConns = n } +} + +func WithLogger(l *slog.Logger) Option { + return func(s *Server) { s.logger = l } +} + +func NewServer(addr string, opts ...Option) *Server { + s := &Server{ + addr: addr, + readTimeout: 5 * time.Second, // sensible defaults + writeTimeout: 10 * time.Second, + maxConns: 100, + logger: slog.Default(), + } + for _, opt := range opts { + opt(s) + } + return s +} + +// Usage: +srv := NewServer(":8080", + WithReadTimeout(10*time.Second), + WithLogger(myLogger), +) +``` + +This is clean, extensible, and self-documenting. Prefer it over config structs when options are genuinely optional. + +--- + +## 2. Concurrency + +### Goroutines + Channels + +#### Basic patterns + +```go +// Fire-and-forget (careful — no error handling) +go processEvent(event) + +// Channel for result +ch := make(chan Result, 1) +go func() { + ch <- expensiveComputation() +}() +result := <-ch +``` + +#### Always know how a goroutine ends + +Every goroutine you start should have a clear termination path. Leaked goroutines are memory leaks. + +```go +// Bad — goroutine leaks if ctx is never cancelled and ch is never read +go func() { + ch <- doWork(ctx) +}() + +// Good — select with context +go func() { + select { + case ch <- doWork(ctx): + case <-ctx.Done(): + } +}() +``` + +### sync.WaitGroup + +For waiting on a known set of goroutines: + +```go +func processAll(ctx context.Context, items []Item) error { + var wg sync.WaitGroup + errs := make(chan error, len(items)) + + for _, item := range items { + wg.Add(1) + go func() { + defer wg.Done() + if err := process(ctx, item); err != nil { + errs <- err + } + }() + } + + wg.Wait() + close(errs) + + for err := range errs { + return err // return first error + } + return nil +} +``` + +### Mutex vs Channels + +**Rule of thumb**: +- **Mutex**: Protecting shared state (maps, counters, caches) +- **Channels**: Communicating between goroutines, coordinating work + +```go +// Mutex — protecting a shared map +type SafeMap struct { + mu sync.RWMutex + m map[string]int +} + +func (s *SafeMap) Get(key string) (int, bool) { + s.mu.RLock() + defer s.mu.RUnlock() + v, ok := s.m[key] + return v, ok +} + +func (s *SafeMap) Set(key string, val int) { + s.mu.Lock() + defer s.mu.Unlock() + s.m[key] = val +} + +// Channel — pipeline stage +func square(in <-chan int) <-chan int { + out := make(chan int) + go func() { + defer close(out) + for n := range in { + out <- n * n + } + }() + return out +} +``` + +Use `sync.RWMutex` when reads vastly outnumber writes. + +### context.Context + +#### Cancellation + +```go +func main() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + go worker(ctx) + + // Cancel after signal + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) + <-sigCh + cancel() // all workers using ctx will see Done() +} + +func worker(ctx context.Context) { + for { + select { + case <-ctx.Done(): + log.Println("worker shutting down:", ctx.Err()) + return + default: + doWork() + } + } +} +``` + +#### Timeouts + +```go +ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) +defer cancel() // always defer cancel to release resources + +result, err := fetchFromAPI(ctx, url) +if errors.Is(err, context.DeadlineExceeded) { + log.Println("API call timed out") +} +``` + +#### Context values — use sparingly + +Only for request-scoped data that crosses API boundaries (trace IDs, auth tokens). Never for optional parameters. + +```go +type ctxKey string + +const requestIDKey ctxKey = "request_id" + +func WithRequestID(ctx context.Context, id string) context.Context { + return context.WithValue(ctx, requestIDKey, id) +} + +func RequestID(ctx context.Context) string { + id, _ := ctx.Value(requestIDKey).(string) + return id +} +``` + +### errgroup + +`golang.org/x/sync/errgroup` — the standard tool for concurrent work with error propagation: + +```go +import "golang.org/x/sync/errgroup" + +func fetchAll(ctx context.Context, urls []string) ([]Response, error) { + g, ctx := errgroup.WithContext(ctx) + responses := make([]Response, len(urls)) + + for i, url := range urls { + g.Go(func() error { + resp, err := fetch(ctx, url) + if err != nil { + return fmt.Errorf("fetch %s: %w", url, err) + } + responses[i] = resp // safe — each goroutine writes to unique index + return nil + }) + } + + if err := g.Wait(); err != nil { + return nil, err + } + return responses, nil +} +``` + +With concurrency limit: + +```go +g, ctx := errgroup.WithContext(ctx) +g.SetLimit(10) // max 10 concurrent goroutines +``` + +### Worker Pool + +```go +func workerPool(ctx context.Context, jobs <-chan Job, numWorkers int) <-chan Result { + results := make(chan Result, numWorkers) + var wg sync.WaitGroup + + for range numWorkers { + wg.Add(1) + go func() { + defer wg.Done() + for job := range jobs { + select { + case <-ctx.Done(): + return + case results <- process(job): + } + } + }() + } + + go func() { + wg.Wait() + close(results) + }() + + return results +} +``` + +### Fan-Out / Fan-In + +```go +// Fan-out: distribute work to multiple goroutines +func fanOut(ctx context.Context, input <-chan Task, workers int) []<-chan Result { + channels := make([]<-chan Result, workers) + for i := range workers { + channels[i] = worker(ctx, input) + } + return channels +} + +// Fan-in: merge multiple channels into one +func fanIn(ctx context.Context, channels ...<-chan Result) <-chan Result { + merged := make(chan Result) + var wg sync.WaitGroup + + for _, ch := range channels { + wg.Add(1) + go func() { + defer wg.Done() + for result := range ch { + select { + case merged <- result: + case <-ctx.Done(): + return + } + } + }() + } + + go func() { + wg.Wait() + close(merged) + }() + + return merged +} +``` + +--- + +## 3. Project Structure + +### Standard Layout + +``` +myproject/ +├── cmd/ +│ ├── server/ # main application entry point +│ │ └── main.go +│ └── cli/ # CLI tool +│ └── main.go +├── internal/ # private packages — can't be imported by other modules +│ ├── server/ # HTTP/gRPC server setup +│ ├── handler/ # request handlers +│ ├── service/ # business logic +│ ├── repo/ # data access +│ ├── model/ # domain types +│ └── config/ # configuration +├── proto/ # protobuf definitions +│ └── v1/ +├── migrations/ # database migrations +├── docs/ +├── scripts/ +├── Makefile +├── Dockerfile +├── go.mod +└── go.sum +``` + +#### The `/pkg` debate + +**Skip `/pkg` for most projects.** It was popularized by early Go projects but adds no real value — `internal/` enforces visibility; `pkg/` is just convention. If you need to export packages for other modules, put them at the root or in a clearly named directory. + +#### `internal/` is your friend + +Anything in `internal/` can't be imported by other Go modules. Use it aggressively — you can always promote later. + +### Dependency Injection with Wire + +[Wire](https://github.com/google/wire) generates DI code at compile time — no runtime reflection. + +```go +// internal/server/wire.go +//go:build wireinject + +package server + +import "github.com/google/wire" + +func InitializeApp(cfg *config.Config) (*App, error) { + wire.Build( + NewApp, + handler.NewDeviceHandler, + service.NewDeviceService, + repo.NewPostgresRepo, + db.NewPool, + ) + return nil, nil +} +``` + +Run `wire ./internal/server/` to generate the actual initialization code. + +**Alternative**: For simpler projects, manual DI in `main()` is fine. Don't use Wire until you have enough components to justify it. + +### Config Management + +#### `envconfig` — simple, env-var-based + +```go +import "github.com/kelseyhightower/envconfig" + +type Config struct { + Port int `envconfig:"PORT" default:"8080"` + DatabaseURL string `envconfig:"DATABASE_URL" required:"true"` + LogLevel string `envconfig:"LOG_LEVEL" default:"info"` + Timeout time.Duration `envconfig:"TIMEOUT" default:"30s"` +} + +func LoadConfig() (*Config, error) { + var cfg Config + if err := envconfig.Process("", &cfg); err != nil { + return nil, fmt.Errorf("load config: %w", err) + } + return &cfg, nil +} +``` + +#### Alternatives + +| Library | Best for | +|---------|----------| +| `envconfig` | Simple env-var config, 12-factor apps | +| `koanf` | Multi-source config (env, files, flags), lighter than Viper | +| `viper` | Kitchen-sink config (heavy, pulls in many deps) | +| `ff` | Flags-first config with env var fallback | + +**Recommendation**: Start with `envconfig`. Graduate to `koanf` if you need config files or multiple sources. Avoid Viper unless you're already using it. + +### Makefile Patterns + +```makefile +.PHONY: build test lint run clean proto generate + +# Variables +BINARY := myapp +GO := go +GOFLAGS := -trimpath +LDFLAGS := -s -w -X main.version=$(shell git describe --tags --always --dirty) +CGO_ENABLED := 0 + +## Build +build: + $(GO) build $(GOFLAGS) -ldflags "$(LDFLAGS)" -o bin/$(BINARY) ./cmd/server + +## Run locally +run: + $(GO) run ./cmd/server + +## Test +test: + $(GO) test -race -count=1 ./... + +test-integration: + $(GO) test -race -count=1 -tags=integration -timeout=5m ./... + +test-coverage: + $(GO) test -race -coverprofile=coverage.out ./... + $(GO) tool cover -html=coverage.out -o coverage.html + +## Lint +lint: + golangci-lint run ./... + +## Proto generation +proto: + buf generate + +## Clean +clean: + rm -rf bin/ coverage.out coverage.html + +## Generate (wire, mocks, etc.) +generate: + $(GO) generate ./... + +## All — CI pipeline +all: lint test build +``` + +--- + +## 4. Networking + +### net/http Server Best Practices + +#### Always set timeouts + +```go +srv := &http.Server{ + Addr: ":8080", + Handler: router, + ReadTimeout: 5 * time.Second, + WriteTimeout: 10 * time.Second, + IdleTimeout: 120 * time.Second, + // For request body size limits: + MaxHeaderBytes: 1 << 20, // 1 MB +} +``` + +Never use `http.ListenAndServe(":8080", nil)` in production — no timeouts = DDoS target. + +#### Graceful shutdown + +```go +func run(ctx context.Context) error { + srv := &http.Server{ + Addr: ":8080", + Handler: newRouter(), + ReadTimeout: 5 * time.Second, + WriteTimeout: 10 * time.Second, + } + + // Start server in goroutine + errCh := make(chan error, 1) + go func() { + if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed { + errCh <- err + } + }() + + // Wait for interrupt + select { + case err := <-errCh: + return fmt.Errorf("server error: %w", err) + case <-ctx.Done(): + } + + // Graceful shutdown with timeout + shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + + if err := srv.Shutdown(shutdownCtx); err != nil { + return fmt.Errorf("shutdown: %w", err) + } + return nil +} +``` + +#### Middleware chains + +```go +// Middleware type +type Middleware func(http.Handler) http.Handler + +// Chain composes middleware +func Chain(h http.Handler, mw ...Middleware) http.Handler { + for i := len(mw) - 1; i >= 0; i-- { + h = mw[i](h) + } + return h +} + +// Logging middleware +func LoggingMiddleware(logger *slog.Logger) Middleware { + return func(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + start := time.Now() + wrapped := &responseWriter{ResponseWriter: w, statusCode: 200} + next.ServeHTTP(wrapped, r) + logger.Info("request", + "method", r.Method, + "path", r.URL.Path, + "status", wrapped.statusCode, + "duration", time.Since(start), + ) + }) + } +} + +// Recovery middleware +func RecoveryMiddleware(logger *slog.Logger) Middleware { + return func(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + defer func() { + if err := recover(); err != nil { + logger.Error("panic recovered", + "error", err, + "stack", string(debug.Stack()), + ) + http.Error(w, "internal server error", http.StatusInternalServerError) + } + }() + next.ServeHTTP(w, r) + }) + } +} + +// Usage +handler := Chain(router, + RecoveryMiddleware(logger), + LoggingMiddleware(logger), + CORSMiddleware(), +) +``` + +#### responseWriter wrapper for capturing status + +```go +type responseWriter struct { + http.ResponseWriter + statusCode int +} + +func (rw *responseWriter) WriteHeader(code int) { + rw.statusCode = code + rw.ResponseWriter.WriteHeader(code) +} +``` + +### gRPC + Protobuf + +#### Proto file conventions + +```protobuf +syntax = "proto3"; + +package towerops.device.v1; + +option go_package = "github.com/towerops/towerops/gen/device/v1;devicev1"; + +service DeviceService { + rpc GetDevice(GetDeviceRequest) returns (GetDeviceResponse); + rpc ListDevices(ListDevicesRequest) returns (ListDevicesResponse); + rpc WatchDevices(WatchDevicesRequest) returns (stream DeviceEvent); +} + +message GetDeviceRequest { + string id = 1; +} + +message GetDeviceResponse { + Device device = 1; +} +``` + +#### Code generation with buf + +```yaml +# buf.gen.yaml +version: v2 +plugins: + - remote: buf.build/protocolbuffers/go + out: gen + opt: paths=source_relative + - remote: buf.build/grpc/go + out: gen + opt: paths=source_relative +``` + +#### gRPC interceptors (middleware equivalent) + +```go +import ( + "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// Unary interceptor for logging +func loggingInterceptor(logger *slog.Logger) grpc.UnaryServerInterceptor { + return func( + ctx context.Context, + req any, + info *grpc.UnaryServerInfo, + handler grpc.UnaryHandler, + ) (any, error) { + start := time.Now() + resp, err := handler(ctx, req) + logger.Info("grpc", + "method", info.FullMethod, + "duration", time.Since(start), + "error", err, + ) + return resp, err + } +} + +// Recovery interceptor +func recoveryInterceptor(logger *slog.Logger) grpc.UnaryServerInterceptor { + return func( + ctx context.Context, + req any, + info *grpc.UnaryServerInfo, + handler grpc.UnaryHandler, + ) (resp any, err error) { + defer func() { + if r := recover(); r != nil { + logger.Error("grpc panic", "error", r, "stack", string(debug.Stack())) + err = status.Errorf(codes.Internal, "internal error") + } + }() + return handler(ctx, req) + } +} + +// Server setup +srv := grpc.NewServer( + grpc.ChainUnaryInterceptor( + recoveryInterceptor(logger), + loggingInterceptor(logger), + ), + grpc.ChainStreamInterceptor( + // stream interceptors... + ), +) +``` + +#### Server-side streaming + +```go +func (s *server) WatchDevices( + req *pb.WatchDevicesRequest, + stream pb.DeviceService_WatchDevicesServer, +) error { + ctx := stream.Context() + ch := s.deviceEvents.Subscribe() + defer s.deviceEvents.Unsubscribe(ch) + + for { + select { + case <-ctx.Done(): + return ctx.Err() + case event := <-ch: + if err := stream.Send(event); err != nil { + return fmt.Errorf("send event: %w", err) + } + } + } +} +``` + +### HTTP Client Patterns + +#### Reuse clients — never create per-request + +```go +// Package-level or injected client with connection pooling +var httpClient = &http.Client{ + Timeout: 30 * time.Second, + Transport: &http.Transport{ + MaxIdleConns: 100, + MaxIdleConnsPerHost: 10, + IdleConnTimeout: 90 * time.Second, + TLSHandshakeTimeout: 10 * time.Second, + }, +} +``` + +#### Retry with exponential backoff + +```go +func doWithRetry(ctx context.Context, req *http.Request, maxRetries int) (*http.Response, error) { + var resp *http.Response + var err error + backoff := 100 * time.Millisecond + + for attempt := range maxRetries { + resp, err = httpClient.Do(req.Clone(ctx)) + if err != nil { + // Network error — retry + } else if resp.StatusCode < 500 && resp.StatusCode != 429 { + return resp, nil + } else { + resp.Body.Close() // must close before retry + } + + if attempt == maxRetries-1 { + break + } + + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-time.After(backoff): + } + backoff *= 2 // exponential + } + + if err != nil { + return nil, fmt.Errorf("after %d retries: %w", maxRetries, err) + } + return resp, nil +} +``` + +#### Always close response bodies + +```go +resp, err := httpClient.Do(req) +if err != nil { + return err +} +defer resp.Body.Close() + +// Drain body even if you don't need it — allows connection reuse +io.Copy(io.Discard, resp.Body) +``` + +--- + +## 5. Testing + +### Table-Driven Tests + +The Go standard for test organization: + +```go +func TestParsePort(t *testing.T) { + tests := []struct { + name string + input string + want int + wantErr bool + }{ + {name: "valid port", input: "8080", want: 8080}, + {name: "zero", input: "0", want: 0}, + {name: "max", input: "65535", want: 65535}, + {name: "negative", input: "-1", wantErr: true}, + {name: "too high", input: "65536", wantErr: true}, + {name: "not a number", input: "abc", wantErr: true}, + {name: "empty", input: "", wantErr: true}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := ParsePort(tt.input) + if tt.wantErr { + if err == nil { + t.Fatal("expected error, got nil") + } + return + } + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if got != tt.want { + t.Errorf("got %d, want %d", got, tt.want) + } + }) + } +} +``` + +### testify vs stdlib + +**stdlib** is fine for most cases. `t.Errorf` / `t.Fatalf` with clear messages work great. + +**testify** helps when assertions get complex: + +```go +import ( + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestDeviceService(t *testing.T) { + svc := NewDeviceService(mockRepo) + + device, err := svc.Get(ctx, "dev_123") + require.NoError(t, err) // stops test on failure + assert.Equal(t, "switch-01", device.Name) + assert.NotEmpty(t, device.ID) + assert.WithinDuration(t, time.Now(), device.CreatedAt, time.Minute) +} +``` + +`require` = fatal on failure (test stops). `assert` = log and continue. Use `require` for setup/preconditions, `assert` for actual checks. + +### httptest + +Test HTTP handlers without a real server: + +```go +func TestGetDevice(t *testing.T) { + // Setup + repo := &mockRepo{ + devices: map[string]*Device{ + "dev_1": {ID: "dev_1", Name: "switch-01"}, + }, + } + handler := NewDeviceHandler(repo) + + // Create request + req := httptest.NewRequest(http.MethodGet, "/devices/dev_1", nil) + rec := httptest.NewRecorder() + + // Serve + handler.ServeHTTP(rec, req) + + // Assert + assert.Equal(t, http.StatusOK, rec.Code) + + var got Device + err := json.NewDecoder(rec.Body).Decode(&got) + require.NoError(t, err) + assert.Equal(t, "switch-01", got.Name) +} +``` + +For full integration tests with routing: + +```go +func TestAPI(t *testing.T) { + srv := httptest.NewServer(newRouter()) + defer srv.Close() + + resp, err := http.Get(srv.URL + "/health") + require.NoError(t, err) + defer resp.Body.Close() + assert.Equal(t, 200, resp.StatusCode) +} +``` + +### Mocking with Interfaces (not frameworks) + +Define the interface where it's consumed, implement a mock manually: + +```go +// In handler package +type DeviceRepo interface { + Get(ctx context.Context, id string) (*model.Device, error) + List(ctx context.Context) ([]*model.Device, error) +} + +// In handler test file +type mockRepo struct { + devices map[string]*model.Device + err error +} + +func (m *mockRepo) Get(ctx context.Context, id string) (*model.Device, error) { + if m.err != nil { + return nil, m.err + } + d, ok := m.devices[id] + if !ok { + return nil, repo.ErrNotFound + } + return d, nil +} + +func (m *mockRepo) List(ctx context.Context) ([]*model.Device, error) { + if m.err != nil { + return nil, m.err + } + var result []*model.Device + for _, d := range m.devices { + result = append(result, d) + } + return result, nil +} +``` + +This is simpler, more readable, and more maintainable than mock frameworks like gomock. For large interfaces, consider `moq` or `mockery` for code generation — but keep mocks simple. + +### Integration Tests with testcontainers-go + +```go +//go:build integration + +package repo_test + +import ( + "context" + "testing" + + "github.com/testcontainers/testcontainers-go" + "github.com/testcontainers/testcontainers-go/modules/postgres" + "github.com/testcontainers/testcontainers-go/wait" +) + +func TestPostgresRepo(t *testing.T) { + ctx := context.Background() + + pgContainer, err := postgres.Run(ctx, + "postgres:16-alpine", + postgres.WithDatabase("testdb"), + postgres.WithUsername("test"), + postgres.WithPassword("test"), + testcontainers.WithWaitStrategy( + wait.ForLog("database system is ready to accept connections"). + WithOccurrence(2), + ), + ) + require.NoError(t, err) + t.Cleanup(func() { pgContainer.Terminate(ctx) }) + + connStr, err := pgContainer.ConnectionString(ctx, "sslmode=disable") + require.NoError(t, err) + + // Run migrations + runMigrations(t, connStr) + + // Create repo with real DB + r, err := repo.NewPostgresRepo(ctx, connStr) + require.NoError(t, err) + + // Test actual queries + t.Run("create and get", func(t *testing.T) { + device := &model.Device{Name: "test-switch"} + err := r.Create(ctx, device) + require.NoError(t, err) + assert.NotEmpty(t, device.ID) + + got, err := r.Get(ctx, device.ID) + require.NoError(t, err) + assert.Equal(t, "test-switch", got.Name) + }) +} +``` + +### go test Flags + +```bash +# Race detector — always use in CI +go test -race ./... + +# Disable test caching (important for flaky test detection) +go test -count=1 ./... + +# Run tests in parallel (default is GOMAXPROCS) +go test -parallel=4 ./... + +# Only run matching tests +go test -run TestGetDevice ./internal/handler/ + +# Verbose output +go test -v ./... + +# Short mode (skip slow tests) +go test -short ./... + +# Timeout +go test -timeout=5m ./... + +# Integration tests only (with build tag) +go test -tags=integration ./... + +# Combined for CI: +go test -race -count=1 -timeout=10m ./... +``` + +In test code, respect `-short`: + +```go +func TestSlowIntegration(t *testing.T) { + if testing.Short() { + t.Skip("skipping in short mode") + } + // slow test... +} +``` + +--- + +## 6. Performance + +### pprof Profiling + +#### Enable in HTTP server + +```go +import _ "net/http/pprof" + +// If using default mux, pprof is auto-registered. +// For custom mux, register explicitly: +mux.HandleFunc("/debug/pprof/", pprof.Index) +mux.HandleFunc("/debug/pprof/profile", pprof.Profile) +mux.HandleFunc("/debug/pprof/heap", pprof.Handler("heap").ServeHTTP) +mux.HandleFunc("/debug/pprof/goroutine", pprof.Handler("goroutine").ServeHTTP) +``` + +#### Capture and analyze + +```bash +# CPU profile (30 seconds) +go tool pprof http://localhost:8080/debug/pprof/profile?seconds=30 + +# Heap profile +go tool pprof http://localhost:8080/debug/pprof/heap + +# Goroutine dump +curl http://localhost:8080/debug/pprof/goroutine?debug=2 + +# Interactive commands in pprof: +# top20 — top 20 functions by CPU/memory +# list funcName — show annotated source +# web — open flamegraph in browser +``` + +### Benchmarking + +```go +func BenchmarkProcessEvent(b *testing.B) { + event := createTestEvent() + b.ResetTimer() + + for b.Loop() { + processEvent(event) + } +} + +// With sub-benchmarks for different sizes +func BenchmarkParse(b *testing.B) { + sizes := []int{10, 100, 1000, 10000} + for _, size := range sizes { + b.Run(fmt.Sprintf("size=%d", size), func(b *testing.B) { + data := generateData(size) + b.ResetTimer() + for b.Loop() { + parse(data) + } + }) + } +} + +// Report allocations +func BenchmarkBuildString(b *testing.B) { + b.ReportAllocs() + for b.Loop() { + buildString(1000) + } +} +``` + +```bash +# Run benchmarks +go test -bench=. -benchmem ./... + +# Compare benchmarks (install benchstat) +go test -bench=. -count=10 > old.txt +# ... make changes ... +go test -bench=. -count=10 > new.txt +benchstat old.txt new.txt +``` + +### Memory Allocation Patterns + +#### sync.Pool — reuse temporary objects + +```go +var bufPool = sync.Pool{ + New: func() any { + return new(bytes.Buffer) + }, +} + +func processRequest(data []byte) string { + buf := bufPool.Get().(*bytes.Buffer) + defer func() { + buf.Reset() + bufPool.Put(buf) + }() + + buf.Write(data) + // ... process ... + return buf.String() +} +``` + +#### Pre-allocation + +```go +// Bad — grows the slice repeatedly +var result []string +for _, item := range items { + result = append(result, item.Name) +} + +// Good — allocate once +result := make([]string, 0, len(items)) +for _, item := range items { + result = append(result, item.Name) +} + +// Same for maps +m := make(map[string]int, len(items)) +``` + +### Escape Analysis + +Check what escapes to the heap: + +```bash +go build -gcflags='-m -m' ./... 2>&1 | grep 'escapes to heap' +``` + +Tips to reduce heap allocations: +- Return values instead of pointers when the struct is small +- Use value receivers for small types +- Avoid closures that capture variables (they often escape) +- Pre-size slices and maps + +```go +// This might stay on stack (small struct, returned by value) +func newPoint(x, y int) Point { + return Point{X: x, Y: y} +} + +// This always escapes to heap (pointer returned) +func newPoint(x, y int) *Point { + return &Point{X: x, Y: y} +} +``` + +### String Building + +```go +// Bad — O(n²) concatenation +s := "" +for _, item := range items { + s += item.Name + "," +} + +// Good — strings.Builder +var b strings.Builder +b.Grow(len(items) * 20) // estimate capacity +for i, item := range items { + if i > 0 { + b.WriteByte(',') + } + b.WriteString(item.Name) +} +result := b.String() + +// For simple joins +result := strings.Join(names, ",") +``` + +--- + +## 7. Observability + +### Structured Logging with slog (Go 1.21+) + +```go +import "log/slog" + +// Setup +func setupLogger(level string) *slog.Logger { + var lvl slog.Level + switch level { + case "debug": + lvl = slog.LevelDebug + case "warn": + lvl = slog.LevelWarn + case "error": + lvl = slog.LevelError + default: + lvl = slog.LevelInfo + } + + return slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{ + Level: lvl, + AddSource: true, // adds file:line to log entries + })) +} + +// Usage +logger := setupLogger("info") + +logger.Info("server started", "addr", ":8080", "version", version) + +logger.Error("query failed", + "error", err, + "query", query, + "duration", time.Since(start), +) + +// With context (carries request ID, trace ID, etc.) +logger.InfoContext(ctx, "device updated", + "device_id", device.ID, + "changes", changes, +) + +// Child logger with common fields +reqLogger := logger.With( + "request_id", requestID, + "method", r.Method, + "path", r.URL.Path, +) +``` + +#### slog groups for structured fields + +```go +logger.Info("request", + slog.Group("http", + slog.String("method", r.Method), + slog.String("path", r.URL.Path), + slog.Int("status", status), + ), + slog.Group("timing", + slog.Duration("total", totalDuration), + slog.Duration("db", dbDuration), + ), +) +// Output: {"http":{"method":"GET","path":"/devices","status":200},"timing":{"total":"12ms","db":"3ms"}} +``` + +### OpenTelemetry + +#### Tracing setup + +```go +import ( + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" + "go.opentelemetry.io/otel/sdk/resource" + sdktrace "go.opentelemetry.io/otel/sdk/trace" + semconv "go.opentelemetry.io/otel/semconv/v1.24.0" +) + +func initTracer(ctx context.Context, serviceName, version string) (func(), error) { + exporter, err := otlptracegrpc.New(ctx) + if err != nil { + return nil, fmt.Errorf("create exporter: %w", err) + } + + res, err := resource.New(ctx, + resource.WithAttributes( + semconv.ServiceNameKey.String(serviceName), + semconv.ServiceVersionKey.String(version), + ), + ) + if err != nil { + return nil, fmt.Errorf("create resource: %w", err) + } + + tp := sdktrace.NewTracerProvider( + sdktrace.WithBatcher(exporter), + sdktrace.WithResource(res), + sdktrace.WithSampler(sdktrace.AlwaysSample()), // tune in prod + ) + otel.SetTracerProvider(tp) + + cleanup := func() { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + tp.Shutdown(ctx) + } + return cleanup, nil +} +``` + +#### Using traces + +```go +var tracer = otel.Tracer("myapp/service") + +func (s *DeviceService) GetDevice(ctx context.Context, id string) (*Device, error) { + ctx, span := tracer.Start(ctx, "DeviceService.GetDevice", + trace.WithAttributes(attribute.String("device.id", id)), + ) + defer span.End() + + device, err := s.repo.Get(ctx, id) + if err != nil { + span.RecordError(err) + span.SetStatus(codes.Error, err.Error()) + return nil, err + } + + return device, nil +} +``` + +### Prometheus Metrics + +```go +import ( + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/promauto" + "github.com/prometheus/client_golang/prometheus/promhttp" +) + +var ( + httpRequestsTotal = promauto.NewCounterVec( + prometheus.CounterOpts{ + Name: "http_requests_total", + Help: "Total HTTP requests", + }, + []string{"method", "path", "status"}, + ) + + httpRequestDuration = promauto.NewHistogramVec( + prometheus.HistogramOpts{ + Name: "http_request_duration_seconds", + Help: "HTTP request duration in seconds", + Buckets: prometheus.DefBuckets, + }, + []string{"method", "path"}, + ) + + activeConnections = promauto.NewGauge( + prometheus.GaugeOpts{ + Name: "active_connections", + Help: "Number of active connections", + }, + ) +) + +// Metrics middleware +func MetricsMiddleware(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + start := time.Now() + wrapped := &responseWriter{ResponseWriter: w, statusCode: 200} + + next.ServeHTTP(wrapped, r) + + duration := time.Since(start).Seconds() + httpRequestsTotal.WithLabelValues(r.Method, r.URL.Path, strconv.Itoa(wrapped.statusCode)).Inc() + httpRequestDuration.WithLabelValues(r.Method, r.URL.Path).Observe(duration) + }) +} + +// Expose metrics endpoint +mux.Handle("/metrics", promhttp.Handler()) +``` + +### Health Checks + +```go +type HealthChecker struct { + db *sql.DB + redis *redis.Client +} + +type HealthStatus struct { + Status string `json:"status"` + Checks map[string]string `json:"checks"` +} + +// Liveness — is the process alive? Keep it simple. +func (h *HealthChecker) Liveness(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + w.Write([]byte(`{"status":"ok"}`)) +} + +// Readiness — can the process handle traffic? +func (h *HealthChecker) Readiness(w http.ResponseWriter, r *http.Request) { + ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second) + defer cancel() + + status := HealthStatus{ + Status: "ok", + Checks: make(map[string]string), + } + + if err := h.db.PingContext(ctx); err != nil { + status.Status = "degraded" + status.Checks["database"] = err.Error() + } else { + status.Checks["database"] = "ok" + } + + if err := h.redis.Ping(ctx).Err(); err != nil { + status.Status = "degraded" + status.Checks["redis"] = err.Error() + } else { + status.Checks["redis"] = "ok" + } + + code := http.StatusOK + if status.Status != "ok" { + code = http.StatusServiceUnavailable + } + + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(code) + json.NewEncoder(w).Encode(status) +} + +// Register +mux.HandleFunc("GET /healthz", health.Liveness) +mux.HandleFunc("GET /readyz", health.Readiness) +``` + +--- + +## 8. Build & Deploy + +### Multi-Stage Docker Builds + +```dockerfile +# Stage 1: Build +FROM golang:1.24-alpine AS builder + +WORKDIR /app + +# Cache dependencies +COPY go.mod go.sum ./ +RUN go mod download + +# Copy source and build +COPY . . +RUN CGO_ENABLED=0 GOOS=linux go build \ + -trimpath \ + -ldflags="-s -w -X main.version=$(cat VERSION)" \ + -o /app/server \ + ./cmd/server + +# Stage 2: Runtime +FROM gcr.io/distroless/static-debian12:nonroot + +COPY --from=builder /app/server /server + +EXPOSE 8080 +USER nonroot:nonroot +ENTRYPOINT ["/server"] +``` + +**Key points**: +- `CGO_ENABLED=0` for static binary (no libc dependency) +- `-trimpath` removes local paths from binary +- `-ldflags="-s -w"` strips debug info (smaller binary) +- `distroless` or `scratch` for minimal attack surface +- `nonroot` user for security + +### CGO Considerations + +**Default: Avoid CGO.** `CGO_ENABLED=0` gives you a static binary that runs anywhere. + +When you need CGO (SQLite, certain crypto libs): + +```dockerfile +FROM golang:1.24-alpine AS builder +RUN apk add --no-cache gcc musl-dev +# CGO_ENABLED=1 is default when gcc is available +RUN go build -o /app/server ./cmd/server + +FROM alpine:3.20 +RUN apk add --no-cache ca-certificates +COPY --from=builder /app/server /server +ENTRYPOINT ["/server"] +``` + +### Cross-Compilation + +```bash +# Build for Linux AMD64 +GOOS=linux GOARCH=amd64 go build -o myapp-linux-amd64 ./cmd/server + +# Build for Linux ARM64 +GOOS=linux GOARCH=arm64 go build -o myapp-linux-arm64 ./cmd/server + +# Build for macOS ARM64 (Apple Silicon) +GOOS=darwin GOARCH=arm64 go build -o myapp-darwin-arm64 ./cmd/server + +# Build for Windows +GOOS=windows GOARCH=amd64 go build -o myapp-windows-amd64.exe ./cmd/server +``` + +Cross-compilation only works cleanly with `CGO_ENABLED=0`. If you need CGO + cross-compile, use Docker or [zig as CC](https://dev.to/kristoff/zig-makes-go-cross-compilation-just-work-29ho). + +### GoReleaser + +```yaml +# .goreleaser.yaml +version: 2 + +builds: + - id: server + main: ./cmd/server + binary: myapp + env: + - CGO_ENABLED=0 + goos: + - linux + - darwin + goarch: + - amd64 + - arm64 + ldflags: + - -s -w + - -X main.version={{.Version}} + - -X main.commit={{.Commit}} + - -X main.date={{.Date}} + +archives: + - format: tar.gz + name_template: "{{ .ProjectName }}_{{ .Version }}_{{ .Os }}_{{ .Arch }}" + +dockers: + - image_templates: + - "ghcr.io/myorg/myapp:{{ .Version }}-amd64" + dockerfile: Dockerfile + use: buildx + build_flag_templates: + - "--platform=linux/amd64" + +changelog: + sort: asc + filters: + exclude: + - "^docs:" + - "^test:" + - "^ci:" +``` + +```bash +# Test locally +goreleaser release --snapshot --clean + +# Release (triggered by git tag) +git tag v1.0.0 +git push origin v1.0.0 +# CI runs: goreleaser release +``` + +### Linting with golangci-lint + +```yaml +# .golangci.yml +run: + timeout: 5m + +linters: + enable: + - errcheck # unchecked errors + - govet # vet checks + - staticcheck # comprehensive static analysis + - unused # unused code + - gosimple # simplifications + - ineffassign # useless assignments + - typecheck # type checking + - gocritic # opinionated checks + - revive # flexible linter (replaces golint) + - misspell # spelling + - prealloc # pre-allocation suggestions + - unconvert # unnecessary conversions + - noctx # HTTP requests without context + - bodyclose # unclosed HTTP response bodies + - errname # error naming conventions + - errorlint # error wrapping checks + - exhaustive # exhaustive enum switches + - copyloopvar # loop variable capture bugs (Go <1.22) + +linters-settings: + gocritic: + enabled-tags: + - diagnostic + - performance + disabled-checks: + - ifElseChain + revive: + rules: + - name: exported + arguments: [checkPrivateReceivers] + - name: blank-imports + - name: context-as-argument + - name: error-return + - name: error-naming + - name: increment-decrement + - name: var-naming + govet: + enable-all: true + +issues: + exclude-rules: + - path: _test\.go + linters: + - errcheck + - gocritic + max-issues-per-linter: 0 + max-same-issues: 0 +``` + +```bash +# Run +golangci-lint run ./... + +# Fix auto-fixable issues +golangci-lint run --fix ./... + +# Only new issues (great for CI on PRs) +golangci-lint run --new-from-rev=main ./... +``` + +--- + +## Quick Reference: Go Proverbs + +- Don't communicate by sharing memory; share memory by communicating. +- Concurrency is not parallelism. +- The bigger the interface, the weaker the abstraction. +- Make the zero value useful. +- `interface{}` says nothing (use generics or concrete types). +- A little copying is better than a little dependency. +- Clear is better than clever. +- Errors are values — program with them. +- Don't just check errors, handle them gracefully. + +## Further Reading + +- [Effective Go](https://go.dev/doc/effective_go) +- [Go Code Review Comments](https://go.dev/wiki/CodeReviewComments) +- [Go Proverbs](https://go-proverbs.github.io/) +- [Uber Go Style Guide](https://github.com/uber-go/guide/blob/master/style.md) +- [100 Go Mistakes](https://100go.co/) +- [Go Blog](https://go.dev/blog/) diff --git a/docs/references/sre-devops-best-practices.md b/docs/references/sre-devops-best-practices.md new file mode 100644 index 00000000..191d654b --- /dev/null +++ b/docs/references/sre-devops-best-practices.md @@ -0,0 +1,2758 @@ +# SRE/DevOps Best Practices Reference + +> Practical patterns, real configs, and battle-tested commands. Not academic theory. + +--- + +## Table of Contents + +1. [Reliability Engineering](#1-reliability-engineering) +2. [Observability](#2-observability) +3. [Infrastructure as Code](#3-infrastructure-as-code) +4. [Kubernetes](#4-kubernetes) +5. [CI/CD](#5-cicd) +6. [Networking & Security](#6-networking--security) +7. [Database Operations](#7-database-operations) +8. [On-Call & Alerting](#8-on-call--alerting) + +--- + +## 1. Reliability Engineering + +### SLIs, SLOs, and SLAs + +**SLI (Service Level Indicator)** — A quantitative measure of some aspect of service. +**SLO (Service Level Objective)** — A target value or range for an SLI. +**SLA (Service Level Agreement)** — A contract with consequences if the SLO is missed. + +Rule of thumb: SLA threshold < SLO threshold (always have internal margin). + +#### Common SLIs + +| Service Type | SLI | How to Measure | +|---|---|---| +| HTTP API | Availability | `successful_requests / total_requests` | +| HTTP API | Latency | `requests < 300ms / total_requests` | +| Data pipeline | Freshness | `time_since_last_successful_run` | +| Storage | Durability | `intact_objects / total_objects` | +| Batch job | Throughput | `records_processed / expected_records` | + +#### Defining SLOs — Concrete Example + +```yaml +# slo.yaml — Structured SLO definition +service: towerops-api +slos: + - name: availability + sli: "ratio of HTTP requests with status < 500" + target: 99.9% # 43.8 min/month downtime budget + window: 30d rolling + + - name: latency-p99 + sli: "99th percentile response time" + target: 500ms + window: 30d rolling + + - name: latency-p50 + sli: "50th percentile response time" + target: 100ms + window: 30d rolling +``` + +#### Error Budgets + +Error budget = `1 - SLO`. For a 99.9% availability SLO over 30 days: + +``` +Budget = 0.1% × 30 days × 24 hours × 60 minutes = 43.2 minutes of downtime +``` + +**Error budget policy:** + +| Budget Remaining | Action | +|---|---| +| > 50% | Ship features freely | +| 20–50% | Require extra review, feature flag new code | +| 5–20% | Freeze non-critical changes, focus on reliability | +| < 5% | Full freeze — only reliability work ships | + +PromQL to track error budget consumption: + +```promql +# Error budget remaining (0 = exhausted, 1 = full) +1 - ( + ( + sum(rate(http_requests_total{status=~"5.."}[30d])) + / + sum(rate(http_requests_total[30d])) + ) / (1 - 0.999) +) +``` + +### Toil Reduction + +**Toil** = manual, repetitive, automatable, tactical, devoid of long-term value, scales linearly with service growth. + +#### Identifying Toil + +Track it explicitly. Every on-call rotation, log toil: + +```markdown +## Toil Log — Week of 2026-03-09 + +| Task | Time Spent | Frequency | Automatable? | Ticket | +|---|---|---|---|---| +| Restart stuck workers | 15 min | 3x/week | Yes — add health check + auto-restart | TOIL-042 | +| Manually rotate certs | 30 min | Monthly | Yes — cert-manager | TOIL-043 | +| Resize PVCs | 20 min | 2x/month | Yes — auto-expand policy | TOIL-044 | +``` + +**Target: Keep toil below 50% of on-call time.** If it's above that, you're not doing engineering — you're doing ops. + +#### Automation Priority Matrix + +``` +High frequency + Easy to automate → DO IT NOW +High frequency + Hard to automate → Invest time, high ROI +Low frequency + Easy to automate → Quick win, do it +Low frequency + Hard to automate → Probably not worth it +``` + +### Incident Response + +#### Severity Levels + +| Severity | Criteria | Response Time | Examples | +|---|---|---|---| +| **SEV-1** | Complete outage, data loss risk, security breach | 15 min | API down, DB corruption, credentials leaked | +| **SEV-2** | Major feature degraded, significant user impact | 30 min | Auth broken, payments failing, 50%+ error rate | +| **SEV-3** | Minor feature degraded, workaround exists | 4 hours | Slow queries, non-critical service down | +| **SEV-4** | Cosmetic, minimal impact | Next business day | Dashboard glitch, log noise | + +#### Incident Response Process + +``` +1. DETECT → Alert fires or user reports +2. TRIAGE → Assign severity, page on-call if needed +3. MITIGATE → Stop the bleeding (rollback, scale, redirect) +4. DIAGNOSE → Find root cause (but mitigation comes first!) +5. RESOLVE → Fix the underlying issue +6. FOLLOWUP → Postmortem within 48 hours +``` + +**Key principle: Mitigate first, diagnose second.** Rolling back a deploy takes 2 minutes. Finding the bug in the deploy might take 2 hours. Your users don't care about root cause during an outage. + +#### Incident Commander Checklist + +```markdown +## Incident Commander Checklist + +- [ ] Declare incident and severity in #incidents channel +- [ ] Assign roles: IC (you), Communications Lead, Ops Lead +- [ ] Start incident doc (copy from template) +- [ ] Set up war room (video call link in channel) +- [ ] Post status update every 15 minutes (SEV-1) or 30 minutes (SEV-2) +- [ ] Communicate to stakeholders via status page +- [ ] When resolved: announce resolution, schedule postmortem +``` + +#### Runbook Template + +See [Section 8](#runbook-template) for the full runbook template. + +### Capacity Planning + +#### The Four Signals to Monitor + +```promql +# 1. Utilization — How full is the resource? +avg(node_cpu_seconds_total{mode!="idle"}) by (instance) +sum(container_memory_working_set_bytes) / sum(machine_memory_bytes) + +# 2. Saturation — How queued/overloaded is it? +rate(node_disk_io_time_weighted_seconds_total[5m]) +avg_over_time(node_load15[1h]) / count(node_cpu_seconds_total{mode="idle"}) by (instance) + +# 3. Rate of growth +predict_linear(node_filesystem_avail_bytes[7d], 30*24*3600) # 30-day projection + +# 4. Headroom — How much buffer remains? +(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes +``` + +#### Capacity Planning Process + +```bash +# Export resource usage trends for capacity review +# Run monthly, review in capacity planning meeting + +# CPU: average and peak over past 30 days +curl -s "http://prometheus:9090/api/v1/query?query=\ + quantile_over_time(0.95, instance:node_cpu_utilisation:rate5m[30d])" | jq '.data.result' + +# Memory: project when we hit 80% +curl -s "http://prometheus:9090/api/v1/query?query=\ + predict_linear(node_memory_MemAvailable_bytes[30d], 90*24*3600)" | jq '.data.result' + +# Disk: days until full +curl -s "http://prometheus:9090/api/v1/query?query=\ + node_filesystem_avail_bytes / ignoring(mountpoint) \ + rate(node_filesystem_avail_bytes[7d]) / -86400" | jq '.data.result' +``` + +**Rule of thumb:** Plan for 2x current peak. If you're above 60% sustained utilization, start provisioning. + +--- + +## 2. Observability + +The three pillars: **Metrics**, **Logs**, **Traces**. Each answers different questions: + +- **Metrics** → "What is broken?" (symptoms) +- **Logs** → "Why is it broken?" (details) +- **Traces** → "Where is it broken?" (request path) + +### Prometheus + Grafana + +#### Prometheus Configuration + +```yaml +# prometheus.yml +global: + scrape_interval: 15s + evaluation_interval: 15s + external_labels: + cluster: production + environment: prod + +rule_files: + - "/etc/prometheus/rules/*.yml" + +alerting: + alertmanagers: + - static_configs: + - targets: ["alertmanager:9093"] + +scrape_configs: + - job_name: "kubernetes-pods" + kubernetes_sd_configs: + - role: pod + relabel_configs: + # Only scrape pods with annotation prometheus.io/scrape: "true" + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] + action: keep + regex: true + # Use custom port if annotated + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port] + action: replace + target_label: __address__ + regex: (.+) + replacement: ${1}:${2} + # Use custom path if annotated + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] + action: replace + target_label: __metrics_path__ + regex: (.+) + + - job_name: "node-exporter" + static_configs: + - targets: ["node-exporter:9100"] + + - job_name: "postgres" + static_configs: + - targets: ["postgres-exporter:9187"] +``` + +#### Essential PromQL Queries + +```promql +# --- HTTP Request Rate --- +# Requests per second by status code +sum(rate(http_requests_total[5m])) by (status_code) + +# Error rate (5xx / total) +sum(rate(http_requests_total{status_code=~"5.."}[5m])) +/ +sum(rate(http_requests_total[5m])) + +# --- Latency --- +# P50/P95/P99 from histogram +histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) +histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) +histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) + +# Apdex score (satisfied < 0.3s, tolerating < 1.2s) +( + sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m])) + + sum(rate(http_request_duration_seconds_bucket{le="1.2"}[5m])) +) / 2 / sum(rate(http_request_duration_seconds_count[5m])) + +# --- Saturation --- +# CPU saturation (load average > CPU count) +node_load1 / count without(cpu, mode) (node_cpu_seconds_total{mode="idle"}) + +# Memory pressure +1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) + +# Disk I/O saturation +rate(node_disk_io_time_weighted_seconds_total[5m]) + +# --- Kubernetes-specific --- +# Pod restart rate (crash loops) +rate(kube_pod_container_status_restarts_total[15m]) > 0 + +# Container OOMKills +kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} + +# HPA at max replicas (can't scale further) +kube_horizontalpodautoscaler_status_current_replicas + == kube_horizontalpodautoscaler_spec_max_replicas + +# Pending pods (scheduling issues) +kube_pod_status_phase{phase="Pending"} > 0 + +# PVC usage approaching limits +kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.85 + +# --- PostgreSQL --- +# Active connections vs max +pg_stat_activity_count / pg_settings_max_connections + +# Slow queries (from pg_stat_statements exporter) +topk(10, pg_stat_statements_mean_time_seconds) + +# Replication lag +pg_replication_lag_seconds > 30 + +# Cache hit ratio (should be > 99%) +pg_stat_database_blks_hit / (pg_stat_database_blks_hit + pg_stat_database_blks_read) +``` + +#### Alert Rules + +```yaml +# alerts.yml +groups: + - name: slo-alerts + rules: + # Multi-window multi-burn-rate alert (Google SRE book pattern) + # Fast burn: 14.4x burn rate over 1h (uses 2% of budget in 1h) + - alert: HighErrorBudgetBurn_Fast + expr: | + ( + sum(rate(http_requests_total{status_code=~"5.."}[1h])) + / sum(rate(http_requests_total[1h])) + ) > (14.4 * 0.001) + and + ( + sum(rate(http_requests_total{status_code=~"5.."}[5m])) + / sum(rate(http_requests_total[5m])) + ) > (14.4 * 0.001) + for: 2m + labels: + severity: critical + annotations: + summary: "High error budget burn rate" + description: "Burning error budget at 14.4x. {{ $value | humanizePercentage }} error rate over 1h." + runbook_url: "https://wiki.internal/runbooks/high-error-rate" + + # Slow burn: 1x burn rate over 3d (steady degradation) + - alert: HighErrorBudgetBurn_Slow + expr: | + ( + sum(rate(http_requests_total{status_code=~"5.."}[3d])) + / sum(rate(http_requests_total[3d])) + ) > (1 * 0.001) + and + ( + sum(rate(http_requests_total{status_code=~"5.."}[6h])) + / sum(rate(http_requests_total[6h])) + ) > (1 * 0.001) + for: 1h + labels: + severity: warning + annotations: + summary: "Slow error budget burn detected" + description: "Steady error rate consuming budget. Review recent changes." + + - name: infrastructure + rules: + - alert: NodeHighCPU + expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85 + for: 15m + labels: + severity: warning + annotations: + summary: "High CPU on {{ $labels.instance }}" + description: "CPU usage above 85% for 15 minutes. Current: {{ $value }}%" + + - alert: NodeHighMemory + expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90 + for: 10m + labels: + severity: warning + annotations: + summary: "High memory on {{ $labels.instance }}" + description: "Memory usage above 90%. Current: {{ $value }}%" + + - alert: DiskWillFillIn24h + expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.lxcfs"}[6h], 24*3600) < 0 + for: 30m + labels: + severity: warning + annotations: + summary: "Disk will fill within 24h on {{ $labels.instance }}" + description: "Filesystem {{ $labels.mountpoint }} projected to run out of space." + + - alert: PodCrashLooping + expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 3 + for: 5m + labels: + severity: warning + annotations: + summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} crash looping" + description: "Container {{ $labels.container }} restarted {{ $value }} times in 15m." + + - name: postgres + rules: + - alert: PostgresHighConnections + expr: pg_stat_activity_count / pg_settings_max_connections > 0.8 + for: 5m + labels: + severity: warning + annotations: + summary: "PostgreSQL connections at {{ $value | humanizePercentage }}" + + - alert: PostgresReplicationLag + expr: pg_replication_lag_seconds > 60 + for: 5m + labels: + severity: critical + annotations: + summary: "Replication lag {{ $value }}s on {{ $labels.instance }}" + + - alert: PostgresLowCacheHitRatio + expr: | + pg_stat_database_blks_hit{datname!~"template.*"} + / (pg_stat_database_blks_hit{datname!~"template.*"} + pg_stat_database_blks_read{datname!~"template.*"}) + < 0.99 + for: 15m + labels: + severity: warning + annotations: + summary: "Low cache hit ratio on {{ $labels.datname }}: {{ $value }}" +``` + +#### Alertmanager Configuration + +```yaml +# alertmanager.yml +global: + resolve_timeout: 5m + slack_api_url: "https://hooks.slack.com/services/T00/B00/XXXX" + +route: + receiver: "default" + group_by: ["alertname", "namespace"] + group_wait: 30s # Wait to batch related alerts + group_interval: 5m # Wait before sending updated group + repeat_interval: 4h # Re-notify interval + routes: + - match: + severity: critical + receiver: "pagerduty-critical" + group_wait: 10s + repeat_interval: 1h + - match: + severity: warning + receiver: "slack-warnings" + repeat_interval: 8h + +receivers: + - name: "default" + slack_configs: + - channel: "#alerts" + title: '{{ .GroupLabels.alertname }}' + text: >- + {{ range .Alerts }} + *{{ .Labels.severity | toUpper }}*: {{ .Annotations.summary }} + {{ .Annotations.description }} + {{ end }} + + - name: "pagerduty-critical" + pagerduty_configs: + - service_key: "" + description: '{{ .GroupLabels.alertname }}: {{ (index .Alerts 0).Annotations.summary }}' + severity: critical + + - name: "slack-warnings" + slack_configs: + - channel: "#alerts-warnings" + title: '⚠️ {{ .GroupLabels.alertname }}' + text: '{{ (index .Alerts 0).Annotations.description }}' + +inhibit_rules: + # Don't alert on warning if critical is already firing + - source_match: + severity: critical + target_match: + severity: warning + equal: ["alertname", "namespace"] +``` + +### Structured Logging + +**Stop logging unstructured strings.** Use JSON with consistent fields: + +```json +{ + "timestamp": "2026-03-13T21:27:00.000Z", + "level": "error", + "message": "Failed to process payment", + "service": "billing-api", + "trace_id": "abc123def456", + "span_id": "span789", + "request_id": "req-uuid-here", + "user_id": "usr_12345", + "error": { + "type": "PaymentGatewayTimeout", + "message": "Connection timed out after 30s", + "stack": "..." + }, + "context": { + "payment_amount": 49.99, + "currency": "USD", + "gateway": "stripe", + "retry_count": 2 + } +} +``` + +#### Required Log Fields + +Every log line should have at minimum: + +``` +timestamp — ISO 8601 (with timezone) +level — debug, info, warn, error, fatal +message — Human-readable description +service — Which service produced this +trace_id — For correlation across services +``` + +#### Elixir Structured Logging Example + +```elixir +# config/config.exs +config :logger, :default_formatter, + format: {LoggerJSON.Formatters.Basic, :format}, + metadata: [:request_id, :trace_id, :span_id, :user_id] + +# In application code +Logger.error("Payment failed", + user_id: user.id, + payment_id: payment.id, + error: inspect(reason), + gateway: "stripe", + amount: amount +) +``` + +### OpenTelemetry Tracing + +#### Elixir Setup + +```elixir +# mix.exs +defp deps do + [ + {:opentelemetry, "~> 1.4"}, + {:opentelemetry_exporter, "~> 1.7"}, + {:opentelemetry_phoenix, "~> 1.2"}, + {:opentelemetry_ecto, "~> 1.2"}, + ] +end + +# config/runtime.exs +config :opentelemetry, + span_processor: :batch, + exporter: {:opentelemetry_exporter, %{ + protocol: :grpc, + endpoints: [System.get_env("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")] + }} + +config :opentelemetry, :resource, + service: [name: "towerops-api", version: Mix.Project.config()[:version]] + +# application.ex — instrument Phoenix and Ecto +def start(_type, _args) do + OpentelemetryPhoenix.setup(adapter: :cowboy2) + OpentelemetryEcto.setup([:towerops, :repo]) + # ... +end +``` + +#### Custom Spans + +```elixir +require OpenTelemetry.Tracer, as: Tracer + +def process_device_command(device_id, command) do + Tracer.with_span "device.process_command", %{ + attributes: [ + {"device.id", device_id}, + {"command.type", command.type} + ] + } do + with {:ok, device} <- lookup_device(device_id), + {:ok, result} <- send_command(device, command) do + Tracer.set_attribute("command.result", "success") + {:ok, result} + else + {:error, reason} -> + Tracer.set_status(:error, inspect(reason)) + Tracer.set_attribute("command.result", "failure") + {:error, reason} + end + end +end +``` + +### Log Aggregation with Loki + +#### Promtail Configuration (ships logs to Loki) + +```yaml +# promtail-config.yml +server: + http_listen_port: 9080 + +positions: + filename: /tmp/positions.yaml + +clients: + - url: http://loki:3100/loki/api/v1/push + +scrape_configs: + - job_name: kubernetes-pods + kubernetes_sd_configs: + - role: pod + pipeline_stages: + # Parse JSON logs + - json: + expressions: + level: level + message: message + trace_id: trace_id + # Set log level as label + - labels: + level: + # Drop debug logs in production + - match: + selector: '{level="debug"}' + action: drop + # Extract timestamp from log line + - timestamp: + source: timestamp + format: RFC3339Nano + relabel_configs: + - source_labels: [__meta_kubernetes_pod_label_app] + target_label: app + - source_labels: [__meta_kubernetes_namespace] + target_label: namespace +``` + +#### LogQL Query Examples + +```logql +# All errors from billing service in last hour +{app="billing-api"} |= "error" | json | level="error" + +# Errors with specific trace ID (incident investigation) +{namespace="production"} | json | trace_id="abc123def456" + +# Top 10 error messages +sum by (message) (count_over_time({app="towerops-api"} | json | level="error" [1h])) + +# Latency from logs (if you log request duration) +{app="towerops-api"} | json | duration > 1s + +# Rate of errors per service +sum(rate({namespace="production"} | json | level="error" [5m])) by (app) +``` + +--- + +## 3. Infrastructure as Code + +### Terraform Patterns + +#### Project Structure + +``` +terraform/ +├── modules/ +│ ├── vpc/ +│ │ ├── main.tf +│ │ ├── variables.tf +│ │ ├── outputs.tf +│ │ └── versions.tf +│ ├── k8s-cluster/ +│ ├── database/ +│ └── cdn/ +├── environments/ +│ ├── prod/ +│ │ ├── main.tf # Composes modules +│ │ ├── terraform.tfvars +│ │ └── backend.tf +│ ├── staging/ +│ └── dev/ +├── global/ # Shared resources (DNS, IAM) +│ ├── main.tf +│ └── backend.tf +└── scripts/ + └── plan-all.sh +``` + +#### Module Pattern — VPC Example + +```hcl +# modules/vpc/variables.tf +variable "name" { + type = string + description = "VPC name" +} + +variable "cidr" { + type = string + default = "10.0.0.0/16" +} + +variable "availability_zones" { + type = list(string) + description = "AZs for subnet distribution" +} + +variable "enable_nat_gateway" { + type = bool + default = true +} + +# modules/vpc/main.tf +resource "aws_vpc" "main" { + cidr_block = var.cidr + enable_dns_hostnames = true + enable_dns_support = true + + tags = { + Name = var.name + ManagedBy = "terraform" + Environment = var.name + } +} + +resource "aws_subnet" "public" { + count = length(var.availability_zones) + vpc_id = aws_vpc.main.id + cidr_block = cidrsubnet(var.cidr, 8, count.index) + availability_zone = var.availability_zones[count.index] + + map_public_ip_on_launch = true + + tags = { + Name = "${var.name}-public-${var.availability_zones[count.index]}" + Tier = "public" + } +} + +resource "aws_subnet" "private" { + count = length(var.availability_zones) + vpc_id = aws_vpc.main.id + cidr_block = cidrsubnet(var.cidr, 8, count.index + length(var.availability_zones)) + availability_zone = var.availability_zones[count.index] + + tags = { + Name = "${var.name}-private-${var.availability_zones[count.index]}" + Tier = "private" + } +} + +# modules/vpc/outputs.tf +output "vpc_id" { + value = aws_vpc.main.id +} + +output "public_subnet_ids" { + value = aws_subnet.public[*].id +} + +output "private_subnet_ids" { + value = aws_subnet.private[*].id +} +``` + +#### Environment Composition + +```hcl +# environments/prod/main.tf +module "vpc" { + source = "../../modules/vpc" + + name = "prod" + cidr = "10.0.0.0/16" + availability_zones = ["us-east-1a", "us-east-1b", "us-east-1c"] + enable_nat_gateway = true +} + +module "k8s" { + source = "../../modules/k8s-cluster" + + cluster_name = "prod" + vpc_id = module.vpc.vpc_id + subnet_ids = module.vpc.private_subnet_ids + node_count = 5 + instance_type = "m6i.xlarge" + k8s_version = "1.29" +} + +module "database" { + source = "../../modules/database" + + name = "prod" + vpc_id = module.vpc.vpc_id + subnet_ids = module.vpc.private_subnet_ids + instance_class = "db.r6g.xlarge" + engine_version = "16.2" + multi_az = true + backup_retention = 30 +} +``` + +#### Remote State Configuration + +```hcl +# environments/prod/backend.tf +terraform { + backend "s3" { + bucket = "towerops-terraform-state" + key = "prod/terraform.tfstate" + region = "us-east-1" + dynamodb_table = "terraform-locks" + encrypt = true + } +} + +# Reading state from another workspace +data "terraform_remote_state" "global" { + backend = "s3" + config = { + bucket = "towerops-terraform-state" + key = "global/terraform.tfstate" + region = "us-east-1" + } +} + +# Use: data.terraform_remote_state.global.outputs.route53_zone_id +``` + +#### Terraform Best Practices — Quick Reference + +```bash +# Always plan before apply +terraform plan -out=plan.tfplan +terraform apply plan.tfplan + +# Import existing resources +terraform import aws_instance.web i-1234567890abcdef0 + +# Target specific resources (use sparingly) +terraform plan -target=module.database + +# Move state (rename without destroy/recreate) +terraform state mv aws_instance.old aws_instance.new + +# Detect drift +terraform plan -detailed-exitcode # Exit 2 = drift detected + +# Workspace management (alternative to directory-per-env) +terraform workspace new staging +terraform workspace select staging +terraform workspace list +``` + +**State locking is non-negotiable.** Always use DynamoDB (AWS) or equivalent. Two engineers running `terraform apply` simultaneously = destroyed infrastructure. + +### NixOS for Reproducible Servers + +#### NixOS Server Configuration + +```nix +# /etc/nixos/configuration.nix +{ config, pkgs, ... }: + +{ + imports = [ + ./hardware-configuration.nix + ./services/prometheus.nix + ./services/postgres.nix + ./networking.nix + ]; + + # System basics + system.stateVersion = "24.11"; + time.timeZone = "America/Chicago"; + + # Automatic updates with rollback capability + system.autoUpgrade = { + enable = true; + allowReboot = false; # Manual reboot approval + channel = "https://nixos.org/channels/nixos-24.11"; + }; + + # Minimal packages + environment.systemPackages = with pkgs; [ + vim + git + htop + curl + jq + ripgrep + ]; + + # Firewall + networking.firewall = { + enable = true; + allowedTCPPorts = [ 22 80 443 ]; + allowedUDPPorts = [ ]; + }; + + # SSH hardening + services.openssh = { + enable = true; + settings = { + PasswordAuthentication = false; + PermitRootLogin = "no"; + KbdInteractiveAuthentication = false; + }; + extraConfig = '' + AllowUsers deploy + MaxAuthTries 3 + ''; + }; + + # Automatic garbage collection + nix.gc = { + automatic = true; + dates = "weekly"; + options = "--delete-older-than 30d"; + }; + + # Users + users.users.deploy = { + isNormalUser = true; + extraGroups = [ "wheel" "docker" ]; + openssh.authorizedKeys.keys = [ + "ssh-ed25519 AAAA... deploy@towerops" + ]; + }; + + # Security + security.sudo.wheelNeedsPassword = true; +} +``` + +#### NixOS Service Module Example + +```nix +# services/prometheus.nix +{ config, pkgs, ... }: + +{ + services.prometheus = { + enable = true; + port = 9090; + retentionTime = "90d"; + + globalConfig = { + scrape_interval = "15s"; + evaluation_interval = "15s"; + }; + + scrapeConfigs = [ + { + job_name = "node"; + static_configs = [{ + targets = [ "localhost:9100" ]; + }]; + } + { + job_name = "postgres"; + static_configs = [{ + targets = [ "localhost:9187" ]; + }]; + } + ]; + + rules = [ + (builtins.toJSON { + groups = [{ + name = "system"; + rules = [{ + alert = "HighCPU"; + expr = "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100) > 85"; + for = "15m"; + labels.severity = "warning"; + }]; + }]; + }) + ]; + }; + + services.prometheus.exporters.node = { + enable = true; + port = 9100; + enabledCollectors = [ "systemd" "processes" ]; + }; + + services.grafana = { + enable = true; + settings = { + server = { + http_port = 3000; + domain = "grafana.towerops.net"; + }; + security = { + admin_password = "$__file{/run/secrets/grafana-admin-password}"; + }; + }; + }; +} +``` + +#### NixOS Deployment + +```bash +# Build without switching (test first) +nixos-rebuild build --flake .#my-server + +# Switch to new configuration +nixos-rebuild switch --flake .#my-server + +# Rollback to previous generation +nixos-rebuild switch --rollback + +# List generations (your deployment history) +nix-env --list-generations --profile /nix/var/nix/profiles/system + +# Remote deployment +nixos-rebuild switch --flake .#prod-server \ + --target-host deploy@prod.towerops.net \ + --use-remote-sudo +``` + +**Key NixOS advantage:** Every deployment is atomic and rollbackable. If generation 42 breaks, `nixos-rebuild switch --rollback` takes you back to generation 41 in seconds. No Ansible "undo" scripts needed. + +--- + +## 4. Kubernetes + +### Deployment with Best Practices + +```yaml +# deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: towerops-api + namespace: production + labels: + app: towerops-api + version: v1.4.2 +spec: + replicas: 3 + revisionHistoryLimit: 5 # Keep 5 old ReplicaSets for rollback + selector: + matchLabels: + app: towerops-api + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 # 1 extra pod during rollout + maxUnavailable: 0 # Never drop below desired count + template: + metadata: + labels: + app: towerops-api + version: v1.4.2 + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "4000" + prometheus.io/path: "/metrics" + spec: + serviceAccountName: towerops-api + terminationGracePeriodSeconds: 60 + + # Don't schedule all replicas on the same node + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: + app: towerops-api + + containers: + - name: towerops-api + image: ghcr.io/towerops/api:v1.4.2 # Always pin exact tags + ports: + - containerPort: 4000 + protocol: TCP + + # Resource limits — ALWAYS set these + resources: + requests: + cpu: 250m # Guaranteed minimum + memory: 256Mi + limits: + cpu: "1" # Hard ceiling + memory: 512Mi # OOMKilled if exceeded + + # Liveness: is the process alive? Restart if not. + livenessProbe: + httpGet: + path: /healthz + port: 4000 + initialDelaySeconds: 15 + periodSeconds: 10 + timeoutSeconds: 3 + failureThreshold: 3 + + # Readiness: can it serve traffic? Remove from LB if not. + readinessProbe: + httpGet: + path: /readyz + port: 4000 + initialDelaySeconds: 5 + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 2 + + # Startup: for slow-starting apps. Liveness/readiness wait until this passes. + startupProbe: + httpGet: + path: /healthz + port: 4000 + failureThreshold: 30 # 30 × 2s = 60s to start + periodSeconds: 2 + + # Graceful shutdown + lifecycle: + preStop: + exec: + command: ["/bin/sh", "-c", "sleep 5"] # Let LB drain connections + + env: + - name: MIX_ENV + value: "prod" + - name: PHX_HOST + value: "api.towerops.net" + - name: PORT + value: "4000" + - name: DATABASE_URL + valueFrom: + secretKeyRef: + name: towerops-secrets + key: database-url + - name: SECRET_KEY_BASE + valueFrom: + secretKeyRef: + name: towerops-secrets + key: secret-key-base + + volumeMounts: + - name: tmp + mountPath: /tmp + + volumes: + - name: tmp + emptyDir: + sizeLimit: 100Mi + + # Security context + securityContext: + runAsNonRoot: true + runAsUser: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault +``` + +### Health Check Implementation + +```elixir +# In your Phoenix router — separate liveness and readiness +scope "/", TowerOpsWeb do + # Liveness: "is the BEAM VM running?" + get "/healthz", HealthController, :liveness + + # Readiness: "can we serve traffic?" + # Checks DB connection, required services, etc. + get "/readyz", HealthController, :readiness +end + +# lib/towerops_web/controllers/health_controller.ex +defmodule TowerOpsWeb.HealthController do + use TowerOpsWeb, :controller + + def liveness(conn, _params) do + conn |> put_status(200) |> json(%{status: "ok"}) + end + + def readiness(conn, _params) do + checks = %{ + database: check_database(), + redis: check_redis() + } + + if Enum.all?(Map.values(checks), &(&1 == :ok)) do + conn |> put_status(200) |> json(%{status: "ready", checks: checks}) + else + conn |> put_status(503) |> json(%{status: "not_ready", checks: checks}) + end + end + + defp check_database do + case Ecto.Adapters.SQL.query(TowerOps.Repo, "SELECT 1", []) do + {:ok, _} -> :ok + _ -> :error + end + end + + defp check_redis do + case Redix.command(:redix, ["PING"]) do + {:ok, "PONG"} -> :ok + _ -> :error + end + end +end +``` + +### Service & Ingress + +```yaml +# service.yaml +apiVersion: v1 +kind: Service +metadata: + name: towerops-api + namespace: production +spec: + type: ClusterIP + selector: + app: towerops-api + ports: + - port: 80 + targetPort: 4000 + protocol: TCP + +--- +# ingress.yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: towerops-api + namespace: production + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod + nginx.ingress.kubernetes.io/rate-limit: "100" + nginx.ingress.kubernetes.io/rate-limit-window: "1m" + nginx.ingress.kubernetes.io/proxy-body-size: "10m" + nginx.ingress.kubernetes.io/ssl-redirect: "true" + nginx.ingress.kubernetes.io/force-ssl-redirect: "true" +spec: + ingressClassName: nginx + tls: + - hosts: + - api.towerops.net + secretName: towerops-api-tls + rules: + - host: api.towerops.net + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: towerops-api + port: + number: 80 +``` + +### Horizontal Pod Autoscaler + +```yaml +# hpa.yaml +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: towerops-api + namespace: production +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: towerops-api + minReplicas: 3 + maxReplicas: 20 + behavior: + scaleUp: + stabilizationWindowSeconds: 60 # Prevent flapping + policies: + - type: Pods + value: 4 # Add max 4 pods at a time + periodSeconds: 60 + scaleDown: + stabilizationWindowSeconds: 300 # Wait 5 min before scaling down + policies: + - type: Percent + value: 25 # Remove max 25% at a time + periodSeconds: 60 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 + - type: Resource + resource: + name: memory + target: + type: Utilization + averageUtilization: 80 + # Custom metric: requests per second per pod + - type: Pods + pods: + metric: + name: http_requests_per_second + target: + type: AverageValue + averageValue: "100" +``` + +### Pod Disruption Budget + +```yaml +# pdb.yaml — Ensure minimum availability during voluntary disruptions +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: towerops-api + namespace: production +spec: + # At least 2 pods must always be running + minAvailable: 2 + # OR: at most 1 pod can be disrupted at a time + # maxUnavailable: 1 + selector: + matchLabels: + app: towerops-api +``` + +### Helm vs Kustomize + +#### When to Use Each + +| Criteria | Helm | Kustomize | +|---|---|---| +| Third-party charts | ✅ Great | ❌ Not designed for this | +| Environment overlays | Works (values files) | ✅ Built for this | +| Templating | Full Go templates | Patches only (no templates) | +| Complexity | Higher | Lower | +| Learning curve | Steeper | Gentler | +| Debugging | `helm template` + squint | WYSIWYG patches | + +**Recommendation:** Use Helm for third-party dependencies (nginx-ingress, cert-manager, prometheus). Use Kustomize for your own application manifests. + +#### Kustomize Example + +``` +k8s/ +├── base/ +│ ├── kustomization.yaml +│ ├── deployment.yaml +│ ├── service.yaml +│ └── ingress.yaml +├── overlays/ +│ ├── staging/ +│ │ ├── kustomization.yaml +│ │ ├── replicas-patch.yaml +│ │ └── env-patch.yaml +│ └── production/ +│ ├── kustomization.yaml +│ ├── replicas-patch.yaml +│ └── hpa.yaml +``` + +```yaml +# base/kustomization.yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - deployment.yaml + - service.yaml + - ingress.yaml +commonLabels: + app: towerops-api + +# overlays/production/kustomization.yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: production +resources: + - ../../base + - hpa.yaml +patches: + - path: replicas-patch.yaml +images: + - name: ghcr.io/towerops/api + newTag: v1.4.2 + +# overlays/production/replicas-patch.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: towerops-api +spec: + replicas: 3 +``` + +```bash +# Preview what will be applied +kubectl kustomize overlays/production + +# Apply +kubectl apply -k overlays/production + +# Diff against live cluster +kubectl diff -k overlays/production +``` + +### Kubernetes Operations — Quick Reference + +```bash +# --- Deployments --- +kubectl rollout status deployment/towerops-api -n production +kubectl rollout history deployment/towerops-api -n production +kubectl rollout undo deployment/towerops-api -n production # Rollback to previous +kubectl rollout undo deployment/towerops-api -n production --to-revision=3 # Specific revision + +# --- Debugging --- +kubectl get pods -n production -o wide # Pod status + node +kubectl describe pod -n production # Events, conditions +kubectl logs -n production --tail=100 -f # Stream logs +kubectl logs -n production -c --previous # Crashed container logs +kubectl top pods -n production --sort-by=memory # Resource usage +kubectl get events -n production --sort-by=.lastTimestamp # Recent events + +# --- Emergency --- +kubectl scale deployment/towerops-api -n production --replicas=10 # Scale manually +kubectl cordon # Prevent scheduling +kubectl drain --ignore-daemonsets --delete-emptydir-data # Evacuate node +``` + +--- + +## 5. CI/CD + +### Pipeline Design Principles + +1. **Fast feedback first** — Lint and unit tests run in < 2 minutes +2. **Fail fast** — Don't run slow integration tests if linting fails +3. **Immutable artifacts** — Build once, promote to each environment +4. **No manual steps** — If a human has to click something, automate it + +#### GitHub Actions Pipeline + +```yaml +# .github/workflows/ci.yml +name: CI/CD + +on: + push: + branches: [main] + pull_request: + branches: [main] + +env: + REGISTRY: ghcr.io + IMAGE_NAME: ${{ github.repository }} + +jobs: + lint-and-test: + runs-on: ubuntu-latest + services: + postgres: + image: postgres:16 + env: + POSTGRES_PASSWORD: test + POSTGRES_DB: towerops_test + ports: ["5432:5432"] + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + steps: + - uses: actions/checkout@v4 + + - uses: erlef/setup-beam@v1 + with: + elixir-version: "1.19" + otp-version: "28" + + - uses: actions/cache@v4 + with: + path: | + deps + _build + key: ${{ runner.os }}-mix-${{ hashFiles('mix.lock') }} + + - run: mix deps.get + - run: mix format --check-formatted + - run: mix compile --warnings-as-errors + - run: mix credo --strict + - run: mix dialyzer + - run: mix test --cover + env: + DATABASE_URL: postgres://postgres:test@localhost/towerops_test + + build-and-push: + needs: lint-and-test + if: github.ref == 'refs/heads/main' + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + outputs: + image-tag: ${{ steps.meta.outputs.version }} + steps: + - uses: actions/checkout@v4 + + - uses: docker/login-action@v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - id: meta + uses: docker/metadata-action@v5 + with: + images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} + tags: | + type=sha,prefix= + type=semver,pattern={{version}} + + - uses: docker/build-push-action@v6 + with: + context: . + push: true + tags: ${{ steps.meta.outputs.tags }} + cache-from: type=gha + cache-to: type=gha,mode=max + + deploy-staging: + needs: build-and-push + runs-on: ubuntu-latest + environment: staging + steps: + - uses: actions/checkout@v4 + + - name: Update image tag in kustomization + run: | + cd k8s/overlays/staging + kustomize edit set image \ + ghcr.io/towerops/api=ghcr.io/towerops/api:${{ needs.build-and-push.outputs.image-tag }} + + - name: Apply to staging + run: kubectl apply -k k8s/overlays/staging + env: + KUBECONFIG: ${{ secrets.STAGING_KUBECONFIG }} + + - name: Wait for rollout + run: kubectl rollout status deployment/towerops-api -n staging --timeout=300s + env: + KUBECONFIG: ${{ secrets.STAGING_KUBECONFIG }} + + - name: Smoke test + run: | + for i in {1..10}; do + status=$(curl -s -o /dev/null -w '%{http_code}' https://staging-api.towerops.net/healthz) + if [ "$status" = "200" ]; then echo "✅ Healthy"; exit 0; fi + sleep 5 + done + echo "❌ Smoke test failed"; exit 1 + + deploy-production: + needs: [build-and-push, deploy-staging] + runs-on: ubuntu-latest + environment: production # Requires manual approval in GitHub + steps: + - uses: actions/checkout@v4 + + - name: Update image tag + run: | + cd k8s/overlays/production + kustomize edit set image \ + ghcr.io/towerops/api=ghcr.io/towerops/api:${{ needs.build-and-push.outputs.image-tag }} + + - name: Apply to production + run: kubectl apply -k k8s/overlays/production + env: + KUBECONFIG: ${{ secrets.PROD_KUBECONFIG }} + + - name: Wait for rollout + run: kubectl rollout status deployment/towerops-api -n production --timeout=600s + env: + KUBECONFIG: ${{ secrets.PROD_KUBECONFIG }} +``` + +### GitOps with Flux + +```yaml +# flux-system/gotk-components.yaml — installed via: +# flux bootstrap github \ +# --owner=towerops \ +# --repository=fleet \ +# --branch=main \ +# --path=clusters/production + +# clusters/production/towerops-api.yaml +--- +apiVersion: source.toolkit.fluxcd.io/v1 +kind: GitRepository +metadata: + name: towerops-api + namespace: flux-system +spec: + interval: 1m + url: https://github.com/towerops/towerops + ref: + branch: main + secretRef: + name: github-token + +--- +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: towerops-api + namespace: flux-system +spec: + interval: 5m + path: ./k8s/overlays/production + prune: true # Delete resources removed from git + sourceRef: + kind: GitRepository + name: towerops-api + healthChecks: + - apiVersion: apps/v1 + kind: Deployment + name: towerops-api + namespace: production + timeout: 10m +``` + +### GitOps with ArgoCD + +```yaml +# argocd/applications/towerops-api.yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: towerops-api + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: default + source: + repoURL: https://github.com/towerops/towerops + targetRevision: main + path: k8s/overlays/production + destination: + server: https://kubernetes.default.svc + namespace: production + syncPolicy: + automated: + prune: true + selfHeal: true # Revert manual kubectl changes + syncOptions: + - CreateNamespace=true + retry: + limit: 3 + backoff: + duration: 5s + maxDuration: 3m + factor: 2 +``` + +### Secrets Management + +#### SOPS + Age (Simple, Git-friendly) + +```bash +# Install +# nix-env -iA nixpkgs.sops nixpkgs.age + +# Generate age key +age-keygen -o ~/.config/sops/age/keys.txt + +# .sops.yaml — tells SOPS which keys to use +cat > .sops.yaml << 'EOF' +creation_rules: + - path_regex: secrets/.*\.yaml$ + age: >- + age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p +EOF + +# Create encrypted secret +sops secrets/production.yaml +# Opens $EDITOR — write plaintext YAML, SOPS encrypts on save + +# Example encrypted file (values encrypted, keys readable) +cat secrets/production.yaml +# database_url: ENC[AES256_GCM,data:abc123...,iv:...,tag:...,type:str] +# secret_key_base: ENC[AES256_GCM,data:def456...,iv:...,tag:...,type:str] + +# Decrypt and use +sops -d secrets/production.yaml + +# Use with kubectl +sops -d secrets/production.yaml | kubectl create secret generic towerops-secrets \ + --from-file=database-url=/dev/stdin \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +#### HashiCorp Vault + +```bash +# --- Vault Setup --- +# Enable KV secrets engine +vault secrets enable -path=secret kv-v2 + +# Store a secret +vault kv put secret/towerops/production \ + database_url="postgres://user:pass@db:5432/towerops" \ + secret_key_base="$(openssl rand -hex 64)" + +# Read a secret +vault kv get -field=database_url secret/towerops/production + +# --- Kubernetes Auth --- +vault auth enable kubernetes +vault write auth/kubernetes/config \ + kubernetes_host="https://kubernetes.default.svc" + +# Policy: towerops-api can only read its own secrets +vault policy write towerops-api - < interval '5 minutes';") +if [ "$LONG_TX" -gt 0 ]; then + echo "⚠️ $LONG_TX long-running transactions detected. Proceeding with caution." +fi + +echo "=== Running migrations ===" +# For Elixir/Ecto: +mix ecto.migrate + +# Verify migration status +echo "=== Post-migration status ===" +mix ecto.migrations + +echo "✅ Migration complete" +``` + +**Golden rule for zero-downtime migrations:** Never remove a column or rename a column in a single deploy. Always use expand-contract: + +``` +Deploy 1: Add new column (nullable or with default) +Deploy 2: Backfill data, update code to write to both old and new +Deploy 3: Switch reads to new column +Deploy 4: Stop writing to old column +Deploy 5: Drop old column +``` + +--- + +## 6. Networking & Security + +### TLS / mTLS + +#### cert-manager with Let's Encrypt + +```yaml +# cert-manager ClusterIssuer +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: letsencrypt-prod +spec: + acme: + server: https://acme-v02.api.letsencrypt.org/directory + email: ops@towerops.net + privateKeySecretRef: + name: letsencrypt-prod-key + solvers: + - http01: + ingress: + class: nginx + # Or DNS challenge for wildcard certs: + - dns01: + cloudflare: + email: ops@towerops.net + apiTokenSecretRef: + name: cloudflare-api-token + key: api-token + selector: + dnsZones: + - towerops.net + +--- +# Certificate resource (or use ingress annotation instead) +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: towerops-wildcard + namespace: production +spec: + secretName: towerops-wildcard-tls + issuerRef: + name: letsencrypt-prod + kind: ClusterIssuer + dnsNames: + - "*.towerops.net" + - towerops.net +``` + +#### mTLS Between Services (Istio / Linkerd) + +```yaml +# Istio PeerAuthentication — enforce mTLS for all services in namespace +apiVersion: security.istio.io/v1 +kind: PeerAuthentication +metadata: + name: default + namespace: production +spec: + mtls: + mode: STRICT # All traffic must be mTLS + +--- +# Or with Linkerd (simpler): +# Just annotate the namespace +apiVersion: v1 +kind: Namespace +metadata: + name: production + annotations: + linkerd.io/inject: enabled # Auto-injects sidecar proxy with mTLS +``` + +### Zero-Trust Networking + +**Principle: Never trust the network. Always verify identity.** + +```yaml +# Kubernetes NetworkPolicy — default deny all +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-deny-all + namespace: production +spec: + podSelector: {} + policyTypes: + - Ingress + - Egress + +--- +# Allow specific traffic +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: towerops-api-policy + namespace: production +spec: + podSelector: + matchLabels: + app: towerops-api + policyTypes: + - Ingress + - Egress + ingress: + # Allow from ingress controller only + - from: + - namespaceSelector: + matchLabels: + name: ingress-nginx + ports: + - port: 4000 + protocol: TCP + egress: + # Allow to PostgreSQL + - to: + - podSelector: + matchLabels: + app: postgresql + ports: + - port: 5432 + protocol: TCP + # Allow to Redis + - to: + - podSelector: + matchLabels: + app: redis + ports: + - port: 6379 + protocol: TCP + # Allow DNS + - to: + - namespaceSelector: {} + ports: + - port: 53 + protocol: UDP + - port: 53 + protocol: TCP +``` + +### RBAC + +#### Kubernetes RBAC + +```yaml +# Least-privilege service account for CI/CD deployments +apiVersion: v1 +kind: ServiceAccount +metadata: + name: ci-deployer + namespace: production + +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: deployer + namespace: production +rules: + - apiGroups: ["apps"] + resources: ["deployments"] + verbs: ["get", "list", "patch", "update"] + - apiGroups: [""] + resources: ["configmaps", "secrets"] + verbs: ["get", "list", "create", "update", "patch"] + - apiGroups: [""] + resources: ["pods"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["pods/log"] + verbs: ["get"] + +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: ci-deployer-binding + namespace: production +subjects: + - kind: ServiceAccount + name: ci-deployer + namespace: production +roleRef: + kind: Role + name: deployer + apiGroup: rbac.authorization.k8s.io +``` + +### DNS Patterns + +```bash +# DNS record types and when to use them +# A — Direct IP mapping (static servers) +# CNAME — Alias to another domain (CDN, load balancer) +# AAAA — IPv6 address +# MX — Mail routing +# TXT — Verification, SPF, DKIM, DMARC +# SRV — Service discovery (rare outside K8s) +# CAA — Certificate Authority Authorization (security) + +# Example zone file for towerops.net +# Always set CAA to restrict who can issue certs +towerops.net. CAA 0 issue "letsencrypt.org" +towerops.net. CAA 0 issuewild "letsencrypt.org" + +# Primary domains +towerops.net. A 203.0.113.10 +www.towerops.net. CNAME towerops.net. +api.towerops.net. CNAME lb.towerops.net. # Points to load balancer + +# Low TTL for things that might change during incidents +api.towerops.net. 300 CNAME lb.towerops.net. # 5 min TTL + +# High TTL for stable records +towerops.net. 86400 A 203.0.113.10 # 24h TTL + +# Email security +towerops.net. TXT "v=spf1 include:_spf.google.com ~all" +_dmarc.towerops.net. TXT "v=DMARC1; p=reject; rua=mailto:dmarc@towerops.net" +``` + +### CDN Patterns + +```bash +# Cloudflare as CDN + WAF + DDoS protection +# Key headers to set: + +# Cache static assets aggressively +Cache-Control: public, max-age=31536000, immutable # For hashed assets (/app-abc123.js) + +# Don't cache API responses +Cache-Control: private, no-store + +# Don't cache HTML (or use short TTL) +Cache-Control: public, max-age=300, s-maxage=60 # Browser: 5min, CDN: 1min + +# Purge cache (Cloudflare API) +curl -X POST "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/purge_cache" \ + -H "Authorization: Bearer ${CF_API_TOKEN}" \ + -H "Content-Type: application/json" \ + --data '{"purge_everything": true}' + +# Purge specific URLs +curl -X POST "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/purge_cache" \ + -H "Authorization: Bearer ${CF_API_TOKEN}" \ + -H "Content-Type: application/json" \ + --data '{"files": ["https://towerops.net/styles.css"]}' +``` + +--- + +## 7. Database Operations + +### PostgreSQL Backup — WAL Archiving + +#### Continuous Archiving Setup + +```bash +# postgresql.conf — WAL archiving settings +wal_level = replica # Required for archiving +archive_mode = on +archive_command = 'pgbackrest --stanza=main archive-push %p' +# Or with plain file copy: +# archive_command = 'test ! -f /backup/wal/%f && cp %p /backup/wal/%f' + +max_wal_senders = 3 # For streaming replication +wal_keep_size = 1GB # Keep WAL segments for slow replicas +``` + +#### pgBackRest Configuration + +```ini +# /etc/pgbackrest/pgbackrest.conf +[global] +repo1-path=/backup/pgbackrest +repo1-retention-full=4 # Keep 4 full backups +repo1-retention-diff=14 # Keep 14 differential backups +repo1-cipher-type=aes-256-cbc +repo1-cipher-pass= +compress-type=zst +compress-level=6 +process-max=4 # Parallel backup processes + +# S3 storage (alternative to local) +# repo1-type=s3 +# repo1-s3-bucket=towerops-backups +# repo1-s3-region=us-east-1 +# repo1-s3-endpoint=s3.amazonaws.com + +[main] +pg1-path=/var/lib/postgresql/16/main +pg1-port=5432 +``` + +```bash +# Initialize backup stanza +pgbackrest --stanza=main stanza-create + +# Full backup (weekly cron) +pgbackrest --stanza=main --type=full backup + +# Differential backup (daily cron) +pgbackrest --stanza=main --type=diff backup + +# Verify backups +pgbackrest --stanza=main check +pgbackrest --stanza=main info + +# Point-in-time recovery (PITR) +# Restore to specific timestamp +pgbackrest --stanza=main --type=time \ + --target="2026-03-13 15:30:00-05" \ + --target-action=promote \ + restore + +# Crontab +0 2 * * 0 pgbackrest --stanza=main --type=full backup # Weekly full at 2 AM Sunday +0 2 * * 1-6 pgbackrest --stanza=main --type=diff backup # Daily diff at 2 AM Mon-Sat +``` + +#### Backup Verification (Test Your Restores!) + +```bash +#!/bin/bash +# scripts/verify-backup.sh — Run weekly, alert on failure +set -euo pipefail + +RESTORE_DIR="/tmp/pg-restore-test" +rm -rf "$RESTORE_DIR" + +echo "Restoring latest backup to temp directory..." +pgbackrest --stanza=main --pg1-path="$RESTORE_DIR" restore + +echo "Starting temporary PostgreSQL instance..." +pg_ctl -D "$RESTORE_DIR" -o "-p 5433" -l /tmp/pg-restore.log start + +echo "Running verification queries..." +psql -p 5433 -d towerops -c "SELECT count(*) FROM users;" || { + echo "❌ Backup verification FAILED" + pg_ctl -D "$RESTORE_DIR" stop + exit 1 +} + +echo "✅ Backup verification passed" +pg_ctl -D "$RESTORE_DIR" stop +rm -rf "$RESTORE_DIR" +``` + +### PgBouncer — Connection Pooling + +```ini +# pgbouncer.ini +[databases] +towerops = host=127.0.0.1 port=5432 dbname=towerops + +[pgbouncer] +listen_addr = 0.0.0.0 +listen_port = 6432 +auth_type = scram-sha-256 +auth_file = /etc/pgbouncer/userlist.txt + +# Pool settings +pool_mode = transaction # Best for web apps (release conn after each tx) +default_pool_size = 25 # Connections per user/database pair +max_client_conn = 1000 # Max incoming connections +min_pool_size = 5 # Keep warm connections +reserve_pool_size = 5 # Emergency overflow +reserve_pool_timeout = 3 # Wait before using reserve pool + +# Timeouts +server_idle_timeout = 600 # Close idle server connections after 10 min +client_idle_timeout = 0 # No client idle timeout +query_timeout = 30 # Kill queries running > 30s +client_login_timeout = 60 + +# Logging +log_connections = 0 +log_disconnections = 0 +log_pooler_errors = 1 +stats_period = 60 + +# TLS +client_tls_sslmode = prefer +client_tls_key_file = /etc/pgbouncer/server.key +client_tls_cert_file = /etc/pgbouncer/server.crt +``` + +```bash +# Connect to PgBouncer admin console +psql -p 6432 -U pgbouncer pgbouncer + +# Useful admin commands: +SHOW POOLS; # Pool status per database/user +SHOW CLIENTS; # Connected clients +SHOW SERVERS; # Backend connections +SHOW STATS; # Request stats +SHOW CONFIG; # Current config + +# Graceful restart (finishes active transactions) +RELOAD; # Reload config +PAUSE; # Pause new queries, finish active +RESUME; # Resume after pause +``` + +### Zero-Downtime Migrations + +#### Safe Migration Patterns + +```sql +-- ❌ DANGEROUS: Locks entire table for write +ALTER TABLE devices ADD COLUMN firmware_version TEXT NOT NULL DEFAULT 'unknown'; + +-- ✅ SAFE: Add nullable column (instant, no lock) +ALTER TABLE devices ADD COLUMN firmware_version TEXT; + +-- ✅ SAFE: Add default in separate step (PG 11+ is instant for defaults) +ALTER TABLE devices ALTER COLUMN firmware_version SET DEFAULT 'unknown'; + +-- ✅ SAFE: Backfill in batches, not all at once +DO $$ +DECLARE + batch_size INT := 10000; + rows_updated INT; +BEGIN + LOOP + UPDATE devices + SET firmware_version = 'unknown' + WHERE id IN ( + SELECT id FROM devices + WHERE firmware_version IS NULL + LIMIT batch_size + FOR UPDATE SKIP LOCKED + ); + GET DIAGNOSTICS rows_updated = ROW_COUNT; + EXIT WHEN rows_updated = 0; + RAISE NOTICE 'Updated % rows', rows_updated; + PERFORM pg_sleep(0.1); -- Brief pause to reduce load + COMMIT; + END LOOP; +END $$; + +-- ✅ SAFE: Add NOT NULL constraint after backfill +ALTER TABLE devices ALTER COLUMN firmware_version SET NOT NULL; + +-- ❌ DANGEROUS: CREATE INDEX locks writes +CREATE INDEX idx_devices_firmware ON devices(firmware_version); + +-- ✅ SAFE: CONCURRENTLY doesn't block writes (takes longer) +CREATE INDEX CONCURRENTLY idx_devices_firmware ON devices(firmware_version); + +-- ❌ DANGEROUS: Renaming column breaks running code +ALTER TABLE devices RENAME COLUMN name TO display_name; + +-- ✅ SAFE: Use a view or add new column, migrate, drop old +-- Step 1: Add new column +ALTER TABLE devices ADD COLUMN display_name TEXT; +-- Step 2: Backfill (app writes to both during transition) +UPDATE devices SET display_name = name WHERE display_name IS NULL; +-- Step 3: (Next deploy) Switch reads to display_name +-- Step 4: (Next deploy) Stop writing to name +-- Step 5: (Next deploy) Drop old column +ALTER TABLE devices DROP COLUMN name; +``` + +#### Lock Timeout Safety Net + +```sql +-- Always set lock_timeout for DDL in production +SET lock_timeout = '5s'; +-- If we can't get the lock in 5s, fail rather than queue behind long transactions + +BEGIN; +SET lock_timeout = '5s'; +ALTER TABLE devices ADD COLUMN new_col TEXT; +COMMIT; +``` + +### PostgreSQL Monitoring Queries + +```sql +-- === Connection Health === +-- Current connections by state +SELECT state, count(*) +FROM pg_stat_activity +WHERE pid != pg_backend_pid() +GROUP BY state +ORDER BY count DESC; + +-- Long-running queries (potential problems) +SELECT pid, now() - pg_stat_activity.query_start AS duration, + query, state, wait_event_type, wait_event +FROM pg_stat_activity +WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes' + AND state != 'idle' +ORDER BY duration DESC; + +-- Kill a stuck query +SELECT pg_cancel_backend(pid); -- Graceful (cancels query) +SELECT pg_terminate_backend(pid); -- Force (kills connection) + +-- === Performance === +-- Table bloat (dead tuples needing VACUUM) +SELECT schemaname, relname, + n_live_tup, n_dead_tup, + round(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 1) AS dead_pct, + last_vacuum, last_autovacuum +FROM pg_stat_user_tables +WHERE n_dead_tup > 1000 +ORDER BY n_dead_tup DESC +LIMIT 20; + +-- Index usage (find unused indexes to drop) +SELECT schemaname, tablename, indexname, + idx_scan AS times_used, + pg_size_pretty(pg_relation_size(indexrelid)) AS index_size +FROM pg_stat_user_indexes +JOIN pg_index USING (indexrelid) +WHERE idx_scan = 0 + AND NOT indisunique -- Don't drop unique constraints +ORDER BY pg_relation_size(indexrelid) DESC; + +-- Missing indexes (sequential scans on large tables) +SELECT relname, seq_scan, seq_tup_read, + idx_scan, idx_tup_fetch, + seq_tup_read / NULLIF(seq_scan, 0) AS avg_rows_per_seq_scan +FROM pg_stat_user_tables +WHERE seq_scan > 100 + AND seq_tup_read / NULLIF(seq_scan, 0) > 10000 +ORDER BY seq_tup_read DESC; + +-- Slowest queries (requires pg_stat_statements extension) +SELECT query, + calls, + round(total_exec_time::numeric, 2) AS total_time_ms, + round(mean_exec_time::numeric, 2) AS avg_time_ms, + round((100 * total_exec_time / sum(total_exec_time) OVER ())::numeric, 2) AS pct_total +FROM pg_stat_statements +ORDER BY total_exec_time DESC +LIMIT 20; + +-- Cache hit ratio (should be > 99%) +SELECT + sum(heap_blks_read) AS heap_read, + sum(heap_blks_hit) AS heap_hit, + round(sum(heap_blks_hit) / NULLIF(sum(heap_blks_hit) + sum(heap_blks_read), 0)::numeric * 100, 2) AS hit_ratio +FROM pg_statio_user_tables; + +-- === Replication === +-- Replication status (on primary) +SELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, + (sent_lsn - replay_lsn) AS replication_lag +FROM pg_stat_replication; + +-- Replication lag in seconds (on replica) +SELECT CASE WHEN pg_last_wal_receive_lsn() = pg_last_wal_replay_lsn() + THEN 0 + ELSE EXTRACT(EPOCH FROM now() - pg_last_xact_replay_timestamp()) + END AS replication_lag_seconds; + +-- === Locks === +-- Blocked queries (waiting for locks) +SELECT blocked_locks.pid AS blocked_pid, + blocked_activity.usename AS blocked_user, + blocking_locks.pid AS blocking_pid, + blocking_activity.usename AS blocking_user, + blocked_activity.query AS blocked_statement, + blocking_activity.query AS blocking_statement +FROM pg_catalog.pg_locks blocked_locks +JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid +JOIN pg_catalog.pg_locks blocking_locks + ON blocking_locks.locktype = blocked_locks.locktype + AND blocking_locks.relation = blocked_locks.relation + AND blocking_locks.pid != blocked_locks.pid +JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid +WHERE NOT blocked_locks.granted; + +-- === Disk Usage === +-- Largest tables +SELECT relname, + pg_size_pretty(pg_total_relation_size(relid)) AS total_size, + pg_size_pretty(pg_relation_size(relid)) AS table_size, + pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) AS index_size +FROM pg_catalog.pg_statio_user_tables +ORDER BY pg_total_relation_size(relid) DESC +LIMIT 20; +``` + +--- + +## 8. On-Call & Alerting + +### Alert Design Principles + +**Every alert must be:** + +1. **Actionable** — Someone can do something about it right now +2. **Urgent** — It needs attention within the response SLA +3. **Real** — It indicates an actual problem (< 5% false positive rate) +4. **Unique** — Not duplicated by another alert + +**Bad alerts → alert fatigue → ignored alerts → outages.** + +#### Alert Anti-Patterns + +```yaml +# ❌ BAD: Too sensitive, will flap +- alert: HighCPU + expr: node_cpu_usage > 80 + for: 1m # 1 minute is too short + +# ✅ GOOD: Sustained high CPU +- alert: HighCPU + expr: node_cpu_usage > 85 + for: 15m # Must sustain for 15 min + +# ❌ BAD: Not actionable +- alert: DiskUsageAbove50Percent + expr: disk_usage > 50 # What am I supposed to do at 50%? + +# ✅ GOOD: Predictive and actionable +- alert: DiskWillFillIn24h + expr: predict_linear(disk_free[6h], 24*3600) < 0 + # Clear action: expand disk or clean up + annotations: + runbook_url: "https://wiki/runbooks/disk-full" + +# ❌ BAD: Causes alert storm (one per pod) +- alert: PodHighMemory + expr: container_memory_usage > 100Mi + +# ✅ GOOD: Aggregate and threshold appropriately +- alert: NamespaceHighMemory + expr: | + sum(container_memory_usage_bytes{namespace="production"}) + / sum(kube_resourcequota{resource="limits.memory", namespace="production"}) + > 0.85 +``` + +### Escalation Policy + +``` +┌─────────────────────────────────────────────────────────┐ +│ Time │ Who Gets Paged │ Channel │ +├─────────────────────────────────────────────────────────┤ +│ 0 min │ Primary on-call │ PagerDuty + SMS │ +│ 10 min │ Secondary on-call │ PagerDuty + SMS │ +│ 20 min │ Engineering manager │ Phone call │ +│ 30 min │ VP Engineering │ Phone call │ +│ 45 min │ CTO │ Phone call │ +└─────────────────────────────────────────────────────────┘ + +SEV-1: Start at top, escalate every 10 min if not acknowledged +SEV-2: Start at primary on-call, escalate after 30 min +SEV-3: Slack notification only, addressed next business day +SEV-4: Ticket, addressed during sprint planning +``` + +### Runbook Template + +````markdown +# Runbook: [Alert Name] + +## Alert + +- **Name:** HighErrorBudgetBurn_Fast +- **Severity:** Critical +- **Dashboard:** [Link to Grafana dashboard] +- **SLO:** 99.9% availability, 30d rolling + +## What This Means + +The API error rate is consuming our error budget at 14.4x the sustainable rate. +At this burn rate, we'll exhaust the entire monthly budget within ~2 hours. + +## Impact + +- Users may see 5xx errors on API calls +- Dependent services (mobile app, dashboard) will be degraded + +## Diagnosis Steps + +1. **Check deployment history:** + ```bash + kubectl rollout history deployment/towerops-api -n production + # Was there a recent deploy? If so, rollback is likely the fix. + ``` + +2. **Check error logs:** + ```bash + kubectl logs -l app=towerops-api -n production --tail=50 | grep -i error + ``` + +3. **Check dependencies:** + ```bash + # Database + psql "$DATABASE_URL" -c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state;" + + # Redis + redis-cli -h redis ping + + # External APIs + curl -s -o /dev/null -w '%{http_code}' https://api.stripe.com/v1/charges -H "Authorization: Bearer $STRIPE_KEY" + ``` + +4. **Check resource usage:** + ```bash + kubectl top pods -n production -l app=towerops-api + ``` + +## Mitigation + +### If caused by a recent deployment: +```bash +kubectl rollout undo deployment/towerops-api -n production +kubectl rollout status deployment/towerops-api -n production +``` + +### If caused by traffic spike: +```bash +kubectl scale deployment/towerops-api -n production --replicas=10 +``` + +### If caused by database issues: +```bash +# Kill long-running queries +psql "$DATABASE_URL" -c "SELECT pg_cancel_backend(pid) FROM pg_stat_activity WHERE state = 'active' AND now() - query_start > interval '1 minute';" +``` + +### If caused by external dependency: +- Enable circuit breaker / fallback mode +- Update status page + +## Resolution Verification + +```bash +# Verify error rate is back to normal +curl -s "http://prometheus:9090/api/v1/query?query=sum(rate(http_requests_total{status_code=~\"5..\"}[5m]))/sum(rate(http_requests_total[5m]))" + +# Verify pods are healthy +kubectl get pods -n production -l app=towerops-api +``` + +## Escalation + +If not resolved within 30 minutes, escalate to the team lead. + +## Related + +- [Architecture diagram](link) +- [Previous incidents](link) +- [Service dependencies](link) +```` + +### Blameless Postmortem Template + +````markdown +# Postmortem: [Incident Title] + +**Date:** 2026-03-13 +**Duration:** 14:30 – 15:15 CDT (45 minutes) +**Severity:** SEV-2 +**Authors:** [Names of postmortem authors] +**Status:** Action items in progress + +## Summary + +Brief, 2-3 sentence description. What happened, who was impacted, how was it resolved. + +> A misconfigured database migration caused connection exhaustion in the PostgreSQL +> connection pool, resulting in 503 errors for 40% of API requests over 45 minutes. +> Resolved by rolling back the migration and restarting PgBouncer. + +## Impact + +- **Users affected:** ~2,400 (40% of active users) +- **Revenue impact:** ~$3,200 in failed transactions +- **Error budget consumed:** 8.3% of monthly budget +- **Support tickets:** 47 + +## Timeline (all times CDT) + +| Time | Event | +|------|-------| +| 14:28 | Deploy #847 rolls out (includes DB migration) | +| 14:30 | `HighErrorBudgetBurn_Fast` alert fires | +| 14:32 | On-call engineer (Alice) acknowledges alert | +| 14:35 | Alice identifies high 503 rate, checks recent deploys | +| 14:38 | Correlation with deploy #847 confirmed | +| 14:40 | Alice initiates rollback of deploy #847 | +| 14:42 | Rollback complete, but errors persist | +| 14:45 | Bob (secondary) joins, identifies PgBouncer pool exhaustion | +| 14:48 | PgBouncer restarted, connection pool drains | +| 14:52 | Error rate drops to normal | +| 15:00 | Monitoring confirms sustained recovery | +| 15:15 | Incident declared resolved | + +## Root Cause + +The migration added a `NOT NULL` constraint to a heavily-used column, which acquired +an `ACCESS EXCLUSIVE` lock on the `devices` table. This blocked all queries touching +the table, causing connection pool exhaustion as requests queued up waiting for the lock. + +The migration was tested in staging, but staging has ~1% of production data and the +lock was released in < 1 second. In production with 2.3M rows, the lock validation +took > 5 minutes. + +## Contributing Factors + +1. No lock timeout set for DDL operations in migration scripts +2. Staging data volume doesn't reflect production +3. Migration ran inside the application deploy, not as a separate step +4. PgBouncer didn't release connections after the migration was rolled back (stale pool state) + +## What Went Well + +- Alert fired within 2 minutes of impact starting +- On-call responded in under 5 minutes +- Rollback procedure worked as documented +- Team communicated clearly in #incidents channel + +## What Went Wrong + +- Migration wasn't tested against production-like data volume +- Rollback didn't fully resolve the issue (PgBouncer needed manual intervention) +- Status page wasn't updated until 15 minutes into the incident + +## Action Items + +| Action | Owner | Priority | Ticket | Due | +|--------|-------|----------|--------|-----| +| Add `SET lock_timeout = '5s'` to all migration scripts | Alice | P1 | OPS-234 | 2026-03-17 | +| Create production-sized staging dataset (anonymized) | Bob | P2 | OPS-235 | 2026-03-27 | +| Separate migration step from application deploy in CI/CD | Carol | P2 | OPS-236 | 2026-03-27 | +| Add PgBouncer pool drain to rollback runbook | Alice | P3 | OPS-237 | 2026-03-20 | +| Automate status page updates on SEV-1/2 alerts | Dave | P3 | OPS-238 | 2026-04-03 | + +## Lessons Learned + +1. **DDL operations on large tables need special handling.** Always use `SET lock_timeout`, + always test against production-scale data, always run migrations separately from deploys. + +2. **Connection poolers have state.** Rolling back the root cause doesn't always clear + downstream effects. Include dependent service restarts in rollback procedures. + +3. **Staging ≠ Production.** When the difference in data volume is 100x, some issues + will only manifest in production. Consider periodic production data snapshots for staging. + +--- + +*This postmortem is blameless. It focuses on systemic improvements, not individual fault. +The goal is to prevent recurrence, not assign blame.* +```` + +--- + +## Quick Reference Cards + +### The RED Method (Request-driven services) + +``` +R — Rate: Request throughput (req/s) +E — Errors: Failed requests (count or rate) +D — Duration: Request latency (histograms) +``` + +### The USE Method (Infrastructure resources) + +``` +U — Utilization: % time resource is busy +S — Saturation: Queue depth / backlog +E — Errors: Error events on the resource +``` + +### The Four Golden Signals (Google SRE) + +``` +1. Latency — Time to serve a request +2. Traffic — Demand on the system (req/s) +3. Errors — Rate of failed requests +4. Saturation — How "full" the system is +``` + +### Emergency Commands + +```bash +# === Kubernetes === +kubectl rollout undo deployment/NAME -n NAMESPACE # Rollback deploy +kubectl scale deployment/NAME --replicas=N -n NAMESPACE # Scale manually +kubectl delete pod NAME -n NAMESPACE # Force restart pod +kubectl cordon NODE # Stop scheduling +kubectl drain NODE --ignore-daemonsets # Evacuate node + +# === PostgreSQL === +# Kill all active queries on a table +SELECT pg_cancel_backend(pid) FROM pg_stat_activity +WHERE query LIKE '%tablename%' AND state = 'active'; + +# Kill all connections to a database +SELECT pg_terminate_backend(pid) FROM pg_stat_activity +WHERE datname = 'dbname' AND pid <> pg_backend_pid(); + +# Emergency VACUUM (when autovacuum is too slow) +VACUUM (VERBOSE, PARALLEL 4) tablename; + +# === System === +# Find what's eating disk +du -h --max-depth=2 / | sort -rh | head -20 + +# Find what's eating CPU +ps aux --sort=-%cpu | head -10 + +# Find what's eating memory +ps aux --sort=-%mem | head -10 + +# Network: who's connected +ss -tunlp # Listening ports +ss -tun state established # Active connections +``` + +--- + +*Last updated: 2026-03-13* +*Maintained by: TowerOps SRE Team* diff --git a/docs/references/ux-frontend-best-practices.md b/docs/references/ux-frontend-best-practices.md new file mode 100644 index 00000000..14bde186 --- /dev/null +++ b/docs/references/ux-frontend-best-practices.md @@ -0,0 +1,1992 @@ +# Modern UX & Front-End Design Best Practices + +> Reference guide for B2B SaaS dashboard applications, with emphasis on ISP/network monitoring tools. +> All examples use Tailwind CSS. Phoenix LiveView patterns included where relevant. + +--- + +## Table of Contents + +1. [Design Systems](#1-design-systems) +2. [Layout Patterns](#2-layout-patterns) +3. [Data Display](#3-data-display) +4. [Forms & Input](#4-forms--input) +5. [Navigation](#5-navigation) +6. [Accessibility](#6-accessibility) +7. [Animation & Transitions](#7-animation--transitions) +8. [Phoenix LiveView Specific](#8-phoenix-liveview-specific) +9. [Color & Typography](#9-color--typography) +10. [Anti-Patterns](#10-anti-patterns) + +--- + +## 1. Design Systems + +### Component Hierarchy + +Organize components in three tiers: + +| Tier | Examples | Characteristics | +|------|----------|-----------------| +| **Primitives** | Button, Badge, Input, Label | Stateless, no business logic, maximum reuse | +| **Composites** | Card, DataTable, FormGroup, Modal | Combine primitives, may have local state | +| **Features** | DevicePanel, AlertFeed, BandwidthChart | Domain-specific, connected to data | + +``` +components/ +├── ui/ # Primitives — button, badge, input, tooltip +├── composite/ # Composites — card, data_table, form_group +└── features/ # Features — device_panel, alert_feed +``` + +### Design Tokens + +Define tokens in `tailwind.config.js` — never use raw hex in templates: + +```js +// tailwind.config.js +module.exports = { + theme: { + extend: { + colors: { + // Semantic tokens + surface: { + DEFAULT: 'var(--color-surface)', // white / gray-900 + raised: 'var(--color-surface-raised)', // gray-50 / gray-800 + overlay: 'var(--color-surface-overlay)', // white / gray-800 + }, + border: { + DEFAULT: 'var(--color-border)', // gray-200 / gray-700 + strong: 'var(--color-border-strong)', // gray-300 / gray-600 + }, + status: { + healthy: '#10b981', // emerald-500 + warning: '#f59e0b', // amber-500 + critical: '#ef4444', // red-500 + unknown: '#6b7280', // gray-500 + maint: '#3b82f6', // blue-500 + }, + }, + spacing: { + 'sidebar': '16rem', // 256px collapsed + 'sidebar-sm': '4rem', // 64px collapsed + 'topbar': '4rem', // 64px + }, + fontSize: { + 'kpi': ['2rem', { lineHeight: '2.5rem', fontWeight: '700' }], + 'metric': ['1.5rem', { lineHeight: '2rem', fontWeight: '600' }], + }, + }, + }, +} +``` + +### Tailwind Best Practices + +**Avoid class soup — extract components, not utilities:** + +```html + + + + +<.button variant="primary" size="md">Save +``` + +**The component function:** + +```elixir +# core_components.ex +attr :variant, :string, default: "primary", values: ~w(primary secondary ghost danger) +attr :size, :string, default: "md", values: ~w(sm md lg) +attr :disabled, :boolean, default: false +slot :inner_block, required: true + +def button(assigns) do + ~H""" + + """ +end +``` + +### Dark Mode + +Use Tailwind's `dark:` variant with CSS custom properties for seamless theming: + +```css +/* app.css */ +:root { + --color-surface: theme('colors.white'); + --color-surface-raised: theme('colors.gray.50'); + --color-surface-overlay: theme('colors.white'); + --color-border: theme('colors.gray.200'); + --color-border-strong: theme('colors.gray.300'); + --color-text-primary: theme('colors.gray.900'); + --color-text-secondary: theme('colors.gray.500'); +} + +.dark { + --color-surface: theme('colors.gray.900'); + --color-surface-raised: theme('colors.gray.800'); + --color-surface-overlay: theme('colors.gray.800'); + --color-border: theme('colors.gray.700'); + --color-border-strong: theme('colors.gray.600'); + --color-text-primary: theme('colors.gray.100'); + --color-text-secondary: theme('colors.gray.400'); +} +``` + +**Toggle strategy:** + +```js +// Respect system preference, allow manual override +const theme = localStorage.getItem('theme'); +if (theme === 'dark' || (!theme && window.matchMedia('(prefers-color-scheme: dark)').matches)) { + document.documentElement.classList.add('dark'); +} +``` + +--- + +## 2. Layout Patterns + +### App Shell + +The canonical B2B SaaS layout: fixed sidebar + top bar + scrollable content area. + +```html +
    + + + + +
    + +
    + + + +
    + Network + / + Devices +
    + +
    + + + + +
    +
    + + +
    + +
    +
    +
    +``` + +### Collapsible Sidebar + +```html + + +``` + +### Mobile Sidebar (Slide-over) + +```html + +
    + +
    + + +
    +``` + +### Dashboard Grid + +Use CSS Grid with responsive breakpoints for KPI cards and widgets: + +```html + +
    +
    +

    Total Devices

    +

    1,247

    +

    + + +12 this week +

    +
    + +
    + + +
    + +
    +

    Bandwidth Usage (24h)

    +
    +
    + +
    +

    Recent Alerts

    + +
    +
    +``` + +### Card Component + +```html + +
    + +
    +

    Device Health

    + +
    + +
    + +
    + +
    +

    Last updated 2 min ago

    +
    +
    +``` + +### Skeleton Loading + +```html + +
    +
    +
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    + +
    +``` + +**Skeleton shimmer effect (optional CSS):** + +```css +@keyframes shimmer { + 0% { background-position: -200% 0; } + 100% { background-position: 200% 0; } +} +.skeleton-shimmer { + background: linear-gradient(90deg, + theme('colors.gray.200') 25%, + theme('colors.gray.100') 50%, + theme('colors.gray.200') 75% + ); + background-size: 200% 100%; + animation: shimmer 1.5s infinite; +} +.dark .skeleton-shimmer { + background: linear-gradient(90deg, + theme('colors.gray.700') 25%, + theme('colors.gray.600') 50%, + theme('colors.gray.700') 75% + ); +} +``` + +--- + +## 3. Data Display + +### Tables + +Tables are the backbone of network monitoring UIs. Get them right. + +```html +
    + +
    + +
    + + +
    + +
    + + +
    +
    + + + + + +
    + + + + + + + + + + + + + + + + + + + + + + + + +
    + + +
    + Hostname + +
    +     		IP AddressStatusUptimeLast Seen
    + + core-router-0110.0.1.1 + + + Online + + 142d 7h2 min ago + +
    +
    + + +
    +

    + Showing 1 to 25 + of 1,247 +

    +
    + + + + + ... + + +
    +
    +
    +``` + +**Table design rules:** + +- Left-align text, right-align numbers +- Use `font-mono text-xs` for IPs, MACs, serial numbers +- Status columns: colored badge with dot indicator +- Always provide a row hover state +- Sticky header on long tables: `sticky top-0 z-10` +- Avoid wrapping — use `truncate` and `max-w-*` with tooltips + +### Charts — Which Type When + +| Data Type | Chart | Example | +|-----------|-------|---------| +| **Time series** | Line/Area | Bandwidth, latency, CPU over time | +| **Composition** | Stacked area/bar | Traffic by protocol over time | +| **Comparison** | Horizontal bar | Top 10 talkers by bandwidth | +| **Distribution** | Histogram | Latency distribution | +| **Single value + trend** | KPI card + sparkline | Current throughput | +| **Part of whole** | Donut (never pie) | Device status breakdown | +| **Geographic** | Map with pins/heatmap | Tower locations, coverage | +| **Correlation** | Scatter plot | Packet loss vs. distance | + +**Chart guidelines for monitoring dashboards:** + +- Default to **24h** time range with **auto-refresh** +- Always show the y-axis scale and units (Mbps, ms, %) +- Use **area fills** with low opacity (`fill-opacity: 0.1`) for single-line charts +- Color thresholds: green < 70%, amber 70–90%, red > 90% +- Provide time range selector: 1h | 6h | 24h | 7d | 30d +- Tooltip should show exact timestamp + value +- Loading state: show axes + gray placeholder area + +### KPI Widget + +```html +
    +
    +

    Packet Loss

    + + +
    +
    +

    0.02%

    + + + + -0.01% + +
    +

    vs. 0.03% yesterday

    +
    +``` + +### Empty / Error / Loading States + +Every data view needs all three. **Never show a blank screen.** + +```html + +
    +
    + +
    +

    No devices found

    +

    + Add your first device to start monitoring your network. +

    + +
    + + +
    +
    + +
    +

    Failed to load devices

    +

    + Something went wrong. Please try again or contact support if the problem persists. +

    + +
    + + + +
    + + + + +
    +``` + +**State hierarchy:** +1. **Loading** → skeleton (content areas) or spinner (actions) +2. **Error** → message + retry action +3. **Empty** → illustration + explanation + primary action +4. **Data** → the actual content + +--- + +## 4. Forms & Input + +### Validation Patterns + +**Inline validation — validate on blur, show errors immediately:** + +```html +
    + + +

    + + Hostname must contain only lowercase letters, numbers, and hyphens. +

    +
    + + +
    + +
    + + +
    +
    +``` + +**Validation rules:** + +- Validate on blur (not on every keystroke) +- Show success states for complex fields (IP addresses, CIDR notation) +- Group related errors at the top of the form for long forms +- Never clear the user's input on error +- For LiveView: use `phx-change` with debounce for real-time validation + +### Radio Card Groups + +Perfect for selecting device types, plan tiers, or monitoring profiles: + +```html +
    + Device Type +
    + + + + + + + + +
    +
    +``` + +### Search & Filter Patterns + +**Combo filter bar:** + +```html +
    + + + Status: Online + + + + Type: Router + + + + +
    +``` + +### Auto-Save Indicator + +```html + +
    + + + Saving... + + + + Saved + + + + Failed to save + +
    +``` + +### Confirmation Dialogs + +Use for destructive actions only. **Never confirm routine saves.** + +```html + +
    +
    +
    + +
    + +
    +

    Delete Device

    +

    + Are you sure you want to delete core-router-01? + All monitoring data will be permanently removed. This action cannot be undone. +

    +
    +
    + + +
    +
    +
    +``` + +**Rules for confirmation dialogs:** + +- Destructive button on the **right**, cancel on the **left** +- Name the action specifically ("Delete Device" not "Are you sure?") +- Include the resource name in the message +- Red color for destructive action button +- For high-stakes actions, require typing the resource name + +--- + +## 5. Navigation + +### Breadcrumbs + +```html + +``` + +### Tabs + +```html +
    + +
    +``` + +### Command Palette (Cmd+K) + +Essential for power users in network monitoring tools: + +```html + +
    +
    + +
    + + + + + ESC +
    + + +
    + +

    + Devices +

    + +
    + +
    +

    core-router-01

    +

    10.0.1.1 · Router

    +
    + ↵ +
    + +
    + +
    +

    core-router-02

    +

    10.0.1.2 · Router

    +
    +
    + + +

    + Actions +

    +
    + +

    Add new device

    +
    + ⌘ + N +
    +
    +
    + + +
    + + ↑↓ Navigate + + + ↵ Open + + + ESC Close + +
    +
    +
    +``` + +### Notifications + +**Toast notifications** — for transient feedback: + +```html + +
    + + + + + +
    +``` + +**Toast rules:** +- Auto-dismiss success toasts after 5 seconds +- Error toasts persist until dismissed +- Max 3 visible toasts; queue the rest +- Position: bottom-right for desktop, top-center for mobile + +**Banner notifications** — for system-wide alerts: + +```html + +
    + +

    + Scheduled maintenance: + Core network upgrade on March 15 at 02:00 UTC. + Learn more +

    + +
    +``` + +**Notification badge:** + +```html + + +
    + + Alerts +
    + + 12 + +     +``` + +### Onboarding + +**Setup checklist:** + +```html +
    +
    +

    Getting Started

    + 2 of 4 complete +
    + +
    +
    +
    +
      + +
    • +
      + +
      + Create your account +
      • + +
    • +
      + +
      + Install the agent +
      • + +
    • +
      +
      +
      + + Add your first device + +
      • + +
    • +
      + Configure alerting +
      • +
    +
    +``` + +**Progressive disclosure:** Show complexity only when needed. + +```html + +
    + +
    + +
    +
    +``` + +--- + +## 6. Accessibility + +### ARIA Essentials + +```html + +
    + Device core-router-01 status changed to offline +
    + + + + + Online + + + + + + + + + + + +
    + + +
    + + +
    + + +
    +
    + +
    +``` + +### Keyboard Navigation + +| Component | Keys | Behavior | +|-----------|------|----------| +| **Modal** | `Escape` | Close | +| **Tabs** | `←` `→` | Switch tabs | +| **Dropdown** | `↑` `↓` `Enter` `Escape` | Navigate, select, close | +| **Table** | `Space` | Toggle row selection | +| **Command palette** | `⌘K` / `Ctrl+K` | Open | +| **Toast** | Auto-focus close button | Dismiss | + +### Focus Management + +```html + + + + + + + + Skip to main content + + + + + +``` + +### Color Contrast + +**WCAG 2.1 AA minimum contrast ratios:** + +| Element | Ratio | Example | +|---------|-------|---------| +| Normal text (< 18px) | **4.5:1** | `gray-600` on white = 5.4:1 ✅ | +| Large text (≥ 18px bold / 24px) | **3:1** | `gray-500` on white = 4.6:1 ✅ | +| UI components & graphics | **3:1** | Border, icons, form controls | + +**Don't rely on color alone:** +- ❌ Red text for errors, green text for success (color blind users lose info) +- ✅ Red text + error icon + descriptive text +- ✅ Status dot + text label ("Online", "Offline") +- ✅ Chart lines with different patterns/shapes, not just colors + +```html + + + + Critical + + + +``` + +--- + +## 7. Animation & Transitions + +### CSS Transitions + +Use for state changes, not entrance animations. Keep them fast (150–200ms). + +```html + + + +
    + +
    + + +