- New notification script (incidentio-notification.sh) that POSTs JSON
payloads to incident.io HTTP alert sources via curl/jq
- New Icinga2 config template defining User, NotificationCommand (host
+ service), and apply Notification rules — follows the PagerDuty
pattern with opt-in via vars.enable_incidentio
- Notification type mapping: PROBLEM/DOWN/CRITICAL/WARNING/UNKNOWN →
firing, RECOVERY/UP/OK → resolved, ACKNOWLEDGEMENT → firing with
metadata
- Deduplication keys: host-{name} for hosts, service-{host}-{service}
for services
- Deployed via Ansible: script copied to /etc/icinga2/scripts/,
config rendered to conf.d/; gated on icinga2_incidentio_token being
defined
- Add mcintire.me zone + all 14 DNS records to tofu/cloudflare.tf (A, MX, CNAME, SRV, TXT)
- Include zone settings, HSTS, bot management, firewall for mcintire.me
- Update Porkbun NS from as393837_ns to cloudflare_ns in tofu/dns.tf
- Remove mcintire.me zone from BIND9 master config (group_vars/bind9_servers/)
- Delete mcintire.me.yml zone file
- Remove mcintire.me secondary zone from git.mcintire.me host_vars
Add a telegraf role (applied to the resolvers) that runs unbound-control
via the inputs.unbound plugin and exposes the stats as a Prometheus
/metrics endpoint on :9273, firewalled to the 204.110.188.0/22 scrape
network. Imports the InfluxData signing key (influxdata-archive.key; the
_compat key expired 2026-01-17) so dnf can validate the package.
- prom.w5isp.com: scrape job "unbound" for resolver1/2:9273
- grafana: "Unbound Resolvers" dashboard (queries, cache hit ratio,
rcodes, query types, recursion time, request list, cache memory)
- resolvers: disable unused shm-enable to silence the startup warning
- Remove forgejo role, play, group, host_vars, and DNS references
- Replace hardcoded /Users/graham/.ssh/ paths with env var lookup
- Remove broken setup-user Makefile target
- Add community.mysql collection to requirements.yml
- Add .ansible/ to .gitignore
- git rm --cached .DS_Store files and .vscode/settings.json
- Update CLAUDE.md and README.md to reflect actual state
- Add findings.md with audit results
- new ansible loki role installs loki 3.5.5 on prometheus_servers as a
systemd unit (single-binary, filesystem store, 30d retention)
- open tcp/3100 on prom.w5isp.com to 10.0.0.0/16
- grafana: provision loki datasource at http://127.0.0.1:3100
- k8s: promtail daemonset + rbac in monitoring namespace, ships pod logs
to http://10.0.15.31:3100 with cluster=home-cluster label
- argocd: Application points at home/cluster/promtail
Codifies the bare-metal Icinga2 master + Icinga Web 2 stack on
monitor.vntx.net (Ubuntu 22.04, Apache + mod_php, MariaDB 10.6, Icinga
2.16, IDO-MySQL backend). Re-running the role against the live host
should produce a near-identical configuration.
Role includes:
- Icinga apt repo + full package set (icinga2, ido-mysql, icingaweb2,
monitoring-plugins, nagios-plugins-contrib, php + apache deps).
- MariaDB DB + user provisioning for both icinga2 (IDO daemon) and
icingaweb2 (UI), plus a separate read-only `icinga` user that the UI
uses to read the IDO. Schema bootstrapped only on first install.
- Templated icinga2 config (constants/zones/icinga2.conf), features-
enabled symlinks (api/checker/command/ido-mysql/mainlog), and the
three secret-bearing conf.d files (api-users, pagerduty x2) wired to
vault vars.
- 23 dormant conf.d/* host + service definitions copied verbatim under
files/. The live install has `include_recursive "conf.d"` commented
out (a node-setup CLI artifact); role mirrors that. Set
`icinga2_load_confd: true` in host_vars to rehydrate them.
- Icinga Web 2 ini files (config/auth/groups/roles/modules) + templated
resources.ini for DB credentials. monitoring module enabled via
symlink. icingaweb2 admin user bootstrapped on first install via a
one-shot insert into icingaweb_user (gated on schema-not-yet-imported).
- Apache rewrite + ssl modules. Vhost itself is left to certbot --apache
(Let's Encrypt cert is operator-managed, role doesn't manage it).
Cluster-wide cleanup:
- Gut roles/general/tasks/debian/icinga2.yml down to the uninstall
safety net (kept the sweep that removes icinga2 from non-monitoring
hosts). The previous master-side install + cert-distribution tasks
referenced templates that never existed in the role; the new monitor
role replaces them properly.
Operator action before first apply:
- Create host_vars/monitor.vntx.net/vault.yml with the seven required
secrets (see roles/monitor/README.md). Live values can be pulled from
/etc/icinga2/conf.d/{api-users,pagerduty-*,ido-mysql}.conf and
/etc/icingaweb2/resources.ini on the live host.
- Generate the icingaweb2 admin password hash via
`php -r 'echo password_hash("plaintext", PASSWORD_DEFAULT);'`.
- Run --check --diff before applying; the role is meant to be idempotent
but hasn't been verified against the live install.
The previous role was netbox-community/netbox-docker (compose-based) but
the live host runs the upstream tarball install directly: system Postgres
17, system Redis 8, gunicorn under systemd, and Caddy reverse-proxy.
Re-running the old role would have torn the working install down.
This rewrite mirrors what's actually deployed:
- defaults: NetBox 4.6.0, /opt/netbox layout, gunicorn 5w/3t/120s, all
configuration.py knobs exposed as ansible vars.
- tasks: install OS deps, ensure netbox user, create Postgres role + db,
download + extract release tarball, run upgrade.sh, render
configuration.py / gunicorn.py / systemd units, enable services.
- templates: configuration.py.j2 covers the same keys the live file sets;
netbox.service / netbox-rq.service are byte-equivalent to the live
units (modulo paths driven by the install-dir var).
- caddy role gets a per-host Caddyfile-netbox.vntx.net.j2 mirroring the
live vhost (25MB body limit, /static/* file_server, reverse_proxy to
127.0.0.1:8001).
- inventory: new netbox_servers group; netbox.vntx.net added to
caddy_servers so the per-host Caddyfile is wired up.
- host_vars/netbox.vntx.net.yml: secrets reference vault_* vars; role
refuses to render config.py until they're set.
Operator action: create host_vars/netbox.vntx.net/vault.yml with
netbox_secret_key, netbox_db_password, and netbox_api_token_peppers
(value visible via `sudo cat /opt/netbox/netbox/netbox/configuration.py`
on the live host). See roles/netbox/README.md.
Provisions prom.w5isp.com (10.0.15.31) as the home-cluster monitoring host.
Prometheus 3.6.0 scrapes the Talos cluster API/nodes/cAdvisor via an external
SA token; Grafana ships with the Prometheus datasource pre-provisioned.
host_vars opens 9090/9093/9100/3000 to 10.0.0.0/16 plus SSH from the home LAN
so the firewall role doesn't lock out admin access.
