Add vars.enable_incidentio = true to every host template (generic-host,
server, mikrotik, ap-ubnt, netonix, edgeswitch, ap-epmp, af11, af24)
and the generic-service template — all ~80 hosts and their services
are now covered via template inheritance.
- New notification script (incidentio-notification.sh) that POSTs JSON
payloads to incident.io HTTP alert sources via curl/jq
- New Icinga2 config template defining User, NotificationCommand (host
+ service), and apply Notification rules — follows the PagerDuty
pattern with opt-in via vars.enable_incidentio
- Notification type mapping: PROBLEM/DOWN/CRITICAL/WARNING/UNKNOWN →
firing, RECOVERY/UP/OK → resolved, ACKNOWLEDGEMENT → firing with
metadata
- Deduplication keys: host-{name} for hosts, service-{host}-{service}
for services
- Deployed via Ansible: script copied to /etc/icinga2/scripts/,
config rendered to conf.d/; gated on icinga2_incidentio_token being
defined
- forgejo role: tasks no longer wipe data on subsequent runs (check for existing
data before resetting)
- runner: use data.forgejo.org/forgejo/runner:12, register with --no-interactive,
run as root for Docker socket access
- runner-config: proper generate-config format
- graham user: password hash in managed_users config
- SSH key added to Forgejo graham user
- direnv in base Debian packages
Butane unit dropins write to /etc/systemd/system/<unit>.d/, but
DNSStubListener is a systemd-resolved config option that belongs in
/etc/systemd/resolved.conf.d/. Use storage.files instead so Ignition
places it in the correct location.
systemd-resolved must run for host DNS (nss/resolv.conf), but its stub
listener on 127.0.0.53:53 conflicts with Unbound's bind to 0.0.0.0:53.
Replace masking with DNSStubListener=no dropin + static resolv.conf
pointing to 1.1.1.1 and 9.9.9.9, so the host can resolve without
occupying port 53.
- Add mcintire.me zone + all 14 DNS records to tofu/cloudflare.tf (A, MX, CNAME, SRV, TXT)
- Include zone settings, HSTS, bot management, firewall for mcintire.me
- Update Porkbun NS from as393837_ns to cloudflare_ns in tofu/dns.tf
- Remove mcintire.me zone from BIND9 master config (group_vars/bind9_servers/)
- Delete mcintire.me.yml zone file
- Remove mcintire.me secondary zone from git.mcintire.me host_vars
- imagePullPolicy: Always so :latest actually tracks upstream (nodes were
running two different cached digests)
- --retries 1: supervisor retries forever; this caps the jittered backoff
at ~2s instead of the exponential 1..32s ramp
- liveness failureThreshold 30: let cloudflared's own fast retry loop
reconnect instead of killing pods into exponential CrashLoopBackOff
(pods had 300+ restarts)
- maxSurge 0 / maxUnavailable 1: required anti-affinity on 3 workers
deadlocked surge-based rollouts
- securityContext for restricted:latest PodSecurity (nonroot 65532,
drop ALL caps, no privilege escalation, RuntimeDefault seccomp)
Co-Authored-By: Claude <noreply@anthropic.com>
- so-reuseport: yes — kernel hashes UDP across the per-thread sockets
instead of waking all threads, balancing load and improving headroom
- edns-buffer-size: 1232 — DNS Flag Day 2020 default, avoids IP
fragmentation (retransmits/TCP fallback)
- serve-expired (RFC 8767) with 1800ms client timeout and 86400s stale
ttl — fast answers from stale cache during upstream slowness/outages
The packaged telegraf.service sets PrivateMounts=true, which requires a
mount namespace that unprivileged LXC containers cannot create, so the
service failed to start on the LXC resolver with status 226/NAMESPACE.
Drop in PrivateMounts=false, gated to container guests so VM hosts keep
the unit's hardening.
Add a telegraf role (applied to the resolvers) that runs unbound-control
via the inputs.unbound plugin and exposes the stats as a Prometheus
/metrics endpoint on :9273, firewalled to the 204.110.188.0/22 scrape
network. Imports the InfluxData signing key (influxdata-archive.key; the
_compat key expired 2026-01-17) so dnf can validate the package.
- prom.w5isp.com: scrape job "unbound" for resolver1/2:9273
- grafana: "Unbound Resolvers" dashboard (queries, cache hit ratio,
rcodes, query types, recursion time, request list, cache memory)
- resolvers: disable unused shm-enable to silence the startup warning
The 'Validate sshd configuration' task had ansible.builtin.command with no
command, so it always errored (one of the following is required:
_raw_params, cmd, argv), tripping the rescue that reverts the drop-in. The
post-quantum KexAlgorithms hardening therefore never applied. Run sshd -t
to validate before restart as intended.
Add an RPZ blocklist on the Unbound resolvers that returns NXDOMAIN for a
set of domains (and subdomains) and logs only those matches under rpz-log
tag "blocklist". Domains are templated from resolver_blocked_domains
(single source of truth) into /etc/unbound/blocklist.rpz; respip module
enabled for RPZ support.
Fix deprecation warnings across the Ansible tree:
- apt_repository -> deb822_repository (monitor, aprsc, uisp, grafana),
removing stale .list files and adding update_cache where deb822 drops it
- community.mysql.* -> ansible.mysql.* (monitor) and requirements.yml
- top-level facts -> ansible_facts[...] (ansible_os_family,
distribution_release, fqdn, architecture, default_ipv4) repo-wide
- Add cloudflare_bot_management for all zones with is_robots_txt_managed=false
so each app's own robots.txt is served instead of Cloudflare's managed one
- Fix zone_setting id zero_rtt -> 0rtt (invalid id rejected by API)
- Fix security_header value: object instead of jsonencode (avoids perpetual diff)
- Fix firewall expression http.request.user_agent -> http.user_agent (invalid field)
- tofu fmt reindentation
