Add vars.enable_incidentio = true to every host template (generic-host,
server, mikrotik, ap-ubnt, netonix, edgeswitch, ap-epmp, af11, af24)
and the generic-service template — all ~80 hosts and their services
are now covered via template inheritance.
- New notification script (incidentio-notification.sh) that POSTs JSON
payloads to incident.io HTTP alert sources via curl/jq
- New Icinga2 config template defining User, NotificationCommand (host
+ service), and apply Notification rules — follows the PagerDuty
pattern with opt-in via vars.enable_incidentio
- Notification type mapping: PROBLEM/DOWN/CRITICAL/WARNING/UNKNOWN →
firing, RECOVERY/UP/OK → resolved, ACKNOWLEDGEMENT → firing with
metadata
- Deduplication keys: host-{name} for hosts, service-{host}-{service}
for services
- Deployed via Ansible: script copied to /etc/icinga2/scripts/,
config rendered to conf.d/; gated on icinga2_incidentio_token being
defined
- forgejo role: tasks no longer wipe data on subsequent runs (check for existing
data before resetting)
- runner: use data.forgejo.org/forgejo/runner:12, register with --no-interactive,
run as root for Docker socket access
- runner-config: proper generate-config format
- graham user: password hash in managed_users config
- SSH key added to Forgejo graham user
- direnv in base Debian packages
- so-reuseport: yes — kernel hashes UDP across the per-thread sockets
instead of waking all threads, balancing load and improving headroom
- edns-buffer-size: 1232 — DNS Flag Day 2020 default, avoids IP
fragmentation (retransmits/TCP fallback)
- serve-expired (RFC 8767) with 1800ms client timeout and 86400s stale
ttl — fast answers from stale cache during upstream slowness/outages
The packaged telegraf.service sets PrivateMounts=true, which requires a
mount namespace that unprivileged LXC containers cannot create, so the
service failed to start on the LXC resolver with status 226/NAMESPACE.
Drop in PrivateMounts=false, gated to container guests so VM hosts keep
the unit's hardening.
Add a telegraf role (applied to the resolvers) that runs unbound-control
via the inputs.unbound plugin and exposes the stats as a Prometheus
/metrics endpoint on :9273, firewalled to the 204.110.188.0/22 scrape
network. Imports the InfluxData signing key (influxdata-archive.key; the
_compat key expired 2026-01-17) so dnf can validate the package.
- prom.w5isp.com: scrape job "unbound" for resolver1/2:9273
- grafana: "Unbound Resolvers" dashboard (queries, cache hit ratio,
rcodes, query types, recursion time, request list, cache memory)
- resolvers: disable unused shm-enable to silence the startup warning
The 'Validate sshd configuration' task had ansible.builtin.command with no
command, so it always errored (one of the following is required:
_raw_params, cmd, argv), tripping the rescue that reverts the drop-in. The
post-quantum KexAlgorithms hardening therefore never applied. Run sshd -t
to validate before restart as intended.
Add an RPZ blocklist on the Unbound resolvers that returns NXDOMAIN for a
set of domains (and subdomains) and logs only those matches under rpz-log
tag "blocklist". Domains are templated from resolver_blocked_domains
(single source of truth) into /etc/unbound/blocklist.rpz; respip module
enabled for RPZ support.
Fix deprecation warnings across the Ansible tree:
- apt_repository -> deb822_repository (monitor, aprsc, uisp, grafana),
removing stale .list files and adding update_cache where deb822 drops it
- community.mysql.* -> ansible.mysql.* (monitor) and requirements.yml
- top-level facts -> ansible_facts[...] (ansible_os_family,
distribution_release, fqdn, architecture, default_ipv4) repo-wide
- Remove duplicate firewall include_role from base role (#18)
- Move remote_user from [ssh_connection] to [defaults] in ansible.cfg (#20)
- Add set -euo pipefail to run.sh (#21)
- Delete orphaned playbook_handlers.yml (#22)
- Fix changed_when: true in debian tasks (#23)
- Add tailscale_authkey env var lookup with TAILSCALE_KEY fallback (#25)
- Document -K requirement in bootstrap.yml (#26)
- Replace bare ansible_host=mail/git with IP addresses (#27)
- Update findings.md
- Add |default() fallbacks for ntpserver, template_run_date,
deb_mirror, and sshd.* to prevent template crash
- Remove include_role: name=alpine from base tasks (no such role)
- Fix SSH -t flag placement in Makefile reboot targets
- Update findings.md
- Convert ~290 short module names to FQCNs across all roles
- Change become: yes/no to become: true/false, gather_facts: no to false
- Remove deprecated inject_facts_as_vars from ansible.cfg
- Add missing task names to import_tasks/include_role calls
- Add become: yes to librenms clone task (partial-become fix)
- Add pipefail to risky shell command in udp-gro-fix.yml
- Quote all octal mode values
- Harden .ansible-lint with production profile and empty skip_list
- Update findings.md with all fixes
- Remove forgejo role, play, group, host_vars, and DNS references
- Replace hardcoded /Users/graham/.ssh/ paths with env var lookup
- Remove broken setup-user Makefile target
- Add community.mysql collection to requirements.yml
- Add .ansible/ to .gitignore
- git rm --cached .DS_Store files and .vscode/settings.json
- Update CLAUDE.md and README.md to reflect actual state
- Add findings.md with audit results
- unbound: add loopback DoH listener (127.0.0.1@8080, http-notls-downstream);
interface-automatic-ports lists 8080 so interface-automatic does not silently
drop the non-53 listener; access-control driven by resolver_allowed_netblocks
- caddy: add RedHat/COPR install path, caddy_template var, /var/log/caddy dir;
new Caddyfile-resolver.j2 terminates LE TLS and h2c-proxies to unbound,
enforcing the client allow-list via remote_ip
- resolvers: open 80/443 in firewall; prefer post-quantum SSH key exchange
(sntrup761x25519-sha512) via validated sshd_config.d drop-in
- bind9: zone.j2 emits AAAA records; add ipv6 for resolver1/2 (2606:1c80::240/250)
- playbook: run caddy role on the resolvers group
Listen on ::0, set do-ip6: yes for IPv6 upstream resolution, and add
IPv6 access-control: default-deny ::0/0, allow ::1 and the VNTX
2606:1c80::/32 allocation.
Without this, a renamed dashboard left the old file on disk and the
provisioner refused to write any dashboards because of a duplicate UID
(seen with towerops-phoenix-live-view vs towerops-phoenix_live_view).
- new ansible loki role installs loki 3.5.5 on prometheus_servers as a
systemd unit (single-binary, filesystem store, 30d retention)
- open tcp/3100 on prom.w5isp.com to 10.0.0.0/16
- grafana: provision loki datasource at http://127.0.0.1:3100
- k8s: promtail daemonset + rbac in monitoring namespace, ships pod logs
to http://10.0.15.31:3100 with cluster=home-cluster label
- argocd: Application points at home/cluster/promtail
The dashboards referenced `towerops_prom_ex_status_info` but PromEx exports
this as `towerops_prom_ex_prom_ex_status_info`. The template job dropdown
was empty because Grafana couldn't find any job values for the wrong metric.
joshuasing/starlink_exporter 0.9.0 (Docker Hub; ghcr.io lags behind
upstream releases). Targets the dish's gRPC management interface at
192.168.100.1:9200, reachable from cluster pods via 10.0.15.254 while
the Starlink router is in bypass mode.
Grafana dashboard "Starlink" covers throughput, PoP ping latency/drop,
obstruction state + 24h total, GPS lock, and dish position/info — synced
to /var/lib/grafana/dashboards/ by the existing grafana role.
Replaces the vendored Go exporter with two pre-built upstream images:
- quay.io/prometheuscommunity/json-exporter:v0.7.0 maps the gateway JSON
into prometheus metrics via JSONPath. The G4AR keys signal blocks as
"5g" / "4g" (digit-prefixed), which the k8s client-go JSONPath dialect
only reaches via bracket-with-single-quote notation.
- nginxinc/nginx-unprivileged:1.27-alpine-slim sidecar rewrites
GET /metrics → GET /probe?module=tmhi&target=http://192.168.12.1/...
so the existing kubernetes-services scrape job picks it up over the
apiserver proxy with no Prometheus-side config changes.
Avoiding caddy here: its binary carries cap_net_bind_service file caps,
which the kernel refuses to exec under no_new_privs as a non-root user.
Dashboard "Gateway up" panel now reads up{service="tmobile-exporter"}
(Prometheus's built-in scrape-success gauge) since json_exporter's
/probe endpoint does not emit probe_success.
15 panels covering gateway up/uptime, 4G/5G bars, and per-band
RSRP/RSRQ/SNR/RSSI time series. Sources metrics from the new
tmobile-exporter (job=tmobile_gateway).
