Add a telegraf role (applied to the resolvers) that runs unbound-control
via the inputs.unbound plugin and exposes the stats as a Prometheus
/metrics endpoint on :9273, firewalled to the 204.110.188.0/22 scrape
network. Imports the InfluxData signing key (influxdata-archive.key; the
_compat key expired 2026-01-17) so dnf can validate the package.
- prom.w5isp.com: scrape job "unbound" for resolver1/2:9273
- grafana: "Unbound Resolvers" dashboard (queries, cache hit ratio,
rcodes, query types, recursion time, request list, cache memory)
- resolvers: disable unused shm-enable to silence the startup warning
Add an RPZ blocklist on the Unbound resolvers that returns NXDOMAIN for a
set of domains (and subdomains) and logs only those matches under rpz-log
tag "blocklist". Domains are templated from resolver_blocked_domains
(single source of truth) into /etc/unbound/blocklist.rpz; respip module
enabled for RPZ support.
Fix deprecation warnings across the Ansible tree:
- apt_repository -> deb822_repository (monitor, aprsc, uisp, grafana),
removing stale .list files and adding update_cache where deb822 drops it
- community.mysql.* -> ansible.mysql.* (monitor) and requirements.yml
- top-level facts -> ansible_facts[...] (ansible_os_family,
distribution_release, fqdn, architecture, default_ipv4) repo-wide
Without this, a renamed dashboard left the old file on disk and the
provisioner refused to write any dashboards because of a duplicate UID
(seen with towerops-phoenix-live-view vs towerops-phoenix_live_view).
- new ansible loki role installs loki 3.5.5 on prometheus_servers as a
systemd unit (single-binary, filesystem store, 30d retention)
- open tcp/3100 on prom.w5isp.com to 10.0.0.0/16
- grafana: provision loki datasource at http://127.0.0.1:3100
- k8s: promtail daemonset + rbac in monitoring namespace, ships pod logs
to http://10.0.15.31:3100 with cluster=home-cluster label
- argocd: Application points at home/cluster/promtail
The dashboards referenced `towerops_prom_ex_status_info` but PromEx exports
this as `towerops_prom_ex_prom_ex_status_info`. The template job dropdown
was empty because Grafana couldn't find any job values for the wrong metric.
joshuasing/starlink_exporter 0.9.0 (Docker Hub; ghcr.io lags behind
upstream releases). Targets the dish's gRPC management interface at
192.168.100.1:9200, reachable from cluster pods via 10.0.15.254 while
the Starlink router is in bypass mode.
Grafana dashboard "Starlink" covers throughput, PoP ping latency/drop,
obstruction state + 24h total, GPS lock, and dish position/info — synced
to /var/lib/grafana/dashboards/ by the existing grafana role.
Replaces the vendored Go exporter with two pre-built upstream images:
- quay.io/prometheuscommunity/json-exporter:v0.7.0 maps the gateway JSON
into prometheus metrics via JSONPath. The G4AR keys signal blocks as
"5g" / "4g" (digit-prefixed), which the k8s client-go JSONPath dialect
only reaches via bracket-with-single-quote notation.
- nginxinc/nginx-unprivileged:1.27-alpine-slim sidecar rewrites
GET /metrics → GET /probe?module=tmhi&target=http://192.168.12.1/...
so the existing kubernetes-services scrape job picks it up over the
apiserver proxy with no Prometheus-side config changes.
Avoiding caddy here: its binary carries cap_net_bind_service file caps,
which the kernel refuses to exec under no_new_privs as a non-root user.
Dashboard "Gateway up" panel now reads up{service="tmobile-exporter"}
(Prometheus's built-in scrape-success gauge) since json_exporter's
/probe endpoint does not emit probe_success.
15 panels covering gateway up/uptime, 4G/5G bars, and per-band
RSRP/RSRQ/SNR/RSSI time series. Sources metrics from the new
tmobile-exporter (job=tmobile_gateway).
- grafana role: empty grafana_admin_password by default, and only emit
admin_password into grafana.ini when the var is set. Previously the
default 'admin' was rewritten on every run, clobbering passwords
rotated via the Grafana UI.
- bootstrap.yml: probe for sudo state before attempting su-as-root.
The first play was using become_method=su unconditionally; on hosts
where graham already has passwordless sudo (or where -K isn't passed),
ansible blocked indefinitely waiting for a root password prompt.
Now the install-via-su task only runs when sudo is actually missing,
and the python3 task uses sudo when available.
Also bring in latest microwaveprop dashboard tweaks.
Dashboards: aprs-me, towerops-{application,beam,ecto,oban,phoenix,phoenix-live-view}.
prometheus.yml.j2: kubernetes-pods job now honors a `prometheus.io/job`
pod annotation so apps can claim a named job label (e.g. towerops)
instead of bucketing under job=kubernetes-pods.
Cluster-side:
- Deploy kube-state-metrics (kube-system) for cluster-state series.
- Deploy node-exporter DaemonSet on all 7 nodes via the new monitoring ns
(privileged PSS, hostNetwork, read-only host mounts).
- Both managed by ArgoCD apps under home/cluster/argocd/apps/.
Prometheus-side:
- Replace direct pod-IP scraping (which can't reach the 10.244.0.0/16 pod
network from outside the cluster) with API-server-proxy scraping for
both Services and Pods. Now one target per pod, all healthy.
- Extend external Prometheus RBAC with services/proxy + pods/proxy.
Grafana-side:
- Add file-based dashboard provisioner; ship 5 standard dashboards
(Node Exporter Full, k8s cluster-via-prometheus, k8s all-in-one,
k8s monitoring, Prometheus 2.0 overview).
prom.w5isp.com:3000 now shows live data for all 7 cluster nodes plus
the prom host itself.
Provisions prom.w5isp.com (10.0.15.31) as the home-cluster monitoring host.
Prometheus 3.6.0 scrapes the Talos cluster API/nodes/cAdvisor via an external
SA token; Grafana ships with the Prometheus datasource pre-provisioned.
host_vars opens 9090/9093/9100/3000 to 10.0.0.0/16 plus SSH from the home LAN
so the firewall role doesn't lock out admin access.
