- new ansible loki role installs loki 3.5.5 on prometheus_servers as a
systemd unit (single-binary, filesystem store, 30d retention)
- open tcp/3100 on prom.w5isp.com to 10.0.0.0/16
- grafana: provision loki datasource at http://127.0.0.1:3100
- k8s: promtail daemonset + rbac in monitoring namespace, ships pod logs
to http://10.0.15.31:3100 with cluster=home-cluster label
- argocd: Application points at home/cluster/promtail
The dashboards referenced `towerops_prom_ex_status_info` but PromEx exports
this as `towerops_prom_ex_prom_ex_status_info`. The template job dropdown
was empty because Grafana couldn't find any job values for the wrong metric.
joshuasing/starlink_exporter 0.9.0 (Docker Hub; ghcr.io lags behind
upstream releases). Targets the dish's gRPC management interface at
192.168.100.1:9200, reachable from cluster pods via 10.0.15.254 while
the Starlink router is in bypass mode.
Grafana dashboard "Starlink" covers throughput, PoP ping latency/drop,
obstruction state + 24h total, GPS lock, and dish position/info — synced
to /var/lib/grafana/dashboards/ by the existing grafana role.
Replaces the vendored Go exporter with two pre-built upstream images:
- quay.io/prometheuscommunity/json-exporter:v0.7.0 maps the gateway JSON
into prometheus metrics via JSONPath. The G4AR keys signal blocks as
"5g" / "4g" (digit-prefixed), which the k8s client-go JSONPath dialect
only reaches via bracket-with-single-quote notation.
- nginxinc/nginx-unprivileged:1.27-alpine-slim sidecar rewrites
GET /metrics → GET /probe?module=tmhi&target=http://192.168.12.1/...
so the existing kubernetes-services scrape job picks it up over the
apiserver proxy with no Prometheus-side config changes.
Avoiding caddy here: its binary carries cap_net_bind_service file caps,
which the kernel refuses to exec under no_new_privs as a non-root user.
Dashboard "Gateway up" panel now reads up{service="tmobile-exporter"}
(Prometheus's built-in scrape-success gauge) since json_exporter's
/probe endpoint does not emit probe_success.
Replaces tpaulus/tmobile-internet-exporter upstream image, which crashes
on every scrape against the G4AR because the gateway returns `bars` as a
JSON float (`4.0`) and the upstream struct types it as int.
The vendored exporter:
- decodes all radio numerics as float64 so `4.0` round-trips
- treats the 4g / 5g signal blocks as optional (the G4AR omits 4g)
- evicts stale (type, band) series when a band disappears between scrapes
- adds a scrape_errors_total counter and /healthz endpoint
Deployment now pulls git.mcintire.me/graham/tmobile-exporter:v1 via the
forgejo-registry imagePullSecret (matching the convention used by aprs,
prop, towerops). Build + push the image before Argo can pull it; the
imagePullSecret must also be copied into the monitoring namespace.
15 panels covering gateway up/uptime, 4G/5G bars, and per-band
RSRP/RSRQ/SNR/RSSI time series. Sources metrics from the new
tmobile-exporter (job=tmobile_gateway).
Deploys tpaulus/tmobile-internet-exporter against the G4AR at
192.168.12.1, picked up by the existing kubernetes-services scrape
job via prometheus.io/scrape annotations on the Service.
Codifies the bare-metal Icinga2 master + Icinga Web 2 stack on
monitor.vntx.net (Ubuntu 22.04, Apache + mod_php, MariaDB 10.6, Icinga
2.16, IDO-MySQL backend). Re-running the role against the live host
should produce a near-identical configuration.
Role includes:
- Icinga apt repo + full package set (icinga2, ido-mysql, icingaweb2,
monitoring-plugins, nagios-plugins-contrib, php + apache deps).
- MariaDB DB + user provisioning for both icinga2 (IDO daemon) and
icingaweb2 (UI), plus a separate read-only `icinga` user that the UI
uses to read the IDO. Schema bootstrapped only on first install.
- Templated icinga2 config (constants/zones/icinga2.conf), features-
enabled symlinks (api/checker/command/ido-mysql/mainlog), and the
three secret-bearing conf.d files (api-users, pagerduty x2) wired to
vault vars.
- 23 dormant conf.d/* host + service definitions copied verbatim under
files/. The live install has `include_recursive "conf.d"` commented
out (a node-setup CLI artifact); role mirrors that. Set
`icinga2_load_confd: true` in host_vars to rehydrate them.
- Icinga Web 2 ini files (config/auth/groups/roles/modules) + templated
resources.ini for DB credentials. monitoring module enabled via
symlink. icingaweb2 admin user bootstrapped on first install via a
one-shot insert into icingaweb_user (gated on schema-not-yet-imported).
- Apache rewrite + ssl modules. Vhost itself is left to certbot --apache
(Let's Encrypt cert is operator-managed, role doesn't manage it).
Cluster-wide cleanup:
- Gut roles/general/tasks/debian/icinga2.yml down to the uninstall
safety net (kept the sweep that removes icinga2 from non-monitoring
hosts). The previous master-side install + cert-distribution tasks
referenced templates that never existed in the role; the new monitor
role replaces them properly.
Operator action before first apply:
- Create host_vars/monitor.vntx.net/vault.yml with the seven required
secrets (see roles/monitor/README.md). Live values can be pulled from
/etc/icinga2/conf.d/{api-users,pagerduty-*,ido-mysql}.conf and
/etc/icingaweb2/resources.ini on the live host.
- Generate the icingaweb2 admin password hash via
`php -r 'echo password_hash("plaintext", PASSWORD_DEFAULT);'`.
- Run --check --diff before applying; the role is meant to be idempotent
but hasn't been verified against the live install.
The previous role was netbox-community/netbox-docker (compose-based) but
the live host runs the upstream tarball install directly: system Postgres
17, system Redis 8, gunicorn under systemd, and Caddy reverse-proxy.
Re-running the old role would have torn the working install down.
This rewrite mirrors what's actually deployed:
- defaults: NetBox 4.6.0, /opt/netbox layout, gunicorn 5w/3t/120s, all
configuration.py knobs exposed as ansible vars.
- tasks: install OS deps, ensure netbox user, create Postgres role + db,
download + extract release tarball, run upgrade.sh, render
configuration.py / gunicorn.py / systemd units, enable services.
- templates: configuration.py.j2 covers the same keys the live file sets;
netbox.service / netbox-rq.service are byte-equivalent to the live
units (modulo paths driven by the install-dir var).
- caddy role gets a per-host Caddyfile-netbox.vntx.net.j2 mirroring the
live vhost (25MB body limit, /static/* file_server, reverse_proxy to
127.0.0.1:8001).
- inventory: new netbox_servers group; netbox.vntx.net added to
caddy_servers so the per-host Caddyfile is wired up.
- host_vars/netbox.vntx.net.yml: secrets reference vault_* vars; role
refuses to render config.py until they're set.
Operator action: create host_vars/netbox.vntx.net/vault.yml with
netbox_secret_key, netbox_db_password, and netbox_api_token_peppers
(value visible via `sudo cat /opt/netbox/netbox/netbox/configuration.py`
on the live host). See roles/netbox/README.md.
caddy_ansible.caddy_ansible was only referenced by the (commented) netbox play,
so removing the disabled play also makes the role unused. ns play was already
superseded by the BIND9 play. Tailscale-on-home-hosts was never re-enabled
post-Talos-migration — drop the stub; restore from git history if needed.
- grafana role: empty grafana_admin_password by default, and only emit
admin_password into grafana.ini when the var is set. Previously the
default 'admin' was rewritten on every run, clobbering passwords
rotated via the Grafana UI.
- bootstrap.yml: probe for sudo state before attempting su-as-root.
The first play was using become_method=su unconditionally; on hosts
where graham already has passwordless sudo (or where -K isn't passed),
ansible blocked indefinitely waiting for a root password prompt.
Now the install-via-su task only runs when sudo is actually missing,
and the python3 task uses sudo when available.
Also bring in latest microwaveprop dashboard tweaks.
Dashboards: aprs-me, towerops-{application,beam,ecto,oban,phoenix,phoenix-live-view}.
prometheus.yml.j2: kubernetes-pods job now honors a `prometheus.io/job`
pod annotation so apps can claim a named job label (e.g. towerops)
instead of bucketing under job=kubernetes-pods.
Cluster-side:
- Deploy kube-state-metrics (kube-system) for cluster-state series.
- Deploy node-exporter DaemonSet on all 7 nodes via the new monitoring ns
(privileged PSS, hostNetwork, read-only host mounts).
- Both managed by ArgoCD apps under home/cluster/argocd/apps/.
Prometheus-side:
- Replace direct pod-IP scraping (which can't reach the 10.244.0.0/16 pod
network from outside the cluster) with API-server-proxy scraping for
both Services and Pods. Now one target per pod, all healthy.
- Extend external Prometheus RBAC with services/proxy + pods/proxy.
Grafana-side:
- Add file-based dashboard provisioner; ship 5 standard dashboards
(Node Exporter Full, k8s cluster-via-prometheus, k8s all-in-one,
k8s monitoring, Prometheus 2.0 overview).
prom.w5isp.com:3000 now shows live data for all 7 cluster nodes plus
the prom host itself.
Provisions prom.w5isp.com (10.0.15.31) as the home-cluster monitoring host.
Prometheus 3.6.0 scrapes the Talos cluster API/nodes/cAdvisor via an external
SA token; Grafana ships with the Prometheus datasource pre-provisioned.
host_vars opens 9090/9093/9100/3000 to 10.0.0.0/16 plus SSH from the home LAN
so the firewall role doesn't lock out admin access.
The towerops-web repo got renamed to codeberg.org/gmcintire/towerops
and its container image follows the same rename
(codeberg.org/gmcintire/towerops). Application source repo and
image-updater registry list updated.
The prop and prop-grid-rs container images now live on
codeberg.org/gmcintire/* (see prop repo commit 8a9c0ba6 for the
workflow flip). Application source repo and image-updater registry
list follow.
Cluster-side state still required for this to roll out, applied
out-of-band via kubectl:
- Secret prop/forgejo-registry — dockerconfigjson with codeberg.org
auth (also keeps git.mcintire.me auth during the transition).
- Secret argocd/forgejo-registry — same shape, used by the
argocd-image-updater registry config.
- Secret argocd/codeberg-credentials-gmcintire (label
argocd.argoproj.io/secret-type=repo-creds) — credentials template
matching url=https://codeberg.org/gmcintire so future codeberg
repos under that owner pick up auth automatically.
- ConfigMap argocd/argocd-image-updater-config — registries.conf
extended with codeberg.org entry referencing the new pull secret.
annotates the 4 Applications so image-updater scans git.mcintire.me
on a 2min loop and writes new tags to spec.source.kustomize.images
(no git commits, no CI sed-and-bump).
Configure passthrough NVMe drives (/dev/sdb) on each worker with
XFS partition mounted at /var/mnt/longhorn. Switch worker installer
image to Talos Image Factory build with iscsi-tools and util-linux-tools
extensions required by Longhorn. Remove conflicting grubUseUKICmdline
setting.
- Upgrade all nodes from v1.12.1/v1.12.2 to v1.12.4 installer image
- Set explicit nameservers (9.9.9.9, 149.112.112.112, 1.1.1.1) on
workers to replace unreliable DHCP-provided resolver at 10.0.15.254
- Fix talos-worker2 missing ExtensionServiceConfig for ext-tailscale
which was causing node reboots every ~35 minutes
- Update tailscale.patch.yaml with working auth key
