- new ansible loki role installs loki 3.5.5 on prometheus_servers as a
systemd unit (single-binary, filesystem store, 30d retention)
- open tcp/3100 on prom.w5isp.com to 10.0.0.0/16
- grafana: provision loki datasource at http://127.0.0.1:3100
- k8s: promtail daemonset + rbac in monitoring namespace, ships pod logs
to http://10.0.15.31:3100 with cluster=home-cluster label
- argocd: Application points at home/cluster/promtail
joshuasing/starlink_exporter 0.9.0 (Docker Hub; ghcr.io lags behind
upstream releases). Targets the dish's gRPC management interface at
192.168.100.1:9200, reachable from cluster pods via 10.0.15.254 while
the Starlink router is in bypass mode.
Grafana dashboard "Starlink" covers throughput, PoP ping latency/drop,
obstruction state + 24h total, GPS lock, and dish position/info — synced
to /var/lib/grafana/dashboards/ by the existing grafana role.
Replaces the vendored Go exporter with two pre-built upstream images:
- quay.io/prometheuscommunity/json-exporter:v0.7.0 maps the gateway JSON
into prometheus metrics via JSONPath. The G4AR keys signal blocks as
"5g" / "4g" (digit-prefixed), which the k8s client-go JSONPath dialect
only reaches via bracket-with-single-quote notation.
- nginxinc/nginx-unprivileged:1.27-alpine-slim sidecar rewrites
GET /metrics → GET /probe?module=tmhi&target=http://192.168.12.1/...
so the existing kubernetes-services scrape job picks it up over the
apiserver proxy with no Prometheus-side config changes.
Avoiding caddy here: its binary carries cap_net_bind_service file caps,
which the kernel refuses to exec under no_new_privs as a non-root user.
Dashboard "Gateway up" panel now reads up{service="tmobile-exporter"}
(Prometheus's built-in scrape-success gauge) since json_exporter's
/probe endpoint does not emit probe_success.
Replaces tpaulus/tmobile-internet-exporter upstream image, which crashes
on every scrape against the G4AR because the gateway returns `bars` as a
JSON float (`4.0`) and the upstream struct types it as int.
The vendored exporter:
- decodes all radio numerics as float64 so `4.0` round-trips
- treats the 4g / 5g signal blocks as optional (the G4AR omits 4g)
- evicts stale (type, band) series when a band disappears between scrapes
- adds a scrape_errors_total counter and /healthz endpoint
Deployment now pulls git.mcintire.me/graham/tmobile-exporter:v1 via the
forgejo-registry imagePullSecret (matching the convention used by aprs,
prop, towerops). Build + push the image before Argo can pull it; the
imagePullSecret must also be copied into the monitoring namespace.
Deploys tpaulus/tmobile-internet-exporter against the G4AR at
192.168.12.1, picked up by the existing kubernetes-services scrape
job via prometheus.io/scrape annotations on the Service.
Cluster-side:
- Deploy kube-state-metrics (kube-system) for cluster-state series.
- Deploy node-exporter DaemonSet on all 7 nodes via the new monitoring ns
(privileged PSS, hostNetwork, read-only host mounts).
- Both managed by ArgoCD apps under home/cluster/argocd/apps/.
Prometheus-side:
- Replace direct pod-IP scraping (which can't reach the 10.244.0.0/16 pod
network from outside the cluster) with API-server-proxy scraping for
both Services and Pods. Now one target per pod, all healthy.
- Extend external Prometheus RBAC with services/proxy + pods/proxy.
Grafana-side:
- Add file-based dashboard provisioner; ship 5 standard dashboards
(Node Exporter Full, k8s cluster-via-prometheus, k8s all-in-one,
k8s monitoring, Prometheus 2.0 overview).
prom.w5isp.com:3000 now shows live data for all 7 cluster nodes plus
the prom host itself.
The towerops-web repo got renamed to codeberg.org/gmcintire/towerops
and its container image follows the same rename
(codeberg.org/gmcintire/towerops). Application source repo and
image-updater registry list updated.
The prop and prop-grid-rs container images now live on
codeberg.org/gmcintire/* (see prop repo commit 8a9c0ba6 for the
workflow flip). Application source repo and image-updater registry
list follow.
Cluster-side state still required for this to roll out, applied
out-of-band via kubectl:
- Secret prop/forgejo-registry — dockerconfigjson with codeberg.org
auth (also keeps git.mcintire.me auth during the transition).
- Secret argocd/forgejo-registry — same shape, used by the
argocd-image-updater registry config.
- Secret argocd/codeberg-credentials-gmcintire (label
argocd.argoproj.io/secret-type=repo-creds) — credentials template
matching url=https://codeberg.org/gmcintire so future codeberg
repos under that owner pick up auth automatically.
- ConfigMap argocd/argocd-image-updater-config — registries.conf
extended with codeberg.org entry referencing the new pull secret.
annotates the 4 Applications so image-updater scans git.mcintire.me
on a 2min loop and writes new tags to spec.source.kustomize.images
(no git commits, no CI sed-and-bump).
- Upgrade all nodes from v1.12.1/v1.12.2 to v1.12.4 installer image
- Set explicit nameservers (9.9.9.9, 149.112.112.112, 1.1.1.1) on
workers to replace unreliable DHCP-provided resolver at 10.0.15.254
- Fix talos-worker2 missing ExtensionServiceConfig for ext-tailscale
which was causing node reboots every ~35 minutes
- Update tailscale.patch.yaml with working auth key
- Ensures Phoenix knows the original request was HTTPS
- Complements externalTrafficPolicy:Local on Traefik service
- Service was patched with: kubectl patch service traefik -n traefik -p '{"spec":{"externalTrafficPolicy":"Local"}}'
- This preserves real client IPs for geolocation
