Replaces the vendored Go exporter with two pre-built upstream images:
- quay.io/prometheuscommunity/json-exporter:v0.7.0 maps the gateway JSON
into prometheus metrics via JSONPath. The G4AR keys signal blocks as
"5g" / "4g" (digit-prefixed), which the k8s client-go JSONPath dialect
only reaches via bracket-with-single-quote notation.
- nginxinc/nginx-unprivileged:1.27-alpine-slim sidecar rewrites
GET /metrics → GET /probe?module=tmhi&target=http://192.168.12.1/...
so the existing kubernetes-services scrape job picks it up over the
apiserver proxy with no Prometheus-side config changes.
Avoiding caddy here: its binary carries cap_net_bind_service file caps,
which the kernel refuses to exec under no_new_privs as a non-root user.
Dashboard "Gateway up" panel now reads up{service="tmobile-exporter"}
(Prometheus's built-in scrape-success gauge) since json_exporter's
/probe endpoint does not emit probe_success.
Replaces tpaulus/tmobile-internet-exporter upstream image, which crashes
on every scrape against the G4AR because the gateway returns `bars` as a
JSON float (`4.0`) and the upstream struct types it as int.
The vendored exporter:
- decodes all radio numerics as float64 so `4.0` round-trips
- treats the 4g / 5g signal blocks as optional (the G4AR omits 4g)
- evicts stale (type, band) series when a band disappears between scrapes
- adds a scrape_errors_total counter and /healthz endpoint
Deployment now pulls git.mcintire.me/graham/tmobile-exporter:v1 via the
forgejo-registry imagePullSecret (matching the convention used by aprs,
prop, towerops). Build + push the image before Argo can pull it; the
imagePullSecret must also be copied into the monitoring namespace.
Deploys tpaulus/tmobile-internet-exporter against the G4AR at
192.168.12.1, picked up by the existing kubernetes-services scrape
job via prometheus.io/scrape annotations on the Service.
Cluster-side:
- Deploy kube-state-metrics (kube-system) for cluster-state series.
- Deploy node-exporter DaemonSet on all 7 nodes via the new monitoring ns
(privileged PSS, hostNetwork, read-only host mounts).
- Both managed by ArgoCD apps under home/cluster/argocd/apps/.
Prometheus-side:
- Replace direct pod-IP scraping (which can't reach the 10.244.0.0/16 pod
network from outside the cluster) with API-server-proxy scraping for
both Services and Pods. Now one target per pod, all healthy.
- Extend external Prometheus RBAC with services/proxy + pods/proxy.
Grafana-side:
- Add file-based dashboard provisioner; ship 5 standard dashboards
(Node Exporter Full, k8s cluster-via-prometheus, k8s all-in-one,
k8s monitoring, Prometheus 2.0 overview).
prom.w5isp.com:3000 now shows live data for all 7 cluster nodes plus
the prom host itself.
The towerops-web repo got renamed to codeberg.org/gmcintire/towerops
and its container image follows the same rename
(codeberg.org/gmcintire/towerops). Application source repo and
image-updater registry list updated.
The prop and prop-grid-rs container images now live on
codeberg.org/gmcintire/* (see prop repo commit 8a9c0ba6 for the
workflow flip). Application source repo and image-updater registry
list follow.
Cluster-side state still required for this to roll out, applied
out-of-band via kubectl:
- Secret prop/forgejo-registry — dockerconfigjson with codeberg.org
auth (also keeps git.mcintire.me auth during the transition).
- Secret argocd/forgejo-registry — same shape, used by the
argocd-image-updater registry config.
- Secret argocd/codeberg-credentials-gmcintire (label
argocd.argoproj.io/secret-type=repo-creds) — credentials template
matching url=https://codeberg.org/gmcintire so future codeberg
repos under that owner pick up auth automatically.
- ConfigMap argocd/argocd-image-updater-config — registries.conf
extended with codeberg.org entry referencing the new pull secret.
annotates the 4 Applications so image-updater scans git.mcintire.me
on a 2min loop and writes new tags to spec.source.kustomize.images
(no git commits, no CI sed-and-bump).
- Upgrade all nodes from v1.12.1/v1.12.2 to v1.12.4 installer image
- Set explicit nameservers (9.9.9.9, 149.112.112.112, 1.1.1.1) on
workers to replace unreliable DHCP-provided resolver at 10.0.15.254
- Fix talos-worker2 missing ExtensionServiceConfig for ext-tailscale
which was causing node reboots every ~35 minutes
- Update tailscale.patch.yaml with working auth key
- Ensures Phoenix knows the original request was HTTPS
- Complements externalTrafficPolicy:Local on Traefik service
- Service was patched with: kubectl patch service traefik -n traefik -p '{"spec":{"externalTrafficPolicy":"Local"}}'
- This preserves real client IPs for geolocation
Node1's Tailscale IP changed from 100.102.235.20 to 100.111.4.102 after
reapplying the Tailscale configuration. Updated the kubeconfig server
address to use the new Tailscale IP for remote kubectl access.
Generated with Claude Code https://claude.com/claude-code
