- new ansible loki role installs loki 3.5.5 on prometheus_servers as a
systemd unit (single-binary, filesystem store, 30d retention)
- open tcp/3100 on prom.w5isp.com to 10.0.0.0/16
- grafana: provision loki datasource at http://127.0.0.1:3100
- k8s: promtail daemonset + rbac in monitoring namespace, ships pod logs
to http://10.0.15.31:3100 with cluster=home-cluster label
- argocd: Application points at home/cluster/promtail
joshuasing/starlink_exporter 0.9.0 (Docker Hub; ghcr.io lags behind
upstream releases). Targets the dish's gRPC management interface at
192.168.100.1:9200, reachable from cluster pods via 10.0.15.254 while
the Starlink router is in bypass mode.
Grafana dashboard "Starlink" covers throughput, PoP ping latency/drop,
obstruction state + 24h total, GPS lock, and dish position/info — synced
to /var/lib/grafana/dashboards/ by the existing grafana role.
Deploys tpaulus/tmobile-internet-exporter against the G4AR at
192.168.12.1, picked up by the existing kubernetes-services scrape
job via prometheus.io/scrape annotations on the Service.
Cluster-side:
- Deploy kube-state-metrics (kube-system) for cluster-state series.
- Deploy node-exporter DaemonSet on all 7 nodes via the new monitoring ns
(privileged PSS, hostNetwork, read-only host mounts).
- Both managed by ArgoCD apps under home/cluster/argocd/apps/.
Prometheus-side:
- Replace direct pod-IP scraping (which can't reach the 10.244.0.0/16 pod
network from outside the cluster) with API-server-proxy scraping for
both Services and Pods. Now one target per pod, all healthy.
- Extend external Prometheus RBAC with services/proxy + pods/proxy.
Grafana-side:
- Add file-based dashboard provisioner; ship 5 standard dashboards
(Node Exporter Full, k8s cluster-via-prometheus, k8s all-in-one,
k8s monitoring, Prometheus 2.0 overview).
prom.w5isp.com:3000 now shows live data for all 7 cluster nodes plus
the prom host itself.
The towerops-web repo got renamed to codeberg.org/gmcintire/towerops
and its container image follows the same rename
(codeberg.org/gmcintire/towerops). Application source repo and
image-updater registry list updated.
The prop and prop-grid-rs container images now live on
codeberg.org/gmcintire/* (see prop repo commit 8a9c0ba6 for the
workflow flip). Application source repo and image-updater registry
list follow.
Cluster-side state still required for this to roll out, applied
out-of-band via kubectl:
- Secret prop/forgejo-registry — dockerconfigjson with codeberg.org
auth (also keeps git.mcintire.me auth during the transition).
- Secret argocd/forgejo-registry — same shape, used by the
argocd-image-updater registry config.
- Secret argocd/codeberg-credentials-gmcintire (label
argocd.argoproj.io/secret-type=repo-creds) — credentials template
matching url=https://codeberg.org/gmcintire so future codeberg
repos under that owner pick up auth automatically.
- ConfigMap argocd/argocd-image-updater-config — registries.conf
extended with codeberg.org entry referencing the new pull secret.
annotates the 4 Applications so image-updater scans git.mcintire.me
on a 2min loop and writes new tags to spec.source.kustomize.images
(no git commits, no CI sed-and-bump).
