graham/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
infra/ansible/roles/prometheus/README.md
Graham McIntire 39bedb08d1
renumber infrastructure from 10.0.15.0/24 to 10.0.16.0/22 (hosts in 10.0.19.x) 
Proxmox: node1-3 → 10.0.19.101-103
Talos cp1-3 → 10.0.19.1-3, workers → 10.0.19.4-6
K8s endpoint → VIP https://10.0.19.10:6443
Ansible: prom → 10.0.19.31, db → 10.0.19.30
Talos: added VIP block to controlplane.yaml base config
Promtail: Loki URL → 10.0.19.31
Docs: all references updated, talos4 removed
2026-07-18 08:28:47 -05:00

2.1 KiB
Raw Blame History

prometheus role

Installs Prometheus from upstream tarballs, configures scraping for the home-cluster Kubernetes API, nodes, cAdvisor, and pods/services with prometheus.io/scrape: "true" annotations.

One-time Kubernetes bootstrap

Prometheus runs outside the cluster on prom.w5isp.com, so it needs a ServiceAccount token + the cluster's CA cert.

  1. Apply the RBAC manifest from your workstation:

    kubectl apply -f roles/prometheus/files/k8s-rbac.yaml

  2. Pull the token + CA out of the cluster:

    TOKEN=$(kubectl -n prometheus-external get secret prometheus-token \
  -o jsonpath='{.data.token}' | base64 -d)
CA=$(kubectl -n prometheus-external get secret prometheus-token \
  -o jsonpath='{.data.ca\.crt}' | base64 -d)

  3. Copy them onto the prom host:

    ssh ansible@10.0.19.31 'sudo install -d -o prometheus -g prometheus -m 0750 /etc/prometheus/k8s'
printf '%s' "$TOKEN" | ssh ansible@10.0.19.31 \
  'sudo tee /etc/prometheus/k8s/token >/dev/null && \
   sudo chown prometheus:prometheus /etc/prometheus/k8s/token && \
   sudo chmod 0640 /etc/prometheus/k8s/token'
printf '%s' "$CA" | ssh ansible@10.0.19.31 \
  'sudo tee /etc/prometheus/k8s/ca.crt >/dev/null && \
   sudo chown prometheus:prometheus /etc/prometheus/k8s/ca.crt && \
   sudo chmod 0640 /etc/prometheus/k8s/ca.crt'

  4. Reload Prometheus:

    ssh ansible@10.0.19.31 'sudo systemctl reload prometheus'

The token persists across pod restarts (it's a long-lived kubernetes.io/service-account-token Secret, not a projected token), so it does not need to be rotated unless you delete the Secret.

Verifying scrape targets

Open http://10.0.19.31:9090/targets — the kubernetes-apiservers, kubernetes-nodes, and kubernetes-cadvisor jobs should all be UP.

If kubernetes-pods / kubernetes-service-endpoints are empty, that's expected until you annotate workloads with prometheus.io/scrape: "true".

Disabling k8s scraping

Set prometheus_k8s_enabled: false in host_vars/prom.w5isp.com.yml and re-run the playbook.