infra/ansible/roles/prometheus

prometheus role

Installs Prometheus from upstream tarballs, configures scraping for the home-cluster Kubernetes API, nodes, cAdvisor, and pods/services with prometheus.io/scrape: "true" annotations.

One-time Kubernetes bootstrap

Prometheus runs outside the cluster on prom.w5isp.com, so it needs a ServiceAccount token + the cluster's CA cert.

1. Apply the RBAC manifest from your workstation:

   kubectl apply -f roles/prometheus/files/k8s-rbac.yaml
   

2. Pull the token + CA out of the cluster:

   TOKEN=$(kubectl -n prometheus-external get secret prometheus-token \
     -o jsonpath='{.data.token}' | base64 -d)
   CA=$(kubectl -n prometheus-external get secret prometheus-token \
     -o jsonpath='{.data.ca\.crt}' | base64 -d)
   

3. Copy them onto the prom host:

   ssh ansible@10.0.15.31 'sudo install -d -o prometheus -g prometheus -m 0750 /etc/prometheus/k8s'
   printf '%s' "$TOKEN" | ssh ansible@10.0.15.31 \
     'sudo tee /etc/prometheus/k8s/token >/dev/null && \
      sudo chown prometheus:prometheus /etc/prometheus/k8s/token && \
      sudo chmod 0640 /etc/prometheus/k8s/token'
   printf '%s' "$CA" | ssh ansible@10.0.15.31 \
     'sudo tee /etc/prometheus/k8s/ca.crt >/dev/null && \
      sudo chown prometheus:prometheus /etc/prometheus/k8s/ca.crt && \
      sudo chmod 0640 /etc/prometheus/k8s/ca.crt'
   

4. Reload Prometheus:

   ssh ansible@10.0.15.31 'sudo systemctl reload prometheus'
   

The token persists across pod restarts (it's a long-lived kubernetes.io/service-account-token Secret, not a projected token), so it does not need to be rotated unless you delete the Secret.

Verifying scrape targets

Open http://10.0.15.31:9090/targets — the kubernetes-apiservers, kubernetes-nodes, and kubernetes-cadvisor jobs should all be UP.

If kubernetes-pods / kubernetes-service-endpoints are empty, that's expected until you annotate workloads with prometheus.io/scrape: "true".

Disabling k8s scraping

Set prometheus_k8s_enabled: false in host_vars/prom.w5isp.com.yml and re-run the playbook.