graham/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
infra/ansible/roles/prometheus/README.md
Graham McIntire 922552059f
prom: add prometheus stack (prometheus, alertmanager, grafana, node_exporter) 
Provisions prom.w5isp.com (10.0.15.31) as the home-cluster monitoring host.
Prometheus 3.6.0 scrapes the Talos cluster API/nodes/cAdvisor via an external
SA token; Grafana ships with the Prometheus datasource pre-provisioned.

host_vars opens 9090/9093/9100/3000 to 10.0.0.0/16 plus SSH from the home LAN
so the firewall role doesn't lock out admin access.
2026-05-08 09:50:31 -05:00

2.1 KiB
Raw Blame History

prometheus role

Installs Prometheus from upstream tarballs, configures scraping for the home-cluster Kubernetes API, nodes, cAdvisor, and pods/services with prometheus.io/scrape: "true" annotations.

One-time Kubernetes bootstrap

Prometheus runs outside the cluster on prom.w5isp.com, so it needs a ServiceAccount token + the cluster's CA cert.

  1. Apply the RBAC manifest from your workstation:

    kubectl apply -f roles/prometheus/files/k8s-rbac.yaml

  2. Pull the token + CA out of the cluster:

    TOKEN=$(kubectl -n prometheus-external get secret prometheus-token \
  -o jsonpath='{.data.token}' | base64 -d)
CA=$(kubectl -n prometheus-external get secret prometheus-token \
  -o jsonpath='{.data.ca\.crt}' | base64 -d)

  3. Copy them onto the prom host:

    ssh ansible@10.0.15.31 'sudo install -d -o prometheus -g prometheus -m 0750 /etc/prometheus/k8s'
printf '%s' "$TOKEN" | ssh ansible@10.0.15.31 \
  'sudo tee /etc/prometheus/k8s/token >/dev/null && \
   sudo chown prometheus:prometheus /etc/prometheus/k8s/token && \
   sudo chmod 0640 /etc/prometheus/k8s/token'
printf '%s' "$CA" | ssh ansible@10.0.15.31 \
  'sudo tee /etc/prometheus/k8s/ca.crt >/dev/null && \
   sudo chown prometheus:prometheus /etc/prometheus/k8s/ca.crt && \
   sudo chmod 0640 /etc/prometheus/k8s/ca.crt'

  4. Reload Prometheus:

    ssh ansible@10.0.15.31 'sudo systemctl reload prometheus'

The token persists across pod restarts (it's a long-lived kubernetes.io/service-account-token Secret, not a projected token), so it does not need to be rotated unless you delete the Secret.

Verifying scrape targets

Open http://10.0.15.31:9090/targets — the kubernetes-apiservers, kubernetes-nodes, and kubernetes-cadvisor jobs should all be UP.

If kubernetes-pods / kubernetes-service-endpoints are empty, that's expected until you annotate workloads with prometheus.io/scrape: "true".

Disabling k8s scraping

Set prometheus_k8s_enabled: false in host_vars/prom.w5isp.com.yml and re-run the playbook.