Graham McIntire
39bedb08d1
Proxmox: node1-3 → 10.0.19.101-103 Talos cp1-3 → 10.0.19.1-3, workers → 10.0.19.4-6 K8s endpoint → VIP https://10.0.19.10:6443 Ansible: prom → 10.0.19.31, db → 10.0.19.30 Talos: added VIP block to controlplane.yaml base config Promtail: Loki URL → 10.0.19.31 Docs: all references updated, talos4 removed
61 lines
2.1 KiB
Markdown
61 lines
2.1 KiB
Markdown
# prometheus role
|
|
|
|
Installs Prometheus from upstream tarballs, configures scraping for the
|
|
home-cluster Kubernetes API, nodes, cAdvisor, and pods/services with
|
|
`prometheus.io/scrape: "true"` annotations.
|
|
|
|
## One-time Kubernetes bootstrap
|
|
|
|
Prometheus runs *outside* the cluster on `prom.w5isp.com`, so it needs a
|
|
ServiceAccount token + the cluster's CA cert.
|
|
|
|
1. Apply the RBAC manifest from your workstation:
|
|
|
|
```bash
|
|
kubectl apply -f roles/prometheus/files/k8s-rbac.yaml
|
|
```
|
|
|
|
2. Pull the token + CA out of the cluster:
|
|
|
|
```bash
|
|
TOKEN=$(kubectl -n prometheus-external get secret prometheus-token \
|
|
-o jsonpath='{.data.token}' | base64 -d)
|
|
CA=$(kubectl -n prometheus-external get secret prometheus-token \
|
|
-o jsonpath='{.data.ca\.crt}' | base64 -d)
|
|
```
|
|
|
|
3. Copy them onto the prom host:
|
|
|
|
```bash
|
|
ssh ansible@10.0.19.31 'sudo install -d -o prometheus -g prometheus -m 0750 /etc/prometheus/k8s'
|
|
printf '%s' "$TOKEN" | ssh ansible@10.0.19.31 \
|
|
'sudo tee /etc/prometheus/k8s/token >/dev/null && \
|
|
sudo chown prometheus:prometheus /etc/prometheus/k8s/token && \
|
|
sudo chmod 0640 /etc/prometheus/k8s/token'
|
|
printf '%s' "$CA" | ssh ansible@10.0.19.31 \
|
|
'sudo tee /etc/prometheus/k8s/ca.crt >/dev/null && \
|
|
sudo chown prometheus:prometheus /etc/prometheus/k8s/ca.crt && \
|
|
sudo chmod 0640 /etc/prometheus/k8s/ca.crt'
|
|
```
|
|
|
|
4. Reload Prometheus:
|
|
|
|
```bash
|
|
ssh ansible@10.0.19.31 'sudo systemctl reload prometheus'
|
|
```
|
|
|
|
The token persists across pod restarts (it's a long-lived `kubernetes.io/service-account-token`
|
|
Secret, not a projected token), so it does not need to be rotated unless you delete the Secret.
|
|
|
|
## Verifying scrape targets
|
|
|
|
Open `http://10.0.19.31:9090/targets` — the `kubernetes-apiservers`,
|
|
`kubernetes-nodes`, and `kubernetes-cadvisor` jobs should all be UP.
|
|
|
|
If `kubernetes-pods` / `kubernetes-service-endpoints` are empty, that's expected
|
|
until you annotate workloads with `prometheus.io/scrape: "true"`.
|
|
|
|
## Disabling k8s scraping
|
|
|
|
Set `prometheus_k8s_enabled: false` in `host_vars/prom.w5isp.com.yml` and re-run
|
|
the playbook.
|