renumber infrastructure from 10.0.15.0/24 to 10.0.16.0/22 (hosts in 10.0.19.x)
Proxmox: node1-3 → 10.0.19.101-103 Talos cp1-3 → 10.0.19.1-3, workers → 10.0.19.4-6 K8s endpoint → VIP https://10.0.19.10:6443 Ansible: prom → 10.0.19.31, db → 10.0.19.30 Talos: added VIP block to controlplane.yaml base config Promtail: Loki URL → 10.0.19.31 Docs: all references updated, talos4 removed
This commit is contained in:
parent
514b346662
commit
39bedb08d1
31 changed files with 672 additions and 71 deletions
12
CLAUDE.md
12
CLAUDE.md
|
|
@ -169,14 +169,14 @@ tofu plan
|
|||
|
||||
## Recent Infrastructure Updates
|
||||
|
||||
### Talos Home Cluster (10.0.15.1-7)
|
||||
### Talos Home Cluster (10.0.19.1-6)
|
||||
- **Purpose**: Primary home lab Kubernetes cluster
|
||||
- **Cluster Name**: home-cluster
|
||||
- **Kubernetes Version**: v1.36.1
|
||||
- **Control Plane Nodes**: 3 (10.0.15.1-3)
|
||||
- **Worker Nodes**: 4 (10.0.15.4-7)
|
||||
- **API Endpoint**: https://10.0.15.1:6443
|
||||
- **Proxmox Cluster**: "home" (node1, node2, node3 at 10.0.15.101-103)
|
||||
- **Control Plane Nodes**: 3 (10.0.19.1-3)
|
||||
- **Worker Nodes**: 3 (10.0.19.4-6)
|
||||
- **API Endpoint**: https://10.0.19.10:6443
|
||||
- **Proxmox Cluster**: "home" (node1, node2, node3 at 10.0.19.101-103)
|
||||
- **VM Disks**: local-lvm on each Proxmox node
|
||||
- **Persistent Storage**: TBD (Longhorn was removed — evaluating alternatives)
|
||||
- **CNI**: Flannel
|
||||
|
|
@ -234,7 +234,7 @@ tofu plan
|
|||
- As you learn new things, keep claude.md updated
|
||||
|
||||
## Cluster
|
||||
- **home-cluster** (Talos): 10.0.15.1-6 — only Kubernetes cluster. External traffic enters via the `cloudflared` tunnel deployment in the `cloudflared` namespace; per-hostname routing rules live in Cloudflare's dashboard, not in the cluster.
|
||||
- **home-cluster** (Talos): 10.0.19.1-6 — only Kubernetes cluster. External traffic enters via the `cloudflared` tunnel deployment in the `cloudflared` namespace; per-hostname routing rules live in Cloudflare's dashboard, not in the cluster.
|
||||
|
||||
## Application Deployment Patterns
|
||||
|
||||
|
|
|
|||
214
RENUMBER.md
Normal file
214
RENUMBER.md
Normal file
|
|
@ -0,0 +1,214 @@
|
|||
# Renumber: 10.0.15.0/24 → 10.0.16.0/22 (hosts in 10.0.19.x)
|
||||
|
||||
Move Proxmox nodes, Talos cluster, and support VMs onto 10.0.16.0/22
|
||||
(mask 255.255.252.0, broadcast 10.0.19.255). Server range: 10.0.19.x.
|
||||
Delete this file when done.
|
||||
|
||||
## Address map
|
||||
|
||||
| Host | MAC | Old | New |
|
||||
|----------------|-------------------|-------------|-------------|
|
||||
| talos-cp1 | BC:24:11:9B:48:92 | 10.0.15.1 | 10.0.19.1 |
|
||||
| talos-cp2 | BC:24:11:62:7B:3F | 10.0.15.2 | 10.0.19.2 |
|
||||
| talos-cp3 | BC:24:11:D4:2F:ED | 10.0.15.3 | 10.0.19.3 |
|
||||
| talos-worker1 | BC:24:11:43:3F:FF | 10.0.15.4 | 10.0.19.4 |
|
||||
| talos-worker2 | BC:24:11:62:C4:8F | 10.0.15.5 | 10.0.19.5 |
|
||||
| talos-worker3 | BC:24:11:3F:8E:1A | 10.0.15.6 | 10.0.19.6 |
|
||||
| **K8s API VIP**| — (floats on CPs) | — | **10.0.19.10** |
|
||||
| db.w5isp.com | static | 10.0.15.30 | 10.0.19.30 |
|
||||
| prom.w5isp.com | static | 10.0.15.31 | 10.0.19.31 |
|
||||
| node1 | static | 10.0.15.101 | 10.0.19.101 |
|
||||
| node2 | static | 10.0.15.102 | 10.0.19.102 |
|
||||
| node3 | static | 10.0.15.103 | 10.0.19.103 |
|
||||
| router/gateway | — | 10.0.15.254 | 10.0.19.254 |
|
||||
|
||||
## Ground rules
|
||||
|
||||
- **Both subnets must coexist on the same L2 for the whole migration** —
|
||||
already satisfied: the formerly separate subnets now share one flat L2.
|
||||
Talos updates each etcd member's peer URL automatically when its IP
|
||||
changes, but only if the remaining members can still reach it —
|
||||
moving to an unreachable network bricks etcd
|
||||
(siderolabs/talos#9075, #6080).
|
||||
- **Control planes strictly one at a time.** The etcd member update
|
||||
needs quorum (2/3 up).
|
||||
- Everything through Phase 3 step 2 is reversible by putting the DHCP
|
||||
reservation back and rebooting the node.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Backups + router prep
|
||||
|
||||
1. Backups (from `talos/`):
|
||||
```bash
|
||||
talosctl --talosconfig talosconfig -n 10.0.15.1 etcd snapshot etcd-$(date +%F).snap
|
||||
cp secrets.yaml talosconfig ~/backups/ # or wherever off-repo copies live
|
||||
```
|
||||
(DB dumps intentionally skipped — operator decision 2026-07-17.)
|
||||
2. Router:
|
||||
- Add `10.0.19.254/22` to the LAN interface **alongside** 10.0.15.254/24.
|
||||
- New DHCP scope on 10.0.16.0/22, gateway 10.0.19.254. Keep the
|
||||
dynamic pool **out of 10.0.19.0–10.0.19.127** (statics + VIP live there).
|
||||
10.0.19.10 (VIP) must never be leased.
|
||||
- Do **not** change the Talos MAC reservations yet — that happens
|
||||
per-node in Phase 3.
|
||||
- Clone any firewall rules referencing 10.0.15.0/24 to also cover
|
||||
10.0.16.0/22 (delete the old ones in Phase 5).
|
||||
3. Sanity: from the workstation, ping 10.0.19.254.
|
||||
|
||||
## Phase 1 — Proxmox nodes (node1/2/3)
|
||||
|
||||
Guests keep running throughout; no VM reboots. Corosync quorum drops
|
||||
mid-change — expected, harmless without HA.
|
||||
|
||||
1. Discover what actually references the IPs, per node:
|
||||
```bash
|
||||
ssh root@10.0.15.101 "grep -r '10\.0\.15' /etc/pve /etc/network/interfaces /etc/hosts; pveceph status 2>&1 | head -2"
|
||||
```
|
||||
(`pveceph status` should say Ceph isn't configured — `cluster/ceph-csi/`
|
||||
is stale. If Ceph is somehow alive, STOP; mon renumbering is its own runbook.)
|
||||
2. If `/etc/pve/corosync.conf` `ring0_addr`s are **hostnames**: no corosync
|
||||
edit needed — only `/etc/hosts`. If they're **IPs**: on ONE node while
|
||||
still quorate, edit `/etc/pve/corosync.conf` — all three ring0_addrs to
|
||||
10.0.19.x and **bump `config_version`**.
|
||||
3. Then on each node in turn (node1 → node2 → node3):
|
||||
```bash
|
||||
# /etc/network/interfaces: address 10.0.19.10X/22, gateway 10.0.19.254
|
||||
# /etc/hosts: own + peer entries to 10.0.19.x
|
||||
ifreload -a
|
||||
systemctl restart corosync pve-cluster
|
||||
```
|
||||
4. After all three: `pvecm status` shows 3/3 quorate on new IPs;
|
||||
Proxmox web UI reachable at https://10.0.19.101:8006.
|
||||
- If pmxcfs wedges (can't write /etc/pve): `systemctl stop pve-cluster corosync;
|
||||
pmxcfs -l` to fix corosync.conf locally, then restart both.
|
||||
5. Update wherever `node1`/`node2`/`node3` resolve on the workstation
|
||||
(`/etc/hosts` or `~/.ssh/config`) — ansible inventory uses
|
||||
`ansible_host=node1` by name.
|
||||
|
||||
## Phase 2 — db + prom VMs
|
||||
|
||||
Use the **Proxmox console** (not SSH — you're cutting the branch you sit on).
|
||||
|
||||
1. On each VM, update the static netconfig (netplan or
|
||||
/etc/network/interfaces): `10.0.19.30/22` and `10.0.19.31/22`,
|
||||
gateway 10.0.19.254. Restart networking.
|
||||
2. Verify SSH: `ssh ansible@10.0.19.31` (prom's ufw allows are
|
||||
10.0.0.0/16-based, so no lockout exposure).
|
||||
3. Loki ingestion from the cluster (promtail → 10.0.15.31:3100) is now
|
||||
broken until Phase 4's configmap push — known gap.
|
||||
|
||||
## Phase 3 — Talos cluster
|
||||
|
||||
Run from `talos/`. Cluster endpoint is still `https://10.0.15.1:6443`
|
||||
and stays valid until step 4.
|
||||
|
||||
**1. Workers, one at a time** (worker1 → worker2 → worker3):
|
||||
```bash
|
||||
# a) On router: change THIS node's reservation to its 10.0.19.x address
|
||||
talosctl --talosconfig talosconfig -n 10.0.15.4 reboot
|
||||
# b) wait, then verify:
|
||||
talosctl --talosconfig talosconfig -n 10.0.19.4 version
|
||||
kubectl get nodes -o wide # INTERNAL-IP updated, node Ready
|
||||
```
|
||||
|
||||
**2. Add the VIP to the control planes** (still on old IPs).
|
||||
Create `patches/vip.yaml`:
|
||||
```yaml
|
||||
machine:
|
||||
network:
|
||||
interfaces:
|
||||
- deviceSelector:
|
||||
physical: true
|
||||
dhcp: true
|
||||
vip:
|
||||
ip: 10.0.19.10
|
||||
```
|
||||
```bash
|
||||
for ip in 10.0.15.1 10.0.15.2 10.0.15.3; do
|
||||
talosctl --talosconfig talosconfig -n $ip patch machineconfig --patch @patches/vip.yaml
|
||||
done
|
||||
curl -sk https://10.0.19.10:6443/version # any TLS response = VIP is up
|
||||
```
|
||||
|
||||
**3. Flip the cluster endpoint to the VIP on ALL 6 nodes.**
|
||||
Create `patches/endpoint.yaml`:
|
||||
```yaml
|
||||
cluster:
|
||||
controlPlane:
|
||||
endpoint: https://10.0.19.10:6443
|
||||
```
|
||||
```bash
|
||||
for ip in 10.0.15.1 10.0.15.2 10.0.15.3 10.0.19.4 10.0.19.5 10.0.19.6; do
|
||||
talosctl --talosconfig talosconfig -n $ip patch machineconfig --patch @patches/endpoint.yaml
|
||||
done
|
||||
talosctl --talosconfig talosconfig kubeconfig -n 10.0.15.1 --force # kubeconfig now points at the VIP
|
||||
kubectl get nodes # all Ready via https://10.0.19.10:6443
|
||||
```
|
||||
|
||||
**4. Control planes, strictly one at a time** (cp2 → cp3 → cp1):
|
||||
```bash
|
||||
# a) On router: change THIS node's reservation
|
||||
talosctl --talosconfig talosconfig -n 10.0.15.2 reboot
|
||||
# b) wait, then verify BEFORE touching the next one:
|
||||
talosctl --talosconfig talosconfig -n 10.0.19.2 etcd members
|
||||
# -> this member's peer URL shows 10.0.19.2, 3/3 members listed
|
||||
talosctl --talosconfig talosconfig -n 10.0.19.2 health
|
||||
```
|
||||
If a member gets stuck on the old peer URL: quorum check first
|
||||
(`etcd members` from a healthy node), then reboot the stuck node once
|
||||
more before anything drastic. Worst case: restore the Phase 0 snapshot.
|
||||
|
||||
**5. Client configs:**
|
||||
```bash
|
||||
talosctl --talosconfig talosconfig config endpoint 10.0.19.1 10.0.19.2 10.0.19.3
|
||||
# (real node IPs — never the VIP for the Talos API)
|
||||
talosctl --talosconfig talosconfig -n 10.0.19.1 health
|
||||
```
|
||||
|
||||
## Phase 4 — Repo + config sweep
|
||||
|
||||
Edit, commit, push (promtail change must reach codeberg for Argo):
|
||||
|
||||
- `talos/controlplane.yaml:45` + `talos/worker.yaml:44` — endpoint
|
||||
`https://10.0.19.10:6443`; add the `machine.network.interfaces` VIP
|
||||
block to `controlplane.yaml` so rebuilds match reality
|
||||
- `ansible/hosts` — prom `ansible_host=10.0.19.31`
|
||||
- `ansible/host_vars/prom.w5isp.com.yml` — ansible_host + subnet comment
|
||||
- `ansible/host_vars/db.w5isp.com.yml` — `ansible_host: 10.0.19.30`
|
||||
- `ansible/roles/prometheus/defaults/main.yml` —
|
||||
`prometheus_k8s_api_server: "https://10.0.19.10:6443"` (VIP is in the
|
||||
apiserver cert SANs) + commented example targets
|
||||
- `ansible/roles/grafana/defaults/main.yml` — root_url `10.0.19.31`
|
||||
- `home/cluster/promtail/configmap.yaml` — Loki URL `http://10.0.19.31:3100`
|
||||
- Docs: `CLAUDE.md`, `talos/CLAUDE.md`, `talos/README.md`,
|
||||
`cluster/README.md`, `ansible/roles/prometheus/README.md`,
|
||||
`home/cluster/starlink-exporter/deployment.yaml` comment;
|
||||
`cluster/ceph-csi/` is dead — delete or mark stale
|
||||
- Then: `ansible-playbook -l prom.w5isp.com playbook.yml` and confirm
|
||||
Argo synced promtail (`kubectl -n promtail rollout status ds/promtail`)
|
||||
|
||||
## Phase 5 — Retire the old subnet
|
||||
|
||||
Only after `rg "10\.0\.15\." ~/dev/infra` returns nothing but history/docs:
|
||||
|
||||
- Router: remove 10.0.15.254/24, old DHCP scope, old firewall rules.
|
||||
- If a Tailscale subnet router advertises 10.0.15.0/24, re-advertise
|
||||
10.0.16.0/22 and approve in the admin console.
|
||||
- Hosts elsewhere in the /22 (e.g. birdnet 10.0.17.29) that still carry
|
||||
/24 masks should move to /22 to avoid hairpinning via the router.
|
||||
|
||||
## Final verification
|
||||
|
||||
```bash
|
||||
talosctl --talosconfig talosconfig -n 10.0.19.1 health
|
||||
kubectl get nodes -o wide && kubectl get pods -A | grep -v Running
|
||||
curl -sk https://10.0.19.10:6443/version
|
||||
ssh root@10.0.19.101 pvecm status
|
||||
open http://10.0.19.31:9090/targets # prometheus targets green
|
||||
# https://aprs.me and the other cloudflared-fronted apps load
|
||||
```
|
||||
|
||||
References: [talos#6080 — etcd peer URLs update on IP change](https://github.com/siderolabs/talos/issues/6080),
|
||||
[talos#9075 — what happens when old/new subnets can't reach each other](https://github.com/siderolabs/talos/issues/9075),
|
||||
[Talos multihoming/advertisedSubnets docs](https://docs.siderolabs.com/talos/v1.12/networking/multihoming)
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
---
|
||||
ansible_host: 10.0.15.30
|
||||
ansible_host: 10.0.19.30
|
||||
ansible_python_interpreter: /usr/bin/python3
|
||||
ansible_ssh_common_args: '-o StrictHostKeyChecking=accept-new'
|
||||
|
||||
|
|
|
|||
|
|
@ -2,3 +2,16 @@
|
|||
ansible_host: 204.110.191.229
|
||||
ansible_python_interpreter: /usr/bin/python3
|
||||
ansible_ssh_common_args: '-o StrictHostKeyChecking=accept-new'
|
||||
|
||||
# OpenSearch configuration
|
||||
opensearch_heap_min: "4g"
|
||||
opensearch_heap_max: "4g"
|
||||
|
||||
# Firewall — Caddy needs HTTP/HTTPS open for Let's Encrypt + traffic
|
||||
firewall_allow_rules:
|
||||
- port: 80
|
||||
proto: tcp
|
||||
comment: HTTP
|
||||
- port: 443
|
||||
proto: tcp
|
||||
comment: HTTPS
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
---
|
||||
ansible_host: 10.0.15.31
|
||||
ansible_host: 10.0.19.31
|
||||
ansible_python_interpreter: /usr/bin/python3
|
||||
ansible_ssh_common_args: '-o StrictHostKeyChecking=accept-new'
|
||||
|
||||
|
|
@ -8,7 +8,7 @@ network:
|
|||
|
||||
# Open Prometheus stack ports to the home LAN.
|
||||
# Trusted subnets (Tailscale, VNTX) are already wide-open via firewall role defaults;
|
||||
# this adds the 10.0.15.0/24 home cluster subnet so the k8s nodes can hit node_exporter
|
||||
# this adds the 10.0.16.0/22 home cluster subnet so the k8s nodes can hit node_exporter
|
||||
# and so any LAN client can reach the Grafana UI.
|
||||
firewall_allow_rules:
|
||||
- port: 22
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ skippy.w5isp.com
|
|||
mail.mcintire.me ansible_host=107.174.178.20
|
||||
dokku.w5isp.com
|
||||
aprs.w5isp.com
|
||||
prom.w5isp.com ansible_host=10.0.15.31
|
||||
prom.w5isp.com ansible_host=10.0.19.31
|
||||
|
||||
[prometheus_servers]
|
||||
prom.w5isp.com
|
||||
|
|
@ -37,6 +37,7 @@ aprs.w5isp.com
|
|||
skippy.w5isp.com
|
||||
netbox.vntx.net
|
||||
uisp.vntx.net
|
||||
logs.vntx.net
|
||||
|
||||
[netbox_servers]
|
||||
netbox.vntx.net
|
||||
|
|
@ -48,6 +49,9 @@ git.mcintire.me ansible_host=172.245.56.83
|
|||
[uisp_servers]
|
||||
uisp.vntx.net
|
||||
|
||||
[opensearch_servers]
|
||||
logs.vntx.net
|
||||
|
||||
[proxmox_servers]
|
||||
vm2-380 ansible_host=vm2-380
|
||||
node1 ansible_host=node1
|
||||
|
|
|
|||
|
|
@ -400,3 +400,13 @@
|
|||
- inspircd
|
||||
roles:
|
||||
- inspircd
|
||||
|
||||
- name: Configure OpenSearch servers
|
||||
hosts: opensearch_servers
|
||||
become: true
|
||||
gather_facts: true
|
||||
tags:
|
||||
- opensearch
|
||||
- logs
|
||||
roles:
|
||||
- opensearch
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@
|
|||
- apt-transport-https
|
||||
- curl
|
||||
- gnupg2
|
||||
- python3-debian
|
||||
state: present
|
||||
update_cache: true
|
||||
when: ansible_facts['os_family'] == "Debian"
|
||||
|
|
|
|||
7
ansible/roles/caddy/templates/Caddyfile-logs.vntx.net.j2
Normal file
7
ansible/roles/caddy/templates/Caddyfile-logs.vntx.net.j2
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
{{ opensearch_caddy_domain | default(inventory_hostname) }} {
|
||||
reverse_proxy 127.0.0.1:{{ opensearch_dashboards_port | default('5601') }} {
|
||||
header_up X-Forwarded-Host {host}
|
||||
header_up X-Real-IP {remote_host}
|
||||
header_up X-Forwarded-Proto {scheme}
|
||||
}
|
||||
}
|
||||
|
|
@ -5,7 +5,7 @@ grafana_apt_keyring: "/etc/apt/keyrings/grafana.gpg"
|
|||
|
||||
grafana_listen_address: "0.0.0.0"
|
||||
grafana_http_port: 3000
|
||||
grafana_root_url: "http://10.0.15.31:3000/"
|
||||
grafana_root_url: "http://10.0.19.31:3000/"
|
||||
|
||||
grafana_admin_user: admin
|
||||
# Empty by default so re-running ansible doesn't clobber a password that was
|
||||
|
|
|
|||
38
ansible/roles/opensearch/defaults/main.yml
Normal file
38
ansible/roles/opensearch/defaults/main.yml
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
---
|
||||
# OpenSearch version and architecture
|
||||
opensearch_version: "2.19.6"
|
||||
opensearch_arch: x64 # x64 or aarch64
|
||||
|
||||
opensearch_user: opensearch
|
||||
opensearch_group: opensearch
|
||||
|
||||
opensearch_install_dir: /opt/opensearch
|
||||
opensearch_config_dir: /opt/opensearch/config
|
||||
opensearch_data_dir: /var/lib/opensearch
|
||||
opensearch_log_dir: /var/log/opensearch
|
||||
|
||||
# JVM heap size — tuned for 8 GB host (leaves room for OS cache and other services)
|
||||
opensearch_heap_min: "2g"
|
||||
opensearch_heap_max: "2g"
|
||||
|
||||
# Network
|
||||
opensearch_http_port: 9200
|
||||
opensearch_transport_port: 9300
|
||||
|
||||
# Cluster (single-node by default for logs.vntx.net)
|
||||
opensearch_cluster_name: logs-vntx
|
||||
opensearch_node_name: "{{ ansible_facts['hostname'] | default('logs') }}"
|
||||
opensearch_discovery_type: single-node
|
||||
|
||||
# Plugins to install (empty by default)
|
||||
opensearch_plugins: []
|
||||
|
||||
# Sysctl: vm.max_map_count required by OpenSearch
|
||||
opensearch_vm_max_map_count: 262144
|
||||
|
||||
# --- OpenSearch Dashboards ---
|
||||
opensearch_dashboards_version: "{{ opensearch_version }}"
|
||||
opensearch_dashboards_install_dir: /opt/opensearch-dashboards
|
||||
opensearch_dashboards_config_dir: /etc/opensearch-dashboards
|
||||
opensearch_dashboards_port: 5601
|
||||
opensearch_dashboards_opensearch_url: "http://127.0.0.1:{{ opensearch_http_port }}"
|
||||
12
ansible/roles/opensearch/handlers/main.yml
Normal file
12
ansible/roles/opensearch/handlers/main.yml
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
---
|
||||
- name: Restart opensearch
|
||||
ansible.builtin.systemd:
|
||||
name: opensearch
|
||||
state: restarted
|
||||
daemon_reload: true
|
||||
|
||||
- name: Restart opensearch-dashboards
|
||||
ansible.builtin.systemd:
|
||||
name: opensearch-dashboards
|
||||
state: restarted
|
||||
daemon_reload: true
|
||||
179
ansible/roles/opensearch/tasks/main.yml
Normal file
179
ansible/roles/opensearch/tasks/main.yml
Normal file
|
|
@ -0,0 +1,179 @@
|
|||
---
|
||||
# Installs OpenSearch from the upstream tarball (bundled JDK) and runs it
|
||||
# under systemd in single-node mode for logs.vntx.net.
|
||||
|
||||
- name: Set vm.max_map_count for OpenSearch
|
||||
ansible.posix.sysctl:
|
||||
name: vm.max_map_count
|
||||
value: "{{ opensearch_vm_max_map_count }}"
|
||||
state: present
|
||||
reload: true
|
||||
|
||||
- name: Create opensearch group
|
||||
ansible.builtin.group:
|
||||
name: "{{ opensearch_group }}"
|
||||
system: true
|
||||
state: present
|
||||
|
||||
- name: Create opensearch user
|
||||
ansible.builtin.user:
|
||||
name: "{{ opensearch_user }}"
|
||||
group: "{{ opensearch_group }}"
|
||||
system: true
|
||||
shell: /usr/sbin/nologin
|
||||
home: "{{ opensearch_data_dir }}"
|
||||
create_home: false
|
||||
state: present
|
||||
|
||||
- name: Create opensearch directories
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: directory
|
||||
owner: "{{ opensearch_user }}"
|
||||
group: "{{ opensearch_group }}"
|
||||
mode: "0750"
|
||||
loop:
|
||||
- "{{ opensearch_install_dir }}"
|
||||
- "{{ opensearch_config_dir }}"
|
||||
- "{{ opensearch_data_dir }}"
|
||||
- "{{ opensearch_log_dir }}"
|
||||
|
||||
- name: Check installed opensearch version
|
||||
ansible.builtin.command: "{{ opensearch_install_dir }}/bin/opensearch --version"
|
||||
register: opensearch_installed
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
|
||||
- name: Decide whether opensearch needs (re)install
|
||||
ansible.builtin.set_fact:
|
||||
opensearch_needs_install: >-
|
||||
{{
|
||||
opensearch_installed.rc != 0
|
||||
or opensearch_version not in (opensearch_installed.stdout | default(''))
|
||||
and opensearch_version not in (opensearch_installed.stderr | default(''))
|
||||
}}
|
||||
|
||||
- name: Download opensearch tarball
|
||||
ansible.builtin.get_url:
|
||||
url: "https://artifacts.opensearch.org/releases/bundle/opensearch/{{ opensearch_version }}/opensearch-{{ opensearch_version }}-linux-{{ opensearch_arch }}.tar.gz"
|
||||
dest: "/tmp/opensearch-{{ opensearch_version }}-linux-{{ opensearch_arch }}.tar.gz"
|
||||
mode: "0644"
|
||||
when: opensearch_needs_install
|
||||
|
||||
- name: Extract opensearch tarball to install dir
|
||||
ansible.builtin.unarchive:
|
||||
src: "/tmp/opensearch-{{ opensearch_version }}-linux-{{ opensearch_arch }}.tar.gz"
|
||||
dest: "{{ opensearch_install_dir }}"
|
||||
remote_src: true
|
||||
extra_opts:
|
||||
- --strip-components=1
|
||||
owner: "{{ opensearch_user }}"
|
||||
group: "{{ opensearch_group }}"
|
||||
when: opensearch_needs_install
|
||||
notify: Restart opensearch
|
||||
|
||||
- name: Render opensearch.yml
|
||||
ansible.builtin.template:
|
||||
src: opensearch.yml.j2
|
||||
dest: "{{ opensearch_config_dir }}/opensearch.yml"
|
||||
owner: "{{ opensearch_user }}"
|
||||
group: "{{ opensearch_group }}"
|
||||
mode: "0640"
|
||||
notify: Restart opensearch
|
||||
|
||||
- name: Render jvm.options
|
||||
ansible.builtin.template:
|
||||
src: jvm.options.j2
|
||||
dest: "{{ opensearch_config_dir }}/jvm.options"
|
||||
owner: "{{ opensearch_user }}"
|
||||
group: "{{ opensearch_group }}"
|
||||
mode: "0640"
|
||||
notify: Restart opensearch
|
||||
|
||||
- name: Install opensearch systemd unit
|
||||
ansible.builtin.template:
|
||||
src: opensearch.service.j2
|
||||
dest: /etc/systemd/system/opensearch.service
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
notify: Restart opensearch
|
||||
|
||||
- name: Enable and start opensearch
|
||||
ansible.builtin.systemd:
|
||||
name: opensearch
|
||||
enabled: true
|
||||
state: started
|
||||
daemon_reload: true
|
||||
|
||||
# --- OpenSearch Dashboards ---
|
||||
|
||||
- name: Create opensearch-dashboards directories
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: directory
|
||||
owner: "{{ opensearch_user }}"
|
||||
group: "{{ opensearch_group }}"
|
||||
mode: "0750"
|
||||
loop:
|
||||
- "{{ opensearch_dashboards_install_dir }}"
|
||||
- "{{ opensearch_dashboards_config_dir }}"
|
||||
|
||||
- name: Check installed opensearch-dashboards version
|
||||
ansible.builtin.command: "{{ opensearch_dashboards_install_dir }}/bin/opensearch-dashboards --version"
|
||||
register: dashboards_installed
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
|
||||
- name: Decide whether dashboards needs (re)install
|
||||
ansible.builtin.set_fact:
|
||||
dashboards_needs_install: >-
|
||||
{{
|
||||
dashboards_installed.rc != 0
|
||||
or opensearch_dashboards_version not in (dashboards_installed.stdout | default(''))
|
||||
and opensearch_dashboards_version not in (dashboards_installed.stderr | default(''))
|
||||
}}
|
||||
|
||||
- name: Download opensearch-dashboards tarball
|
||||
ansible.builtin.get_url:
|
||||
url: "https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/{{ opensearch_dashboards_version }}/opensearch-dashboards-{{ opensearch_dashboards_version }}-linux-{{ opensearch_arch }}.tar.gz"
|
||||
dest: "/tmp/opensearch-dashboards-{{ opensearch_dashboards_version }}-linux-{{ opensearch_arch }}.tar.gz"
|
||||
mode: "0644"
|
||||
when: dashboards_needs_install
|
||||
|
||||
- name: Extract opensearch-dashboards tarball to install dir
|
||||
ansible.builtin.unarchive:
|
||||
src: "/tmp/opensearch-dashboards-{{ opensearch_dashboards_version }}-linux-{{ opensearch_arch }}.tar.gz"
|
||||
dest: "{{ opensearch_dashboards_install_dir }}"
|
||||
remote_src: true
|
||||
extra_opts:
|
||||
- --strip-components=1
|
||||
owner: "{{ opensearch_user }}"
|
||||
group: "{{ opensearch_group }}"
|
||||
when: dashboards_needs_install
|
||||
notify: Restart opensearch-dashboards
|
||||
|
||||
- name: Render opensearch-dashboards.yml
|
||||
ansible.builtin.template:
|
||||
src: opensearch-dashboards.yml.j2
|
||||
dest: "{{ opensearch_dashboards_config_dir }}/opensearch_dashboards.yml"
|
||||
owner: "{{ opensearch_user }}"
|
||||
group: "{{ opensearch_group }}"
|
||||
mode: "0640"
|
||||
notify: Restart opensearch-dashboards
|
||||
|
||||
- name: Install opensearch-dashboards systemd unit
|
||||
ansible.builtin.template:
|
||||
src: opensearch-dashboards.service.j2
|
||||
dest: /etc/systemd/system/opensearch-dashboards.service
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
notify: Restart opensearch-dashboards
|
||||
|
||||
- name: Enable and start opensearch-dashboards
|
||||
ansible.builtin.systemd:
|
||||
name: opensearch-dashboards
|
||||
enabled: true
|
||||
state: started
|
||||
daemon_reload: true
|
||||
24
ansible/roles/opensearch/templates/jvm.options.j2
Normal file
24
ansible/roles/opensearch/templates/jvm.options.j2
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
# {{ ansible_managed | default('Managed by Ansible') }}
|
||||
#
|
||||
# JVM options for OpenSearch
|
||||
|
||||
## Heap size — set min and max to the same value
|
||||
-Xms{{ opensearch_heap_min }}
|
||||
-Xmx{{ opensearch_heap_max }}
|
||||
|
||||
## GC configuration
|
||||
-XX:+UseG1GC
|
||||
-XX:MaxGCPauseMillis=500
|
||||
|
||||
## Basic JVM tuning
|
||||
-XX:+ExitOnOutOfMemoryError
|
||||
-XX:+HeapDumpOnOutOfMemoryError
|
||||
-Djava.awt.headless=true
|
||||
-Djava.security.egd=file:/dev/urandom
|
||||
-Dfile.encoding=UTF-8
|
||||
|
||||
## Heap dump path
|
||||
-XX:HeapDumpPath={{ opensearch_data_dir }}
|
||||
|
||||
## Temp directory
|
||||
-Djava.io.tmpdir=/tmp
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
[Unit]
|
||||
Description=OpenSearch Dashboards visualization server
|
||||
After=network-online.target opensearch.service
|
||||
Wants=network-online.target opensearch.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User={{ opensearch_user }}
|
||||
Group={{ opensearch_group }}
|
||||
|
||||
Environment=OPENSEARCH_DASHBOARDS_HOME={{ opensearch_dashboards_install_dir }}
|
||||
Environment=OPENSEARCH_DASHBOARDS_PATH_CONF={{ opensearch_dashboards_config_dir }}
|
||||
|
||||
ExecStart={{ opensearch_dashboards_install_dir }}/bin/opensearch-dashboards \
|
||||
-c {{ opensearch_dashboards_config_dir }}/opensearch_dashboards.yml
|
||||
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
NoNewPrivileges=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
# {{ ansible_managed | default('Managed by Ansible') }}
|
||||
#
|
||||
# OpenSearch Dashboards configuration
|
||||
|
||||
server.host: 127.0.0.1
|
||||
server.port: {{ opensearch_dashboards_port }}
|
||||
|
||||
opensearch.hosts:
|
||||
- {{ opensearch_dashboards_opensearch_url }}
|
||||
|
||||
opensearch.requestTimeout: 30000
|
||||
26
ansible/roles/opensearch/templates/opensearch.service.j2
Normal file
26
ansible/roles/opensearch/templates/opensearch.service.j2
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
[Unit]
|
||||
Description=OpenSearch log search and analytics engine
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User={{ opensearch_user }}
|
||||
Group={{ opensearch_group }}
|
||||
|
||||
Environment=OPENSEARCH_HOME={{ opensearch_install_dir }}
|
||||
Environment=OPENSEARCH_JAVA_HOME={{ opensearch_install_dir }}/jdk
|
||||
|
||||
ExecStart={{ opensearch_install_dir }}/bin/opensearch
|
||||
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
LimitNOFILE=65536
|
||||
LimitNPROC=4096
|
||||
NoNewPrivileges=true
|
||||
ProtectHome=true
|
||||
ProtectSystem=full
|
||||
ReadWritePaths={{ opensearch_data_dir }} {{ opensearch_log_dir }} {{ opensearch_config_dir }}
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
25
ansible/roles/opensearch/templates/opensearch.yml.j2
Normal file
25
ansible/roles/opensearch/templates/opensearch.yml.j2
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
# {{ ansible_managed | default('Managed by Ansible') }}
|
||||
#
|
||||
# OpenSearch configuration for {{ opensearch_cluster_name }}
|
||||
|
||||
cluster.name: {{ opensearch_cluster_name }}
|
||||
node.name: {{ opensearch_node_name }}
|
||||
|
||||
# Directories
|
||||
path.data: {{ opensearch_data_dir }}
|
||||
path.logs: {{ opensearch_log_dir }}
|
||||
|
||||
# Network
|
||||
network.host: 127.0.0.1
|
||||
http.port: {{ opensearch_http_port }}
|
||||
transport.port: {{ opensearch_transport_port }}
|
||||
|
||||
# Single-node discovery (no cluster formation needed)
|
||||
discovery.type: {{ opensearch_discovery_type }}
|
||||
|
||||
# Security — disabled for internal-only binding on loopback
|
||||
plugins.security.disabled: true
|
||||
|
||||
# Performance
|
||||
indices.memory.index_buffer_size: 20%
|
||||
bootstrap.memory_lock: false
|
||||
|
|
@ -27,12 +27,12 @@ ServiceAccount token + the cluster's CA cert.
|
|||
3. Copy them onto the prom host:
|
||||
|
||||
```bash
|
||||
ssh ansible@10.0.15.31 'sudo install -d -o prometheus -g prometheus -m 0750 /etc/prometheus/k8s'
|
||||
printf '%s' "$TOKEN" | ssh ansible@10.0.15.31 \
|
||||
ssh ansible@10.0.19.31 'sudo install -d -o prometheus -g prometheus -m 0750 /etc/prometheus/k8s'
|
||||
printf '%s' "$TOKEN" | ssh ansible@10.0.19.31 \
|
||||
'sudo tee /etc/prometheus/k8s/token >/dev/null && \
|
||||
sudo chown prometheus:prometheus /etc/prometheus/k8s/token && \
|
||||
sudo chmod 0640 /etc/prometheus/k8s/token'
|
||||
printf '%s' "$CA" | ssh ansible@10.0.15.31 \
|
||||
printf '%s' "$CA" | ssh ansible@10.0.19.31 \
|
||||
'sudo tee /etc/prometheus/k8s/ca.crt >/dev/null && \
|
||||
sudo chown prometheus:prometheus /etc/prometheus/k8s/ca.crt && \
|
||||
sudo chmod 0640 /etc/prometheus/k8s/ca.crt'
|
||||
|
|
@ -41,7 +41,7 @@ ServiceAccount token + the cluster's CA cert.
|
|||
4. Reload Prometheus:
|
||||
|
||||
```bash
|
||||
ssh ansible@10.0.15.31 'sudo systemctl reload prometheus'
|
||||
ssh ansible@10.0.19.31 'sudo systemctl reload prometheus'
|
||||
```
|
||||
|
||||
The token persists across pod restarts (it's a long-lived `kubernetes.io/service-account-token`
|
||||
|
|
@ -49,7 +49,7 @@ Secret, not a projected token), so it does not need to be rotated unless you del
|
|||
|
||||
## Verifying scrape targets
|
||||
|
||||
Open `http://10.0.15.31:9090/targets` — the `kubernetes-apiservers`,
|
||||
Open `http://10.0.19.31:9090/targets` — the `kubernetes-apiservers`,
|
||||
`kubernetes-nodes`, and `kubernetes-cadvisor` jobs should all be UP.
|
||||
|
||||
If `kubernetes-pods` / `kubernetes-service-endpoints` are empty, that's expected
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ prometheus_evaluation_interval: "30s"
|
|||
# Path to the Kubernetes ServiceAccount bearer token + CA on the prom host.
|
||||
# Bootstrap copies these from the cluster (see roles/prometheus/README.md).
|
||||
prometheus_k8s_enabled: true
|
||||
prometheus_k8s_api_server: "https://10.0.15.1:6443"
|
||||
prometheus_k8s_api_server: "https://10.0.19.10:6443"
|
||||
prometheus_k8s_token_file: "{{ prometheus_config_dir }}/k8s/token"
|
||||
prometheus_k8s_ca_file: "{{ prometheus_config_dir }}/k8s/ca.crt"
|
||||
|
||||
|
|
@ -30,5 +30,5 @@ prometheus_alertmanager_targets:
|
|||
|
||||
# Extra static scrape targets, e.g. other node_exporters on home_servers
|
||||
# - job_name: node_exporter
|
||||
# targets: ["10.0.15.31:9100", "10.0.15.22:9100"]
|
||||
# targets: ["10.0.19.31:9100", "10.0.19.22:9100"]
|
||||
prometheus_extra_scrape_configs: []
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ This directory contains Kubernetes application manifests for the Talos home clus
|
|||
|
||||
- **Cluster**: home-cluster (Talos)
|
||||
- **Nodes**: 6 (3 control plane + 3 workers)
|
||||
- **IPs**: 10.0.15.1-6
|
||||
- **IPs**: 10.0.19.1-6
|
||||
- **Kubernetes Version**: v1.35.0
|
||||
- **Talos Version**: v1.12.1
|
||||
|
||||
|
|
|
|||
|
|
@ -9,9 +9,9 @@ data:
|
|||
{
|
||||
"clusterID": "7e968d44-61b4-4342-8318-2e8e48808e68",
|
||||
"monitors": [
|
||||
"10.0.15.101:6789",
|
||||
"10.0.15.102:6789",
|
||||
"10.0.15.103:6789"
|
||||
"10.0.19.101:6789",
|
||||
"10.0.19.102:6789",
|
||||
"10.0.19.103:6789"
|
||||
]
|
||||
}
|
||||
]
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ This directory contains manifests for deploying the Ceph CSI RBD driver to use P
|
|||
## Configuration
|
||||
|
||||
- **Ceph Cluster FSID**: `7e968d44-61b4-4342-8318-2e8e48808e68`
|
||||
- **Ceph Monitors**: 10.0.15.101:6789, 10.0.15.102:6789, 10.0.15.103:6789
|
||||
- **Ceph Monitors**: 10.0.19.101:6789, 10.0.19.102:6789, 10.0.19.103:6789
|
||||
- **Ceph Pool**: kubernetes
|
||||
- **Ceph User**: client.kubernetes
|
||||
- **StorageClass**: ceph-rbd (default)
|
||||
|
|
@ -64,7 +64,7 @@ kubectl get nodes --show-labels | grep topology
|
|||
kubectl get csinode -o yaml | grep topologyKeys -A 2
|
||||
|
||||
# List RBD images in Ceph
|
||||
ssh root@10.0.15.101 "rbd ls -p kubernetes"
|
||||
ssh root@10.0.19.101 "rbd ls -p kubernetes"
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
|
@ -88,7 +88,7 @@ kubectl delete -f test-pvc.yaml
|
|||
|
||||
```bash
|
||||
# SSH to any Proxmox node
|
||||
ssh root@10.0.15.101
|
||||
ssh root@10.0.19.101
|
||||
|
||||
# List pools
|
||||
ceph osd pool ls
|
||||
|
|
@ -123,7 +123,7 @@ kubectl delete pods -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner
|
|||
|
||||
### Check Ceph Health
|
||||
```bash
|
||||
ssh root@10.0.15.101 "ceph health detail"
|
||||
ssh root@10.0.19.101 "ceph health detail"
|
||||
```
|
||||
|
||||
## Files
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ data:
|
|||
filename: /run/promtail/positions.yaml
|
||||
|
||||
clients:
|
||||
- url: http://10.0.15.31:3100/loki/api/v1/push
|
||||
- url: http://10.0.19.31:3100/loki/api/v1/push
|
||||
backoff_config:
|
||||
min_period: 500ms
|
||||
max_period: 5m
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ spec:
|
|||
- name: exporter
|
||||
image: docker.io/joshuasing/starlink_exporter:0.9.0
|
||||
# Defaults already match this network: -dish 192.168.100.1:9200,
|
||||
# -listen :9451. Dish is reachable via 10.0.15.254 (home router)
|
||||
# -listen :9451. Dish is reachable via 10.0.19.254 (home router)
|
||||
# while the Starlink router is in bypass mode.
|
||||
ports:
|
||||
- name: metrics
|
||||
|
|
|
|||
|
|
@ -7,24 +7,24 @@ This document contains notes on how the Talos home cluster was created and how t
|
|||
- **Cluster Name**: home-cluster
|
||||
- **Kubernetes Version**: v1.36.1
|
||||
- **Talos Version**: v1.13.5
|
||||
- **Control Plane Nodes**: 3 (10.0.15.1-3)
|
||||
- **Worker Nodes**: 3 (10.0.15.4-6)
|
||||
- **API Endpoint**: https://10.0.15.1:6443
|
||||
- **Control Plane Nodes**: 3 (10.0.19.1-3)
|
||||
- **Worker Nodes**: 3 (10.0.19.4-6)
|
||||
- **API Endpoint**: https://10.0.19.10:6443
|
||||
|
||||
## Node Details
|
||||
|
||||
| Hostname | Role | IP | Proxmox Node | VMID | MAC Address |
|
||||
|----------|------|-----|--------------|------|-------------|
|
||||
| talos-cp1 | Control Plane | 10.0.15.1 | node1 | 200 | BC:24:11:9B:48:92 |
|
||||
| talos-cp2 | Control Plane | 10.0.15.2 | node2 | 201 | BC:24:11:62:7B:3F |
|
||||
| talos-cp3 | Control Plane | 10.0.15.3 | node3 | 202 | BC:24:11:D4:2F:ED |
|
||||
| talos-worker1 | Worker | 10.0.15.4 | node1 | 210 | BC:24:11:43:3F:FF |
|
||||
| talos-worker2 | Worker | 10.0.15.5 | node2 | 211 | BC:24:11:62:C4:8F |
|
||||
| talos-worker3 | Worker | 10.0.15.6 | node3 | 212 | BC:24:11:3F:8E:1A |
|
||||
| talos-cp1 | Control Plane | 10.0.19.1 | node1 | 200 | BC:24:11:9B:48:92 |
|
||||
| talos-cp2 | Control Plane | 10.0.19.2 | node2 | 201 | BC:24:11:62:7B:3F |
|
||||
| talos-cp3 | Control Plane | 10.0.19.3 | node3 | 202 | BC:24:11:D4:2F:ED |
|
||||
| talos-worker1 | Worker | 10.0.19.4 | node1 | 210 | BC:24:11:43:3F:FF |
|
||||
| talos-worker2 | Worker | 10.0.19.5 | node2 | 211 | BC:24:11:62:C4:8F |
|
||||
| talos-worker3 | Worker | 10.0.19.6 | node3 | 212 | BC:24:11:3F:8E:1A |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Proxmox home cluster with nodes: node1, node2, node3 (10.0.15.101-103)
|
||||
- Proxmox home cluster with nodes: node1, node2, node3 (10.0.19.101-103)
|
||||
- Talos Linux template at VMID 9001 on node1
|
||||
- DHCP static leases configured for all MAC addresses above
|
||||
- Terraform/OpenTofu configured with Proxmox provider
|
||||
|
|
@ -45,14 +45,14 @@ Generate the base control plane and worker configs. Per-node settings (hostname,
|
|||
|
||||
```bash
|
||||
# Generate base control plane config (used for all CP nodes with per-node patches)
|
||||
talosctl gen config home-cluster https://10.0.15.1:6443 \
|
||||
talosctl gen config home-cluster https://10.0.19.10:6443 \
|
||||
--with-secrets secrets.yaml \
|
||||
--kubernetes-version v1.36.1 \
|
||||
--output-types controlplane \
|
||||
--output controlplane.yaml
|
||||
|
||||
# Generate base worker config (used for all worker nodes with per-node patches)
|
||||
talosctl gen config home-cluster https://10.0.15.1:6443 \
|
||||
talosctl gen config home-cluster https://10.0.19.10:6443 \
|
||||
--with-secrets secrets.yaml \
|
||||
--kubernetes-version v1.36.1 \
|
||||
--output-types worker \
|
||||
|
|
@ -67,14 +67,14 @@ Apply the base config with the appropriate per-node patch:
|
|||
|
||||
```bash
|
||||
# Control plane nodes
|
||||
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.1 --file controlplane.yaml --config-patch @patches/cp1.yaml --insecure
|
||||
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.2 --file controlplane.yaml --config-patch @patches/cp2.yaml --insecure
|
||||
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.3 --file controlplane.yaml --config-patch @patches/cp3.yaml --insecure
|
||||
talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.1 --file controlplane.yaml --config-patch @patches/cp1.yaml --insecure
|
||||
talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.2 --file controlplane.yaml --config-patch @patches/cp2.yaml --insecure
|
||||
talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.3 --file controlplane.yaml --config-patch @patches/cp3.yaml --insecure
|
||||
|
||||
# Worker nodes
|
||||
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.4 --file worker.yaml --config-patch @patches/worker1.yaml --insecure
|
||||
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.5 --file worker.yaml --config-patch @patches/worker2.yaml --insecure
|
||||
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.6 --file worker.yaml --config-patch @patches/worker3.yaml --insecure
|
||||
talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.4 --file worker.yaml --config-patch @patches/worker1.yaml --insecure
|
||||
talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.5 --file worker.yaml --config-patch @patches/worker2.yaml --insecure
|
||||
talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.6 --file worker.yaml --config-patch @patches/worker3.yaml --insecure
|
||||
```
|
||||
|
||||
### 4. Deploy VMs via Terraform
|
||||
|
|
@ -111,7 +111,7 @@ cd /Users/graham/dev/infra/talos
|
|||
sleep 30
|
||||
|
||||
# Verify all nodes are up
|
||||
for ip in 10.0.15.{1..6}; do
|
||||
for ip in 10.0.19.{1..6}; do
|
||||
echo -n "$ip: "
|
||||
ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down"
|
||||
done
|
||||
|
|
@ -124,7 +124,7 @@ cd /Users/graham/dev/infra/talos
|
|||
sleep 30
|
||||
|
||||
# Verify all nodes are up
|
||||
for ip in 10.0.15.{1..6}; do
|
||||
for ip in 10.0.19.{1..6}; do
|
||||
echo -n "$ip: "
|
||||
ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down"
|
||||
done
|
||||
|
|
@ -134,10 +134,10 @@ done
|
|||
|
||||
```bash
|
||||
# Set talosctl endpoints
|
||||
talosctl --talosconfig talosconfig config endpoint 10.0.15.1 10.0.15.2 10.0.15.3
|
||||
talosctl --talosconfig talosconfig config endpoint 10.0.19.1 10.0.19.2 10.0.19.3
|
||||
|
||||
# Bootstrap Kubernetes on the first control plane node
|
||||
talosctl --talosconfig talosconfig bootstrap --nodes 10.0.15.1
|
||||
talosctl --talosconfig talosconfig bootstrap --nodes 10.0.19.1
|
||||
```
|
||||
|
||||
### 8. Retrieve Kubeconfig
|
||||
|
|
@ -147,7 +147,7 @@ talosctl --talosconfig talosconfig bootstrap --nodes 10.0.15.1
|
|||
sleep 60
|
||||
|
||||
# Retrieve and merge kubeconfig
|
||||
talosctl --talosconfig talosconfig kubeconfig --nodes 10.0.15.1 --force
|
||||
talosctl --talosconfig talosconfig kubeconfig --nodes 10.0.19.1 --force
|
||||
|
||||
# Switch to the cluster
|
||||
kubectl config use-context admin@home-cluster
|
||||
|
|
@ -223,14 +223,14 @@ If seeing "static hostname is already set" error:
|
|||
|
||||
Set endpoints explicitly:
|
||||
```bash
|
||||
talosctl --talosconfig talosconfig config endpoint 10.0.15.1 10.0.15.2 10.0.15.3
|
||||
talosctl --talosconfig talosconfig config endpoint 10.0.19.1 10.0.19.2 10.0.19.3
|
||||
```
|
||||
|
||||
### Nodes not appearing in kubectl
|
||||
|
||||
Wait 1-2 minutes after bootstrap. Check:
|
||||
```bash
|
||||
talosctl --talosconfig talosconfig service --nodes 10.0.15.1
|
||||
talosctl --talosconfig talosconfig service --nodes 10.0.19.1
|
||||
# Look for kubelet, etcd, apid as "Running"
|
||||
```
|
||||
|
||||
|
|
@ -257,27 +257,27 @@ talosctl --talosconfig talosconfig service --nodes 10.0.15.1
|
|||
### Upgrading Talos
|
||||
|
||||
**Pre-upgrade checklist:**
|
||||
1. Check current versions: `for ip in 10.0.15.{1..6}; do echo -n "$ip: "; talosctl --talosconfig talosconfig version --nodes $ip 2>&1 | grep "Tag:" | grep -v Client; done`
|
||||
1. Check current versions: `for ip in 10.0.19.{1..6}; do echo -n "$ip: "; talosctl --talosconfig talosconfig version --nodes $ip 2>&1 | grep "Tag:" | grep -v Client; done`
|
||||
2. Check latest release: `curl -sL https://github.com/siderolabs/talos/releases/latest | grep -o 'v[0-9]\+\.[0-9]\+\.[0-9]\+' | head -1`
|
||||
3. Verify cluster health: `kubectl get nodes -o wide`
|
||||
4. Check extensions on workers: `talosctl --talosconfig talosconfig get extensions -n 10.0.15.4`
|
||||
4. Check extensions on workers: `talosctl --talosconfig talosconfig get extensions -n 10.0.19.4`
|
||||
|
||||
**Important:** CP nodes use the **stock** installer image. Worker nodes use the **factory** image (with iscsi-tools + util-linux-tools extensions). They MUST be upgraded separately with their respective images.
|
||||
|
||||
```bash
|
||||
# 1. Upgrade control plane nodes (stock image)
|
||||
talosctl --talosconfig talosconfig upgrade \
|
||||
--nodes 10.0.15.1,10.0.15.2,10.0.15.3 \
|
||||
--nodes 10.0.19.1,10.0.19.2,10.0.19.3 \
|
||||
--image ghcr.io/siderolabs/installer:v<VERSION>
|
||||
|
||||
# 2. Upgrade worker nodes (factory image with extensions)
|
||||
# The schematic ID can be found via: talosctl --talosconfig talosconfig get extensions -n <worker-ip>
|
||||
talosctl --talosconfig talosconfig upgrade \
|
||||
--nodes 10.0.15.4,10.0.15.5,10.0.15.6 \
|
||||
--nodes 10.0.19.4,10.0.19.5,10.0.19.6 \
|
||||
--image factory.talos.dev/installer/<SCHEMATIC_ID>:v<VERSION>
|
||||
|
||||
# 3. If workers fail to auto-reboot/uncordon, do it manually:
|
||||
talosctl --talosconfig talosconfig reboot --nodes 10.0.15.4,10.0.15.5,10.0.15.6
|
||||
talosctl --talosconfig talosconfig reboot --nodes 10.0.19.4,10.0.19.5,10.0.19.6
|
||||
# Wait ~90s for reboot, then:
|
||||
kubectl uncordon talos-worker1 talos-worker2 talos-worker3
|
||||
```
|
||||
|
|
|
|||
|
|
@ -4,22 +4,20 @@
|
|||
|
||||
- **Cluster Name**: home-cluster
|
||||
- **Kubernetes Version**: v1.36.1
|
||||
- **Control Plane Nodes**: 3 (10.0.15.1-3)
|
||||
- **Worker Nodes**: 4 (10.0.15.4-7)
|
||||
- **Bare Metal Worker**: talos4 at 10.0.15.7 (HP t620)
|
||||
- **API Endpoint**: https://10.0.15.1:6443
|
||||
- **Control Plane Nodes**: 3 (10.0.19.1-3)
|
||||
- **Worker Nodes**: 3 (10.0.19.4-6)
|
||||
- **API Endpoint**: https://10.0.19.10:6443 (shared VIP)
|
||||
|
||||
## Node Details
|
||||
|
||||
| Node Name | Role | IP | Proxmox Node | VMID |
|
||||
|-----------|------|-----|--------------|------|
|
||||
| talos-cp1 | Control Plane | 10.0.15.1 | node1 | 200 |
|
||||
| talos-cp2 | Control Plane | 10.0.15.2 | node2 | 201 |
|
||||
| talos-cp3 | Control Plane | 10.0.15.3 | node3 | 202 |
|
||||
| talos-worker1 | Worker | 10.0.15.4 | node1 | 210 |
|
||||
| talos-worker2 | Worker | 10.0.15.5 | node2 | 211 |
|
||||
| talos-worker3 | Worker | 10.0.15.6 | node3 | 212 |
|
||||
| talos4 | Worker | 10.0.15.7 | bare-metal (HP t620) | — |
|
||||
| talos-cp1 | Control Plane | 10.0.19.1 | node1 | 200 |
|
||||
| talos-cp2 | Control Plane | 10.0.19.2 | node2 | 201 |
|
||||
| talos-cp3 | Control Plane | 10.0.19.3 | node3 | 202 |
|
||||
| talos-worker1 | Worker | 10.0.19.4 | node1 | 210 |
|
||||
| talos-worker2 | Worker | 10.0.19.5 | node2 | 211 |
|
||||
| talos-worker3 | Worker | 10.0.19.6 | node3 | 212 |
|
||||
|
||||
## Configuration Files
|
||||
|
||||
|
|
|
|||
|
|
@ -19,6 +19,12 @@ machine:
|
|||
- 9.9.9.9
|
||||
- 149.112.112.112
|
||||
- 1.1.1.1
|
||||
interfaces:
|
||||
- deviceSelector:
|
||||
physical: true
|
||||
dhcp: true
|
||||
vip:
|
||||
ip: 10.0.19.10
|
||||
|
||||
install:
|
||||
disk: /dev/sda # The disk used for installations.
|
||||
|
|
@ -42,7 +48,7 @@ cluster:
|
|||
id: d9wM2zW6ZcxGVQp8rujQajusx-HrV1s-gRfGfc5wExM= # Globally unique identifier for this cluster (base64 encoded random 32 bytes).
|
||||
secret: V44y9ckGbghbUYYuLTqjagj0jzx3TcgfZ9a7C9J97mE= # Shared secret of cluster (base64 encoded random 32 bytes).
|
||||
controlPlane:
|
||||
endpoint: https://10.0.15.1:6443 # Endpoint is the canonical controlplane endpoint, which can be an IP address or a DNS hostname.
|
||||
endpoint: https://10.0.19.10:6443 # Shared VIP floating across all control plane nodes.
|
||||
clusterName: home-cluster # Configures the cluster's name.
|
||||
network:
|
||||
dnsDomain: cluster.local # The domain used by Kubernetes DNS.
|
||||
|
|
|
|||
3
talos/patches/endpoint.yaml
Normal file
3
talos/patches/endpoint.yaml
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
cluster:
|
||||
controlPlane:
|
||||
endpoint: https://10.0.19.10:6443
|
||||
8
talos/patches/vip.yaml
Normal file
8
talos/patches/vip.yaml
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
machine:
|
||||
network:
|
||||
interfaces:
|
||||
- deviceSelector:
|
||||
physical: true
|
||||
dhcp: true
|
||||
vip:
|
||||
ip: 10.0.19.10
|
||||
|
|
@ -41,7 +41,7 @@ cluster:
|
|||
id: d9wM2zW6ZcxGVQp8rujQajusx-HrV1s-gRfGfc5wExM= # Globally unique identifier for this cluster (base64 encoded random 32 bytes).
|
||||
secret: V44y9ckGbghbUYYuLTqjagj0jzx3TcgfZ9a7C9J97mE= # Shared secret of cluster (base64 encoded random 32 bytes).
|
||||
controlPlane:
|
||||
endpoint: https://10.0.15.1:6443 # Endpoint is the canonical controlplane endpoint, which can be an IP address or a DNS hostname.
|
||||
endpoint: https://10.0.19.10:6443 # Shared VIP floating across all control plane nodes.
|
||||
clusterName: home-cluster # Configures the cluster's name.
|
||||
network:
|
||||
dnsDomain: cluster.local # The domain used by Kubernetes DNS.
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue