graham/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

renumber infrastructure from 10.0.15.0/24 to 10.0.16.0/22 (hosts in 10.0.19.x)

Browse source 
Proxmox: node1-3 → 10.0.19.101-103
Talos cp1-3 → 10.0.19.1-3, workers → 10.0.19.4-6
K8s endpoint → VIP https://10.0.19.10:6443
Ansible: prom → 10.0.19.31, db → 10.0.19.30
Talos: added VIP block to controlplane.yaml base config
Promtail: Loki URL → 10.0.19.31
Docs: all references updated, talos4 removed
This commit is contained in:
Graham McIntire 2026-07-18 08:28:16 -05:00
parent 514b346662
commit 39bedb08d1
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
31 changed files with 672 additions and 71 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
CLAUDE.md
View file

@ -169,14 +169,14 @@ tofu plan
## Recent Infrastructure Updates ## Recent Infrastructure Updates
### Talos Home Cluster (10.0.15.1-7) ### Talos Home Cluster (10.0.19.1-6)
- **Purpose**: Primary home lab Kubernetes cluster - **Purpose**: Primary home lab Kubernetes cluster
- **Cluster Name**: home-cluster - **Cluster Name**: home-cluster
- **Kubernetes Version**: v1.36.1 - **Kubernetes Version**: v1.36.1
- **Control Plane Nodes**: 3 (10.0.15.1-3) - **Control Plane Nodes**: 3 (10.0.19.1-3)
- **Worker Nodes**: 4 (10.0.15.4-7) - **Worker Nodes**: 3 (10.0.19.4-6)
- **API Endpoint**: https://10.0.15.1:6443 - **API Endpoint**: https://10.0.19.10:6443
- **Proxmox Cluster**: "home" (node1, node2, node3 at 10.0.15.101-103) - **Proxmox Cluster**: "home" (node1, node2, node3 at 10.0.19.101-103)
- **VM Disks**: local-lvm on each Proxmox node - **VM Disks**: local-lvm on each Proxmox node
- **Persistent Storage**: TBD (Longhorn was removed — evaluating alternatives) - **Persistent Storage**: TBD (Longhorn was removed — evaluating alternatives)
- **CNI**: Flannel - **CNI**: Flannel
@ -234,7 +234,7 @@ tofu plan
- As you learn new things, keep claude.md updated - As you learn new things, keep claude.md updated
## Cluster ## Cluster
- **home-cluster** (Talos): 10.0.15.1-6 — only Kubernetes cluster. External traffic enters via the `cloudflared` tunnel deployment in the `cloudflared` namespace; per-hostname routing rules live in Cloudflare's dashboard, not in the cluster. - **home-cluster** (Talos): 10.0.19.1-6 — only Kubernetes cluster. External traffic enters via the `cloudflared` tunnel deployment in the `cloudflared` namespace; per-hostname routing rules live in Cloudflare's dashboard, not in the cluster.
## Application Deployment Patterns ## Application Deployment Patterns

214
RENUMBER.md Normal file
View file

@ -0,0 +1,214 @@
# Renumber: 10.0.15.0/24 → 10.0.16.0/22 (hosts in 10.0.19.x)
Move Proxmox nodes, Talos cluster, and support VMs onto 10.0.16.0/22
(mask 255.255.252.0, broadcast 10.0.19.255). Server range: 10.0.19.x.
Delete this file when done.
## Address map
| Host | MAC | Old | New |
|----------------|-------------------|-------------|-------------|
| talos-cp1 | BC:24:11:9B:48:92 | 10.0.15.1 | 10.0.19.1 |
| talos-cp2 | BC:24:11:62:7B:3F | 10.0.15.2 | 10.0.19.2 |
| talos-cp3 | BC:24:11:D4:2F:ED | 10.0.15.3 | 10.0.19.3 |
| talos-worker1 | BC:24:11:43:3F:FF | 10.0.15.4 | 10.0.19.4 |
| talos-worker2 | BC:24:11:62:C4:8F | 10.0.15.5 | 10.0.19.5 |
| talos-worker3 | BC:24:11:3F:8E:1A | 10.0.15.6 | 10.0.19.6 |
| **K8s API VIP**| — (floats on CPs) | — | **10.0.19.10** |
| db.w5isp.com | static | 10.0.15.30 | 10.0.19.30 |
| prom.w5isp.com | static | 10.0.15.31 | 10.0.19.31 |
| node1 | static | 10.0.15.101 | 10.0.19.101 |
| node2 | static | 10.0.15.102 | 10.0.19.102 |
| node3 | static | 10.0.15.103 | 10.0.19.103 |
| router/gateway | — | 10.0.15.254 | 10.0.19.254 |
## Ground rules
- **Both subnets must coexist on the same L2 for the whole migration** —
already satisfied: the formerly separate subnets now share one flat L2.
Talos updates each etcd member's peer URL automatically when its IP
changes, but only if the remaining members can still reach it —
moving to an unreachable network bricks etcd
(siderolabs/talos#9075, #6080).
- **Control planes strictly one at a time.** The etcd member update
needs quorum (2/3 up).
- Everything through Phase 3 step 2 is reversible by putting the DHCP
reservation back and rebooting the node.
---
## Phase 0 — Backups + router prep
1. Backups (from `talos/`):
```bash
talosctl --talosconfig talosconfig -n 10.0.15.1 etcd snapshot etcd-$(date +%F).snap
cp secrets.yaml talosconfig ~/backups/ # or wherever off-repo copies live
```
(DB dumps intentionally skipped — operator decision 2026-07-17.)
2. Router:
- Add `10.0.19.254/22` to the LAN interface **alongside** 10.0.15.254/24.
- New DHCP scope on 10.0.16.0/22, gateway 10.0.19.254. Keep the
dynamic pool **out of 10.0.19.0–10.0.19.127** (statics + VIP live there).
10.0.19.10 (VIP) must never be leased.
- Do **not** change the Talos MAC reservations yet — that happens
per-node in Phase 3.
- Clone any firewall rules referencing 10.0.15.0/24 to also cover
10.0.16.0/22 (delete the old ones in Phase 5).
3. Sanity: from the workstation, ping 10.0.19.254.
## Phase 1 — Proxmox nodes (node1/2/3)
Guests keep running throughout; no VM reboots. Corosync quorum drops
mid-change — expected, harmless without HA.
1. Discover what actually references the IPs, per node:
```bash
ssh root@10.0.15.101 "grep -r '10\.0\.15' /etc/pve /etc/network/interfaces /etc/hosts; pveceph status 2>&1 | head -2"
```
(`pveceph status` should say Ceph isn't configured — `cluster/ceph-csi/`
is stale. If Ceph is somehow alive, STOP; mon renumbering is its own runbook.)
2. If `/etc/pve/corosync.conf` `ring0_addr`s are **hostnames**: no corosync
edit needed — only `/etc/hosts`. If they're **IPs**: on ONE node while
still quorate, edit `/etc/pve/corosync.conf` — all three ring0_addrs to
10.0.19.x and **bump `config_version`**.
3. Then on each node in turn (node1 → node2 → node3):
```bash
# /etc/network/interfaces: address 10.0.19.10X/22, gateway 10.0.19.254
# /etc/hosts: own + peer entries to 10.0.19.x
ifreload -a
systemctl restart corosync pve-cluster
```
4. After all three: `pvecm status` shows 3/3 quorate on new IPs;
Proxmox web UI reachable at https://10.0.19.101:8006.
- If pmxcfs wedges (can't write /etc/pve): `systemctl stop pve-cluster corosync;
pmxcfs -l` to fix corosync.conf locally, then restart both.
5. Update wherever `node1`/`node2`/`node3` resolve on the workstation
(`/etc/hosts` or `~/.ssh/config`) — ansible inventory uses
`ansible_host=node1` by name.
## Phase 2 — db + prom VMs
Use the **Proxmox console** (not SSH — you're cutting the branch you sit on).
1. On each VM, update the static netconfig (netplan or
/etc/network/interfaces): `10.0.19.30/22` and `10.0.19.31/22`,
gateway 10.0.19.254. Restart networking.
2. Verify SSH: `ssh ansible@10.0.19.31` (prom's ufw allows are
10.0.0.0/16-based, so no lockout exposure).
3. Loki ingestion from the cluster (promtail → 10.0.15.31:3100) is now
broken until Phase 4's configmap push — known gap.
## Phase 3 — Talos cluster
Run from `talos/`. Cluster endpoint is still `https://10.0.15.1:6443`
and stays valid until step 4.
**1. Workers, one at a time** (worker1 → worker2 → worker3):
```bash
# a) On router: change THIS node's reservation to its 10.0.19.x address
talosctl --talosconfig talosconfig -n 10.0.15.4 reboot
# b) wait, then verify:
talosctl --talosconfig talosconfig -n 10.0.19.4 version
kubectl get nodes -o wide # INTERNAL-IP updated, node Ready
```
**2. Add the VIP to the control planes** (still on old IPs).
Create `patches/vip.yaml`:
```yaml
machine:
network:
interfaces:
- deviceSelector:
physical: true
dhcp: true
vip:
ip: 10.0.19.10
```
```bash
for ip in 10.0.15.1 10.0.15.2 10.0.15.3; do
talosctl --talosconfig talosconfig -n $ip patch machineconfig --patch @patches/vip.yaml
done
curl -sk https://10.0.19.10:6443/version # any TLS response = VIP is up
```
**3. Flip the cluster endpoint to the VIP on ALL 6 nodes.**
Create `patches/endpoint.yaml`:
```yaml
cluster:
controlPlane:
endpoint: https://10.0.19.10:6443
```
```bash
for ip in 10.0.15.1 10.0.15.2 10.0.15.3 10.0.19.4 10.0.19.5 10.0.19.6; do
talosctl --talosconfig talosconfig -n $ip patch machineconfig --patch @patches/endpoint.yaml
done
talosctl --talosconfig talosconfig kubeconfig -n 10.0.15.1 --force # kubeconfig now points at the VIP
kubectl get nodes # all Ready via https://10.0.19.10:6443
```
**4. Control planes, strictly one at a time** (cp2 → cp3 → cp1):
```bash
# a) On router: change THIS node's reservation
talosctl --talosconfig talosconfig -n 10.0.15.2 reboot
# b) wait, then verify BEFORE touching the next one:
talosctl --talosconfig talosconfig -n 10.0.19.2 etcd members
# -> this member's peer URL shows 10.0.19.2, 3/3 members listed
talosctl --talosconfig talosconfig -n 10.0.19.2 health
```
If a member gets stuck on the old peer URL: quorum check first
(`etcd members` from a healthy node), then reboot the stuck node once
more before anything drastic. Worst case: restore the Phase 0 snapshot.
**5. Client configs:**
```bash
talosctl --talosconfig talosconfig config endpoint 10.0.19.1 10.0.19.2 10.0.19.3
# (real node IPs — never the VIP for the Talos API)
talosctl --talosconfig talosconfig -n 10.0.19.1 health
```
## Phase 4 — Repo + config sweep
Edit, commit, push (promtail change must reach codeberg for Argo):
- `talos/controlplane.yaml:45` + `talos/worker.yaml:44` — endpoint
`https://10.0.19.10:6443`; add the `machine.network.interfaces` VIP
block to `controlplane.yaml` so rebuilds match reality
- `ansible/hosts` — prom `ansible_host=10.0.19.31`
- `ansible/host_vars/prom.w5isp.com.yml` — ansible_host + subnet comment
- `ansible/host_vars/db.w5isp.com.yml` — `ansible_host: 10.0.19.30`
- `ansible/roles/prometheus/defaults/main.yml` —
`prometheus_k8s_api_server: "https://10.0.19.10:6443"` (VIP is in the
apiserver cert SANs) + commented example targets
- `ansible/roles/grafana/defaults/main.yml` — root_url `10.0.19.31`
- `home/cluster/promtail/configmap.yaml` — Loki URL `http://10.0.19.31:3100`
- Docs: `CLAUDE.md`, `talos/CLAUDE.md`, `talos/README.md`,
`cluster/README.md`, `ansible/roles/prometheus/README.md`,
`home/cluster/starlink-exporter/deployment.yaml` comment;
`cluster/ceph-csi/` is dead — delete or mark stale
- Then: `ansible-playbook -l prom.w5isp.com playbook.yml` and confirm
Argo synced promtail (`kubectl -n promtail rollout status ds/promtail`)
## Phase 5 — Retire the old subnet
Only after `rg "10\.0\.15\." ~/dev/infra` returns nothing but history/docs:
- Router: remove 10.0.15.254/24, old DHCP scope, old firewall rules.
- If a Tailscale subnet router advertises 10.0.15.0/24, re-advertise
10.0.16.0/22 and approve in the admin console.
- Hosts elsewhere in the /22 (e.g. birdnet 10.0.17.29) that still carry
/24 masks should move to /22 to avoid hairpinning via the router.
## Final verification
```bash
talosctl --talosconfig talosconfig -n 10.0.19.1 health
kubectl get nodes -o wide && kubectl get pods -A | grep -v Running
curl -sk https://10.0.19.10:6443/version
ssh root@10.0.19.101 pvecm status
open http://10.0.19.31:9090/targets # prometheus targets green
# https://aprs.me and the other cloudflared-fronted apps load
```
References: [talos#6080 — etcd peer URLs update on IP change](https://github.com/siderolabs/talos/issues/6080),
[talos#9075 — what happens when old/new subnets can't reach each other](https://github.com/siderolabs/talos/issues/9075),
[Talos multihoming/advertisedSubnets docs](https://docs.siderolabs.com/talos/v1.12/networking/multihoming)

2
ansible/host_vars/db.w5isp.com.yml
View file

@ -1,5 +1,5 @@
--- ---
ansible_host: 10.0.15.30 ansible_host: 10.0.19.30
ansible_python_interpreter: /usr/bin/python3 ansible_python_interpreter: /usr/bin/python3
ansible_ssh_common_args: '-o StrictHostKeyChecking=accept-new' ansible_ssh_common_args: '-o StrictHostKeyChecking=accept-new'

13
ansible/host_vars/logs.vntx.net.yml
View file

@ -2,3 +2,16 @@
ansible_host: 204.110.191.229 ansible_host: 204.110.191.229
ansible_python_interpreter: /usr/bin/python3 ansible_python_interpreter: /usr/bin/python3
ansible_ssh_common_args: '-o StrictHostKeyChecking=accept-new' ansible_ssh_common_args: '-o StrictHostKeyChecking=accept-new'
# OpenSearch configuration
opensearch_heap_min: "4g"
opensearch_heap_max: "4g"
# Firewall — Caddy needs HTTP/HTTPS open for Let's Encrypt + traffic
firewall_allow_rules:
- port: 80
proto: tcp
comment: HTTP
- port: 443
proto: tcp
comment: HTTPS

4
ansible/host_vars/prom.w5isp.com.yml
View file

@ -1,5 +1,5 @@
--- ---
ansible_host: 10.0.15.31 ansible_host: 10.0.19.31
ansible_python_interpreter: /usr/bin/python3 ansible_python_interpreter: /usr/bin/python3
ansible_ssh_common_args: '-o StrictHostKeyChecking=accept-new' ansible_ssh_common_args: '-o StrictHostKeyChecking=accept-new'
@ -8,7 +8,7 @@ network:
# Open Prometheus stack ports to the home LAN. # Open Prometheus stack ports to the home LAN.
# Trusted subnets (Tailscale, VNTX) are already wide-open via firewall role defaults; # Trusted subnets (Tailscale, VNTX) are already wide-open via firewall role defaults;
# this adds the 10.0.15.0/24 home cluster subnet so the k8s nodes can hit node_exporter # this adds the 10.0.16.0/22 home cluster subnet so the k8s nodes can hit node_exporter
# and so any LAN client can reach the Grafana UI. # and so any LAN client can reach the Grafana UI.
firewall_allow_rules: firewall_allow_rules:
- port: 22 - port: 22

6
ansible/hosts
View file

@ -20,7 +20,7 @@ skippy.w5isp.com
mail.mcintire.me ansible_host=107.174.178.20 mail.mcintire.me ansible_host=107.174.178.20
dokku.w5isp.com dokku.w5isp.com
aprs.w5isp.com aprs.w5isp.com
prom.w5isp.com ansible_host=10.0.15.31 prom.w5isp.com ansible_host=10.0.19.31
[prometheus_servers] [prometheus_servers]
prom.w5isp.com prom.w5isp.com
@ -37,6 +37,7 @@ aprs.w5isp.com
skippy.w5isp.com skippy.w5isp.com
netbox.vntx.net netbox.vntx.net
uisp.vntx.net uisp.vntx.net
logs.vntx.net
[netbox_servers] [netbox_servers]
netbox.vntx.net netbox.vntx.net
@ -48,6 +49,9 @@ git.mcintire.me ansible_host=172.245.56.83
[uisp_servers] [uisp_servers]
uisp.vntx.net uisp.vntx.net
[opensearch_servers]
logs.vntx.net
[proxmox_servers] [proxmox_servers]
vm2-380 ansible_host=vm2-380 vm2-380 ansible_host=vm2-380
node1 ansible_host=node1 node1 ansible_host=node1

10
ansible/playbook.yml
View file

@ -400,3 +400,13 @@
- inspircd - inspircd
roles: roles:
- inspircd - inspircd
- name: Configure OpenSearch servers
hosts: opensearch_servers
become: true
gather_facts: true
tags:
- opensearch
- logs
roles:
- opensearch

1
ansible/roles/caddy/tasks/main.yml
View file

@ -15,6 +15,7 @@
- apt-transport-https - apt-transport-https
- curl - curl
- gnupg2 - gnupg2
- python3-debian
state: present state: present
update_cache: true update_cache: true
when: ansible_facts['os_family'] == "Debian" when: ansible_facts['os_family'] == "Debian"

7
ansible/roles/caddy/templates/Caddyfile-logs.vntx.net.j2 Normal file
View file

@ -0,0 +1,7 @@
{{ opensearch_caddy_domain | default(inventory_hostname) }} {
reverse_proxy 127.0.0.1:{{ opensearch_dashboards_port | default('5601') }} {
header_up X-Forwarded-Host {host}
header_up X-Real-IP {remote_host}
header_up X-Forwarded-Proto {scheme}
}
}

2
ansible/roles/grafana/defaults/main.yml
View file

@ -5,7 +5,7 @@ grafana_apt_keyring: "/etc/apt/keyrings/grafana.gpg"
grafana_listen_address: "0.0.0.0" grafana_listen_address: "0.0.0.0"
grafana_http_port: 3000 grafana_http_port: 3000
grafana_root_url: "http://10.0.15.31:3000/" grafana_root_url: "http://10.0.19.31:3000/"
grafana_admin_user: admin grafana_admin_user: admin
# Empty by default so re-running ansible doesn't clobber a password that was # Empty by default so re-running ansible doesn't clobber a password that was

38
ansible/roles/opensearch/defaults/main.yml Normal file
View file

@ -0,0 +1,38 @@
---
# OpenSearch version and architecture
opensearch_version: "2.19.6"
opensearch_arch: x64 # x64 or aarch64
opensearch_user: opensearch
opensearch_group: opensearch
opensearch_install_dir: /opt/opensearch
opensearch_config_dir: /opt/opensearch/config
opensearch_data_dir: /var/lib/opensearch
opensearch_log_dir: /var/log/opensearch
# JVM heap size — tuned for 8 GB host (leaves room for OS cache and other services)
opensearch_heap_min: "2g"
opensearch_heap_max: "2g"
# Network
opensearch_http_port: 9200
opensearch_transport_port: 9300
# Cluster (single-node by default for logs.vntx.net)
opensearch_cluster_name: logs-vntx
opensearch_node_name: "{{ ansible_facts['hostname'] | default('logs') }}"
opensearch_discovery_type: single-node
# Plugins to install (empty by default)
opensearch_plugins: []
# Sysctl: vm.max_map_count required by OpenSearch
opensearch_vm_max_map_count: 262144
# --- OpenSearch Dashboards ---
opensearch_dashboards_version: "{{ opensearch_version }}"
opensearch_dashboards_install_dir: /opt/opensearch-dashboards
opensearch_dashboards_config_dir: /etc/opensearch-dashboards
opensearch_dashboards_port: 5601
opensearch_dashboards_opensearch_url: "http://127.0.0.1:{{ opensearch_http_port }}"

12
ansible/roles/opensearch/handlers/main.yml Normal file
View file

@ -0,0 +1,12 @@
---
- name: Restart opensearch
ansible.builtin.systemd:
name: opensearch
state: restarted
daemon_reload: true
- name: Restart opensearch-dashboards
ansible.builtin.systemd:
name: opensearch-dashboards
state: restarted
daemon_reload: true

179
ansible/roles/opensearch/tasks/main.yml Normal file
View file

@ -0,0 +1,179 @@
---
# Installs OpenSearch from the upstream tarball (bundled JDK) and runs it
# under systemd in single-node mode for logs.vntx.net.
- name: Set vm.max_map_count for OpenSearch
ansible.posix.sysctl:
name: vm.max_map_count
value: "{{ opensearch_vm_max_map_count }}"
state: present
reload: true
- name: Create opensearch group
ansible.builtin.group:
name: "{{ opensearch_group }}"
system: true
state: present
- name: Create opensearch user
ansible.builtin.user:
name: "{{ opensearch_user }}"
group: "{{ opensearch_group }}"
system: true
shell: /usr/sbin/nologin
home: "{{ opensearch_data_dir }}"
create_home: false
state: present
- name: Create opensearch directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: "{{ opensearch_user }}"
group: "{{ opensearch_group }}"
mode: "0750"
loop:
- "{{ opensearch_install_dir }}"
- "{{ opensearch_config_dir }}"
- "{{ opensearch_data_dir }}"
- "{{ opensearch_log_dir }}"
- name: Check installed opensearch version
ansible.builtin.command: "{{ opensearch_install_dir }}/bin/opensearch --version"
register: opensearch_installed
changed_when: false
failed_when: false
- name: Decide whether opensearch needs (re)install
ansible.builtin.set_fact:
opensearch_needs_install: >-
{{
opensearch_installed.rc != 0
or opensearch_version not in (opensearch_installed.stdout | default(''))
and opensearch_version not in (opensearch_installed.stderr | default(''))
}}
- name: Download opensearch tarball
ansible.builtin.get_url:
url: "https://artifacts.opensearch.org/releases/bundle/opensearch/{{ opensearch_version }}/opensearch-{{ opensearch_version }}-linux-{{ opensearch_arch }}.tar.gz"
dest: "/tmp/opensearch-{{ opensearch_version }}-linux-{{ opensearch_arch }}.tar.gz"
mode: "0644"
when: opensearch_needs_install
- name: Extract opensearch tarball to install dir
ansible.builtin.unarchive:
src: "/tmp/opensearch-{{ opensearch_version }}-linux-{{ opensearch_arch }}.tar.gz"
dest: "{{ opensearch_install_dir }}"
remote_src: true
extra_opts:
- --strip-components=1
owner: "{{ opensearch_user }}"
group: "{{ opensearch_group }}"
when: opensearch_needs_install
notify: Restart opensearch
- name: Render opensearch.yml
ansible.builtin.template:
src: opensearch.yml.j2
dest: "{{ opensearch_config_dir }}/opensearch.yml"
owner: "{{ opensearch_user }}"
group: "{{ opensearch_group }}"
mode: "0640"
notify: Restart opensearch
- name: Render jvm.options
ansible.builtin.template:
src: jvm.options.j2
dest: "{{ opensearch_config_dir }}/jvm.options"
owner: "{{ opensearch_user }}"
group: "{{ opensearch_group }}"
mode: "0640"
notify: Restart opensearch
- name: Install opensearch systemd unit
ansible.builtin.template:
src: opensearch.service.j2
dest: /etc/systemd/system/opensearch.service
owner: root
group: root
mode: "0644"
notify: Restart opensearch
- name: Enable and start opensearch
ansible.builtin.systemd:
name: opensearch
enabled: true
state: started
daemon_reload: true
# --- OpenSearch Dashboards ---
- name: Create opensearch-dashboards directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: "{{ opensearch_user }}"
group: "{{ opensearch_group }}"
mode: "0750"
loop:
- "{{ opensearch_dashboards_install_dir }}"
- "{{ opensearch_dashboards_config_dir }}"
- name: Check installed opensearch-dashboards version
ansible.builtin.command: "{{ opensearch_dashboards_install_dir }}/bin/opensearch-dashboards --version"
register: dashboards_installed
changed_when: false
failed_when: false
- name: Decide whether dashboards needs (re)install
ansible.builtin.set_fact:
dashboards_needs_install: >-
{{
dashboards_installed.rc != 0
or opensearch_dashboards_version not in (dashboards_installed.stdout | default(''))
and opensearch_dashboards_version not in (dashboards_installed.stderr | default(''))
}}
- name: Download opensearch-dashboards tarball
ansible.builtin.get_url:
url: "https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/{{ opensearch_dashboards_version }}/opensearch-dashboards-{{ opensearch_dashboards_version }}-linux-{{ opensearch_arch }}.tar.gz"
dest: "/tmp/opensearch-dashboards-{{ opensearch_dashboards_version }}-linux-{{ opensearch_arch }}.tar.gz"
mode: "0644"
when: dashboards_needs_install
- name: Extract opensearch-dashboards tarball to install dir
ansible.builtin.unarchive:
src: "/tmp/opensearch-dashboards-{{ opensearch_dashboards_version }}-linux-{{ opensearch_arch }}.tar.gz"
dest: "{{ opensearch_dashboards_install_dir }}"
remote_src: true
extra_opts:
- --strip-components=1
owner: "{{ opensearch_user }}"
group: "{{ opensearch_group }}"
when: dashboards_needs_install
notify: Restart opensearch-dashboards
- name: Render opensearch-dashboards.yml
ansible.builtin.template:
src: opensearch-dashboards.yml.j2
dest: "{{ opensearch_dashboards_config_dir }}/opensearch_dashboards.yml"
owner: "{{ opensearch_user }}"
group: "{{ opensearch_group }}"
mode: "0640"
notify: Restart opensearch-dashboards
- name: Install opensearch-dashboards systemd unit
ansible.builtin.template:
src: opensearch-dashboards.service.j2
dest: /etc/systemd/system/opensearch-dashboards.service
owner: root
group: root
mode: "0644"
notify: Restart opensearch-dashboards
- name: Enable and start opensearch-dashboards
ansible.builtin.systemd:
name: opensearch-dashboards
enabled: true
state: started
daemon_reload: true

24
ansible/roles/opensearch/templates/jvm.options.j2 Normal file
View file

@ -0,0 +1,24 @@
# {{ ansible_managed | default('Managed by Ansible') }}
#
# JVM options for OpenSearch
## Heap size — set min and max to the same value
-Xms{{ opensearch_heap_min }}
-Xmx{{ opensearch_heap_max }}
## GC configuration
-XX:+UseG1GC
-XX:MaxGCPauseMillis=500
## Basic JVM tuning
-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-Djava.awt.headless=true
-Djava.security.egd=file:/dev/urandom
-Dfile.encoding=UTF-8
## Heap dump path
-XX:HeapDumpPath={{ opensearch_data_dir }}
## Temp directory
-Djava.io.tmpdir=/tmp

22
ansible/roles/opensearch/templates/opensearch-dashboards.service.j2 Normal file
View file

@ -0,0 +1,22 @@
[Unit]
Description=OpenSearch Dashboards visualization server
After=network-online.target opensearch.service
Wants=network-online.target opensearch.service
[Service]
Type=simple
User={{ opensearch_user }}
Group={{ opensearch_group }}
Environment=OPENSEARCH_DASHBOARDS_HOME={{ opensearch_dashboards_install_dir }}
Environment=OPENSEARCH_DASHBOARDS_PATH_CONF={{ opensearch_dashboards_config_dir }}
ExecStart={{ opensearch_dashboards_install_dir }}/bin/opensearch-dashboards \
-c {{ opensearch_dashboards_config_dir }}/opensearch_dashboards.yml
Restart=on-failure
RestartSec=10
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target

11
ansible/roles/opensearch/templates/opensearch-dashboards.yml.j2 Normal file
View file

@ -0,0 +1,11 @@
# {{ ansible_managed | default('Managed by Ansible') }}
#
# OpenSearch Dashboards configuration
server.host: 127.0.0.1
server.port: {{ opensearch_dashboards_port }}
opensearch.hosts:
- {{ opensearch_dashboards_opensearch_url }}
opensearch.requestTimeout: 30000

26
ansible/roles/opensearch/templates/opensearch.service.j2 Normal file
View file

@ -0,0 +1,26 @@
[Unit]
Description=OpenSearch log search and analytics engine
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User={{ opensearch_user }}
Group={{ opensearch_group }}
Environment=OPENSEARCH_HOME={{ opensearch_install_dir }}
Environment=OPENSEARCH_JAVA_HOME={{ opensearch_install_dir }}/jdk
ExecStart={{ opensearch_install_dir }}/bin/opensearch
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
LimitNPROC=4096
NoNewPrivileges=true
ProtectHome=true
ProtectSystem=full
ReadWritePaths={{ opensearch_data_dir }} {{ opensearch_log_dir }} {{ opensearch_config_dir }}
[Install]
WantedBy=multi-user.target

25
ansible/roles/opensearch/templates/opensearch.yml.j2 Normal file
View file

@ -0,0 +1,25 @@
# {{ ansible_managed | default('Managed by Ansible') }}
#
# OpenSearch configuration for {{ opensearch_cluster_name }}
cluster.name: {{ opensearch_cluster_name }}
node.name: {{ opensearch_node_name }}
# Directories
path.data: {{ opensearch_data_dir }}
path.logs: {{ opensearch_log_dir }}
# Network
network.host: 127.0.0.1
http.port: {{ opensearch_http_port }}
transport.port: {{ opensearch_transport_port }}
# Single-node discovery (no cluster formation needed)
discovery.type: {{ opensearch_discovery_type }}
# Security — disabled for internal-only binding on loopback
plugins.security.disabled: true
# Performance
indices.memory.index_buffer_size: 20%
bootstrap.memory_lock: false

10
ansible/roles/prometheus/README.md
View file

@ -27,12 +27,12 @@ ServiceAccount token + the cluster's CA cert.
3. Copy them onto the prom host: 3. Copy them onto the prom host:
```bash ```bash
ssh ansible@10.0.15.31 'sudo install -d -o prometheus -g prometheus -m 0750 /etc/prometheus/k8s' ssh ansible@10.0.19.31 'sudo install -d -o prometheus -g prometheus -m 0750 /etc/prometheus/k8s'
printf '%s' "$TOKEN" | ssh ansible@10.0.15.31 \ printf '%s' "$TOKEN" | ssh ansible@10.0.19.31 \
'sudo tee /etc/prometheus/k8s/token >/dev/null && \ 'sudo tee /etc/prometheus/k8s/token >/dev/null && \
sudo chown prometheus:prometheus /etc/prometheus/k8s/token && \ sudo chown prometheus:prometheus /etc/prometheus/k8s/token && \
sudo chmod 0640 /etc/prometheus/k8s/token' sudo chmod 0640 /etc/prometheus/k8s/token'
printf '%s' "$CA" | ssh ansible@10.0.15.31 \ printf '%s' "$CA" | ssh ansible@10.0.19.31 \
'sudo tee /etc/prometheus/k8s/ca.crt >/dev/null && \ 'sudo tee /etc/prometheus/k8s/ca.crt >/dev/null && \
sudo chown prometheus:prometheus /etc/prometheus/k8s/ca.crt && \ sudo chown prometheus:prometheus /etc/prometheus/k8s/ca.crt && \
sudo chmod 0640 /etc/prometheus/k8s/ca.crt' sudo chmod 0640 /etc/prometheus/k8s/ca.crt'
@ -41,7 +41,7 @@ ServiceAccount token + the cluster's CA cert.
4. Reload Prometheus: 4. Reload Prometheus:
```bash ```bash
ssh ansible@10.0.15.31 'sudo systemctl reload prometheus' ssh ansible@10.0.19.31 'sudo systemctl reload prometheus'
``` ```
The token persists across pod restarts (it's a long-lived `kubernetes.io/service-account-token` The token persists across pod restarts (it's a long-lived `kubernetes.io/service-account-token`
@ -49,7 +49,7 @@ Secret, not a projected token), so it does not need to be rotated unless you del
## Verifying scrape targets ## Verifying scrape targets
Open `http://10.0.15.31:9090/targets` — the `kubernetes-apiservers`, Open `http://10.0.19.31:9090/targets` — the `kubernetes-apiservers`,
`kubernetes-nodes`, and `kubernetes-cadvisor` jobs should all be UP. `kubernetes-nodes`, and `kubernetes-cadvisor` jobs should all be UP.
If `kubernetes-pods` / `kubernetes-service-endpoints` are empty, that's expected If `kubernetes-pods` / `kubernetes-service-endpoints` are empty, that's expected

4
ansible/roles/prometheus/defaults/main.yml
View file

@ -20,7 +20,7 @@ prometheus_evaluation_interval: "30s"
# Path to the Kubernetes ServiceAccount bearer token + CA on the prom host. # Path to the Kubernetes ServiceAccount bearer token + CA on the prom host.
# Bootstrap copies these from the cluster (see roles/prometheus/README.md). # Bootstrap copies these from the cluster (see roles/prometheus/README.md).
prometheus_k8s_enabled: true prometheus_k8s_enabled: true
prometheus_k8s_api_server: "https://10.0.15.1:6443" prometheus_k8s_api_server: "https://10.0.19.10:6443"
prometheus_k8s_token_file: "{{ prometheus_config_dir }}/k8s/token" prometheus_k8s_token_file: "{{ prometheus_config_dir }}/k8s/token"
prometheus_k8s_ca_file: "{{ prometheus_config_dir }}/k8s/ca.crt" prometheus_k8s_ca_file: "{{ prometheus_config_dir }}/k8s/ca.crt"
@ -30,5 +30,5 @@ prometheus_alertmanager_targets:
# Extra static scrape targets, e.g. other node_exporters on home_servers # Extra static scrape targets, e.g. other node_exporters on home_servers
# - job_name: node_exporter # - job_name: node_exporter
# targets: ["10.0.15.31:9100", "10.0.15.22:9100"] # targets: ["10.0.19.31:9100", "10.0.19.22:9100"]
prometheus_extra_scrape_configs: [] prometheus_extra_scrape_configs: []

2
cluster/README.md
View file

@ -6,7 +6,7 @@ This directory contains Kubernetes application manifests for the Talos home clus
- **Cluster**: home-cluster (Talos) - **Cluster**: home-cluster (Talos)
- **Nodes**: 6 (3 control plane + 3 workers) - **Nodes**: 6 (3 control plane + 3 workers)
- **IPs**: 10.0.15.1-6 - **IPs**: 10.0.19.1-6
- **Kubernetes Version**: v1.35.0 - **Kubernetes Version**: v1.35.0
- **Talos Version**: v1.12.1 - **Talos Version**: v1.12.1

6
cluster/ceph-csi/01-configmap.yaml
View file

@ -9,9 +9,9 @@ data:
{ {
"clusterID": "7e968d44-61b4-4342-8318-2e8e48808e68", "clusterID": "7e968d44-61b4-4342-8318-2e8e48808e68",
"monitors": [ "monitors": [
"10.0.15.101:6789", "10.0.19.101:6789",
"10.0.15.102:6789", "10.0.19.102:6789",
"10.0.15.103:6789" "10.0.19.103:6789"
] ]
} }
] ]

8
cluster/ceph-csi/README.md
View file

@ -5,7 +5,7 @@ This directory contains manifests for deploying the Ceph CSI RBD driver to use P
## Configuration ## Configuration
- **Ceph Cluster FSID**: `7e968d44-61b4-4342-8318-2e8e48808e68` - **Ceph Cluster FSID**: `7e968d44-61b4-4342-8318-2e8e48808e68`
- **Ceph Monitors**: 10.0.15.101:6789, 10.0.15.102:6789, 10.0.15.103:6789 - **Ceph Monitors**: 10.0.19.101:6789, 10.0.19.102:6789, 10.0.19.103:6789
- **Ceph Pool**: kubernetes - **Ceph Pool**: kubernetes
- **Ceph User**: client.kubernetes - **Ceph User**: client.kubernetes
- **StorageClass**: ceph-rbd (default) - **StorageClass**: ceph-rbd (default)
@ -64,7 +64,7 @@ kubectl get nodes --show-labels | grep topology
kubectl get csinode -o yaml | grep topologyKeys -A 2 kubectl get csinode -o yaml | grep topologyKeys -A 2
# List RBD images in Ceph # List RBD images in Ceph
ssh root@10.0.15.101 "rbd ls -p kubernetes" ssh root@10.0.19.101 "rbd ls -p kubernetes"
``` ```
## Testing ## Testing
@ -88,7 +88,7 @@ kubectl delete -f test-pvc.yaml
```bash ```bash
# SSH to any Proxmox node # SSH to any Proxmox node
ssh root@10.0.15.101 ssh root@10.0.19.101
# List pools # List pools
ceph osd pool ls ceph osd pool ls
@ -123,7 +123,7 @@ kubectl delete pods -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner
### Check Ceph Health ### Check Ceph Health
```bash ```bash
ssh root@10.0.15.101 "ceph health detail" ssh root@10.0.19.101 "ceph health detail"
``` ```
## Files ## Files

2
home/cluster/promtail/configmap.yaml
View file

@ -14,7 +14,7 @@ data:
filename: /run/promtail/positions.yaml filename: /run/promtail/positions.yaml
clients: clients:
- url: http://10.0.15.31:3100/loki/api/v1/push - url: http://10.0.19.31:3100/loki/api/v1/push
backoff_config: backoff_config:
min_period: 500ms min_period: 500ms
max_period: 5m max_period: 5m

2
home/cluster/starlink-exporter/deployment.yaml
View file

@ -25,7 +25,7 @@ spec:
- name: exporter - name: exporter
image: docker.io/joshuasing/starlink_exporter:0.9.0 image: docker.io/joshuasing/starlink_exporter:0.9.0
# Defaults already match this network: -dish 192.168.100.1:9200, # Defaults already match this network: -dish 192.168.100.1:9200,
# -listen :9451. Dish is reachable via 10.0.15.254 (home router) # -listen :9451. Dish is reachable via 10.0.19.254 (home router)
# while the Starlink router is in bypass mode. # while the Starlink router is in bypass mode.
ports: ports:
- name: metrics - name: metrics

60
talos/CLAUDE.md
View file

@ -7,24 +7,24 @@ This document contains notes on how the Talos home cluster was created and how t
- **Cluster Name**: home-cluster - **Cluster Name**: home-cluster
- **Kubernetes Version**: v1.36.1 - **Kubernetes Version**: v1.36.1
- **Talos Version**: v1.13.5 - **Talos Version**: v1.13.5
- **Control Plane Nodes**: 3 (10.0.15.1-3) - **Control Plane Nodes**: 3 (10.0.19.1-3)
- **Worker Nodes**: 3 (10.0.15.4-6) - **Worker Nodes**: 3 (10.0.19.4-6)
- **API Endpoint**: https://10.0.15.1:6443 - **API Endpoint**: https://10.0.19.10:6443
## Node Details ## Node Details
| Hostname | Role | IP | Proxmox Node | VMID | MAC Address | | Hostname | Role | IP | Proxmox Node | VMID | MAC Address |
|----------|------|-----|--------------|------|-------------| |----------|------|-----|--------------|------|-------------|
| talos-cp1 | Control Plane | 10.0.15.1 | node1 | 200 | BC:24:11:9B:48:92 | | talos-cp1 | Control Plane | 10.0.19.1 | node1 | 200 | BC:24:11:9B:48:92 |
| talos-cp2 | Control Plane | 10.0.15.2 | node2 | 201 | BC:24:11:62:7B:3F | | talos-cp2 | Control Plane | 10.0.19.2 | node2 | 201 | BC:24:11:62:7B:3F |
| talos-cp3 | Control Plane | 10.0.15.3 | node3 | 202 | BC:24:11:D4:2F:ED | | talos-cp3 | Control Plane | 10.0.19.3 | node3 | 202 | BC:24:11:D4:2F:ED |
| talos-worker1 | Worker | 10.0.15.4 | node1 | 210 | BC:24:11:43:3F:FF | | talos-worker1 | Worker | 10.0.19.4 | node1 | 210 | BC:24:11:43:3F:FF |
| talos-worker2 | Worker | 10.0.15.5 | node2 | 211 | BC:24:11:62:C4:8F | | talos-worker2 | Worker | 10.0.19.5 | node2 | 211 | BC:24:11:62:C4:8F |
| talos-worker3 | Worker | 10.0.15.6 | node3 | 212 | BC:24:11:3F:8E:1A | | talos-worker3 | Worker | 10.0.19.6 | node3 | 212 | BC:24:11:3F:8E:1A |
## Prerequisites ## Prerequisites
- Proxmox home cluster with nodes: node1, node2, node3 (10.0.15.101-103) - Proxmox home cluster with nodes: node1, node2, node3 (10.0.19.101-103)
- Talos Linux template at VMID 9001 on node1 - Talos Linux template at VMID 9001 on node1
- DHCP static leases configured for all MAC addresses above - DHCP static leases configured for all MAC addresses above
- Terraform/OpenTofu configured with Proxmox provider - Terraform/OpenTofu configured with Proxmox provider
@ -45,14 +45,14 @@ Generate the base control plane and worker configs. Per-node settings (hostname,
```bash ```bash
# Generate base control plane config (used for all CP nodes with per-node patches) # Generate base control plane config (used for all CP nodes with per-node patches)
talosctl gen config home-cluster https://10.0.15.1:6443 \ talosctl gen config home-cluster https://10.0.19.10:6443 \
--with-secrets secrets.yaml \ --with-secrets secrets.yaml \
--kubernetes-version v1.36.1 \ --kubernetes-version v1.36.1 \
--output-types controlplane \ --output-types controlplane \
--output controlplane.yaml --output controlplane.yaml
# Generate base worker config (used for all worker nodes with per-node patches) # Generate base worker config (used for all worker nodes with per-node patches)
talosctl gen config home-cluster https://10.0.15.1:6443 \ talosctl gen config home-cluster https://10.0.19.10:6443 \
--with-secrets secrets.yaml \ --with-secrets secrets.yaml \
--kubernetes-version v1.36.1 \ --kubernetes-version v1.36.1 \
--output-types worker \ --output-types worker \
@ -67,14 +67,14 @@ Apply the base config with the appropriate per-node patch:
```bash ```bash
# Control plane nodes # Control plane nodes
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.1 --file controlplane.yaml --config-patch @patches/cp1.yaml --insecure talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.1 --file controlplane.yaml --config-patch @patches/cp1.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.2 --file controlplane.yaml --config-patch @patches/cp2.yaml --insecure talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.2 --file controlplane.yaml --config-patch @patches/cp2.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.3 --file controlplane.yaml --config-patch @patches/cp3.yaml --insecure talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.3 --file controlplane.yaml --config-patch @patches/cp3.yaml --insecure
# Worker nodes # Worker nodes
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.4 --file worker.yaml --config-patch @patches/worker1.yaml --insecure talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.4 --file worker.yaml --config-patch @patches/worker1.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.5 --file worker.yaml --config-patch @patches/worker2.yaml --insecure talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.5 --file worker.yaml --config-patch @patches/worker2.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.6 --file worker.yaml --config-patch @patches/worker3.yaml --insecure talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.6 --file worker.yaml --config-patch @patches/worker3.yaml --insecure
``` ```
### 4. Deploy VMs via Terraform ### 4. Deploy VMs via Terraform
@ -111,7 +111,7 @@ cd /Users/graham/dev/infra/talos
sleep 30 sleep 30
# Verify all nodes are up # Verify all nodes are up
for ip in 10.0.15.{1..6}; do for ip in 10.0.19.{1..6}; do
echo -n "$ip: " echo -n "$ip: "
ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down" ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down"
done done
@ -124,7 +124,7 @@ cd /Users/graham/dev/infra/talos
sleep 30 sleep 30
# Verify all nodes are up # Verify all nodes are up
for ip in 10.0.15.{1..6}; do for ip in 10.0.19.{1..6}; do
echo -n "$ip: " echo -n "$ip: "
ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down" ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down"
done done
@ -134,10 +134,10 @@ done
```bash ```bash
# Set talosctl endpoints # Set talosctl endpoints
talosctl --talosconfig talosconfig config endpoint 10.0.15.1 10.0.15.2 10.0.15.3 talosctl --talosconfig talosconfig config endpoint 10.0.19.1 10.0.19.2 10.0.19.3
# Bootstrap Kubernetes on the first control plane node # Bootstrap Kubernetes on the first control plane node
talosctl --talosconfig talosconfig bootstrap --nodes 10.0.15.1 talosctl --talosconfig talosconfig bootstrap --nodes 10.0.19.1
``` ```
### 8. Retrieve Kubeconfig ### 8. Retrieve Kubeconfig
@ -147,7 +147,7 @@ talosctl --talosconfig talosconfig bootstrap --nodes 10.0.15.1
sleep 60 sleep 60
# Retrieve and merge kubeconfig # Retrieve and merge kubeconfig
talosctl --talosconfig talosconfig kubeconfig --nodes 10.0.15.1 --force talosctl --talosconfig talosconfig kubeconfig --nodes 10.0.19.1 --force
# Switch to the cluster # Switch to the cluster
kubectl config use-context admin@home-cluster kubectl config use-context admin@home-cluster
@ -223,14 +223,14 @@ If seeing "static hostname is already set" error:
Set endpoints explicitly: Set endpoints explicitly:
```bash ```bash
talosctl --talosconfig talosconfig config endpoint 10.0.15.1 10.0.15.2 10.0.15.3 talosctl --talosconfig talosconfig config endpoint 10.0.19.1 10.0.19.2 10.0.19.3
``` ```
### Nodes not appearing in kubectl ### Nodes not appearing in kubectl
Wait 1-2 minutes after bootstrap. Check: Wait 1-2 minutes after bootstrap. Check:
```bash ```bash
talosctl --talosconfig talosconfig service --nodes 10.0.15.1 talosctl --talosconfig talosconfig service --nodes 10.0.19.1
# Look for kubelet, etcd, apid as "Running" # Look for kubelet, etcd, apid as "Running"
``` ```
@ -257,27 +257,27 @@ talosctl --talosconfig talosconfig service --nodes 10.0.15.1
### Upgrading Talos ### Upgrading Talos
**Pre-upgrade checklist:** **Pre-upgrade checklist:**
1. Check current versions: `for ip in 10.0.15.{1..6}; do echo -n "$ip: "; talosctl --talosconfig talosconfig version --nodes $ip 2>&1 | grep "Tag:" | grep -v Client; done` 1. Check current versions: `for ip in 10.0.19.{1..6}; do echo -n "$ip: "; talosctl --talosconfig talosconfig version --nodes $ip 2>&1 | grep "Tag:" | grep -v Client; done`
2. Check latest release: `curl -sL https://github.com/siderolabs/talos/releases/latest | grep -o 'v[0-9]\+\.[0-9]\+\.[0-9]\+' | head -1` 2. Check latest release: `curl -sL https://github.com/siderolabs/talos/releases/latest | grep -o 'v[0-9]\+\.[0-9]\+\.[0-9]\+' | head -1`
3. Verify cluster health: `kubectl get nodes -o wide` 3. Verify cluster health: `kubectl get nodes -o wide`
4. Check extensions on workers: `talosctl --talosconfig talosconfig get extensions -n 10.0.15.4` 4. Check extensions on workers: `talosctl --talosconfig talosconfig get extensions -n 10.0.19.4`
**Important:** CP nodes use the **stock** installer image. Worker nodes use the **factory** image (with iscsi-tools + util-linux-tools extensions). They MUST be upgraded separately with their respective images. **Important:** CP nodes use the **stock** installer image. Worker nodes use the **factory** image (with iscsi-tools + util-linux-tools extensions). They MUST be upgraded separately with their respective images.
```bash ```bash
# 1. Upgrade control plane nodes (stock image) # 1. Upgrade control plane nodes (stock image)
talosctl --talosconfig talosconfig upgrade \ talosctl --talosconfig talosconfig upgrade \
--nodes 10.0.15.1,10.0.15.2,10.0.15.3 \ --nodes 10.0.19.1,10.0.19.2,10.0.19.3 \
--image ghcr.io/siderolabs/installer:v<VERSION> --image ghcr.io/siderolabs/installer:v<VERSION>
# 2. Upgrade worker nodes (factory image with extensions) # 2. Upgrade worker nodes (factory image with extensions)
# The schematic ID can be found via: talosctl --talosconfig talosconfig get extensions -n <worker-ip> # The schematic ID can be found via: talosctl --talosconfig talosconfig get extensions -n <worker-ip>
talosctl --talosconfig talosconfig upgrade \ talosctl --talosconfig talosconfig upgrade \
--nodes 10.0.15.4,10.0.15.5,10.0.15.6 \ --nodes 10.0.19.4,10.0.19.5,10.0.19.6 \
--image factory.talos.dev/installer/<SCHEMATIC_ID>:v<VERSION> --image factory.talos.dev/installer/<SCHEMATIC_ID>:v<VERSION>
# 3. If workers fail to auto-reboot/uncordon, do it manually: # 3. If workers fail to auto-reboot/uncordon, do it manually:
talosctl --talosconfig talosconfig reboot --nodes 10.0.15.4,10.0.15.5,10.0.15.6 talosctl --talosconfig talosconfig reboot --nodes 10.0.19.4,10.0.19.5,10.0.19.6
# Wait ~90s for reboot, then: # Wait ~90s for reboot, then:
kubectl uncordon talos-worker1 talos-worker2 talos-worker3 kubectl uncordon talos-worker1 talos-worker2 talos-worker3
``` ```

20
talos/README.md
View file

@ -4,22 +4,20 @@
- **Cluster Name**: home-cluster - **Cluster Name**: home-cluster
- **Kubernetes Version**: v1.36.1 - **Kubernetes Version**: v1.36.1
- **Control Plane Nodes**: 3 (10.0.15.1-3) - **Control Plane Nodes**: 3 (10.0.19.1-3)
- **Worker Nodes**: 4 (10.0.15.4-7) - **Worker Nodes**: 3 (10.0.19.4-6)
- **Bare Metal Worker**: talos4 at 10.0.15.7 (HP t620) - **API Endpoint**: https://10.0.19.10:6443 (shared VIP)
- **API Endpoint**: https://10.0.15.1:6443
## Node Details ## Node Details
| Node Name | Role | IP | Proxmox Node | VMID | | Node Name | Role | IP | Proxmox Node | VMID |
|-----------|------|-----|--------------|------| |-----------|------|-----|--------------|------|
| talos-cp1 | Control Plane | 10.0.15.1 | node1 | 200 | | talos-cp1 | Control Plane | 10.0.19.1 | node1 | 200 |
| talos-cp2 | Control Plane | 10.0.15.2 | node2 | 201 | | talos-cp2 | Control Plane | 10.0.19.2 | node2 | 201 |
| talos-cp3 | Control Plane | 10.0.15.3 | node3 | 202 | | talos-cp3 | Control Plane | 10.0.19.3 | node3 | 202 |
| talos-worker1 | Worker | 10.0.15.4 | node1 | 210 | | talos-worker1 | Worker | 10.0.19.4 | node1 | 210 |
| talos-worker2 | Worker | 10.0.15.5 | node2 | 211 | | talos-worker2 | Worker | 10.0.19.5 | node2 | 211 |
| talos-worker3 | Worker | 10.0.15.6 | node3 | 212 | | talos-worker3 | Worker | 10.0.19.6 | node3 | 212 |
| talos4 | Worker | 10.0.15.7 | bare-metal (HP t620) | — |
## Configuration Files ## Configuration Files

8
talos/controlplane.yaml
View file

@ -19,6 +19,12 @@ machine:
- 9.9.9.9 - 9.9.9.9
- 149.112.112.112 - 149.112.112.112
- 1.1.1.1 - 1.1.1.1
interfaces:
- deviceSelector:
physical: true
dhcp: true
vip:
ip: 10.0.19.10
install: install:
disk: /dev/sda # The disk used for installations. disk: /dev/sda # The disk used for installations.
@ -42,7 +48,7 @@ cluster:
id: d9wM2zW6ZcxGVQp8rujQajusx-HrV1s-gRfGfc5wExM= # Globally unique identifier for this cluster (base64 encoded random 32 bytes). id: d9wM2zW6ZcxGVQp8rujQajusx-HrV1s-gRfGfc5wExM= # Globally unique identifier for this cluster (base64 encoded random 32 bytes).
secret: V44y9ckGbghbUYYuLTqjagj0jzx3TcgfZ9a7C9J97mE= # Shared secret of cluster (base64 encoded random 32 bytes). secret: V44y9ckGbghbUYYuLTqjagj0jzx3TcgfZ9a7C9J97mE= # Shared secret of cluster (base64 encoded random 32 bytes).
controlPlane: controlPlane:
endpoint: https://10.0.15.1:6443 # Endpoint is the canonical controlplane endpoint, which can be an IP address or a DNS hostname. endpoint: https://10.0.19.10:6443 # Shared VIP floating across all control plane nodes.
clusterName: home-cluster # Configures the cluster's name. clusterName: home-cluster # Configures the cluster's name.
network: network:
dnsDomain: cluster.local # The domain used by Kubernetes DNS. dnsDomain: cluster.local # The domain used by Kubernetes DNS.

3
talos/patches/endpoint.yaml Normal file
View file

@ -0,0 +1,3 @@
cluster:
controlPlane:
endpoint: https://10.0.19.10:6443

8
talos/patches/vip.yaml Normal file
View file

@ -0,0 +1,8 @@
machine:
network:
interfaces:
- deviceSelector:
physical: true
dhcp: true
vip:
ip: 10.0.19.10

2
talos/worker.yaml
View file

@ -41,7 +41,7 @@ cluster:
id: d9wM2zW6ZcxGVQp8rujQajusx-HrV1s-gRfGfc5wExM= # Globally unique identifier for this cluster (base64 encoded random 32 bytes). id: d9wM2zW6ZcxGVQp8rujQajusx-HrV1s-gRfGfc5wExM= # Globally unique identifier for this cluster (base64 encoded random 32 bytes).
secret: V44y9ckGbghbUYYuLTqjagj0jzx3TcgfZ9a7C9J97mE= # Shared secret of cluster (base64 encoded random 32 bytes). secret: V44y9ckGbghbUYYuLTqjagj0jzx3TcgfZ9a7C9J97mE= # Shared secret of cluster (base64 encoded random 32 bytes).
controlPlane: controlPlane:
endpoint: https://10.0.15.1:6443 # Endpoint is the canonical controlplane endpoint, which can be an IP address or a DNS hostname. endpoint: https://10.0.19.10:6443 # Shared VIP floating across all control plane nodes.
clusterName: home-cluster # Configures the cluster's name. clusterName: home-cluster # Configures the cluster's name.
network: network:
dnsDomain: cluster.local # The domain used by Kubernetes DNS. dnsDomain: cluster.local # The domain used by Kubernetes DNS.