Add Longhorn storage to Talos worker nodes
Configure passthrough NVMe drives (/dev/sdb) on each worker with XFS partition mounted at /var/mnt/longhorn. Switch worker installer image to Talos Image Factory build with iscsi-tools and util-linux-tools extensions required by Longhorn. Remove conflicting grubUseUKICmdline setting.
This commit is contained in:
parent
34aece2126
commit
11c06b13df
5 changed files with 72 additions and 51 deletions
19
CLAUDE.md
19
CLAUDE.md
|
|
@ -183,16 +183,29 @@ tofu plan
|
||||||
- **Worker Nodes**: 3 (10.0.15.4-6)
|
- **Worker Nodes**: 3 (10.0.15.4-6)
|
||||||
- **API Endpoint**: https://10.0.15.1:6443
|
- **API Endpoint**: https://10.0.15.1:6443
|
||||||
- **Proxmox Cluster**: "home" (node1, node2, node3 at 10.0.15.101-103)
|
- **Proxmox Cluster**: "home" (node1, node2, node3 at 10.0.15.101-103)
|
||||||
- **Storage**: Proxmox Ceph for VM disks
|
- **VM Disks**: Proxmox Ceph for VM boot disks
|
||||||
|
- **Persistent Storage**: Longhorn on passthrough NVMe drives (`/dev/sdb` on each worker, mounted at `/var/mnt/longhorn`)
|
||||||
|
- Worker1: 2TB, Worker2: 2TB, Worker3: 1TB
|
||||||
|
- `longhorn` StorageClass is the default
|
||||||
|
- `longhorn-system` namespace requires privileged PodSecurity policy
|
||||||
- **CNI**: Flannel
|
- **CNI**: Flannel
|
||||||
|
- **Talos Extensions** (workers only):
|
||||||
|
- `iscsi-tools` v0.2.0 — required by Longhorn
|
||||||
|
- `util-linux-tools` 2.41.2 — required by Longhorn
|
||||||
|
- Installer image: `factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.12.4`
|
||||||
|
- Control plane nodes use the stock `ghcr.io/siderolabs/installer:v1.12.4`
|
||||||
- **Deployment**: Managed via OpenTofu in `/tofu/`
|
- **Deployment**: Managed via OpenTofu in `/tofu/`
|
||||||
- **Configuration**: `/talos/` directory
|
- **Configuration**: `/talos/` directory
|
||||||
- `talosconfig` - Talosctl client config
|
- `talosconfig` - Talosctl client config
|
||||||
- `secrets.yaml` - Cluster secrets
|
- `secrets.yaml` - Cluster secrets
|
||||||
- `controlplane.yaml` / `worker.yaml` - Node configs
|
- `controlplane.yaml` / `worker.yaml` - Base node configs
|
||||||
|
- `configs/talos-worker{1,2,3}.yaml` - Per-node worker configs (with disk/extension settings)
|
||||||
|
- `configs/talos-cp{1,2,3}.yaml` - Per-node control plane configs
|
||||||
- **Applications**: `/cluster/` directory contains Kubernetes manifests
|
- **Applications**: `/cluster/` directory contains Kubernetes manifests
|
||||||
- `ceph-csi/` - Ceph CSI RBD driver for persistent storage (default StorageClass)
|
|
||||||
- **Documentation**: See `/talos/README.md` and `/cluster/README.md` for detailed operations
|
- **Documentation**: See `/talos/README.md` and `/cluster/README.md` for detailed operations
|
||||||
|
- **Config Notes**:
|
||||||
|
- `install.extraKernelArgs` and `install.grubUseUKICmdline` cannot be used together
|
||||||
|
- Worker disks with existing data (e.g. old Ceph LVM) must be wiped before Talos will partition them: `talosctl reset --user-disks-to-wipe /dev/sdb --wipe-mode user-disks --reboot`
|
||||||
|
|
||||||
### K3s Cluster at 204.110.191.2
|
### K3s Cluster at 204.110.191.2
|
||||||
- **Purpose**: Home lab Kubernetes cluster
|
- **Purpose**: Home lab Kubernetes cluster
|
||||||
|
|
|
||||||
|
|
@ -31,16 +31,14 @@ machine:
|
||||||
# extraArgs:
|
# extraArgs:
|
||||||
# key: value
|
# key: value
|
||||||
|
|
||||||
# # The `extraMounts` field is used to add additional mounts to the kubelet container.
|
extraMounts:
|
||||||
# extraMounts:
|
- destination: /var/mnt/longhorn
|
||||||
# - destination: /var/lib/example # Destination is the absolute path where the mount will be placed in the container.
|
type: bind
|
||||||
# type: bind # Type specifies the mount kind.
|
source: /var/mnt/longhorn
|
||||||
# source: /var/lib/example # Source specifies the source path of the mount.
|
options:
|
||||||
# # Options are fstab style mount options.
|
- bind
|
||||||
# options:
|
- rshared
|
||||||
# - bind
|
- rw
|
||||||
# - rshared
|
|
||||||
# - rw
|
|
||||||
|
|
||||||
# # The `extraConfig` field is used to provide kubelet configuration overrides.
|
# # The `extraConfig` field is used to provide kubelet configuration overrides.
|
||||||
# extraConfig:
|
# extraConfig:
|
||||||
|
|
@ -77,9 +75,8 @@ machine:
|
||||||
# Used to provide instructions for installations.
|
# Used to provide instructions for installations.
|
||||||
install:
|
install:
|
||||||
disk: /dev/sda # The disk used for installations.
|
disk: /dev/sda # The disk used for installations.
|
||||||
image: ghcr.io/siderolabs/installer:v1.12.4 # Allows for supplying the image used to perform the installation.
|
image: factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.12.4 # Talos installer with iscsi-tools and util-linux-tools extensions for Longhorn.
|
||||||
wipe: false # Indicates if the installation disk should be wiped at installation time.
|
wipe: false # Indicates if the installation disk should be wiped at installation time.
|
||||||
grubUseUKICmdline: true # Indicates if legacy GRUB bootloader should use kernel cmdline from the UKI instead of building it on the host.
|
|
||||||
extraKernelArgs:
|
extraKernelArgs:
|
||||||
- console=tty0
|
- console=tty0
|
||||||
|
|
||||||
|
|
@ -88,6 +85,11 @@ machine:
|
||||||
# size: 4GB # Disk size.
|
# size: 4GB # Disk size.
|
||||||
# model: WDC* # Disk model `/sys/block/<dev>/device/model`.
|
# model: WDC* # Disk model `/sys/block/<dev>/device/model`.
|
||||||
# busPath: /pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0 # Disk bus path.
|
# busPath: /pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0 # Disk bus path.
|
||||||
|
# Extra disks to configure (Longhorn storage).
|
||||||
|
disks:
|
||||||
|
- device: /dev/sdb
|
||||||
|
partitions:
|
||||||
|
- mountpoint: /var/mnt/longhorn
|
||||||
registries: {}
|
registries: {}
|
||||||
# Features describe individual Talos features that can be switched on or off.
|
# Features describe individual Talos features that can be switched on or off.
|
||||||
features:
|
features:
|
||||||
|
|
|
||||||
|
|
@ -31,16 +31,14 @@ machine:
|
||||||
# extraArgs:
|
# extraArgs:
|
||||||
# key: value
|
# key: value
|
||||||
|
|
||||||
# # The `extraMounts` field is used to add additional mounts to the kubelet container.
|
extraMounts:
|
||||||
# extraMounts:
|
- destination: /var/mnt/longhorn
|
||||||
# - destination: /var/lib/example # Destination is the absolute path where the mount will be placed in the container.
|
type: bind
|
||||||
# type: bind # Type specifies the mount kind.
|
source: /var/mnt/longhorn
|
||||||
# source: /var/lib/example # Source specifies the source path of the mount.
|
options:
|
||||||
# # Options are fstab style mount options.
|
- bind
|
||||||
# options:
|
- rshared
|
||||||
# - bind
|
- rw
|
||||||
# - rshared
|
|
||||||
# - rw
|
|
||||||
|
|
||||||
# # The `extraConfig` field is used to provide kubelet configuration overrides.
|
# # The `extraConfig` field is used to provide kubelet configuration overrides.
|
||||||
# extraConfig:
|
# extraConfig:
|
||||||
|
|
@ -77,9 +75,8 @@ machine:
|
||||||
# Used to provide instructions for installations.
|
# Used to provide instructions for installations.
|
||||||
install:
|
install:
|
||||||
disk: /dev/sda # The disk used for installations.
|
disk: /dev/sda # The disk used for installations.
|
||||||
image: ghcr.io/siderolabs/installer:v1.12.4 # Allows for supplying the image used to perform the installation.
|
image: factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.12.4 # Talos installer with iscsi-tools and util-linux-tools extensions for Longhorn.
|
||||||
wipe: false # Indicates if the installation disk should be wiped at installation time.
|
wipe: false # Indicates if the installation disk should be wiped at installation time.
|
||||||
grubUseUKICmdline: true # Indicates if legacy GRUB bootloader should use kernel cmdline from the UKI instead of building it on the host.
|
|
||||||
extraKernelArgs:
|
extraKernelArgs:
|
||||||
- console=tty0
|
- console=tty0
|
||||||
|
|
||||||
|
|
@ -88,6 +85,11 @@ machine:
|
||||||
# size: 4GB # Disk size.
|
# size: 4GB # Disk size.
|
||||||
# model: WDC* # Disk model `/sys/block/<dev>/device/model`.
|
# model: WDC* # Disk model `/sys/block/<dev>/device/model`.
|
||||||
# busPath: /pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0 # Disk bus path.
|
# busPath: /pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0 # Disk bus path.
|
||||||
|
# Extra disks to configure (Longhorn storage).
|
||||||
|
disks:
|
||||||
|
- device: /dev/sdb
|
||||||
|
partitions:
|
||||||
|
- mountpoint: /var/mnt/longhorn
|
||||||
registries: {}
|
registries: {}
|
||||||
# Features describe individual Talos features that can be switched on or off.
|
# Features describe individual Talos features that can be switched on or off.
|
||||||
features:
|
features:
|
||||||
|
|
|
||||||
|
|
@ -31,16 +31,14 @@ machine:
|
||||||
# extraArgs:
|
# extraArgs:
|
||||||
# key: value
|
# key: value
|
||||||
|
|
||||||
# # The `extraMounts` field is used to add additional mounts to the kubelet container.
|
extraMounts:
|
||||||
# extraMounts:
|
- destination: /var/mnt/longhorn
|
||||||
# - destination: /var/lib/example # Destination is the absolute path where the mount will be placed in the container.
|
type: bind
|
||||||
# type: bind # Type specifies the mount kind.
|
source: /var/mnt/longhorn
|
||||||
# source: /var/lib/example # Source specifies the source path of the mount.
|
options:
|
||||||
# # Options are fstab style mount options.
|
- bind
|
||||||
# options:
|
- rshared
|
||||||
# - bind
|
- rw
|
||||||
# - rshared
|
|
||||||
# - rw
|
|
||||||
|
|
||||||
# # The `extraConfig` field is used to provide kubelet configuration overrides.
|
# # The `extraConfig` field is used to provide kubelet configuration overrides.
|
||||||
# extraConfig:
|
# extraConfig:
|
||||||
|
|
@ -77,9 +75,8 @@ machine:
|
||||||
# Used to provide instructions for installations.
|
# Used to provide instructions for installations.
|
||||||
install:
|
install:
|
||||||
disk: /dev/sda # The disk used for installations.
|
disk: /dev/sda # The disk used for installations.
|
||||||
image: ghcr.io/siderolabs/installer:v1.12.4 # Allows for supplying the image used to perform the installation.
|
image: factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.12.4 # Talos installer with iscsi-tools and util-linux-tools extensions for Longhorn.
|
||||||
wipe: false # Indicates if the installation disk should be wiped at installation time.
|
wipe: false # Indicates if the installation disk should be wiped at installation time.
|
||||||
grubUseUKICmdline: true # Indicates if legacy GRUB bootloader should use kernel cmdline from the UKI instead of building it on the host.
|
|
||||||
extraKernelArgs:
|
extraKernelArgs:
|
||||||
- console=tty0
|
- console=tty0
|
||||||
|
|
||||||
|
|
@ -88,6 +85,11 @@ machine:
|
||||||
# size: 4GB # Disk size.
|
# size: 4GB # Disk size.
|
||||||
# model: WDC* # Disk model `/sys/block/<dev>/device/model`.
|
# model: WDC* # Disk model `/sys/block/<dev>/device/model`.
|
||||||
# busPath: /pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0 # Disk bus path.
|
# busPath: /pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0 # Disk bus path.
|
||||||
|
# Extra disks to configure (Longhorn storage).
|
||||||
|
disks:
|
||||||
|
- device: /dev/sdb
|
||||||
|
partitions:
|
||||||
|
- mountpoint: /var/mnt/longhorn
|
||||||
registries: {}
|
registries: {}
|
||||||
# Features describe individual Talos features that can be switched on or off.
|
# Features describe individual Talos features that can be switched on or off.
|
||||||
features:
|
features:
|
||||||
|
|
|
||||||
|
|
@ -31,16 +31,14 @@ machine:
|
||||||
# extraArgs:
|
# extraArgs:
|
||||||
# key: value
|
# key: value
|
||||||
|
|
||||||
# # The `extraMounts` field is used to add additional mounts to the kubelet container.
|
extraMounts:
|
||||||
# extraMounts:
|
- destination: /var/mnt/longhorn
|
||||||
# - destination: /var/lib/example # Destination is the absolute path where the mount will be placed in the container.
|
type: bind
|
||||||
# type: bind # Type specifies the mount kind.
|
source: /var/mnt/longhorn
|
||||||
# source: /var/lib/example # Source specifies the source path of the mount.
|
options:
|
||||||
# # Options are fstab style mount options.
|
- bind
|
||||||
# options:
|
- rshared
|
||||||
# - bind
|
- rw
|
||||||
# - rshared
|
|
||||||
# - rw
|
|
||||||
|
|
||||||
# # The `extraConfig` field is used to provide kubelet configuration overrides.
|
# # The `extraConfig` field is used to provide kubelet configuration overrides.
|
||||||
# extraConfig:
|
# extraConfig:
|
||||||
|
|
@ -81,15 +79,19 @@ machine:
|
||||||
# Used to provide instructions for installations.
|
# Used to provide instructions for installations.
|
||||||
install:
|
install:
|
||||||
disk: /dev/sda # The disk used for installations.
|
disk: /dev/sda # The disk used for installations.
|
||||||
image: ghcr.io/siderolabs/installer:v1.12.4 # Allows for supplying the image used to perform the installation.
|
image: factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.12.4 # Talos installer with iscsi-tools and util-linux-tools extensions for Longhorn.
|
||||||
wipe: false # Indicates if the installation disk should be wiped at installation time.
|
wipe: false # Indicates if the installation disk should be wiped at installation time.
|
||||||
grubUseUKICmdline: true # Indicates if legacy GRUB bootloader should use kernel cmdline from the UKI instead of building it on the host.
|
|
||||||
|
|
||||||
# # Look up disk using disk attributes like model, size, serial and others.
|
# # Look up disk using disk attributes like model, size, serial and others.
|
||||||
# diskSelector:
|
# diskSelector:
|
||||||
# size: 4GB # Disk size.
|
# size: 4GB # Disk size.
|
||||||
# model: WDC* # Disk model `/sys/block/<dev>/device/model`.
|
# model: WDC* # Disk model `/sys/block/<dev>/device/model`.
|
||||||
# busPath: /pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0 # Disk bus path.
|
# busPath: /pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0 # Disk bus path.
|
||||||
|
# Extra disks to configure (Longhorn storage).
|
||||||
|
disks:
|
||||||
|
- device: /dev/sdb
|
||||||
|
partitions:
|
||||||
|
- mountpoint: /var/mnt/longhorn
|
||||||
registries: {}
|
registries: {}
|
||||||
# Features describe individual Talos features that can be switched on or off.
|
# Features describe individual Talos features that can be switched on or off.
|
||||||
features:
|
features:
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue