177 lines
No EOL
5.2 KiB
Markdown
177 lines
No EOL
5.2 KiB
Markdown
# High Availability Setup with MikroTik
|
|
|
|
## Overview
|
|
Using MikroTik's built-in features to ensure K3s cluster remains accessible when node1 is down.
|
|
|
|
## Option 1: MikroTik Load Balancing with PCC (Per Connection Classifier)
|
|
|
|
### On MikroTik Router:
|
|
```mikrotik
|
|
# 1. Create address list for K3s nodes
|
|
/ip firewall address-list
|
|
add address=10.0.101.21 list=k3s-nodes comment="node1"
|
|
add address=10.0.101.22 list=k3s-nodes comment="node2"
|
|
add address=10.0.101.23 list=k3s-nodes comment="node3"
|
|
|
|
# 2. Create NAT rules for incoming traffic to distribute to nodes
|
|
# For HTTP/HTTPS traffic (ports 80, 443)
|
|
/ip firewall nat
|
|
add chain=dstnat action=dst-nat protocol=tcp dst-address=204.110.191.2 \
|
|
dst-port=80 to-addresses=10.0.101.21-10.0.101.23 to-ports=80 \
|
|
comment="K3s HTTP load balance"
|
|
|
|
add chain=dstnat action=dst-nat protocol=tcp dst-address=204.110.191.2 \
|
|
dst-port=443 to-addresses=10.0.101.21-10.0.101.23 to-ports=443 \
|
|
comment="K3s HTTPS load balance"
|
|
|
|
# 3. Add connection marking for load balancing
|
|
/ip firewall mangle
|
|
add chain=prerouting action=mark-connection new-connection-mark=k3s-conn-1 \
|
|
dst-address=204.110.191.2 dst-port=80,443 protocol=tcp \
|
|
per-connection-classifier=both-addresses:3/0
|
|
|
|
add chain=prerouting action=mark-connection new-connection-mark=k3s-conn-2 \
|
|
dst-address=204.110.191.2 dst-port=80,443 protocol=tcp \
|
|
per-connection-classifier=both-addresses:3/1
|
|
|
|
add chain=prerouting action=mark-connection new-connection-mark=k3s-conn-3 \
|
|
dst-address=204.110.191.2 dst-port=80,443 protocol=tcp \
|
|
per-connection-classifier=both-addresses:3/2
|
|
```
|
|
|
|
## Option 2: MikroTik with VRRP (Virtual Router Redundancy Protocol)
|
|
|
|
### Prerequisites:
|
|
- Assign additional IPs to each node for VRRP
|
|
- Configure MikroTik to route to VRRP virtual IP
|
|
|
|
### On Each K3s Node:
|
|
```bash
|
|
# Install keepalived
|
|
sudo apt-get update && sudo apt-get install -y keepalived
|
|
|
|
# Create keepalived configuration
|
|
sudo tee /etc/keepalived/keepalived.conf << EOF
|
|
vrrp_instance VI_1 {
|
|
state BACKUP # All nodes start as BACKUP
|
|
interface eth0 # Your network interface
|
|
virtual_router_id 51 # Same on all nodes
|
|
priority $(hostname | grep -o '[0-9]*')0 # node1=10, node2=20, node3=30
|
|
advert_int 1
|
|
authentication {
|
|
auth_type PASS
|
|
auth_pass k3scluster
|
|
}
|
|
virtual_ipaddress {
|
|
10.0.101.100/24 # Virtual IP for cluster
|
|
}
|
|
}
|
|
EOF
|
|
|
|
# Start keepalived
|
|
sudo systemctl enable --now keepalived
|
|
```
|
|
|
|
### On MikroTik:
|
|
```mikrotik
|
|
# Route public IP traffic to VRRP virtual IP
|
|
/ip firewall nat
|
|
add chain=dstnat action=dst-nat protocol=tcp dst-address=204.110.191.2 \
|
|
dst-port=80,443 to-addresses=10.0.101.100 \
|
|
comment="K3s VRRP Virtual IP"
|
|
```
|
|
|
|
## Option 3: MikroTik Netwatch with Failover (Simplest)
|
|
|
|
This monitors nodes and updates NAT dynamically:
|
|
|
|
```mikrotik
|
|
# 1. Create scripts for failover
|
|
/system script
|
|
add name="k3s-use-node1" source={
|
|
/ip firewall nat set [find comment="K3s HTTP"] to-addresses=10.0.101.21
|
|
/ip firewall nat set [find comment="K3s HTTPS"] to-addresses=10.0.101.21
|
|
:log info "K3s traffic switched to node1"
|
|
}
|
|
|
|
add name="k3s-use-node2" source={
|
|
/ip firewall nat set [find comment="K3s HTTP"] to-addresses=10.0.101.22
|
|
/ip firewall nat set [find comment="K3s HTTPS"] to-addresses=10.0.101.22
|
|
:log info "K3s traffic switched to node2"
|
|
}
|
|
|
|
add name="k3s-use-node3" source={
|
|
/ip firewall nat set [find comment="K3s HTTP"] to-addresses=10.0.101.23
|
|
/ip firewall nat set [find comment="K3s HTTPS"] to-addresses=10.0.101.23
|
|
:log info "K3s traffic switched to node3"
|
|
}
|
|
|
|
# 2. Create initial NAT rules
|
|
/ip firewall nat
|
|
add chain=dstnat action=dst-nat protocol=tcp dst-address=204.110.191.2 \
|
|
dst-port=80 to-addresses=10.0.101.21 to-ports=80 comment="K3s HTTP"
|
|
|
|
add chain=dstnat action=dst-nat protocol=tcp dst-address=204.110.191.2 \
|
|
dst-port=443 to-addresses=10.0.101.21 to-ports=443 comment="K3s HTTPS"
|
|
|
|
# 3. Set up Netwatch monitors
|
|
/tool netwatch
|
|
add host=10.0.101.21 interval=10s timeout=5s \
|
|
down-script={
|
|
:if ([/tool netwatch get [find host=10.0.101.22] status]="up") do={
|
|
/system script run k3s-use-node2
|
|
} else={
|
|
:if ([/tool netwatch get [find host=10.0.101.23] status]="up") do={
|
|
/system script run k3s-use-node3
|
|
}
|
|
}
|
|
} \
|
|
up-script="/system script run k3s-use-node1"
|
|
|
|
add host=10.0.101.22 interval=10s timeout=5s
|
|
add host=10.0.101.23 interval=10s timeout=5s
|
|
```
|
|
|
|
## Recommended: Configure K3s Services for Any Node Access
|
|
|
|
Make sure your services use NodePort or LoadBalancer type:
|
|
|
|
```yaml
|
|
# Example service configuration
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: myapp
|
|
spec:
|
|
type: NodePort
|
|
ports:
|
|
- port: 80
|
|
targetPort: 8080
|
|
nodePort: 32080 # This port will be accessible on ALL nodes
|
|
selector:
|
|
app: myapp
|
|
```
|
|
|
|
Then configure MikroTik to forward to the NodePort on any available node.
|
|
|
|
## Testing the Failover
|
|
|
|
1. Verify current routing:
|
|
```bash
|
|
curl -v http://204.110.191.2
|
|
```
|
|
|
|
2. Shut down node1:
|
|
```bash
|
|
ssh node1 sudo shutdown -h now
|
|
```
|
|
|
|
3. Watch MikroTik logs and verify traffic switches:
|
|
```mikrotik
|
|
/log print follow where topics~"info"
|
|
```
|
|
|
|
4. Verify service remains accessible:
|
|
```bash
|
|
curl -v http://204.110.191.2
|
|
``` |