network/mikrotik-tool/mpls.md

# MPLS / LDP enablement: verona ↔ climax ↔ core

Goal: bring up MPLS forwarding with LDP across the verona–climax–core spine
on RouterOS 7, leaving the rest of the network untouched. Once these three
are running labeled forwarding, expanding to other PoPs is incremental
(add the LDP instance + interface on each new router).

## Topology being labeled

```
verona (10.254.254.101)
  └── ether3-climax-11ghz  10.250.1.25/29 ──┐
                                            │  10.250.1.24/29 (AF11 backhaul)
  ┌── ether6-verona-11ghz  10.250.1.30/29 ──┘
climax (10.254.254.102)
  └── ether4-380-airfiber24 10.250.1.94/29 ──┐
                                             │  10.250.1.88/29 (AF24 backhaul)
  ┌── ether5-climax          10.250.1.89/29 ─┘
core (10.254.254.253)
```

OSPFv2 (`backbone-v2`, area 0) already redistributes connected, so all three
loopbacks are reachable. That is the IGP we attach LDP to. Nothing else
changes.

RouterOS versions in play: verona 7.20.8, climax 7.22, core 7.20.6 — all
include MPLS in the system package, no extra `.npk` to install.

## Pre-flight (run on each of the three before configuring)

```
# OSPF must be up and loopbacks reachable end-to-end.
/routing ospf neighbor print
/ip route print where dst-address~"10.254.254.10[12]/32" or dst-address~"10.254.254.253/32"

# Confirm the underlay MTU on the AF11 / AF24 ports — MPLS adds 4 bytes per
# label, so we want headroom. Check both physical (l2mtu) and IP MTU:
/interface print detail where name~"ether3-climax-11ghz|ether6-verona-11ghz|ether4-380-airfiber|ether5-climax"
```

Expected: l2mtu ≥ 1600 on the AF radios (default on CCR ports). If
anything is at 1500 we'll bump `mpls-mtu` to 1500 explicitly so we never
generate a 1504-byte frame the radio drops.

## Step 1 — verona (10.254.254.101)

```
# LDP instance, lsr-id and transport address pinned to the loopback.
/mpls ldp
add disabled=no lsr-id=10.254.254.101 transport-addresses=10.254.254.101 vrf=main

# Enable LDP discovery on the climax-facing link only.
/mpls ldp interface
add interface=ether3-climax-11ghz

# Pin the MPLS MTU explicitly. mpls-mtu is the cap on the *labeled* frame,
# so a 1500-byte IP payload + 4-byte label = 1504 needs at least 1508 to
# pass; 1508 also leaves room for one extra stacked label (VPN/FRR/etc.).
/mpls interface
add interface=ether3-climax-11ghz mpls-mtu=1508
```

## Step 2 — climax (10.254.254.102)

```
/mpls ldp
add disabled=no lsr-id=10.254.254.102 transport-addresses=10.254.254.102 vrf=main

/mpls ldp interface
add interface=ether6-verona-11ghz
add interface=ether4-380-airfiber24

/mpls interface
add interface=ether6-verona-11ghz   mpls-mtu=1508
add interface=ether4-380-airfiber24 mpls-mtu=1508
```

## Step 3 — core (10.254.254.253)

```
/mpls ldp
add disabled=no lsr-id=10.254.254.253 transport-addresses=10.254.254.253 vrf=main

/mpls ldp interface
add interface=ether5-climax

/mpls interface
add interface=ether5-climax mpls-mtu=1508
```

Order doesn't matter — LDP discovery is symmetric. As soon as both ends of
a link have `/mpls ldp interface` populated, hellos start flowing on
224.0.0.2:646 and a TCP session forms between the two transport addresses.

## Verification (the actually-works part)

### A. Control plane: LDP adjacencies

Run on **each** router:

```
/mpls ldp neighbor print detail
```

Required state:

- `peer=` lists the neighbor's lsr-id (a loopback IP),
- `transport=` is the neighbor's loopback IP,
- flag column shows `O` (operational); `D` (dynamic) is fine — `nD` (not
  discovered) or missing entries means hellos aren't reaching the far side
  (firewall, wrong interface, MTU).

Expected counts:

| Router | LDP neighbors |
|--------|---------------|
| verona | 1 (climax)    |
| climax | 2 (verona, core) |
| core   | 1 (climax)    |

### B. Control plane: label bindings exist for the remote loopbacks

```
/mpls ldp local-binding  print where dst-address~"10.254.254.(101|102|253)/32"
/mpls ldp remote-binding print where dst-address~"10.254.254.(101|102|253)/32"
```

Each router should advertise an `implicit-null` (or label 3) for its own
loopback in `local-binding`, and learn labels for the two other loopbacks
in `remote-binding`.

### C. Forwarding plane: FIB has the labels

```
/mpls forwarding-table print
```

You should see entries with `out-label`, `out-interface`, and `nexthop`
matching the OSPF next-hop toward each remote loopback. On climax you'll
see two entries — one swapping toward verona, one toward core. On verona
and core you'll see one entry for each remote loopback (the second loopback
is reached via climax with stacked labels collapsing to a single label
because we have only three nodes).

### D. Data plane: traceroute prints labels

From **verona**, source-routed off the loopback:

```
/tool traceroute 10.254.254.253 src-address=10.254.254.101 count=3
```

A working MPLS path prints `MPLS Label=NNNN E=0 ...` on the first hop
(climax) — that is the smoking gun. Without MPLS the same traceroute
succeeds but no `MPLS Label=` field appears.

Repeat from **core** in the other direction:

```
/tool traceroute 10.254.254.101 src-address=10.254.254.253 count=3
```

### E. Data plane: MTU check

We pinned `mpls-mtu=1508` — verify a full-size packet still passes once
labeled (1500 IP + 4 label = 1504 on the wire, well below both the 1508
cap and the radio l2mtu of 2024):

```
# from verona, send 1500-byte ICMP DF=yes to core's loopback
/ping 10.254.254.253 src-address=10.254.254.101 size=1500 do-not-fragment count=5
```

100% return = path is large enough for label + 1500 payload. Any loss
means the labeled frame (1504 bytes) doesn't fit somewhere — most likely
an l2mtu issue on a backhaul radio. Either bump l2mtu on the radio side,
or *raise* `mpls-mtu` if you increased the IP MTU. **Do not lower
`mpls-mtu` below 1508** — that caps the labeled frame and will cause
ICMP-frag-needed for ordinary 1500-byte DF customer traffic.

## Rollback (per router)

If anything misbehaves, MPLS is purely additive — removing the two tables
returns the box to plain IP forwarding. OSPF is untouched.

```
/mpls ldp interface remove [find]
/mpls ldp           remove [find]
# leave /mpls interface mpls-mtu settings as-is; harmless without LDP.
```

## Notes & caveats

- **Don't enable LDP on customer-facing or CGNAT interfaces.** Only the
  three backbone ports listed above. LDP advertised onto a customer port
  would expose label bindings to anyone who joined the broadcast domain.
- **No BGP-LU, no L3VPN, no TE yet.** This is plain IPv4 LDP only — the
  point is to get labeled IGP forwarding working between these three
  before layering services on top.
- **iBGP next-hop-self is not required** for this phase because we have no
  BGP-painted services riding the LSPs yet. When that changes (e.g.,
  bringing public /24s in over iBGP), confirm next-hop-self on the
  ingress LSR so labeled paths terminate at a router with an LSP.
- **OSPFv3 / IPv6 LDP is intentionally out of scope.** RouterOS 7 LDP is
  IPv4-only; v6 transport would need MPLSv6 / 6PE which we are not
  setting up here.
- **BFD on the AF radios** (already enabled on the verona–climax and
  climax–core OSPF templates with `use-bfd=yes`) is what fast-fails the
  underlay; LDP itself follows OSPF and tears down accordingly.
- **Expanding to other PoPs:** repeat steps 1–3 on the new router with
  its loopback as `lsr-id`/`transport-addresses`, then add `/mpls ldp
  interface` on the link to whichever existing MPLS router it peers with.
  No flag day.

---

# Deferred work (2026-05-09 session)

## Customer traceroute hop-collapse — partially done

**Goal**: customer traceroutes from any tower should look like
`customer → tower → core → edge → TWC → ...` — collapsing the intermediate
spine LSRs (climax, newhope) into invisible transit hops.

**What's done as of 2026-05-09**:

- `propagate-ttl=no` set on all 8 fleet routers (verona/climax/culleoka/
  newhope/lowry/982/494/core). This switches MPLS from the default uniform
  model to the pipe model: ingress LSR sets label-TTL=255 (independent of
  IP-TTL), transit LSRs decrement label-TTL only, egress preserves the
  IP-TTL across the LSP. Effect: any traffic that actually rides an LSP
  collapses the intermediate label-switching hops in traceroute.
- This works **today** for traffic whose destination has an LDP label
  binding — i.e., loopback-to-loopback (`10.254.254.x/32` between any
  two fleet routers), tower-to-tower-customer-prefix (`100.64.x.x/32`,
  the tower /44s), and the connected /29 backbone links. Verified:
  `verona /tool/traceroute 10.254.254.109` returns a single hop (lowry),
  the climax↔core↔newhope spine is invisible.

**What's still missing (the actual goal — customer→internet collapse)**:

ROS picks an outgoing label based on the **destination prefix's FEC**, not
on the recursive next-hop. Internet destinations like `1.1.1.1` have no
LDP label in any tower's forwarding table, so the packet leaves plain-IP
even when the tower's static default points at a labeled prefix
(`10.254.254.253` or `10.254.254.254`). We tested both gateway changes on
494 — neither caused label imposition. Reverted.

**To finish this** the fleet would need **BGP labeled unicast** (RFC 3107,
ROS `address-families=ip-labeled-unicast` on iBGP):

1. Core (and/or edge once MPLS-to-edge is solved) advertises BGP routes
   to towers with labels attached. Simplest version: just the default
   route `0.0.0.0/0` with next-hop=core's loopback and a label.
2. Towers receive labeled BGP routes. When forwarding to any destination
   matching such a route, the tower pushes the BGP-LU label, then the LDP
   transport label on top, and rides the LSP.
3. With `propagate-ttl=no` already set, the LSP is invisible in
   traceroute. Customer sees `tower → core → edge → TWC → ...` (3 hops to
   TWC, regardless of which tower they're on).

**Why we didn't do it today**:
- Three MPLS-related outages already (`asymmetric_ldp_parallel_paths`
  memory entry). Don't add a fourth without a maintenance window.
- BGP-LU isn't tested in this fleet. Need to:
  - Confirm ROS7 syntax for `address-families=ip-labeled-unicast` on the
    existing iBGP-RR mesh.
  - Decide whether to advertise LU labels for just the default route or
    for the full reflected RIB.
  - Decide what core advertises — just `0.0.0.0/0`, or also tower /44s
    so return-path traffic from edge to tower customers can also ride
    a labeled path with hop hiding (currently the return path
    edge → core → tower is plain-IP edge→core then labeled core→tower).
- A future attempt should also revisit whether MPLS-to-edge is desirable
  alongside this. With BGP-LU from core, edge doesn't need to be in LDP
  for towers to ride an LSP toward edge — the LSP terminates at core
  (where the BGP-LU label is popped), and edge sees plain IP exactly
  like today. So **BGP-LU may be a cleaner path than extending LDP to
  edge** for this specific goal.

## Extending LDP to edge — blocked, root cause unknown

See `~/.claude/projects/-Users-graham-dev-network-mikrotik-tool/memory/
asymmetric_ldp_parallel_paths.md` for the full incident notes. Three live
attempts on 2026-05-09 to bring up an LDP session between core and edge
all dropped fleet internet traffic within seconds. Hypotheses tried and
disproven: asymmetric LDP path redirect, BGP-table label flooding (we
tightened edge's advertise-filter to just `10.254.254.254/32`), and the
flood still happened. **Don't attempt again live without lab replication
or a console on edge.**

State left in place that survives across attempts:
- `mpls-mtu=1508` on edge `sfp-sfpplus7-core-direct` and
  `sfp-sfpplus11-preseem`.
- `mpls-mtu=1508` on core `ether3-edge-direct` and
  `sfp-sfpplus1-edge-preseem`.
- Fasttrack-bypass pairs (in/out) on chain=forward for those four
  interfaces on both routers.
- Edge `/mpls ldp advertise-filter` restricted to advertise only
  `10.254.254.254/32`.
- Edge `/mpls ldp set distribute-for-default-route=no`.

These are all harmless and would be required again on the next attempt;
do not roll them back.