# MPLS / LDP enablement: verona ↔ climax ↔ core Goal: bring up MPLS forwarding with LDP across the verona–climax–core spine on RouterOS 7, leaving the rest of the network untouched. Once these three are running labeled forwarding, expanding to other PoPs is incremental (add the LDP instance + interface on each new router). ## Topology being labeled ``` verona (10.254.254.101) └── ether3-climax-11ghz 10.250.1.25/29 ──┐ │ 10.250.1.24/29 (AF11 backhaul) ┌── ether6-verona-11ghz 10.250.1.30/29 ──┘ climax (10.254.254.102) └── ether4-380-airfiber24 10.250.1.94/29 ──┐ │ 10.250.1.88/29 (AF24 backhaul) ┌── ether5-climax 10.250.1.89/29 ─┘ core (10.254.254.253) ``` OSPFv2 (`backbone-v2`, area 0) already redistributes connected, so all three loopbacks are reachable. That is the IGP we attach LDP to. Nothing else changes. RouterOS versions in play: verona 7.20.8, climax 7.22, core 7.20.6 — all include MPLS in the system package, no extra `.npk` to install. ## Pre-flight (run on each of the three before configuring) ``` # OSPF must be up and loopbacks reachable end-to-end. /routing ospf neighbor print /ip route print where dst-address~"10.254.254.10[12]/32" or dst-address~"10.254.254.253/32" # Confirm the underlay MTU on the AF11 / AF24 ports — MPLS adds 4 bytes per # label, so we want headroom. Check both physical (l2mtu) and IP MTU: /interface print detail where name~"ether3-climax-11ghz|ether6-verona-11ghz|ether4-380-airfiber|ether5-climax" ``` Expected: l2mtu ≥ 1600 on the AF radios (default on CCR ports). If anything is at 1500 we'll bump `mpls-mtu` to 1500 explicitly so we never generate a 1504-byte frame the radio drops. ## Step 1 — verona (10.254.254.101) ``` # LDP instance, lsr-id and transport address pinned to the loopback. /mpls ldp add disabled=no lsr-id=10.254.254.101 transport-addresses=10.254.254.101 vrf=main # Enable LDP discovery on the climax-facing link only. /mpls ldp interface add interface=ether3-climax-11ghz # Pin the MPLS MTU explicitly. mpls-mtu is the cap on the *labeled* frame, # so a 1500-byte IP payload + 4-byte label = 1504 needs at least 1508 to # pass; 1508 also leaves room for one extra stacked label (VPN/FRR/etc.). /mpls interface add interface=ether3-climax-11ghz mpls-mtu=1508 ``` ## Step 2 — climax (10.254.254.102) ``` /mpls ldp add disabled=no lsr-id=10.254.254.102 transport-addresses=10.254.254.102 vrf=main /mpls ldp interface add interface=ether6-verona-11ghz add interface=ether4-380-airfiber24 /mpls interface add interface=ether6-verona-11ghz mpls-mtu=1508 add interface=ether4-380-airfiber24 mpls-mtu=1508 ``` ## Step 3 — core (10.254.254.253) ``` /mpls ldp add disabled=no lsr-id=10.254.254.253 transport-addresses=10.254.254.253 vrf=main /mpls ldp interface add interface=ether5-climax /mpls interface add interface=ether5-climax mpls-mtu=1508 ``` Order doesn't matter — LDP discovery is symmetric. As soon as both ends of a link have `/mpls ldp interface` populated, hellos start flowing on 224.0.0.2:646 and a TCP session forms between the two transport addresses. ## Verification (the actually-works part) ### A. Control plane: LDP adjacencies Run on **each** router: ``` /mpls ldp neighbor print detail ``` Required state: - `peer=` lists the neighbor's lsr-id (a loopback IP), - `transport=` is the neighbor's loopback IP, - flag column shows `O` (operational); `D` (dynamic) is fine — `nD` (not discovered) or missing entries means hellos aren't reaching the far side (firewall, wrong interface, MTU). Expected counts: | Router | LDP neighbors | |--------|---------------| | verona | 1 (climax) | | climax | 2 (verona, core) | | core | 1 (climax) | ### B. Control plane: label bindings exist for the remote loopbacks ``` /mpls ldp local-binding print where dst-address~"10.254.254.(101|102|253)/32" /mpls ldp remote-binding print where dst-address~"10.254.254.(101|102|253)/32" ``` Each router should advertise an `implicit-null` (or label 3) for its own loopback in `local-binding`, and learn labels for the two other loopbacks in `remote-binding`. ### C. Forwarding plane: FIB has the labels ``` /mpls forwarding-table print ``` You should see entries with `out-label`, `out-interface`, and `nexthop` matching the OSPF next-hop toward each remote loopback. On climax you'll see two entries — one swapping toward verona, one toward core. On verona and core you'll see one entry for each remote loopback (the second loopback is reached via climax with stacked labels collapsing to a single label because we have only three nodes). ### D. Data plane: traceroute prints labels From **verona**, source-routed off the loopback: ``` /tool traceroute 10.254.254.253 src-address=10.254.254.101 count=3 ``` A working MPLS path prints `MPLS Label=NNNN E=0 ...` on the first hop (climax) — that is the smoking gun. Without MPLS the same traceroute succeeds but no `MPLS Label=` field appears. Repeat from **core** in the other direction: ``` /tool traceroute 10.254.254.101 src-address=10.254.254.253 count=3 ``` ### E. Data plane: MTU check We pinned `mpls-mtu=1508` — verify a full-size packet still passes once labeled (1500 IP + 4 label = 1504 on the wire, well below both the 1508 cap and the radio l2mtu of 2024): ``` # from verona, send 1500-byte ICMP DF=yes to core's loopback /ping 10.254.254.253 src-address=10.254.254.101 size=1500 do-not-fragment count=5 ``` 100% return = path is large enough for label + 1500 payload. Any loss means the labeled frame (1504 bytes) doesn't fit somewhere — most likely an l2mtu issue on a backhaul radio. Either bump l2mtu on the radio side, or *raise* `mpls-mtu` if you increased the IP MTU. **Do not lower `mpls-mtu` below 1508** — that caps the labeled frame and will cause ICMP-frag-needed for ordinary 1500-byte DF customer traffic. ## Rollback (per router) If anything misbehaves, MPLS is purely additive — removing the two tables returns the box to plain IP forwarding. OSPF is untouched. ``` /mpls ldp interface remove [find] /mpls ldp remove [find] # leave /mpls interface mpls-mtu settings as-is; harmless without LDP. ``` ## Notes & caveats - **Don't enable LDP on customer-facing or CGNAT interfaces.** Only the three backbone ports listed above. LDP advertised onto a customer port would expose label bindings to anyone who joined the broadcast domain. - **No BGP-LU, no L3VPN, no TE yet.** This is plain IPv4 LDP only — the point is to get labeled IGP forwarding working between these three before layering services on top. - **iBGP next-hop-self is not required** for this phase because we have no BGP-painted services riding the LSPs yet. When that changes (e.g., bringing public /24s in over iBGP), confirm next-hop-self on the ingress LSR so labeled paths terminate at a router with an LSP. - **OSPFv3 / IPv6 LDP is intentionally out of scope.** RouterOS 7 LDP is IPv4-only; v6 transport would need MPLSv6 / 6PE which we are not setting up here. - **BFD on the AF radios** (already enabled on the verona–climax and climax–core OSPF templates with `use-bfd=yes`) is what fast-fails the underlay; LDP itself follows OSPF and tears down accordingly. - **Expanding to other PoPs:** repeat steps 1–3 on the new router with its loopback as `lsr-id`/`transport-addresses`, then add `/mpls ldp interface` on the link to whichever existing MPLS router it peers with. No flag day. --- # Deferred work (2026-05-09 session) ## Customer traceroute hop-collapse — partially done **Goal**: customer traceroutes from any tower should look like `customer → tower → core → edge → TWC → ...` — collapsing the intermediate spine LSRs (climax, newhope) into invisible transit hops. **What's done as of 2026-05-09**: - `propagate-ttl=no` set on all 8 fleet routers (verona/climax/culleoka/ newhope/lowry/982/494/core). This switches MPLS from the default uniform model to the pipe model: ingress LSR sets label-TTL=255 (independent of IP-TTL), transit LSRs decrement label-TTL only, egress preserves the IP-TTL across the LSP. Effect: any traffic that actually rides an LSP collapses the intermediate label-switching hops in traceroute. - This works **today** for traffic whose destination has an LDP label binding — i.e., loopback-to-loopback (`10.254.254.x/32` between any two fleet routers), tower-to-tower-customer-prefix (`100.64.x.x/32`, the tower /44s), and the connected /29 backbone links. Verified: `verona /tool/traceroute 10.254.254.109` returns a single hop (lowry), the climax↔core↔newhope spine is invisible. **What's still missing (the actual goal — customer→internet collapse)**: ROS picks an outgoing label based on the **destination prefix's FEC**, not on the recursive next-hop. Internet destinations like `1.1.1.1` have no LDP label in any tower's forwarding table, so the packet leaves plain-IP even when the tower's static default points at a labeled prefix (`10.254.254.253` or `10.254.254.254`). We tested both gateway changes on 494 — neither caused label imposition. Reverted. **To finish this** the fleet would need **BGP labeled unicast** (RFC 3107, ROS `address-families=ip-labeled-unicast` on iBGP): 1. Core (and/or edge once MPLS-to-edge is solved) advertises BGP routes to towers with labels attached. Simplest version: just the default route `0.0.0.0/0` with next-hop=core's loopback and a label. 2. Towers receive labeled BGP routes. When forwarding to any destination matching such a route, the tower pushes the BGP-LU label, then the LDP transport label on top, and rides the LSP. 3. With `propagate-ttl=no` already set, the LSP is invisible in traceroute. Customer sees `tower → core → edge → TWC → ...` (3 hops to TWC, regardless of which tower they're on). **Why we didn't do it today**: - Three MPLS-related outages already (`asymmetric_ldp_parallel_paths` memory entry). Don't add a fourth without a maintenance window. - BGP-LU isn't tested in this fleet. Need to: - Confirm ROS7 syntax for `address-families=ip-labeled-unicast` on the existing iBGP-RR mesh. - Decide whether to advertise LU labels for just the default route or for the full reflected RIB. - Decide what core advertises — just `0.0.0.0/0`, or also tower /44s so return-path traffic from edge to tower customers can also ride a labeled path with hop hiding (currently the return path edge → core → tower is plain-IP edge→core then labeled core→tower). - A future attempt should also revisit whether MPLS-to-edge is desirable alongside this. With BGP-LU from core, edge doesn't need to be in LDP for towers to ride an LSP toward edge — the LSP terminates at core (where the BGP-LU label is popped), and edge sees plain IP exactly like today. So **BGP-LU may be a cleaner path than extending LDP to edge** for this specific goal. ## Extending LDP to edge — blocked, root cause unknown See `~/.claude/projects/-Users-graham-dev-network-mikrotik-tool/memory/ asymmetric_ldp_parallel_paths.md` for the full incident notes. Three live attempts on 2026-05-09 to bring up an LDP session between core and edge all dropped fleet internet traffic within seconds. Hypotheses tried and disproven: asymmetric LDP path redirect, BGP-table label flooding (we tightened edge's advertise-filter to just `10.254.254.254/32`), and the flood still happened. **Don't attempt again live without lab replication or a console on edge.** State left in place that survives across attempts: - `mpls-mtu=1508` on edge `sfp-sfpplus7-core-direct` and `sfp-sfpplus11-preseem`. - `mpls-mtu=1508` on core `ether3-edge-direct` and `sfp-sfpplus1-edge-preseem`. - Fasttrack-bypass pairs (in/out) on chain=forward for those four interfaces on both routers. - Edge `/mpls ldp advertise-filter` restricted to advertise only `10.254.254.254/32`. - Edge `/mpls ldp set distribute-for-default-route=no`. These are all harmless and would be required again on the next attempt; do not roll them back.