graham/network
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
network/mikrotik-tool/mpls.md
Graham McIntire f1e8faa797
routers update
2026-05-09 14:56:17 -05:00

12 KiB
Raw Permalink Blame History

MPLS / LDP enablement: verona ↔ climax ↔ core

Goal: bring up MPLS forwarding with LDP across the verona–climax–core spine on RouterOS 7, leaving the rest of the network untouched. Once these three are running labeled forwarding, expanding to other PoPs is incremental (add the LDP instance + interface on each new router).

Topology being labeled

verona (10.254.254.101)
  └── ether3-climax-11ghz  10.250.1.25/29 ──┐
                                            │  10.250.1.24/29 (AF11 backhaul)
  ┌── ether6-verona-11ghz  10.250.1.30/29 ──┘
climax (10.254.254.102)
  └── ether4-380-airfiber24 10.250.1.94/29 ──┐
                                             │  10.250.1.88/29 (AF24 backhaul)
  ┌── ether5-climax          10.250.1.89/29 ─┘
core (10.254.254.253)

OSPFv2 (backbone-v2, area 0) already redistributes connected, so all three loopbacks are reachable. That is the IGP we attach LDP to. Nothing else changes.

RouterOS versions in play: verona 7.20.8, climax 7.22, core 7.20.6 — all include MPLS in the system package, no extra .npk to install.

Pre-flight (run on each of the three before configuring)

# OSPF must be up and loopbacks reachable end-to-end.
/routing ospf neighbor print
/ip route print where dst-address~"10.254.254.10[12]/32" or dst-address~"10.254.254.253/32"

# Confirm the underlay MTU on the AF11 / AF24 ports — MPLS adds 4 bytes per
# label, so we want headroom. Check both physical (l2mtu) and IP MTU:
/interface print detail where name~"ether3-climax-11ghz|ether6-verona-11ghz|ether4-380-airfiber|ether5-climax"

Expected: l2mtu ≥ 1600 on the AF radios (default on CCR ports). If anything is at 1500 we'll bump mpls-mtu to 1500 explicitly so we never generate a 1504-byte frame the radio drops.

Step 1 — verona (10.254.254.101)

# LDP instance, lsr-id and transport address pinned to the loopback.
/mpls ldp
add disabled=no lsr-id=10.254.254.101 transport-addresses=10.254.254.101 vrf=main

# Enable LDP discovery on the climax-facing link only.
/mpls ldp interface
add interface=ether3-climax-11ghz

# Pin the MPLS MTU explicitly. mpls-mtu is the cap on the *labeled* frame,
# so a 1500-byte IP payload + 4-byte label = 1504 needs at least 1508 to
# pass; 1508 also leaves room for one extra stacked label (VPN/FRR/etc.).
/mpls interface
add interface=ether3-climax-11ghz mpls-mtu=1508

Step 2 — climax (10.254.254.102)

/mpls ldp
add disabled=no lsr-id=10.254.254.102 transport-addresses=10.254.254.102 vrf=main

/mpls ldp interface
add interface=ether6-verona-11ghz
add interface=ether4-380-airfiber24

/mpls interface
add interface=ether6-verona-11ghz   mpls-mtu=1508
add interface=ether4-380-airfiber24 mpls-mtu=1508

Step 3 — core (10.254.254.253)

/mpls ldp
add disabled=no lsr-id=10.254.254.253 transport-addresses=10.254.254.253 vrf=main

/mpls ldp interface
add interface=ether5-climax

/mpls interface
add interface=ether5-climax mpls-mtu=1508

Order doesn't matter — LDP discovery is symmetric. As soon as both ends of a link have /mpls ldp interface populated, hellos start flowing on 224.0.0.2:646 and a TCP session forms between the two transport addresses.

Verification (the actually-works part)

A. Control plane: LDP adjacencies

Run on each router:

/mpls ldp neighbor print detail

Required state:

  • peer= lists the neighbor's lsr-id (a loopback IP),
  • transport= is the neighbor's loopback IP,
  • flag column shows O (operational); D (dynamic) is fine — nD (not discovered) or missing entries means hellos aren't reaching the far side (firewall, wrong interface, MTU).

Expected counts:

Router LDP neighbors
verona 1 (climax)
climax 2 (verona, core)
core 1 (climax)

B. Control plane: label bindings exist for the remote loopbacks

/mpls ldp local-binding  print where dst-address~"10.254.254.(101|102|253)/32"
/mpls ldp remote-binding print where dst-address~"10.254.254.(101|102|253)/32"

Each router should advertise an implicit-null (or label 3) for its own loopback in local-binding, and learn labels for the two other loopbacks in remote-binding.

C. Forwarding plane: FIB has the labels

/mpls forwarding-table print

You should see entries with out-label, out-interface, and nexthop matching the OSPF next-hop toward each remote loopback. On climax you'll see two entries — one swapping toward verona, one toward core. On verona and core you'll see one entry for each remote loopback (the second loopback is reached via climax with stacked labels collapsing to a single label because we have only three nodes).

D. Data plane: traceroute prints labels

From verona, source-routed off the loopback:

/tool traceroute 10.254.254.253 src-address=10.254.254.101 count=3

A working MPLS path prints MPLS Label=NNNN E=0 ... on the first hop (climax) — that is the smoking gun. Without MPLS the same traceroute succeeds but no MPLS Label= field appears.

Repeat from core in the other direction:

/tool traceroute 10.254.254.101 src-address=10.254.254.253 count=3

E. Data plane: MTU check

We pinned mpls-mtu=1508 — verify a full-size packet still passes once labeled (1500 IP + 4 label = 1504 on the wire, well below both the 1508 cap and the radio l2mtu of 2024):

# from verona, send 1500-byte ICMP DF=yes to core's loopback
/ping 10.254.254.253 src-address=10.254.254.101 size=1500 do-not-fragment count=5

100% return = path is large enough for label + 1500 payload. Any loss means the labeled frame (1504 bytes) doesn't fit somewhere — most likely an l2mtu issue on a backhaul radio. Either bump l2mtu on the radio side, or raise mpls-mtu if you increased the IP MTU. Do not lower mpls-mtu below 1508 — that caps the labeled frame and will cause ICMP-frag-needed for ordinary 1500-byte DF customer traffic.

Rollback (per router)

If anything misbehaves, MPLS is purely additive — removing the two tables returns the box to plain IP forwarding. OSPF is untouched.

/mpls ldp interface remove [find]
/mpls ldp           remove [find]
# leave /mpls interface mpls-mtu settings as-is; harmless without LDP.

Notes & caveats

  • Don't enable LDP on customer-facing or CGNAT interfaces. Only the three backbone ports listed above. LDP advertised onto a customer port would expose label bindings to anyone who joined the broadcast domain.
  • No BGP-LU, no L3VPN, no TE yet. This is plain IPv4 LDP only — the point is to get labeled IGP forwarding working between these three before layering services on top.
  • iBGP next-hop-self is not required for this phase because we have no BGP-painted services riding the LSPs yet. When that changes (e.g., bringing public /24s in over iBGP), confirm next-hop-self on the ingress LSR so labeled paths terminate at a router with an LSP.
  • OSPFv3 / IPv6 LDP is intentionally out of scope. RouterOS 7 LDP is IPv4-only; v6 transport would need MPLSv6 / 6PE which we are not setting up here.
  • BFD on the AF radios (already enabled on the verona–climax and climax–core OSPF templates with use-bfd=yes) is what fast-fails the underlay; LDP itself follows OSPF and tears down accordingly.
  • Expanding to other PoPs: repeat steps 1–3 on the new router with its loopback as lsr-id/transport-addresses, then add /mpls ldp interface on the link to whichever existing MPLS router it peers with. No flag day.

Deferred work (2026-05-09 session)

Customer traceroute hop-collapse — partially done

Goal: customer traceroutes from any tower should look like customer → tower → core → edge → TWC → ... — collapsing the intermediate spine LSRs (climax, newhope) into invisible transit hops.

What's done as of 2026-05-09:

  • propagate-ttl=no set on all 8 fleet routers (verona/climax/culleoka/ newhope/lowry/982/494/core). This switches MPLS from the default uniform model to the pipe model: ingress LSR sets label-TTL=255 (independent of IP-TTL), transit LSRs decrement label-TTL only, egress preserves the IP-TTL across the LSP. Effect: any traffic that actually rides an LSP collapses the intermediate label-switching hops in traceroute.
  • This works today for traffic whose destination has an LDP label binding — i.e., loopback-to-loopback (10.254.254.x/32 between any two fleet routers), tower-to-tower-customer-prefix (100.64.x.x/32, the tower /44s), and the connected /29 backbone links. Verified: verona /tool/traceroute 10.254.254.109 returns a single hop (lowry), the climax↔core↔newhope spine is invisible.

What's still missing (the actual goal — customer→internet collapse):

ROS picks an outgoing label based on the destination prefix's FEC, not on the recursive next-hop. Internet destinations like 1.1.1.1 have no LDP label in any tower's forwarding table, so the packet leaves plain-IP even when the tower's static default points at a labeled prefix (10.254.254.253 or 10.254.254.254). We tested both gateway changes on 494 — neither caused label imposition. Reverted.

To finish this the fleet would need BGP labeled unicast (RFC 3107, ROS address-families=ip-labeled-unicast on iBGP):

  1. Core (and/or edge once MPLS-to-edge is solved) advertises BGP routes to towers with labels attached. Simplest version: just the default route 0.0.0.0/0 with next-hop=core's loopback and a label.
  2. Towers receive labeled BGP routes. When forwarding to any destination matching such a route, the tower pushes the BGP-LU label, then the LDP transport label on top, and rides the LSP.
  3. With propagate-ttl=no already set, the LSP is invisible in traceroute. Customer sees tower → core → edge → TWC → ... (3 hops to TWC, regardless of which tower they're on).

Why we didn't do it today:

  • Three MPLS-related outages already (asymmetric_ldp_parallel_paths memory entry). Don't add a fourth without a maintenance window.
  • BGP-LU isn't tested in this fleet. Need to:
    • Confirm ROS7 syntax for address-families=ip-labeled-unicast on the existing iBGP-RR mesh.
    • Decide whether to advertise LU labels for just the default route or for the full reflected RIB.
    • Decide what core advertises — just 0.0.0.0/0, or also tower /44s so return-path traffic from edge to tower customers can also ride a labeled path with hop hiding (currently the return path edge → core → tower is plain-IP edge→core then labeled core→tower).
  • A future attempt should also revisit whether MPLS-to-edge is desirable alongside this. With BGP-LU from core, edge doesn't need to be in LDP for towers to ride an LSP toward edge — the LSP terminates at core (where the BGP-LU label is popped), and edge sees plain IP exactly like today. So BGP-LU may be a cleaner path than extending LDP to edge for this specific goal.

Extending LDP to edge — blocked, root cause unknown

See ~/.claude/projects/-Users-graham-dev-network-mikrotik-tool/memory/ asymmetric_ldp_parallel_paths.md for the full incident notes. Three live attempts on 2026-05-09 to bring up an LDP session between core and edge all dropped fleet internet traffic within seconds. Hypotheses tried and disproven: asymmetric LDP path redirect, BGP-table label flooding (we tightened edge's advertise-filter to just 10.254.254.254/32), and the flood still happened. Don't attempt again live without lab replication or a console on edge.

State left in place that survives across attempts:

  • mpls-mtu=1508 on edge sfp-sfpplus7-core-direct and sfp-sfpplus11-preseem.
  • mpls-mtu=1508 on core ether3-edge-direct and sfp-sfpplus1-edge-preseem.
  • Fasttrack-bypass pairs (in/out) on chain=forward for those four interfaces on both routers.
  • Edge /mpls ldp advertise-filter restricted to advertise only 10.254.254.254/32.
  • Edge /mpls ldp set distribute-for-default-route=no.

These are all harmless and would be required again on the next attempt; do not roll them back.