graham/network
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
network/docs/plans/2026-05-14-ipv6-traceroute-collapse-6pe.md
Graham McIntire d0d1667991
update
2026-07-19 14:42:02 -05:00

14 KiB
Raw Blame History

IPv6 Traceroute Collapse (6PE) Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Make IPv6 traffic transit the vntx backbone as MPLS-labeled frames so traceroute6 collapses the internal core to ~1 hop, matching the existing IPv4 behaviour.

Architecture: Today LDP is IPv4-only and /mpls settings propagate-ttl=no is set fleet-wide, so labeled IPv4 hides the core. IPv6 is forwarded natively (hop-limit decremented every hop). To get the same collapse, IPv6 must ride the existing IPv4 LSPs — either via 6PE (RFC 4798: labeled-unicast IPv6 in the global table, BGP next-hop = IPv4 loopback) or VPNv6 (IPv6 in a VRF over the v4 core). Both are unproven on this fleet, so this plan is spike-first: Phase 1 is a disposable lab that produces a hard go/no-go before any production router is touched.

Tech Stack: RouterOS 7.21.4 (8 tower routers, arm64/tile), RouterOS 6.49.18 (edge), MP-BGP, LDP, OSPFv2/v3, Proxmox + CHR for the lab.

Why this plan is spike-first (read before doing anything)

Verified facts that make the production syntax unknowable today:

  1. ROS7 6PE is undocumented. MikroTik's MPLS Overview page never names 6PE. Community testing (forum, v7.8 CHR) reports "mixed/uncertain" results. There is no authoritative config reference.
  2. Possible hard conflict with the existing v6 iBGP. The fleet already runs afi=ipv6 iBGP (tower → core RR → edge) to carry the per-tower /44 aggregates. A third-party note (netlab) states you cannot run 6PE and plain IPv6 BGP-LU on the same node. If that holds on ROS7, 6PE would replace, not augment, the current v6 iBGP — a much larger change.
  3. edge is RouterOS 6.49.18. It is the PE toward TWC (AS11427). ROS6's MPLS/BGP stack is entirely separate from ROS7's, and ROS6 6PE support is at least as questionable. edge may need to stay native-v6 (its single upstream hop is cosmetically irrelevant anyway).

Therefore: no production router is reconfigured until Phase 1 proves the mechanism in a lab and answers conflict-question #2. Phases 2–4 deliberately do not contain exact commands — they are filled in from the spike's findings (Task 1.7).

If you are tempted to skip Phase 1: don't. A half-working 6PE deployment black-holes customer IPv6 fleet-wide.

Phase 1 — Lab spike (disposable, zero production impact)

Build a 3-router CHR lab on graham's home Proxmox that mirrors a slice of the backbone (edge′ — core′ — verona′) plus a CE host at each end, prove that labeled IPv6 collapses traceroute6, and determine whether 6PE coexists with plain IPv6 iBGP.

Topology:

[CE-A]──ce-a/v6──[verona′]──p2p v4 LDP──[core′]──p2p v4 LDP──[edge′]──ce-b/v6──[CE-B]
        2606:db8:a::/64                                              2606:db8:b::/64
   loopbacks: verona′ 10.0.0.1  core′ 10.0.0.2  edge′ 10.0.0.3   (v6: 2606:db8:0:10::1/2/3)

CE-A/CE-B can be CHRs or any Linux VM — they just need an IPv6 address and traceroute6.

Task 1.1: Stand up the CHR lab VMs

Files:

  • Create: docs/plans/lab/6pe-lab-notes.md (running log of what works — this becomes the source for Phases 2–4)

Step 1: On Proxmox, import the RouterOS CHR image (https://mikrotik.com/download → "Cloud Hosted Router" → 7.21.4 stable VMDK/qcow2). Create 5 VMs: verona-lab, core-lab, edge-lab, ce-a, ce-b. 256 MB RAM / 1 vCPU each is plenty. CHR is licence-free at ≤1 Mbps — fine for a lab.

Step 2: Create 4 isolated Proxmox bridges (no uplink): vmbr-cea, vmbr-vc (verona′↔core′), vmbr-ce (core′↔edge′), vmbr-ceb. Wire NICs:

  • verona-lab: eth1→vmbr-cea, eth2→vmbr-vc
  • core-lab: eth1→vmbr-vc, eth2→vmbr-ce
  • edge-lab: eth1→vmbr-ce, eth2→vmbr-ceb
  • ce-a: eth1→vmbr-cea ; ce-b: eth1→vmbr-ceb

Step 3: Boot all 5. Confirm console login on each.

Step 4 (commit):

cd /Users/graham/dev/network
git add docs/plans/lab/6pe-lab-notes.md
git commit -m "6pe lab: stand up 5-VM CHR topology"

Expected: lab notes file committed with the topology + VM IDs recorded.

Task 1.2: Build the IPv4 + LDP core (mirror of production)

Step 1: On each lab router configure loopback, p2p /30s, OSPFv2, LDP, and propagate-ttl=no — a scaled-down copy of the real fleet. Example for core-lab (adapt addresses per router):

/interface bridge add name=lo
/ip address add address=10.0.0.2/32 interface=lo
/ip address add address=10.1.12.2/30 interface=ether1   ;; to verona′
/ip address add address=10.1.23.1/30 interface=ether2   ;; to edge′
/routing ospf instance add name=v2 router-id=10.0.0.2
/routing ospf area add name=backbone instance=v2 area-id=0.0.0.0
/routing ospf interface-template add area=backbone interfaces=lo passive
/routing ospf interface-template add area=backbone interfaces=ether1,ether2 type=ptp
/mpls ldp add lsr-id=10.0.0.2 transport-addresses=10.0.0.2
/mpls ldp interface add interface=ether1
/mpls ldp interface add interface=ether2
/mpls settings set propagate-ttl=no

Step 2: Verify the v4 core. From verona-lab:

/routing ospf neighbor print          ;; expect Full to core′
/mpls ldp neighbor print              ;; expect operational to core′
/mpls forwarding-table print          ;; expect labels for 10.0.0.2, 10.0.0.3
/tool traceroute 10.0.0.3             ;; edge′ loopback — expect 1 hop (LDP + propagate-ttl=no)

Expected: traceroute to the far loopback shows one hop. This proves the lab reproduces the production v4 collapse. If it does not, fix the lab before continuing — Phase 1's whole value is a faithful mirror.

Step 3 (commit): Record the working core configs in 6pe-lab-notes.md, commit.

Task 1.3: Baseline — native IPv6, measure the problem

Step 1: Add native IPv6 + OSPFv3 across the lab core (loopbacks 2606:db8:0:10::1/2/3, p2p 2606:db8:0:1N::/64), and give CE-A/CE-B addresses in 2606:db8:a::/64 and 2606:db8:b::/64 with static routes or OSPFv3.

Step 2: Measure baseline. From CE-A:

traceroute6 2606:db8:b::1

Expected: 3+ hops (verona′, core′, edge′ each appear). Record exact output in lab notes — this is the "before".

Step 3 (commit): Commit lab notes with the native-v6 baseline traceroute.

Task 1.4: Add the existing-style IPv6 iBGP (reproduce the conflict surface)

Step 1: Configure v6 iBGP exactly like production: verona-lab and edge-lab as RR-clients of core-lab, afi=ip,ipv6, sessions over the v6 loopbacks, redistributing connected. This is the config that 6PE may conflict with.

Step 2: Verify v6 iBGP sessions establish and the CE prefixes are learned. Record in lab notes.

Step 3 (commit).

Task 1.5: Attempt 6PE — the actual unknown

This is the discovery task. Do not fabricate commands — find them.

Step 1: Read, in this order, and capture findings in lab notes:

  • MikroTik MPLS Overview + BGP doc pages (help.mikrotik.com) — search for "label", "ipv6", "6PE", "labeled-unicast".
  • /routing bgp connection and /routing bgp template — dump every available property on a lab router (/routing bgp connection print detail on a connection with all afi values; try afi=ipv6 plus any label-related sub-property the CLI tab-completes).
  • The MikroTik forum thread "IPv6-on-MPLS BGP-free Core?" for working/failing examples.

Step 2: On the lab, attempt 6PE: make the v6 iBGP advertise labeled IPv6 with BGP next-hop = the IPv4 loopback (as ::ffff:10.0.0.x). Try it alongside the Task 1.4 plain v6 iBGP first.

Step 3: Decision question A — coexistence. Does enabling 6PE break or get rejected next to the plain afi=ipv6 iBGP? Record the exact behaviour. If they cannot coexist, note that production Phase 3 must replace the v6 iBGP, and re-test 1.5 with the plain v6 iBGP removed.

Step 4 (commit): Lab notes updated with the exact 6PE config that loads without error (or the proof that it can't).

Task 1.6: Verify the collapse (the success criterion)

Step 1: Confirm labels are bound to IPv6 routes:

/mpls forwarding-table print     ;; expect 2606:db8:b::/64 etc. with a label

Step 2: From CE-A:

traceroute6 2606:db8:b::1

Success = 1 hop across the lab core (or core′ invisible), matching the v4 behaviour from Task 1.2.

Step 3: Failure-mode checks — confirm nothing else broke:

  • CE-A ↔ CE-B large-packet ping (ping6 -s 1452) — no new MTU loss (label adds 4 bytes; lab p2p MTU must have headroom, same as the real mpls-mtu=1508).
  • Kill the core-lab↔edge-lab link, confirm v6 reconverges (or fails predictably).

Step 4 (commit): Lab notes updated with the before/after traceroute and the failure-mode results.

Task 1.7: Write the go/no-go and fill in Phases 2–4

Step 1: In 6pe-lab-notes.md, write a GO / NO-GO verdict answering:

  • Does labeled IPv6 collapse traceroute6 on ROS 7.21.4? (Task 1.6)
  • Can 6PE coexist with the existing v6 iBGP, or must it replace it? (Task 1.5 Step 3)
  • What is the exact, tested per-router 6PE config delta?
  • Does anything require an MTU change beyond the existing mpls-mtu=1508?

Step 2: If GO: replace the "TBD pending spike" bodies of Phase 2 and Phase 3 below with the exact tested commands from the lab. If NO-GO: stop here, jump to "Alternatives", and bring the recommendation to graham.

Step 3 (commit):

git add docs/plans/lab/6pe-lab-notes.md docs/plans/2026-05-14-ipv6-traceroute-collapse-6pe.md
git commit -m "6pe lab: go/no-go verdict + fill production phases"

Task 1.8: Tear down the lab

Step 1: Delete the 5 CHR VMs and 4 Proxmox bridges. The lab notes (committed) are the deliverable; the VMs are disposable.

Step 2 (commit): Note teardown in lab notes, commit.

Phase 2 — Production pilot: verona ↔ climax ↔ core (contingent on Phase 1 = GO)

Body TBD — populated from Task 1.7. Constraints the filled-in version must honour:

  • Apply order matters. Bring up the labeled-v6 path core-first, then climax, then verona, so a tower is never advertising labeled v6 toward a router that can't resolve it.
  • One change window, one link triangle. Pilot only verona/climax/core; do not touch the other 5 towers until the pilot soaks ≥24h.
  • Each router task = change → verify → rollback-ready, e.g.:
    • apply 6PE delta on core
    • /mpls forwarding-table print shows labels for the pilot towers' v6
    • /routing bgp advertisements / /routing route print confirms labeled v6 NLRI
    • from graham's home Mac: traceroute6 to an off-net v6 host — core no longer a distinct hop
    • ping6 -s 1452 to 2606:4700:4700::1111 still 0% loss
    • customer-facing: a known v6 PPPoE/wired client still reaches the internet
  • Rollback (per router): the inverse delta, recorded in Task 1.7. Native v6 + OSPFv3 + the existing v6 iBGP must still be present underneath (do not delete them in the pilot — if 6PE coexists, native is the fallback; if it doesn't, Phase 2 rollback = re-add v6 iBGP).
  • Soak gate: 24h with no v6 customer complaints + traceroute stays collapsed before Phase 3.

Phase 3 — Fleet rollout: culleoka, newhope, lowry, 982, 494 (contingent)

Body TBD — populated from Task 1.7. Constraints:

  • Roll out in the CGNAT-renumber order from subnets.md (one tower per change window).
  • Same per-router verify set as Phase 2.
  • newhope carries lowry's transit — do newhope before lowry.
  • After each tower: traceroute6 from home Mac still collapsed, customer v6 spot-check.
  • Keep mikrotik-tool (api/script) as the push mechanism; one .rsc per router or a templated apply.

Phase 4 — edge (RouterOS 6.49.18) (contingent, likely "leave native")

Body TBD. Default expectation from Phase 1 framing point #3: edge stays native IPv6. edge is one hop from core and one hop from TWC — collapsing it changes nothing a user sees. Only revisit if Task 1.7 specifically finds ROS6 6PE both works and is needed. If edge stays native, document that the v6 path is "labeled across the ROS7 core, native for the edge↔TWC hop" so the next operator isn't surprised.

Rollback (whole project)

  • Phase 1: delete VMs. Nothing to roll back.
  • Phase 2/3: per-router inverse delta from Task 1.7, applied in reverse order (towers before core). Native v6 forwarding + OSPFv3 are never removed during pilot/rollout, so the floor is always "back to today's working native v6."
  • Trigger: any v6 customer outage, traceroute6 black-holing (not just collapsing), or MTU loss that won't resolve.
  • The firewall, DNS, and home-router work already done (forward accept src 2606:1c80::/32, v6-dns resolvers, home /56) are independent of this plan and are not touched by any phase.

Alternatives (if Phase 1 = NO-GO)

Option What it gives Cost / risk
Do nothing v6 traceroute keeps showing the core (internal hops currently appear as * — they don't even reply). Everything works. Zero. The collapse is cosmetic.
VPNv6 IPv6 in a VRF over the v4 core — is labeled, so traceroute collapses. ROS7 documents vpnv6. Large: every router's customer v6 (addresses, pools, PPP profiles, firewall, the home /56) moves into a VRF. Much bigger blast radius than 6PE.
Make internal hops reply instead of hiding them Opposite of the goal, but "honest" traceroute — each router answers with ICMPv6 time-exceeded. Small chain=input tweak. Only relevant if the real annoyance is the * * * lines, not the hop count.

Recommendation if NO-GO: do nothing. The IPv4 collapse is a side-effect of MPLS existing for real routing reasons; chasing the same cosmetics for IPv6 via a VRF migration is not worth the blast radius. Revisit only if MikroTik documents real 6PE support in a later ROS7 release.

Open questions for graham (resolve before Phase 2, ideally before Phase 1)

  1. Is collapsing v6 traceroute worth a multi-window backbone change, or is it "nice to have"? (Sets how hard to push past a marginal Phase 1 result.)
  2. Proxmox capacity at home for 5 throwaway CHR VMs — OK to spin up?
  3. edge's eventual ROS6→ROS7 upgrade — if that's on the roadmap anyway, Phase 4 should wait for it rather than attempt ROS6 6PE.