network/docs/plans/2026-05-14-ipv6-traceroute-collapse-6pe.md

# IPv6 Traceroute Collapse (6PE) Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Make IPv6 traffic transit the vntx backbone as MPLS-labeled frames so `traceroute6` collapses the internal core to ~1 hop, matching the existing IPv4 behaviour.

**Architecture:** Today LDP is IPv4-only and `/mpls settings propagate-ttl=no` is set fleet-wide, so labeled IPv4 hides the core. IPv6 is forwarded natively (hop-limit decremented every hop). To get the same collapse, IPv6 must ride the existing IPv4 LSPs — either via **6PE** (RFC 4798: labeled-unicast IPv6 in the global table, BGP next-hop = IPv4 loopback) or **VPNv6** (IPv6 in a VRF over the v4 core). Both are unproven on this fleet, so this plan is **spike-first**: Phase 1 is a disposable lab that produces a hard go/no-go before any production router is touched.

**Tech Stack:** RouterOS 7.21.4 (8 tower routers, arm64/tile), RouterOS 6.49.18 (edge), MP-BGP, LDP, OSPFv2/v3, Proxmox + CHR for the lab.

---

## Why this plan is spike-first (read before doing anything)

Verified facts that make the production syntax **unknowable today**:

1. **ROS7 6PE is undocumented.** MikroTik's MPLS Overview page never names 6PE. Community testing (forum, v7.8 CHR) reports "mixed/uncertain" results. There is no authoritative config reference.
2. **Possible hard conflict with the existing v6 iBGP.** The fleet already runs `afi=ipv6` iBGP (tower → core RR → edge) to carry the per-tower `/44` aggregates. A third-party note (netlab) states you *cannot run 6PE and plain IPv6 BGP-LU on the same node*. If that holds on ROS7, 6PE would **replace**, not augment, the current v6 iBGP — a much larger change.
3. **edge is RouterOS 6.49.18.** It is the PE toward TWC (AS11427). ROS6's MPLS/BGP stack is entirely separate from ROS7's, and ROS6 6PE support is at least as questionable. edge may need to stay native-v6 (its single upstream hop is cosmetically irrelevant anyway).

Therefore: **no production router is reconfigured until Phase 1 proves the mechanism in a lab and answers conflict-question #2.** Phases 2–4 deliberately do not contain exact commands — they are filled in from the spike's findings (Task 1.7).

**If you are tempted to skip Phase 1: don't.** A half-working 6PE deployment black-holes customer IPv6 fleet-wide.

---

## Phase 1 — Lab spike (disposable, zero production impact)

Build a 3-router CHR lab on graham's home Proxmox that mirrors a slice of the
backbone (`edge′ — core′ — verona′`) plus a CE host at each end, prove that
labeled IPv6 collapses `traceroute6`, and determine whether 6PE coexists with
plain IPv6 iBGP.

**Topology:**
```
[CE-A]──ce-a/v6──[verona′]──p2p v4 LDP──[core′]──p2p v4 LDP──[edge′]──ce-b/v6──[CE-B]
        2606:db8:a::/64                                              2606:db8:b::/64
   loopbacks: verona′ 10.0.0.1  core′ 10.0.0.2  edge′ 10.0.0.3   (v6: 2606:db8:0:10::1/2/3)
```
CE-A/CE-B can be CHRs or any Linux VM — they just need an IPv6 address and `traceroute6`.

### Task 1.1: Stand up the CHR lab VMs

**Files:**
- Create: `docs/plans/lab/6pe-lab-notes.md` (running log of what works — this becomes the source for Phases 2–4)

**Step 1:** On Proxmox, import the RouterOS CHR image (https://mikrotik.com/download → "Cloud Hosted Router" → 7.21.4 stable VMDK/qcow2). Create 5 VMs: `verona-lab`, `core-lab`, `edge-lab`, `ce-a`, `ce-b`. 256 MB RAM / 1 vCPU each is plenty. CHR is licence-free at ≤1 Mbps — fine for a lab.

**Step 2:** Create 4 isolated Proxmox bridges (no uplink): `vmbr-cea`, `vmbr-vc` (verona′↔core′), `vmbr-ce` (core′↔edge′), `vmbr-ceb`. Wire NICs:
- verona-lab: eth1→vmbr-cea, eth2→vmbr-vc
- core-lab: eth1→vmbr-vc, eth2→vmbr-ce
- edge-lab: eth1→vmbr-ce, eth2→vmbr-ceb
- ce-a: eth1→vmbr-cea ; ce-b: eth1→vmbr-ceb

**Step 3:** Boot all 5. Confirm console login on each.

**Step 4 (commit):**
```bash
cd /Users/graham/dev/network
git add docs/plans/lab/6pe-lab-notes.md
git commit -m "6pe lab: stand up 5-VM CHR topology"
```
Expected: lab notes file committed with the topology + VM IDs recorded.

### Task 1.2: Build the IPv4 + LDP core (mirror of production)

**Step 1:** On each lab router configure loopback, p2p /30s, OSPFv2, LDP, and `propagate-ttl=no` — a scaled-down copy of the real fleet. Example for `core-lab` (adapt addresses per router):
```rsc
/interface bridge add name=lo
/ip address add address=10.0.0.2/32 interface=lo
/ip address add address=10.1.12.2/30 interface=ether1   ;; to verona′
/ip address add address=10.1.23.1/30 interface=ether2   ;; to edge′
/routing ospf instance add name=v2 router-id=10.0.0.2
/routing ospf area add name=backbone instance=v2 area-id=0.0.0.0
/routing ospf interface-template add area=backbone interfaces=lo passive
/routing ospf interface-template add area=backbone interfaces=ether1,ether2 type=ptp
/mpls ldp add lsr-id=10.0.0.2 transport-addresses=10.0.0.2
/mpls ldp interface add interface=ether1
/mpls ldp interface add interface=ether2
/mpls settings set propagate-ttl=no
```

**Step 2: Verify the v4 core.** From `verona-lab`:
```rsc
/routing ospf neighbor print          ;; expect Full to core′
/mpls ldp neighbor print              ;; expect operational to core′
/mpls forwarding-table print          ;; expect labels for 10.0.0.2, 10.0.0.3
/tool traceroute 10.0.0.3             ;; edge′ loopback — expect 1 hop (LDP + propagate-ttl=no)
```
Expected: traceroute to the far loopback shows **one hop**. This proves the lab reproduces the production v4 collapse. If it does not, fix the lab before continuing — Phase 1's whole value is a faithful mirror.

**Step 3 (commit):** Record the working core configs in `6pe-lab-notes.md`, commit.

### Task 1.3: Baseline — native IPv6, measure the problem

**Step 1:** Add native IPv6 + OSPFv3 across the lab core (loopbacks `2606:db8:0:10::1/2/3`, p2p `2606:db8:0:1N::/64`), and give CE-A/CE-B addresses in `2606:db8:a::/64` and `2606:db8:b::/64` with static routes or OSPFv3.

**Step 2: Measure baseline.** From CE-A:
```
traceroute6 2606:db8:b::1
```
Expected: **3+ hops** (verona′, core′, edge′ each appear). Record exact output in lab notes — this is the "before".

**Step 3 (commit):** Commit lab notes with the native-v6 baseline traceroute.

### Task 1.4: Add the existing-style IPv6 iBGP (reproduce the conflict surface)

**Step 1:** Configure v6 iBGP exactly like production: `verona-lab` and `edge-lab` as RR-clients of `core-lab`, `afi=ip,ipv6`, sessions over the v6 loopbacks, redistributing connected. This is the config that 6PE may conflict with.

**Step 2: Verify** v6 iBGP sessions establish and the CE prefixes are learned. Record in lab notes.

**Step 3 (commit).**

### Task 1.5: Attempt 6PE — the actual unknown

**This is the discovery task. Do not fabricate commands — find them.**

**Step 1:** Read, in this order, and capture findings in lab notes:
- MikroTik MPLS Overview + BGP doc pages (help.mikrotik.com) — search for "label", "ipv6", "6PE", "labeled-unicast".
- `/routing bgp connection` and `/routing bgp template` — dump every available property on a lab router (`/routing bgp connection print detail` on a connection with all afi values; try `afi=ipv6` plus any label-related sub-property the CLI tab-completes).
- The MikroTik forum thread "IPv6-on-MPLS BGP-free Core?" for working/failing examples.

**Step 2:** On the lab, attempt 6PE: make the v6 iBGP advertise labeled IPv6 with BGP next-hop = the IPv4 loopback (as `::ffff:10.0.0.x`). Try it **alongside** the Task 1.4 plain v6 iBGP first.

**Step 3: Decision question A — coexistence.** Does enabling 6PE break or get rejected next to the plain `afi=ipv6` iBGP? Record the exact behaviour. If they cannot coexist, note that production Phase 3 must *replace* the v6 iBGP, and re-test 1.5 with the plain v6 iBGP removed.

**Step 4 (commit):** Lab notes updated with the exact 6PE config that loads without error (or the proof that it can't).

### Task 1.6: Verify the collapse (the success criterion)

**Step 1:** Confirm labels are bound to IPv6 routes:
```rsc
/mpls forwarding-table print     ;; expect 2606:db8:b::/64 etc. with a label
```

**Step 2:** From CE-A:
```
traceroute6 2606:db8:b::1
```
**Success = 1 hop across the lab core** (or core′ invisible), matching the v4 behaviour from Task 1.2.

**Step 3:** Failure-mode checks — confirm nothing else broke:
- CE-A ↔ CE-B large-packet ping (`ping6 -s 1452`) — no new MTU loss (label adds 4 bytes; lab p2p MTU must have headroom, same as the real `mpls-mtu=1508`).
- Kill the `core-lab`↔`edge-lab` link, confirm v6 reconverges (or fails predictably).

**Step 4 (commit):** Lab notes updated with the before/after traceroute and the failure-mode results.

### Task 1.7: Write the go/no-go and fill in Phases 2–4

**Step 1:** In `6pe-lab-notes.md`, write a **GO / NO-GO** verdict answering:
- Does labeled IPv6 collapse `traceroute6` on ROS 7.21.4? (Task 1.6)
- Can 6PE coexist with the existing v6 iBGP, or must it replace it? (Task 1.5 Step 3)
- What is the **exact, tested** per-router 6PE config delta?
- Does anything require an MTU change beyond the existing `mpls-mtu=1508`?

**Step 2:** If **GO**: replace the "TBD pending spike" bodies of Phase 2 and Phase 3 below with the exact tested commands from the lab. If **NO-GO**: stop here, jump to "Alternatives", and bring the recommendation to graham.

**Step 3 (commit):**
```bash
git add docs/plans/lab/6pe-lab-notes.md docs/plans/2026-05-14-ipv6-traceroute-collapse-6pe.md
git commit -m "6pe lab: go/no-go verdict + fill production phases"
```

### Task 1.8: Tear down the lab

**Step 1:** Delete the 5 CHR VMs and 4 Proxmox bridges. The lab notes (committed) are the deliverable; the VMs are disposable.

**Step 2 (commit):** Note teardown in lab notes, commit.

---

## Phase 2 — Production pilot: verona ↔ climax ↔ core *(contingent on Phase 1 = GO)*

**Body TBD — populated from Task 1.7.** Constraints the filled-in version must honour:

- **Apply order matters.** Bring up the labeled-v6 path core-first, then climax, then verona, so a tower is never advertising labeled v6 toward a router that can't resolve it.
- **One change window, one link triangle.** Pilot only verona/climax/core; do not touch the other 5 towers until the pilot soaks ≥24h.
- **Each router task = change → verify → rollback-ready**, e.g.:
  - apply 6PE delta on core
  - `/mpls forwarding-table print` shows labels for the pilot towers' v6
  - `/routing bgp advertisements` / `/routing route print` confirms labeled v6 NLRI
  - from graham's home Mac: `traceroute6` to an off-net v6 host — core no longer a distinct hop
  - `ping6 -s 1452` to `2606:4700:4700::1111` still 0% loss
  - customer-facing: a known v6 PPPoE/wired client still reaches the internet
- **Rollback (per router):** the inverse delta, recorded in Task 1.7. Native v6 + OSPFv3 + the existing v6 iBGP must still be present underneath (do not delete them in the pilot — if 6PE coexists, native is the fallback; if it doesn't, Phase 2 rollback = re-add v6 iBGP).
- **Soak gate:** 24h with no v6 customer complaints + traceroute stays collapsed before Phase 3.

## Phase 3 — Fleet rollout: culleoka, newhope, lowry, 982, 494 *(contingent)*

**Body TBD — populated from Task 1.7.** Constraints:

- Roll out in the CGNAT-renumber order from `subnets.md` (one tower per change window).
- Same per-router verify set as Phase 2.
- newhope carries lowry's transit — do newhope before lowry.
- After each tower: `traceroute6` from home Mac still collapsed, customer v6 spot-check.
- Keep `mikrotik-tool` (`api`/`script`) as the push mechanism; one `.rsc` per router or a templated apply.

## Phase 4 — edge (RouterOS 6.49.18) *(contingent, likely "leave native")*

**Body TBD.** Default expectation from Phase 1 framing point #3: **edge stays native IPv6.** edge is one hop from core and one hop from TWC — collapsing it changes nothing a user sees. Only revisit if Task 1.7 specifically finds ROS6 6PE both works and is needed. If edge stays native, document that the v6 path is "labeled across the ROS7 core, native for the edge↔TWC hop" so the next operator isn't surprised.

---

## Rollback (whole project)

- **Phase 1:** delete VMs. Nothing to roll back.
- **Phase 2/3:** per-router inverse delta from Task 1.7, applied in reverse order (towers before core). Native v6 forwarding + OSPFv3 are never removed during pilot/rollout, so the floor is always "back to today's working native v6."
- **Trigger:** any v6 customer outage, `traceroute6` black-holing (not just collapsing), or MTU loss that won't resolve.
- The firewall, DNS, and home-router work already done (forward `accept src 2606:1c80::/32`, `v6-dns` resolvers, home `/56`) are **independent of this plan** and are not touched by any phase.

---

## Alternatives (if Phase 1 = NO-GO)

| Option | What it gives | Cost / risk |
|---|---|---|
| **Do nothing** | v6 traceroute keeps showing the core (internal hops currently appear as `*` — they don't even reply). Everything works. | Zero. The collapse is cosmetic. |
| **VPNv6** | IPv6 in a VRF over the v4 core — *is* labeled, so traceroute collapses. ROS7 documents `vpnv6`. | Large: every router's customer v6 (addresses, pools, PPP profiles, firewall, the home `/56`) moves into a VRF. Much bigger blast radius than 6PE. |
| **Make internal hops reply instead of hiding them** | Opposite of the goal, but "honest" traceroute — each router answers with ICMPv6 time-exceeded. | Small `chain=input` tweak. Only relevant if the real annoyance is the `* * *` lines, not the hop count. |

**Recommendation if NO-GO:** do nothing. The IPv4 collapse is a side-effect of MPLS existing for real routing reasons; chasing the same cosmetics for IPv6 via a VRF migration is not worth the blast radius. Revisit only if MikroTik documents real 6PE support in a later ROS7 release.

---

## Open questions for graham (resolve before Phase 2, ideally before Phase 1)

1. Is collapsing v6 traceroute worth a multi-window backbone change, or is it "nice to have"? (Sets how hard to push past a marginal Phase 1 result.)
2. Proxmox capacity at home for 5 throwaway CHR VMs — OK to spin up?
3. edge's eventual ROS6→ROS7 upgrade — if that's on the roadmap anyway, Phase 4 should wait for it rather than attempt ROS6 6PE.