# CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Project Overview A small fleet of CLIs for documenting, auditing, and operating the vntx WISP — a multi-tower MikroTik backbone with cnMaestro and UISP managing the radios/CPE behind it. Source of truth for network documentation is NetBox at `https://netbox.vntx.net/` (API token in env var `NETBOX_KEY`); source of truth for the routers is whatever the live `.rsc` exports say. ## Repo Layout The repo is a collection of independent CLIs that each operate on the same physical fleet but talk to different control planes. Each subproject has its own README/CLAUDE.md with details; only repo-wide context lives here. | Subdir | Lang | Talks to | Purpose | |--|--|--|--| | `mikrotik-tool/` | Go | MikroTik API-SSL + SSH | Fan-out to every router in `routers.yaml`: `list`, `export` (saves `.rsc` per router), `inventory` (rewrites `inventory.yaml` from live data + SNMP), `api`/`schema` (ad-hoc API introspection on one router), `script` (push a `.rsc` file via SSH and stream output). Authoritative `inventory.yaml`, `subnets.yaml`, `radios.yaml`, `ipv6.md`, and `mpls.md` live here. | | `mikrotik-tool-rs/` | Rust | MikroTik API-SSL + system `ssh` | Newer rewrite of a subset of `mikrotik-tool` (`list`, `export`, `run `) with `-r name1,name2` filter to scope a fan-out command. Has its own `routers.yaml`. See `mikrotik-tool-rs/CLAUDE.md`. | | `uisp/` | Go (`main.go`) + Python (`uisp.py`) | UISP API at `uisp.vntx.net` | List/approve/upgrade UISP-managed devices. Auth via `UISP_KEY` env var. UNKNOWN-model devices are filtered by default (use `--include-unknown`). | | `cnmaestro/` | Python (uv) | cnMaestro Cloud OAuth2 | Cleanup offline devices and bulk-upgrade firmware. `uv sync` + `.env` with client id/secret. See `cnmaestro/README.md`. | `routers.yaml` at the repo root is the config consumed by `mikrotik-tool/` (Go); `mikrotik-tool-rs/` has its own copy. Keep them aligned when adding/removing routers. `poll_snmp_site.sh` is a one-off helper for SNMP-polling a list of IPs at a site (`./poll_snmp_site.sh ...`). The `*.rsc` files at the repo root are stale exports — fresh ones come out of `mikrotik-tool` or `mikrotik-tool-rs`. ## Common Commands ```bash # mikrotik-tool (Go) cd mikrotik-tool && go build . ./mikrotik-tool list ./mikrotik-tool export # writes .rsc per router into cwd ./mikrotik-tool api verona /ip/route/print ./mikrotik-tool schema verona /routing/bgp/connection ./mikrotik-tool script 494 494-v6-apply.rsc ./mikrotik-tool inventory # re-renders inventory.yaml in place go test ./... # inventory_test.go # mikrotik-tool-rs (Rust) cd mikrotik-tool-rs && cargo build --release cargo run -- list cargo run -- -r verona,982 export # filter to a subset before the verb cargo run -- run /system identity print cargo test # full suite cargo test encode_length # single test by name substring cargo clippy --all-targets && cargo fmt # uisp (Go binary, also a Python script) cd uisp && go build . ./uisp list ./uisp approve ./uisp upgrade # python equivalent: python3 uisp.py list # cnmaestro (Python, uv) cd cnmaestro && uv sync uv run cnmaestro cleanup --dry-run uv run cnmaestro upgrade --product ePMP,PMP --dry-run uv run pytest uv run pytest tests/test_devices.py::test_specific_thing # single test ``` `UISP_KEY` and `NETBOX_KEY` need to be exported in the shell. `cnmaestro` reads `cnmaestro/.env`. ## NetBox - URL: `https://netbox.vntx.net/`, API base `https://netbox.vntx.net/api/` - Auth: `Authorization: Token $NETBOX_KEY` - Convention when creating sites/devices: site slugs are lowercase-hyphenated, status `active`; routers are Manufacturer=MikroTik / Role=Router / primary IP = loopback `/32` (`10.254.254.x/32`). - Prefix roles in use: Infrastructure, Customer, Management, Loopback. ## MikroTik Router Access Read-only fan-out (used by all the above tools): API-SSL on port 8729, user `grahamro` / password in `CLAUDE.md` history. The committed `routers.yaml` actually uses `graham` (not `grahamro`) because `/export` requires the `ftp` policy in addition to `read,api`, which the read-only group lacks. New routers: ensure the user's group has `read,api,ftp` (or extend the read group: `/user group set read add-policy=ftp`). Every router's loopback is `10.254.254.x/32`; see the **Fleet Topology** table below for the full mapping. ## Network Topology Patterns ### Access Point Placement - Access points are always placed in the top /24 of the management subnet for each tower - Example: For management subnet 10.10.16.0/20, APs are in 10.10.31.0/24 (the last /24 in that range) - Formula: For subnet X.Y.Z.0/20, APs are in X.Y.(Z+15).0/24 ### Ubiquiti MAC Prefixes Common MAC address prefixes for Ubiquiti devices: - 00:04:56 (legacy) - 00:27:22 (legacy) - 04:18:D6 - 24:A4:3C - 68:72:51 - 80:2A:A8 - F0:9F:C2 - FC:EC:DA ## MPLS / LDP ### FastTrack is incompatible with MPLS on RouterOS 7 FastTrack bypasses the IP forwarding path that MPLS push/pop runs on, so any flow that gets fasttracked on a router whose path uses an MPLS-enabled interface can break — packets either hit the wrong interface or never get labeled, which presents as black-holing for specific source subnets that weren't fasttracked before. Symptoms: pings/SSH/TCP from one source IP work but the same destination is unreachable from another source on the same router; loopback-sourced traffic works but vlan-interface-sourced doesn't. **Fix:** before each `action=fasttrack-connection` rule in `chain=forward`, add `accept` rules that match the MPLS-bound interface(s) so those flows never enter the fasttrack path: ``` /ip firewall filter add chain=forward action=accept in-interface= comment="bypass fasttrack for MPLS spine (in)" place-before= add chain=forward action=accept out-interface= comment="bypass fasttrack for MPLS spine (out)" place-before= ``` Customer→internet flows continue to fasttrack normally; only flows traversing the MPLS spine bypass it. ### LDP doesn't label OSPF Type-5 externals by default Prefixes redistributed via `redistribute=connected` (e.g., a /27 customer WAN handoff like 204.110.191.0/27) appear as Type-5 external LSAs and don't get LDP label bindings. Forward path to a labeled destination still works, but the return path is plain IP. If you need labeled bidirectional reach for a redistributed prefix, configure an LDP advertise-filter that explicitly includes it. ### MPLS-MTU is the labeled-frame cap, not the IP-payload cap `mpls-mtu=1500` caps the *labeled* frame at 1500 bytes, which means an inner IP payload is limited to 1496 bytes — so 1500-byte DF customer traffic gets icmp-frag-needed. Use `mpls-mtu=1508` for a 1500-byte IP payload + 4-byte label, with 4 bytes of headroom for one more stacked label. The AF11/AF24 radio l2mtu is 2024, so 1508 fits comfortably. ### Fleet-wide MPLS topology LDP runs IPv4-only across every backbone link in the network. Every backbone port has `mpls-mtu=1508` set explicitly and a fasttrack-bypass pair (in/out) above the `fasttrack-connection` rule on both endpoints. Documented in `mikrotik-tool/mpls.md`. ``` verona ──AF11── climax ──AF24── core ──AF11── culleoka │ │ │ │ 5GHz │ AF11 │ AF11 (DOWN: power injector unplugged) │ │ │ 494 newhope ──AF24── lowry │ │ 60 GHz │ 982 ``` Wait — that diagram's links are: climax↔494 (5 GHz airMAX, **not** AF11 — see below), core↔newhope (AF11), core↔982 (60 GHz), newhope↔lowry (AF24). The climax↔culleoka direct AF11 is currently down at the radio (physical issue), so culleoka traffic transits via core. ## Fleet Topology ### Routers and loopbacks All ROS7 routers run RouterOS 7.21.4 long-term (post-2026-05-08 fleet upgrade). Edge runs ROS 6.49.18 (legacy, no MPLS, ignore for the spine). | Router | Loopback (10.254.254.x) | Hardware | Site name | |--|--|--|--| | verona | .101 | CCR2004-16G-2S+ (arm64) | verona | | climax | .102 | CCR2004-16G-2S+ (arm64) | climax | | culleoka | .104 | CCR1009-7G-1C-1S+ (tile) | culleoka | | newhope | .108 | CCR1009-7G-1C-1S+ (tile) | newhope | | lowry | .109 | (tile) | lowrycrossing | | 982 | .110 | (CCR, tile) | 982 | | 494 | .111 | (CCR, tile) | 494 | | core | .253 | (CCR, arm64) at 380 | core/380 | | edge | .254 | (legacy, ROS 6.49.18) | edge | Tile-arch boxes can run MPLS but **not** ZeroTier (no .npk for tile). ### Backbone links Every link below has IPv4 LDP enabled at both ends, `mpls-mtu=1508`, and fasttrack-bypass rules in both directions on both routers. | Link | Type | A-side iface | B-side iface | /29 subnet | l2mtu | |--|--|--|--|--|--| | verona↔climax | AF11 | verona `ether3-climax-11ghz` | climax `ether6-verona-11ghz` | 10.250.1.24/29 | 2024 | | climax↔core | AF24 | climax `ether4-380-airfiber24` | core `ether5-climax` | 10.250.1.88/29 | 2024 | | climax↔494 | 5 GHz airMAX | climax `ether5-494` | 494 `ether2-climax` | 10.250.1.64/29 | 1580 | | climax↔culleoka | AF11 | climax `ether3-culleoka-11ghz` | culleoka `ether1-climax-11ghz` | 10.250.1.8/29 | 2024 (link DOWN) | | core↔culleoka | AF11 | core `ether6-culleoka-11ghz` | culleoka `ether6-380-11ghz` | 10.250.1.48/29 | 2024 | | core↔newhope | AF11 | core `ether4-newhope` | newhope `ether2-380` | 10.250.1.56/29 | 9000 | | core↔982 | 60 GHz | core `ether1-982-60ghz` | 982 `ether7-380` | 10.250.1.32/29 | 9000 | | newhope↔lowry | AF24 | newhope `ether6-lowrycrossing` | lowry `ether1-newhope` | 10.250.1.104/29 | 9000 | | core↔edge | wired | core `sfp-sfpplus1-edge-preseem` + `ether3-edge-direct` | edge ports | 204.110.191.x | n/a | l2mtu mismatches across the fleet are intentional per platform: AF11 base ports default to 2024 on CCR2004 / 1580 on smaller CCRs; jumbo-capable links (60 GHz, AF24-with-jumbo, fiber) go to 9000. **Always raise both sides symmetrically when changing l2mtu** — single-side raises usually work because Ethernet receivers accept anything ≤ their cap, but symmetric is the rule. ### climax↔494 backhaul is 5 GHz airMAX, not AF11 Old docs called this hop AF11; it is actually a Ubiquiti airMAX AC PtP pair (SSID `vntx_pr_494`, WPA2, 40 MHz wide): AP **"Climax to 494"** (PBE-5AC-500) at `10.250.1.69` on the climax side, station **"494 to Climax"** (PBE-5AC-400) at `10.250.1.66` on the 494 side. Both answer SNMP v1 (`kdyyJrT0Mm`, airMAX MIB `1.3.6.1.4.1.41112.1.4`); added to `radios.yaml` 2026-07-17. The station's mgmt IP (and the whole 494 tower) is only reachable **across this RF link** — when changing the channel, first make sure the station's frequency scan list covers the target, then change the AP (climax side), which stays reachable for rollback either way. 2026-07-17 flap incident: link ran on **5260 MHz (DFS)**; associations dropped for 40–90 s every 45–130 min (OSPF 40 s dead-interval timeouts on both routers; ethernet ports never dropped, radios never rebooted — association-uptime SNMP counters matched each OSPF flap). Local 5 GHz survey: climax APs on 5335/5545/5575 (airMAX) + 5750/5775 (ePMP), 494 ePMP omni on 5800 → U-NII-1 (5170–5250) was empty at both towers and is non-DFS, so the fix was moving the PtP there. ### IGP / routing - OSPFv2 area `backbone-v2` (id 0.0.0.0) on all spine links, SHA-512 auth with `auth-id=1` and a shared key. PTP type. **BFD is off on every wireless backbone interface (AF11/AF24/60GHz)** — the global timers (`/routing/bfd/configuration` = 200ms×5 = 1s detection) are too aggressive for backhaul RF: a single >1s burst tore down OSPF on climax↔494 every ~15–40 min until `use-bfd=false` was applied to both ends 2026-05-09. Wired fiber links may keep BFD if desired. If sub-second failover on a wireless link is genuinely needed, also loosen the BFD timers (e.g. min-rx/min-tx=500ms, multiplier=3) — do not flip `use-bfd=true` alone. - OSPFv3 area `backbone-v3` for IPv6 (some interfaces only). - All instances `redistribute=connected` with passthrough filters (`/routing filter rule chain=ospf-out rule="accept;"`). - Verona has a static default to `10.250.1.30` (climax) backing up the OSPF default — keep this; bouncing OSPF on verona doesn't blackhole it. - Distance-1 static routes also exist on climax for `204.110.191.0/27` so the home /27 has guaranteed return path even if OSPF redistribution hiccups. ### Management subnets per tower `10.10.x.0/20` per site, top /24 reserved for APs (see Access Point Placement section). Authoritative mapping is in `mikrotik-tool/inventory.yaml`. Quick reference: - verona: 10.10.0.0/20 - altoga (behind verona, no router): 10.10.16.0/20 - climax: 10.10.48.0/20 - core/380: 10.10.64.0/20 - culleoka: 10.10.96.0/20 - 982: 10.10.128.0/20 - newhope: 10.10.144.0/20 - 494: 10.10.160.0/20 - lowry: 10.10.80.0/20 CGNAT pools: `100.64.x.x/22` per tower (see `inventory.yaml` / `subnets.yaml`). ### graham's home network gotcha graham's home connects to verona via `vlan9_sfpplus1` carrying `204.110.191.0/27` (home router at `.1`, verona at `.30`). This /27 is a subnet of the verona hotspot's covered range (`204.110.188.0/22`). After any verona reboot, ensure `/ip hotspot ip-binding` has an entry: `address=204.110.191.0/27 type=bypassed comment="graham home /27"` — without it, hotspot drops all `204.110.191.x` traffic in `hs-unauth-to` chain with `icmp-host-prohibited`. Symptom is "I can reach verona but nothing past it" from the home network. ### graham's home network — multi-WAN failover (recursive check-gateway) Home (RB5009, `10.0.19.254`, **not** in `routers.yaml` — use a temp config with the same default creds to reach it via mikrotik-tool) fails over between TMO (distance 1), VNTX/verona (distance 2), and Starlink (distance 3, disabled) using the standard ROS recursive pattern, applied 2026-07-17 via `mikrotik-tool/home-recursive-failover.rsc`: - Probe pins: `4.2.2.1/32→192.168.12.1%ether6-tmobile`, `4.2.2.2/32→204.110.191.30%ether5-vntx-static`, `4.2.2.3/32→192.168.1.1%ether7-starlink`, all `scope=10`, **no check-gateway on the pins** — they stay active whenever the interface has link, gluing probes to their WAN regardless of route state. - Defaults: `gateway=4.2.2.x target-scope=11 check-gateway=ping` — the check pings the probe IP through the pin; ~20s to go inactive, first success to return. `tmo table default` (eweka policy routing) has the same check so it falls back to main when TMO upstream dies. **Do not reintroduce netwatch enable/disable failover scripts.** The old design (netwatch 2s/1s single-packet → `/ip route disable`) deadlocked: `disabled=yes` persists across reboots, and the probe was only tied to the VNTX path while its pin was active — so the verona hotspot-binding loss (verona pingable, transit dead) left the route disabled forever, and probes leaked out other WANs giving false "up" (TMO netwatch read up with the TMO interface physically down). Same lesson as the fleet BFD incident: 1-2s detection on AF RF with >1s bursts causes false failover; 10–20s check-gateway damping is intentional. ### IPv6 plan Per-tower /44s + central server LAN at `2606:1c80::/64` on edge. Full allocation plan in `mikrotik-tool/ipv6.md`. NetBox has these as IPAM prefixes. ### IPv6 forward firewall — must accept the whole /32 Every tower router's `/ipv6 firewall filter chain=forward` needs `action=accept src-address=2606:1c80::/32` before the `default deny`. A per-tower `/44` rule is **wrong** — customer traffic transits other routers to reach `edge`, so each transit hop blackholes other towers' `/44`s. Classic symptom: ICMPv6 ping works (there's an `accept protocol=icmpv6` rule) but TCP/HTTP times out at `connect` (forwarded TCP hits `default deny`). The original `ipv6.md` template used `in-interface-list=customer`, but that list was never created on any router so the rule matched nothing — fixed fleet-wide 2026-05-14, and the now-redundant per-tower `/44` and `internal-to-internal` forward rules were removed at the same time. `edge` has a permissive forward chain (no default-deny) so it didn't need the rule. Resolver/DNS roll-out (`2606:1c80::240`/`::250` in `/ip dns` + the `v6-dns` DHCPv6 option) is also fleet-wide as of 2026-05-14. ### graham's home network — IPv6 Home (`RB5009`, identity `graham`, `10.0.19.254`) takes a **`/56`** from verona via DHCPv6-PD on `ether5-vntx-static` (verona's `verona-wired-pd` dhcp-server on `vlan9_sfpplus1`, drawing from `verona-cust-pd-1`). Currently `2606:1c80:1001:e00::/56`. Key config on home: - `/ipv6 dhcp-client` on `ether5-vntx-static`: `request=prefix`, `pool-name=home-pd-vntx`, **`pool-prefix-length=64`** (must be 64, not 56, so the pool yields /64s — changing this only takes effect on a client disable/enable, not live), `default-route-tables=v6-verona`. - `/ipv6 address` with `from-pool=home-pd-vntx address=::1 advertise=yes` on `bridge` (→ `:e00::1/64`) and `ether3-servers` (→ `:e01::1/64`). Remaining 254 /64s of the /56 stay in the pool for internal use. - Egress policy: **two** `/routing rule`s, order matters — (1) `src-address=2606:1c80::/32 dst-address=2606:1c80:1001:e00::/56 action=lookup-only-in-table table=main` so vntx-sourced traffic *destined to the home LANs* resolves via main's connected routes, then (2) `src-address=2606:1c80::/32 action=lookup-only-in-table table=v6-verona` for everything else. The verona /56 is single-homed, so traffic sourced from it must egress via verona only (T-Mobile/Starlink would BCP38-drop it). No v6 WAN failover for the internal /56, by design. The PD client's default lands in the `v6-verona` table, not `main`. **Without rule (1)** the broad rule also catches transit packets (echo replies, ICMPv6 time-exceeded) sourced from internal routers and destined back to the home LAN, and hairpins them to verona in a loop — symptom: home reaches the v6 internet fine but cannot ping any internal router and internal hops show `*` in `traceroute6`. Rule (1)'s `dst-address` is tied to the current delegated /56 — update it if the PD lease ever changes. - Stateless DHCPv6 servers (`home-bridge-stateless`, `home-servers-stateless`) carry the `v6-dns` option (Cloudflare). - Stale `tunnerbroker` `/ipv6 pool` (HE.net `2001:470:ba50::/48`) is unused — leave or remove. - The `mikrotik-tool script` SSH path can't handle bare ROS verbs like `release`/`disable`/`enable` or `:delay` (flattenRSC only knows add/set/remove/print/get) — use `set [find] disabled=yes/no` instead, and sequence delays from the shell side. ## Claude Assistant Guidelines - Any time Claude learns something new, automatically add it to CLAUDE.md ## Development Best Practices - When making scripts, keep them as generic and reusable as possible