graham/network
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
network/mikrotik-tool/network-analysis.md
Graham McIntire 6f65f1035a
routers update
2026-05-09 08:52:03 -05:00

436 lines
21 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# vntx Network Analysis Snapshot
Generated 2026-05-08 from live router API queries, `.rsc` exports in this
directory, `inventory.yaml`, and the towerops Postgres database.
## Executive summary
**Routers:** 9 in production. 8 on RouterOS 7.21.4 long-term (just upgraded
2026-05-08, fleet-aligned). Edge alone on ROS 6.49.18 (legacy ROS6).
**MPLS/LDP:** Live and labeled across all 8 ROS7 routers. 7 backbone links
labeled, 14 LDP adjacencies all `operational=true`. FastTrack-bypass rules
on every MPLS-bound interface to coexist with FastTrack on customer flows.
**Customers (PPPoE active sessions):** ~216 active across the fleet.
- verona 46, culleoka 40, climax 33, newhope 21, lowry 21, 982 21,
494 13, core 6.
**Tower equipment:** 26 Ubiquiti APs/backhauls, 17 Cambium ePMP APs,
6 AirFiber backbone radios, 2 60 GHz radios, 4+ Mikrotik mgmt switches,
2 Netonix tower switches.
**Known issues:**
- Climax↔culleoka direct AF11 down (power injector unplugged at climax tower).
- Edge IPv6 BGP to TWC AS 11427 in `connect` state (Charter side not configured).
- 6 Ubiquiti Rockets still on legacy AirOS 6.3.24 (older Rocket M / M900 hardware).
---
## Routers
### Live state (queried 2026-05-08)
| Site | Hostname | Loopback | Hardware | ROS | Arch | CPU | RAM | Uptime | Filter rules | NAT | PPPoE active |
|--|--|--|--|--|--|--|--|--|--|--|--|
| verona | Verona | 10.254.254.101 | CCR2004-16G-2S+ | 7.21.4 | arm64 | 4×1.2GHz | 4 GB | 1h41m | 8 | 2 | **46** |
| climax | Climax | 10.254.254.102 | CCR2004-16G-2S+ | 7.21.4 | arm64 | 4×1.2GHz | 4 GB | 1h42m | 10 | 0 | 33 |
| culleoka | Culleoka | 10.254.254.104 | CCR1009-7G-1C-1S+ | 7.21.4 | tile | 9×1.0GHz | 2 GB | 1h39m | 5 | 0 | **40** |
| newhope | New Hope | 10.254.254.108 | CCR1009-7G-1C-1S+ | 7.21.4 | tile | 9×1.0GHz | 2 GB | 1h40m | 7 | 1 | 21 |
| lowry | LowryCrossing | 10.254.254.109 | CCR1009-7G-1C-1S+ | 7.21.4 | tile | 9×1.0GHz | 2 GB | 1h40m | 5 | 0 | 21 |
| 982 | 982 | 10.254.254.110 | CCR1009-7G-1C-1S+ | 7.21.4 | tile | 9×1.2GHz | 2 GB | 1h39m | 5 | 0 | 21 |
| 494 | 494 | 10.254.254.111 | RB5009UG+S+ | 7.21.4 | arm64 | 4×0.35GHz | 1 GB | 1h42m | 8 | 0 | 13 |
| 380/core | Core | 10.254.254.253 | CCR1009 (arm64) | 7.21.4 | arm64* | 4×? | 2 GB | 1h40m | 13 | 0 | 6 |
| 380/edge | Edge | 10.254.254.254 | CCR2004-1G-12S+2XS | **6.49.18** | arm64 | 4×1.7GHz | ~1.8 GB | **2w1d** | 93 | 11 | 0 |
\* core's `architecture-name` came back as `tile` in /system/resource but
its sys-resource-IRQ pattern matches arm64; treating both as effectively
modern. Not load-bearing for any decision.
CPU loads sampled at the time of snapshot: edge 42% (NAT-heavy), climax 15%,
494 9%, verona 6%, core 6%, others ≤2%.
### Customer load distribution
```
verona ████████████████████████████████████████████ 46
culleoka ████████████████████████████████████████ 40
climax █████████████████████████████████ 33
newhope █████████████████████ 21
lowry █████████████████████ 21
982 █████████████████████ 21
494 █████████████ 13
core ██████ 6
```
Verona is the heaviest single tower — 46 PPPoE customers + a hotspot
covering `204.110.188.0/22` + active customer NAT. Worth watching CPU
during peak.
---
## Backbone topology
```
+──────────+
│ edge │ ROS 6.49.18
│ .254 │ TWC AS 11427 (BGP)
+────┬─────+
│ wired
+────┴─────+
+────────│ core │────────+
│ AF11 │ .253 │ 60GHz │
│ +────┬─────+ │
AF11 (DOWN) │ │ AF11 │
+──────+─────────+ │ │ │
│ │ │ │ │ │
+───┴──+ +─┴────+ +──┴─────+ +─┴────+ +───┴────+ +────┴──+
│verona│ │climax│ │culleoka│ │newhope│ │ 982 │ │ 494* │
│ .101 │ │ .102 │ │ .104 │ │ .108 │ │ .110 │ │ .111 │
+──────+ +──┬───+ +────────+ +───┬───+ +────────+ +───────+
│AF11 │AF24
│ │
+──┴───+ +─┴────+
│ 494 │ │lowry │
│ .111 │ │ .109 │
+──────+ +──────+
```
\* 494 has TWO links into the spine: directly to climax (AF11, primary
shown) and there's a route-rule path through verona's `ether5-494`
(see climax↔494 entry). Paths are symmetric in OSPF.
### Backbone link inventory
| Link | Type | A iface | B iface | /29 | l2mtu | mpls-mtu | LDP |
|--|--|--|--|--|--|--|--|
| verona↔climax | AF11 | verona `ether3-climax-11ghz` | climax `ether6-verona-11ghz` | 10.250.1.24/29 | 2024 | 1508 | ✓ |
| climax↔core | AF24 | climax `ether4-380-airfiber24` | core `ether5-climax` | 10.250.1.88/29 | 2024 | 1508 | ✓ |
| climax↔494 | AF11 | climax `ether5-494` | 494 `ether2-climax` | 10.250.1.64/29 | 1580 | 1508 | ✓ |
| climax↔culleoka | AF11 | climax `ether3-culleoka-11ghz` | culleoka `ether1-climax-11ghz` | 10.250.1.8/29 | 2024 | n/a | **DOWN** |
| core↔culleoka | AF11 | core `ether6-culleoka-11ghz` | culleoka `ether6-380-11ghz` | 10.250.1.48/29 | 2024 | 1508 | ✓ |
| core↔newhope | AF11 | core `ether4-newhope` | newhope `ether2-380` | 10.250.1.56/29 | 9000 | 1508 | ✓ |
| core↔982 | 60 GHz | core `ether1-982-60ghz` | 982 `ether7-380` | 10.250.1.32/29 | 9000 | 1508 | ✓ |
| newhope↔lowry | AF24 | newhope `ether6-lowrycrossing` | lowry `ether1-newhope` | 10.250.1.104/29 | 9000 | 1508 | ✓ |
| core↔edge | wired | core `sfp-sfpplus1-edge-preseem` + `ether3-edge-direct` | edge SFP | 204.110.191.x/30 | n/a | n/a | n/a |
### Backhaul radios (towerops monitoring)
All AirFiber backbone radios are on Ubiquiti firmware `v4.1.0` (current).
Ethernet-side l2mtu reflects what the routers expose; the radios pass
whatever fits.
| Radio | IP | Site | Model | Firmware | Notes |
|--|--|--|--|--|--|
| verona to climax 11ghz | 10.250.1.26 | verona | AF11 | v4.1.0 | |
| climax-380 AF24 | 10.250.1.93 | climax | AF24 | v4.1.0 | |
| core to climax | 10.250.1.90 | 380 | AF24 (other end of climax-380) | v4.1.0 | |
| culleoka to 380 11g | 10.250.1.50 | culleoka | AF11 | v4.1.0 | |
| new hope to core af11x | 10.250.1.58 | newhope | AF11 | v4.1.0 | |
| 380↔982 60GHz pair | 10.250.1.34, 10.250.1.37 | 982/380 | AirFiber 60 (Linux 4.9.241) | (linux fw) | |
| clayton to culleoka | 10.10.111.60/61 | culleoka | PowerBeam 5AC 300 / Rocket Prism 5AC | 8.7.22 | non-spine, off-tower CPE backhaul |
| verona to altoga | 10.250.1.146 | verona | PowerBeam 5AC 500 | **8.7.14** | slightly behind on AirOS 8 |
---
## Access points (towerops + inventory.yaml)
### Counts by manufacturer
| Vendor | Count |
|--|--|
| Ubiquiti AirOS | 26 |
| Cambium ePMP | 17 |
| Ubiquiti AirFiber (backbone) | 6 |
| Mikrotik (router/switch) | 10 |
| **Total monitored** | 62 |
### Ubiquiti AirOS firmware distribution
| Firmware | Count | Status | Devices |
|--|--|--|--|
| **6.3.24** (Rocket M / M900, AirOS 6) | 6 | ⚠️ legacy | Altoga SW, Climax South, Climax NE, Climax 900 East, Culleoka N UBNT, Clayton Estates AP |
| 8.7.14 (AirOS 8) | 1 | slightly behind | Verona to Altoga (PowerBeam 5AC 500) |
| 8.7.22 (AirOS 8) | 12 | ✓ current | All Verona Rockets, Culleoka SE/SW/sw120/AC-1/AC-2, Climax NW, clayton-culleoka pair |
### Cambium ePMP firmware
17 devices, **none reporting firmware via SNMP** — different sysObjectID
or OID for the ePMP firmware version. Worth fixing in towerops monitoring
(profile_device_oids / profile_sensor_oids).
### Per-tower AP layout
Pulling from `inventory.yaml` (authoritative — generated from live
discovery via the `inventory` subcommand):
| Site | APs | Notes |
|--|--|--|
| Verona | 5 (15.1, 15.2, 15.11, 15.12, 15.13) | 2× 5AC Horn (W, NW) + 3× standard sectors |
| Altoga | 3 (95.20, 95.21, 95.22) | All on legacy AirOS 6 — Rocket M5 |
| Climax | 6 (31.11–13, 31.30, 31.31, 31.40) | Mix: 2× ePMP, 1× Rocket Prism 5AC, 2× Rocket M5, 1× Rocket M900 |
| Culleoka | 11 (111.1, .2, .11, .12, .14, .30–35, .50, etc.) | Mix: 6× Rocket Prism 5AC + 5× ePMP |
| 982 | 7 (63.1–6, plus subscribers? `982-1`..`982-6` listed as Linux 4.9.241, model uncertain) | Investigate — these may be ePMP3000 sectors mis-classified |
| Newhope | 4 (143.11–14) | All Cambium ePMP, all "Verona Networks" sysName |
| Lowry | 4 (159.10–13) | All Cambium ePMP |
| 494 | 1 (175.11) | Single ePMP 3000 Omni |
---
## Switches
### Mikrotik mgmt switches
(From `inventory.yaml`):
| Site | Switch | IP | Model |
|--|--|--|--|
| Verona | Verona Server Switch | 10.0.101.10 | MikroTik |
| Newhope | (mgmt switch on tower) | (in `100.64.x`) | MikroTik |
| 380 | Core uplink switch | n/a | MikroTik |
| 380 | "core to office" Mikrotik | 10.250.2.x | MikroTik |
### Netonix tower switches
| Site | Switch | IP | Model |
|--|--|--|--|
| Verona | Verona Netonix | 100.64.15.251 | Netonix |
| Culleoka | Culleoka Netonix | 100.64.27.250 | Netonix WS-26-400-AC |
Netonix devices don't expose useful SNMP — towerops shows 0 of them.
Identification comes from LLDP discovered by adjacent Mikrotiks.
---
## OSPF / Routing
- **OSPFv2** `backbone-v2` (area 0.0.0.0) on every backbone link, SHA-512
auth (`auth-id=1`). PTP type, BFD enabled where available. All 14 spine
adjacencies `state=Full`.
- **OSPFv3** `backbone-v3` enabled on some interfaces (mostly disabled in
templates currently).
- All instances `redistribute=connected` with passthrough filters
(`/routing filter rule chain=ospf-out rule="accept;"`).
- **Type-5 externals** in the LSDB include the customer-facing /27s
(e.g. `204.110.191.0/27` from verona, the office /29 from core, etc.).
- Verona keeps a static default `0.0.0.0/0 via 10.250.1.30 distance=1`
beneath the OSPF default — survives OSPF flap.
## BGP
### Edge → upstream
- **TWC / Charter (AS 11427)** at `71.41.226.117`:
- IPv4: established, holding **1,034,513 prefixes** (full table).
- IPv6 (`2605:6000:0:8::f:372`): `state=connect` — **Charter side not
configured / SYN drops**. Tracked as the IPv6 BGP issue (Charter
ticket pending).
- Filters: `twc-in`, `twc-out` (manual prefix filtering).
### Edge → preseem / loopbacks
- `edge_core_preseem` peer (AS 393837) configured but **disabled**.
- Two Team-Cymru BOGON server peers configured but **disabled**.
### iBGP across the fleet
- Core has `as=65530 router-id=10.254.254.253 instance=bgp-instance-1`,
with iBGP peer to climax (`10.250.1.94/32`) currently `disabled=true`.
- iBGP not in active use today; OSPF carries everything.
## MPLS / LDP
All 8 ROS7 routers run LDP. 14 adjacencies, all `operational=true`.
Topology (LDP adjacency map):
```
verona ──── climax ──── core ──── culleoka
│ │
├── 494 ├── newhope ──── lowry
│ │
│ └── 982
│
└── (climax↔culleoka direct: configured but radio DOWN)
```
`mpls-mtu=1508` everywhere on the labeled spine. FastTrack-bypass `accept`
rules above each `fasttrack-connection` rule for every MPLS-bound
interface (added during this session — without them, MPLS breaks
non-loopback-sourced traffic via the FastTrack/MPLS incompatibility).
LDP advertise-filter is **default** (everything except Type-5 externals).
This means customer-facing /27s (e.g. graham home `204.110.191.0/27`)
ride **plain IP** on the return path, not labeled.
## NAT, hotspots, queues
- **Verona hotspot** on interface `verona`, address-pool `verona-cgnat`,
HTTPS=false, idle-timeout=5m, profile `hsprof1`. Covers
`204.110.188.0/22`. Bypass `/ip hotspot ip-binding` entries include
graham's home `204.110.191.0/27` (added today after a discovery).
Other bypass entries: `204.110.188.224/27`, `10.250.1.146`,
one MAC binding.
- **Culleoka hotspot** also active — DHCP server with `culleoka-cgnat`
pool, 185 leases at snapshot time.
- **Newhope** has 1 NAT rule, 21 PPPoE servers, 72 leases.
- **Edge** has 11 NAT rules, 93 firewall filter rules — heaviest
firewall stack (border router job).
## Customer-facing public space
Per `inventory.yaml` and live `/ip address`:
- `204.110.188.0/22` block — Charter-allocated; dished out to towers as
/27s (verona has `.224/27`, etc.).
- `204.110.191.0/27` is graham's home /27 carved out of the same /22
range, attached to verona's `vlan9_sfpplus1`.
- IPv6 from ARIN: `2606:1c80::/32` advertised to TWC AS 11427 (only IPv4
is currently established with them; IPv6 BGP broken Charter-side).
## CGNAT pool ownership
From `inventory.yaml`:
- verona: `100.64.0.0/22` + `100.64.12.0/22` (gateway `.3.254`/`.15.254`)
- climax: `100.64.4.0/22` (gw `.7.254`)
- culleoka: `100.64.24.0/22` + `100.64.96.0/20` (gw `.27.254`)
- newhope: `100.64.x.x/22` (separate)
- lowry: `100.64.144.0/20` (gw `.159.254`)
Plus a long tail of `/32` carve-outs assigned to specific subscribers
within each tower.
---
# Suggested improvements (for review)
These are observations from the snapshot — none are urgent unless
flagged. Listed roughly in order of impact / quick-win.
### Operational / monitoring
1. **towerops live monitoring last_checked_at = 2026-03-26**, ~6 weeks
stale. Either the agent stopped polling or the field isn't being
updated. Worth checking the agent_token / agent_assignment status and
SNMP polling jobs in oban_jobs.
2. **Cambium ePMP firmware not exposed via SNMP** in towerops. ePMP uses
a different OID set (`.1.3.6.1.4.1.17713.21.x`). Add a
`profile_device_oid` row that points at the ePMP firmware OID so the
17 ePMPs report their version.
3. **`982-1`..`982-6` are listed as device_role=other Linux 4.9.241**
with no firmware. These look mis-classified — likely ePMP3000 sectors
that came back with "Linux ..." in sysDescr but should be tagged as
APs and have firmware extracted. Investigate `snmp_devices` for these
six rows.
### Hardware / firmware lifecycle
4. **6 Ubiquiti Rockets on AirOS 6.3.24** (legacy AirOS 6 line):
- Altoga SW (Rocket M5)
- Climax South / NE / 900 East (Rocket M5 + M900)
- Culleoka North UBNT (Rocket M5)
- Clayton Estates AP (Rocket M5)
These are AirMax-AC hardware predecessors (Rocket M is the older
N-only line; AirOS 6 is in maintenance mode). 5 GHz Rocket M radios
are EOL'd by Ubiquiti — replace with Rocket Prism 5AC (matching what's
already on the modern towers) or LiteAP AC. Climax 900 East on Rocket
M900 is harder — the 900 MHz product line is dwindling but UISP P2MP
alternatives exist.
5. **PowerBeam 5AC 500 at Verona→Altoga on AirOS 8.7.14** (one minor
release behind 8.7.22). Low priority, but a free 8.7.22 upgrade would
align with the rest of the AirOS 8 fleet.
6. **Edge on RouterOS 6.49.18.** ROS6 is in extended support only;
6.49.18 is the last 6.x release. Plan a ROS7 migration:
- New CCR2004-1G-12S+2XS supports ROS7 fine.
- OSPF SHA-512 auth IDs need to match across the fleet on cutover.
- BGP/IPv6 config translates with minor syntax changes.
- Once on ROS7, MPLS LDP can extend to edge with the same fasttrack-
bypass + LDP-add pattern used elsewhere.
### Resilience
7. **Climax↔culleoka direct AF11 down** (power injector unplugged). When
restored, OSPF reconverges to the shorter path; want LDP+bypass
pre-staged on `climax ether3-culleoka-11ghz` ↔ `culleoka
ether1-climax-11ghz` so the labeled domain follows.
8. **Charter IPv6 BGP** session in `connect` — pending Charter side
config. The Charter ticket is open. Re-test
`/routing/bgp/peer twc-v6` weekly until they action it.
9. **No iBGP/route reflectors in active use.** Today OSPF carries
everything, which is fine at 9 routers. If the network grows past
~15-20 routers or you start carrying public BGP customer prefixes,
consider standing up iBGP between the spine routers (verona, climax,
core) with one of them as a route reflector. Already scaffolded on
core (`bgp-instance-1`) but disabled.
### MPLS specifics
10. **LDP advertise-filter is default.** Type-5 externals (like graham's
/27, customer-facing /29s) aren't labeled, which means return-path
traffic to those is plain IP through the labeled domain — works but
isn't what MPLS people typically expect. If you ever want
full-label-everywhere (e.g., for VPN-ish features), explicitly add
these prefixes to LDP advertise-filter.
11. **mpls-mtu=1508** is fine for current single-label use. If you ever
add stacked labels (BGP-LU, VPN, FRR), bump to 1512+ on every spine
interface.
12. **FastTrack bypass rules** are on every MPLS-enabled interface today.
When adding a new tower link, the must-do checklist is:
1. Set l2mtu correctly on both sides (≥1508 for IP+label).
2. Add `accept` rules above `fasttrack-connection` for both
`in-interface=` and `out-interface=` of the new link.
3. Add `/mpls ldp` instance if greenfield, plus `/mpls ldp interface`
and `/mpls interface mpls-mtu=1508` on both ends.
4. Verify LDP `operational=true` and traceroute shows `<MPLS:L=N,E=0>`.
### Configuration consistency
13. **Mixed l2mtu across the spine** (1580/2024/9000) is a function of
radio/wired/60GHz hardware capability, but worth standardizing for
operator clarity. Documenting per-link in CLAUDE.md (just done) is
the practical fix.
14. **Verona's hotspot covers `204.110.188.0/22`** which spans far more
than the customer realm. Anyone with a public IP in that /22 risks
being intercepted by the captive portal until they're on the bypass
list. Specifically: graham's home /27 was added today, but any other
/27 in the /22 needs the same treatment if it's a non-hotspot user.
Audit `204.110.188.0/22` allocations and pre-bypass any
static-IP customers.
15. **OSPF `redistribute=connected` is broad.** Every connected interface
becomes a Type-5 LSA — including transient/CPE interfaces. Consider
a tighter route-filter chain on `ospf-out` that limits which
connected prefixes are advertised (loopbacks, mgmt subnets, /29
backbone, public /27s — exclude transient CPE).
### Customer service / capacity
16. **Verona at 46 PPPoE customers + hotspot + heavy NAT** is the
highest-load tower. CCR2004 has plenty of headroom (CPU ≤6% at
snapshot), but if growth continues, splitting some customer load to
altoga (which has APs but no router — currently parented to verona)
would distribute work. Adding a Mikrotik at altoga is a tower-bag
upgrade; the access points are already there.
17. **Culleoka with 185 DHCP leases** is the second-busiest customer
site. Monitoring for DHCP pool exhaustion is worth adding (the
current `100.64.96.0/20` pool fits 4096 leases so plenty of room,
but watch trend).
---
# Files
- `/Users/graham/dev/network/mikrotik-tool/*.rsc` — full router exports
- `/Users/graham/dev/network/mikrotik-tool/inventory.yaml` — APs, switches, backhauls, CGNAT subnets per tower
- `/Users/graham/dev/network/mikrotik-tool/subnets.yaml` — per-router /20 + /22 mapping
- `/Users/graham/dev/network/mikrotik-tool/routers.yaml` — connection metadata (host, port, creds source)
- `/Users/graham/dev/network/mikrotik-tool/mpls.md` — MPLS deployment runbook (per-router commands + verification)
- `/Users/graham/dev/network/mikrotik-tool/ipv6.md` — IPv6 allocation plan (in NetBox IPAM as of earlier work)
- `/Users/graham/dev/network/CLAUDE.md` — project-wide guidelines including topology + MPLS/FastTrack caveats
- towerops Postgres — `devices`, `snmp_devices`, `snmp_interfaces`,
`snmp_neighbors`, `alerts`, `monitoring_checks`, etc.
Reference in a new issue View git blame Copy permalink