graham/network
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
network/mikrotik-tool/network-analysis.md
Graham McIntire 6f65f1035a
routers update
2026-05-09 08:52:03 -05:00

21 KiB
Raw Permalink Blame History

vntx Network Analysis Snapshot

Generated 2026-05-08 from live router API queries, .rsc exports in this directory, inventory.yaml, and the towerops Postgres database.

Executive summary

Routers: 9 in production. 8 on RouterOS 7.21.4 long-term (just upgraded 2026-05-08, fleet-aligned). Edge alone on ROS 6.49.18 (legacy ROS6).

MPLS/LDP: Live and labeled across all 8 ROS7 routers. 7 backbone links labeled, 14 LDP adjacencies all operational=true. FastTrack-bypass rules on every MPLS-bound interface to coexist with FastTrack on customer flows.

Customers (PPPoE active sessions): ~216 active across the fleet.

  • verona 46, culleoka 40, climax 33, newhope 21, lowry 21, 982 21, 494 13, core 6.

Tower equipment: 26 Ubiquiti APs/backhauls, 17 Cambium ePMP APs, 6 AirFiber backbone radios, 2 60 GHz radios, 4+ Mikrotik mgmt switches, 2 Netonix tower switches.

Known issues:

  • Climax↔culleoka direct AF11 down (power injector unplugged at climax tower).
  • Edge IPv6 BGP to TWC AS 11427 in connect state (Charter side not configured).
  • 6 Ubiquiti Rockets still on legacy AirOS 6.3.24 (older Rocket M / M900 hardware).

Routers

Live state (queried 2026-05-08)

Site Hostname Loopback Hardware ROS Arch CPU RAM Uptime Filter rules NAT PPPoE active
verona Verona 10.254.254.101 CCR2004-16G-2S+ 7.21.4 arm64 4×1.2GHz 4 GB 1h41m 8 2 46
climax Climax 10.254.254.102 CCR2004-16G-2S+ 7.21.4 arm64 4×1.2GHz 4 GB 1h42m 10 0 33
culleoka Culleoka 10.254.254.104 CCR1009-7G-1C-1S+ 7.21.4 tile 9×1.0GHz 2 GB 1h39m 5 0 40
newhope New Hope 10.254.254.108 CCR1009-7G-1C-1S+ 7.21.4 tile 9×1.0GHz 2 GB 1h40m 7 1 21
lowry LowryCrossing 10.254.254.109 CCR1009-7G-1C-1S+ 7.21.4 tile 9×1.0GHz 2 GB 1h40m 5 0 21
982 982 10.254.254.110 CCR1009-7G-1C-1S+ 7.21.4 tile 9×1.2GHz 2 GB 1h39m 5 0 21
494 494 10.254.254.111 RB5009UG+S+ 7.21.4 arm64 4×0.35GHz 1 GB 1h42m 8 0 13
380/core Core 10.254.254.253 CCR1009 (arm64) 7.21.4 arm64* 4×? 2 GB 1h40m 13 0 6
380/edge Edge 10.254.254.254 CCR2004-1G-12S+2XS 6.49.18 arm64 4×1.7GHz ~1.8 GB 2w1d 93 11 0

* core's architecture-name came back as tile in /system/resource but its sys-resource-IRQ pattern matches arm64; treating both as effectively modern. Not load-bearing for any decision.

CPU loads sampled at the time of snapshot: edge 42% (NAT-heavy), climax 15%, 494 9%, verona 6%, core 6%, others ≤2%.

Customer load distribution

verona    ████████████████████████████████████████████  46
culleoka  ████████████████████████████████████████      40
climax    █████████████████████████████████             33
newhope   █████████████████████                         21
lowry     █████████████████████                         21
982       █████████████████████                         21
494       █████████████                                 13
core      ██████                                         6

Verona is the heaviest single tower — 46 PPPoE customers + a hotspot covering 204.110.188.0/22 + active customer NAT. Worth watching CPU during peak.

Backbone topology

                                           +──────────+
                                           │  edge    │  ROS 6.49.18
                                           │  .254    │  TWC AS 11427 (BGP)
                                           +────┬─────+
                                                │ wired
                                           +────┴─────+
                                  +────────│  core    │────────+
                                  │ AF11   │  .253    │ 60GHz  │
                                  │        +────┬─────+        │
                  AF11 (DOWN)     │             │ AF11         │
       +──────+─────────+         │             │              │
       │      │         │         │             │              │
   +───┴──+ +─┴────+ +──┴─────+ +─┴────+   +───┴────+   +────┴──+
   │verona│ │climax│ │culleoka│ │newhope│   │  982   │   │ 494*  │
   │ .101 │ │ .102 │ │  .104  │ │ .108  │   │ .110   │   │ .111  │
   +──────+ +──┬───+ +────────+ +───┬───+   +────────+   +───────+
               │AF11                │AF24
               │                    │
            +──┴───+              +─┴────+
            │ 494  │              │lowry │
            │ .111 │              │ .109 │
            +──────+              +──────+

* 494 has TWO links into the spine: directly to climax (AF11, primary shown) and there's a route-rule path through verona's ether5-494 (see climax↔494 entry). Paths are symmetric in OSPF.

Backbone link inventory

Link Type A iface B iface /29 l2mtu mpls-mtu LDP
verona↔climax AF11 verona ether3-climax-11ghz climax ether6-verona-11ghz 10.250.1.24/29 2024 1508 ✓
climax↔core AF24 climax ether4-380-airfiber24 core ether5-climax 10.250.1.88/29 2024 1508 ✓
climax↔494 AF11 climax ether5-494 494 ether2-climax 10.250.1.64/29 1580 1508 ✓
climax↔culleoka AF11 climax ether3-culleoka-11ghz culleoka ether1-climax-11ghz 10.250.1.8/29 2024 n/a DOWN
core↔culleoka AF11 core ether6-culleoka-11ghz culleoka ether6-380-11ghz 10.250.1.48/29 2024 1508 ✓
core↔newhope AF11 core ether4-newhope newhope ether2-380 10.250.1.56/29 9000 1508 ✓
core↔982 60 GHz core ether1-982-60ghz 982 ether7-380 10.250.1.32/29 9000 1508 ✓
newhope↔lowry AF24 newhope ether6-lowrycrossing lowry ether1-newhope 10.250.1.104/29 9000 1508 ✓
core↔edge wired core sfp-sfpplus1-edge-preseem + ether3-edge-direct edge SFP 204.110.191.x/30 n/a n/a n/a

Backhaul radios (towerops monitoring)

All AirFiber backbone radios are on Ubiquiti firmware v4.1.0 (current). Ethernet-side l2mtu reflects what the routers expose; the radios pass whatever fits.

Radio IP Site Model Firmware Notes
verona to climax 11ghz 10.250.1.26 verona AF11 v4.1.0
climax-380 AF24 10.250.1.93 climax AF24 v4.1.0
core to climax 10.250.1.90 380 AF24 (other end of climax-380) v4.1.0
culleoka to 380 11g 10.250.1.50 culleoka AF11 v4.1.0
new hope to core af11x 10.250.1.58 newhope AF11 v4.1.0
380↔982 60GHz pair 10.250.1.34, 10.250.1.37 982/380 AirFiber 60 (Linux 4.9.241) (linux fw)
clayton to culleoka 10.10.111.60/61 culleoka PowerBeam 5AC 300 / Rocket Prism 5AC 8.7.22 non-spine, off-tower CPE backhaul
verona to altoga 10.250.1.146 verona PowerBeam 5AC 500 8.7.14 slightly behind on AirOS 8

Access points (towerops + inventory.yaml)

Counts by manufacturer

Vendor Count
Ubiquiti AirOS 26
Cambium ePMP 17
Ubiquiti AirFiber (backbone) 6
Mikrotik (router/switch) 10
Total monitored 62

Ubiquiti AirOS firmware distribution

Firmware Count Status Devices
6.3.24 (Rocket M / M900, AirOS 6) 6 ⚠️ legacy Altoga SW, Climax South, Climax NE, Climax 900 East, Culleoka N UBNT, Clayton Estates AP
8.7.14 (AirOS 8) 1 slightly behind Verona to Altoga (PowerBeam 5AC 500)
8.7.22 (AirOS 8) 12 ✓ current All Verona Rockets, Culleoka SE/SW/sw120/AC-1/AC-2, Climax NW, clayton-culleoka pair

Cambium ePMP firmware

17 devices, none reporting firmware via SNMP — different sysObjectID or OID for the ePMP firmware version. Worth fixing in towerops monitoring (profile_device_oids / profile_sensor_oids).

Per-tower AP layout

Pulling from inventory.yaml (authoritative — generated from live discovery via the inventory subcommand):

Site APs Notes
Verona 5 (15.1, 15.2, 15.11, 15.12, 15.13) 2× 5AC Horn (W, NW) + 3× standard sectors
Altoga 3 (95.20, 95.21, 95.22) All on legacy AirOS 6 — Rocket M5
Climax 6 (31.11–13, 31.30, 31.31, 31.40) Mix: 2× ePMP, 1× Rocket Prism 5AC, 2× Rocket M5, 1× Rocket M900
Culleoka 11 (111.1, .2, .11, .12, .14, .30–35, .50, etc.) Mix: 6× Rocket Prism 5AC + 5× ePMP
982 7 (63.1–6, plus subscribers? 982-1..982-6 listed as Linux 4.9.241, model uncertain) Investigate — these may be ePMP3000 sectors mis-classified
Newhope 4 (143.11–14) All Cambium ePMP, all "Verona Networks" sysName
Lowry 4 (159.10–13) All Cambium ePMP
494 1 (175.11) Single ePMP 3000 Omni

Switches

Mikrotik mgmt switches

(From inventory.yaml):

Site Switch IP Model
Verona Verona Server Switch 10.0.101.10 MikroTik
Newhope (mgmt switch on tower) (in 100.64.x) MikroTik
380 Core uplink switch n/a MikroTik
380 "core to office" Mikrotik 10.250.2.x MikroTik

Netonix tower switches

Site Switch IP Model
Verona Verona Netonix 100.64.15.251 Netonix
Culleoka Culleoka Netonix 100.64.27.250 Netonix WS-26-400-AC

Netonix devices don't expose useful SNMP — towerops shows 0 of them. Identification comes from LLDP discovered by adjacent Mikrotiks.

OSPF / Routing

  • OSPFv2 backbone-v2 (area 0.0.0.0) on every backbone link, SHA-512 auth (auth-id=1). PTP type, BFD enabled where available. All 14 spine adjacencies state=Full.
  • OSPFv3 backbone-v3 enabled on some interfaces (mostly disabled in templates currently).
  • All instances redistribute=connected with passthrough filters (/routing filter rule chain=ospf-out rule="accept;").
  • Type-5 externals in the LSDB include the customer-facing /27s (e.g. 204.110.191.0/27 from verona, the office /29 from core, etc.).
  • Verona keeps a static default 0.0.0.0/0 via 10.250.1.30 distance=1 beneath the OSPF default — survives OSPF flap.

BGP

Edge → upstream

  • TWC / Charter (AS 11427) at 71.41.226.117:
    • IPv4: established, holding 1,034,513 prefixes (full table).
    • IPv6 (2605:6000:0:8::f:372): state=connect — Charter side not configured / SYN drops. Tracked as the IPv6 BGP issue (Charter ticket pending).
  • Filters: twc-in, twc-out (manual prefix filtering).

Edge → preseem / loopbacks

  • edge_core_preseem peer (AS 393837) configured but disabled.
  • Two Team-Cymru BOGON server peers configured but disabled.

iBGP across the fleet

  • Core has as=65530 router-id=10.254.254.253 instance=bgp-instance-1, with iBGP peer to climax (10.250.1.94/32) currently disabled=true.
  • iBGP not in active use today; OSPF carries everything.

MPLS / LDP

All 8 ROS7 routers run LDP. 14 adjacencies, all operational=true.

Topology (LDP adjacency map):

verona ──── climax ──── core ──── culleoka
              │           │
              ├── 494     ├── newhope ──── lowry
              │           │
              │           └── 982
              │
              └── (climax↔culleoka direct: configured but radio DOWN)

mpls-mtu=1508 everywhere on the labeled spine. FastTrack-bypass accept rules above each fasttrack-connection rule for every MPLS-bound interface (added during this session — without them, MPLS breaks non-loopback-sourced traffic via the FastTrack/MPLS incompatibility).

LDP advertise-filter is default (everything except Type-5 externals). This means customer-facing /27s (e.g. graham home 204.110.191.0/27) ride plain IP on the return path, not labeled.

NAT, hotspots, queues

  • Verona hotspot on interface verona, address-pool verona-cgnat, HTTPS=false, idle-timeout=5m, profile hsprof1. Covers 204.110.188.0/22. Bypass /ip hotspot ip-binding entries include graham's home 204.110.191.0/27 (added today after a discovery). Other bypass entries: 204.110.188.224/27, 10.250.1.146, one MAC binding.
  • Culleoka hotspot also active — DHCP server with culleoka-cgnat pool, 185 leases at snapshot time.
  • Newhope has 1 NAT rule, 21 PPPoE servers, 72 leases.
  • Edge has 11 NAT rules, 93 firewall filter rules — heaviest firewall stack (border router job).

Customer-facing public space

Per inventory.yaml and live /ip address:

  • 204.110.188.0/22 block — Charter-allocated; dished out to towers as /27s (verona has .224/27, etc.).
  • 204.110.191.0/27 is graham's home /27 carved out of the same /22 range, attached to verona's vlan9_sfpplus1.
  • IPv6 from ARIN: 2606:1c80::/32 advertised to TWC AS 11427 (only IPv4 is currently established with them; IPv6 BGP broken Charter-side).

CGNAT pool ownership

From inventory.yaml:

  • verona: 100.64.0.0/22 + 100.64.12.0/22 (gateway .3.254/.15.254)
  • climax: 100.64.4.0/22 (gw .7.254)
  • culleoka: 100.64.24.0/22 + 100.64.96.0/20 (gw .27.254)
  • newhope: 100.64.x.x/22 (separate)
  • lowry: 100.64.144.0/20 (gw .159.254)

Plus a long tail of /32 carve-outs assigned to specific subscribers within each tower.

Suggested improvements (for review)

These are observations from the snapshot — none are urgent unless flagged. Listed roughly in order of impact / quick-win.

Operational / monitoring

  1. towerops live monitoring last_checked_at = 2026-03-26, ~6 weeks stale. Either the agent stopped polling or the field isn't being updated. Worth checking the agent_token / agent_assignment status and SNMP polling jobs in oban_jobs.

  2. Cambium ePMP firmware not exposed via SNMP in towerops. ePMP uses a different OID set (.1.3.6.1.4.1.17713.21.x). Add a profile_device_oid row that points at the ePMP firmware OID so the 17 ePMPs report their version.

  3. 982-1..982-6 are listed as device_role=other Linux 4.9.241 with no firmware. These look mis-classified — likely ePMP3000 sectors that came back with "Linux ..." in sysDescr but should be tagged as APs and have firmware extracted. Investigate snmp_devices for these six rows.

Hardware / firmware lifecycle

  1. 6 Ubiquiti Rockets on AirOS 6.3.24 (legacy AirOS 6 line):

    • Altoga SW (Rocket M5)
    • Climax South / NE / 900 East (Rocket M5 + M900)
    • Culleoka North UBNT (Rocket M5)
    • Clayton Estates AP (Rocket M5)

    These are AirMax-AC hardware predecessors (Rocket M is the older N-only line; AirOS 6 is in maintenance mode). 5 GHz Rocket M radios are EOL'd by Ubiquiti — replace with Rocket Prism 5AC (matching what's already on the modern towers) or LiteAP AC. Climax 900 East on Rocket M900 is harder — the 900 MHz product line is dwindling but UISP P2MP alternatives exist.

  2. PowerBeam 5AC 500 at Verona→Altoga on AirOS 8.7.14 (one minor release behind 8.7.22). Low priority, but a free 8.7.22 upgrade would align with the rest of the AirOS 8 fleet.

  3. Edge on RouterOS 6.49.18. ROS6 is in extended support only; 6.49.18 is the last 6.x release. Plan a ROS7 migration:

    • New CCR2004-1G-12S+2XS supports ROS7 fine.
    • OSPF SHA-512 auth IDs need to match across the fleet on cutover.
    • BGP/IPv6 config translates with minor syntax changes.
    • Once on ROS7, MPLS LDP can extend to edge with the same fasttrack- bypass + LDP-add pattern used elsewhere.

Resilience

  1. Climax↔culleoka direct AF11 down (power injector unplugged). When restored, OSPF reconverges to the shorter path; want LDP+bypass pre-staged on climax ether3-culleoka-11ghz ↔ culleoka ether1-climax-11ghz so the labeled domain follows.

  2. Charter IPv6 BGP session in connect — pending Charter side config. The Charter ticket is open. Re-test /routing/bgp/peer twc-v6 weekly until they action it.

  3. No iBGP/route reflectors in active use. Today OSPF carries everything, which is fine at 9 routers. If the network grows past ~15-20 routers or you start carrying public BGP customer prefixes, consider standing up iBGP between the spine routers (verona, climax, core) with one of them as a route reflector. Already scaffolded on core (bgp-instance-1) but disabled.

MPLS specifics

  1. LDP advertise-filter is default. Type-5 externals (like graham's /27, customer-facing /29s) aren't labeled, which means return-path traffic to those is plain IP through the labeled domain — works but isn't what MPLS people typically expect. If you ever want full-label-everywhere (e.g., for VPN-ish features), explicitly add these prefixes to LDP advertise-filter.

  2. mpls-mtu=1508 is fine for current single-label use. If you ever add stacked labels (BGP-LU, VPN, FRR), bump to 1512+ on every spine interface.

  3. FastTrack bypass rules are on every MPLS-enabled interface today. When adding a new tower link, the must-do checklist is:

    1. Set l2mtu correctly on both sides (≥1508 for IP+label).
    2. Add accept rules above fasttrack-connection for both in-interface= and out-interface= of the new link.
    3. Add /mpls ldp instance if greenfield, plus /mpls ldp interface and /mpls interface mpls-mtu=1508 on both ends.
    4. Verify LDP operational=true and traceroute shows <MPLS:L=N,E=0>.

Configuration consistency

  1. Mixed l2mtu across the spine (1580/2024/9000) is a function of radio/wired/60GHz hardware capability, but worth standardizing for operator clarity. Documenting per-link in CLAUDE.md (just done) is the practical fix.

  2. Verona's hotspot covers 204.110.188.0/22 which spans far more than the customer realm. Anyone with a public IP in that /22 risks being intercepted by the captive portal until they're on the bypass list. Specifically: graham's home /27 was added today, but any other /27 in the /22 needs the same treatment if it's a non-hotspot user. Audit 204.110.188.0/22 allocations and pre-bypass any static-IP customers.

  3. OSPF redistribute=connected is broad. Every connected interface becomes a Type-5 LSA — including transient/CPE interfaces. Consider a tighter route-filter chain on ospf-out that limits which connected prefixes are advertised (loopbacks, mgmt subnets, /29 backbone, public /27s — exclude transient CPE).

Customer service / capacity

  1. Verona at 46 PPPoE customers + hotspot + heavy NAT is the highest-load tower. CCR2004 has plenty of headroom (CPU ≤6% at snapshot), but if growth continues, splitting some customer load to altoga (which has APs but no router — currently parented to verona) would distribute work. Adding a Mikrotik at altoga is a tower-bag upgrade; the access points are already there.

  2. Culleoka with 185 DHCP leases is the second-busiest customer site. Monitoring for DHCP pool exhaustion is worth adding (the current 100.64.96.0/20 pool fits 4096 leases so plenty of room, but watch trend).

Files

  • /Users/graham/dev/network/mikrotik-tool/*.rsc — full router exports
  • /Users/graham/dev/network/mikrotik-tool/inventory.yaml — APs, switches, backhauls, CGNAT subnets per tower
  • /Users/graham/dev/network/mikrotik-tool/subnets.yaml — per-router /20 + /22 mapping
  • /Users/graham/dev/network/mikrotik-tool/routers.yaml — connection metadata (host, port, creds source)
  • /Users/graham/dev/network/mikrotik-tool/mpls.md — MPLS deployment runbook (per-router commands + verification)
  • /Users/graham/dev/network/mikrotik-tool/ipv6.md — IPv6 allocation plan (in NetBox IPAM as of earlier work)
  • /Users/graham/dev/network/CLAUDE.md — project-wide guidelines including topology + MPLS/FastTrack caveats
  • towerops Postgres — devices, snmp_devices, snmp_interfaces, snmp_neighbors, alerts, monitoring_checks, etc.