# vntx Network Analysis Snapshot

Generated 2026-05-08 from live router API queries, `.rsc` exports in this
directory, `inventory.yaml`, and the towerops Postgres database.

## Executive summary

**Routers:** 9 in production. 8 on RouterOS 7.21.4 long-term (just upgraded
2026-05-08, fleet-aligned). Edge alone on ROS 6.49.18 (legacy ROS6).

**MPLS/LDP:** Live and labeled across all 8 ROS7 routers. 7 backbone links
labeled, 14 LDP adjacencies all `operational=true`. FastTrack-bypass rules
on every MPLS-bound interface to coexist with FastTrack on customer flows.

**Customers (PPPoE active sessions):** ~216 active across the fleet.
- verona 46, culleoka 40, climax 33, newhope 21, lowry 21, 982 21,
  494 13, core 6.

**Tower equipment:** 26 Ubiquiti APs/backhauls, 17 Cambium ePMP APs,
6 AirFiber backbone radios, 2 60 GHz radios, 4+ Mikrotik mgmt switches,
2 Netonix tower switches.

**Known issues:**
- Climax↔culleoka direct AF11 down (power injector unplugged at climax tower).
- Edge IPv6 BGP to TWC AS 11427 in `connect` state (Charter side not configured).
- 6 Ubiquiti Rockets still on legacy AirOS 6.3.24 (older Rocket M / M900 hardware).

---

## Routers

### Live state (queried 2026-05-08)

| Site | Hostname | Loopback | Hardware | ROS | Arch | CPU | RAM | Uptime | Filter rules | NAT | PPPoE active |
|--|--|--|--|--|--|--|--|--|--|--|--|
| verona | Verona | 10.254.254.101 | CCR2004-16G-2S+ | 7.21.4 | arm64 | 4×1.2GHz | 4 GB | 1h41m | 8 | 2 | **46** |
| climax | Climax | 10.254.254.102 | CCR2004-16G-2S+ | 7.21.4 | arm64 | 4×1.2GHz | 4 GB | 1h42m | 10 | 0 | 33 |
| culleoka | Culleoka | 10.254.254.104 | CCR1009-7G-1C-1S+ | 7.21.4 | tile | 9×1.0GHz | 2 GB | 1h39m | 5 | 0 | **40** |
| newhope | New Hope | 10.254.254.108 | CCR1009-7G-1C-1S+ | 7.21.4 | tile | 9×1.0GHz | 2 GB | 1h40m | 7 | 1 | 21 |
| lowry | LowryCrossing | 10.254.254.109 | CCR1009-7G-1C-1S+ | 7.21.4 | tile | 9×1.0GHz | 2 GB | 1h40m | 5 | 0 | 21 |
| 982 | 982 | 10.254.254.110 | CCR1009-7G-1C-1S+ | 7.21.4 | tile | 9×1.2GHz | 2 GB | 1h39m | 5 | 0 | 21 |
| 494 | 494 | 10.254.254.111 | RB5009UG+S+ | 7.21.4 | arm64 | 4×0.35GHz | 1 GB | 1h42m | 8 | 0 | 13 |
| 380/core | Core | 10.254.254.253 | CCR1009 (arm64) | 7.21.4 | arm64* | 4×? | 2 GB | 1h40m | 13 | 0 | 6 |
| 380/edge | Edge | 10.254.254.254 | CCR2004-1G-12S+2XS | **6.49.18** | arm64 | 4×1.7GHz | ~1.8 GB | **2w1d** | 93 | 11 | 0 |

\* core's `architecture-name` came back as `tile` in /system/resource but
its sys-resource-IRQ pattern matches arm64; treating both as effectively
modern. Not load-bearing for any decision.

CPU loads sampled at the time of snapshot: edge 42% (NAT-heavy), climax 15%,
494 9%, verona 6%, core 6%, others ≤2%.

### Customer load distribution

```
verona    ████████████████████████████████████████████  46
culleoka  ████████████████████████████████████████      40
climax    █████████████████████████████████             33
newhope   █████████████████████                         21
lowry     █████████████████████                         21
982       █████████████████████                         21
494       █████████████                                 13
core      ██████                                         6
```

Verona is the heaviest single tower — 46 PPPoE customers + a hotspot
covering `204.110.188.0/22` + active customer NAT. Worth watching CPU
during peak.

---

## Backbone topology

```
                                           +──────────+
                                           │  edge    │  ROS 6.49.18
                                           │  .254    │  TWC AS 11427 (BGP)
                                           +────┬─────+
                                                │ wired
                                           +────┴─────+
                                  +────────│  core    │────────+
                                  │ AF11   │  .253    │ 60GHz  │
                                  │        +────┬─────+        │
                  AF11 (DOWN)     │             │ AF11         │
       +──────+─────────+         │             │              │
       │      │         │         │             │              │
   +───┴──+ +─┴────+ +──┴─────+ +─┴────+   +───┴────+   +────┴──+
   │verona│ │climax│ │culleoka│ │newhope│   │  982   │   │ 494*  │
   │ .101 │ │ .102 │ │  .104  │ │ .108  │   │ .110   │   │ .111  │
   +──────+ +──┬───+ +────────+ +───┬───+   +────────+   +───────+
               │AF11                │AF24
               │                    │
            +──┴───+              +─┴────+
            │ 494  │              │lowry │
            │ .111 │              │ .109 │
            +──────+              +──────+
```

\* 494 has TWO links into the spine: directly to climax (AF11, primary
shown) and there's a route-rule path through verona's `ether5-494`
(see climax↔494 entry). Paths are symmetric in OSPF.

### Backbone link inventory

| Link | Type | A iface | B iface | /29 | l2mtu | mpls-mtu | LDP |
|--|--|--|--|--|--|--|--|
| verona↔climax | AF11 | verona `ether3-climax-11ghz` | climax `ether6-verona-11ghz` | 10.250.1.24/29 | 2024 | 1508 | ✓ |
| climax↔core | AF24 | climax `ether4-380-airfiber24` | core `ether5-climax` | 10.250.1.88/29 | 2024 | 1508 | ✓ |
| climax↔494 | AF11 | climax `ether5-494` | 494 `ether2-climax` | 10.250.1.64/29 | 1580 | 1508 | ✓ |
| climax↔culleoka | AF11 | climax `ether3-culleoka-11ghz` | culleoka `ether1-climax-11ghz` | 10.250.1.8/29 | 2024 | n/a | **DOWN** |
| core↔culleoka | AF11 | core `ether6-culleoka-11ghz` | culleoka `ether6-380-11ghz` | 10.250.1.48/29 | 2024 | 1508 | ✓ |
| core↔newhope | AF11 | core `ether4-newhope` | newhope `ether2-380` | 10.250.1.56/29 | 9000 | 1508 | ✓ |
| core↔982 | 60 GHz | core `ether1-982-60ghz` | 982 `ether7-380` | 10.250.1.32/29 | 9000 | 1508 | ✓ |
| newhope↔lowry | AF24 | newhope `ether6-lowrycrossing` | lowry `ether1-newhope` | 10.250.1.104/29 | 9000 | 1508 | ✓ |
| core↔edge | wired | core `sfp-sfpplus1-edge-preseem` + `ether3-edge-direct` | edge SFP | 204.110.191.x/30 | n/a | n/a | n/a |

### Backhaul radios (towerops monitoring)

All AirFiber backbone radios are on Ubiquiti firmware `v4.1.0` (current).
Ethernet-side l2mtu reflects what the routers expose; the radios pass
whatever fits.

| Radio | IP | Site | Model | Firmware | Notes |
|--|--|--|--|--|--|
| verona to climax 11ghz | 10.250.1.26 | verona | AF11 | v4.1.0 | |
| climax-380 AF24 | 10.250.1.93 | climax | AF24 | v4.1.0 | |
| core to climax | 10.250.1.90 | 380 | AF24 (other end of climax-380) | v4.1.0 | |
| culleoka to 380 11g | 10.250.1.50 | culleoka | AF11 | v4.1.0 | |
| new hope to core af11x | 10.250.1.58 | newhope | AF11 | v4.1.0 | |
| 380↔982 60GHz pair | 10.250.1.34, 10.250.1.37 | 982/380 | AirFiber 60 (Linux 4.9.241) | (linux fw) | |
| clayton to culleoka | 10.10.111.60/61 | culleoka | PowerBeam 5AC 300 / Rocket Prism 5AC | 8.7.22 | non-spine, off-tower CPE backhaul |
| verona to altoga | 10.250.1.146 | verona | PowerBeam 5AC 500 | **8.7.14** | slightly behind on AirOS 8 |

---

## Access points (towerops + inventory.yaml)

### Counts by manufacturer

| Vendor | Count |
|--|--|
| Ubiquiti AirOS | 26 |
| Cambium ePMP | 17 |
| Ubiquiti AirFiber (backbone) | 6 |
| Mikrotik (router/switch) | 10 |
| **Total monitored** | 62 |

### Ubiquiti AirOS firmware distribution

| Firmware | Count | Status | Devices |
|--|--|--|--|
| **6.3.24** (Rocket M / M900, AirOS 6) | 6 | ⚠️ legacy | Altoga SW, Climax South, Climax NE, Climax 900 East, Culleoka N UBNT, Clayton Estates AP |
| 8.7.14 (AirOS 8) | 1 | slightly behind | Verona to Altoga (PowerBeam 5AC 500) |
| 8.7.22 (AirOS 8) | 12 | ✓ current | All Verona Rockets, Culleoka SE/SW/sw120/AC-1/AC-2, Climax NW, clayton-culleoka pair |

### Cambium ePMP firmware

17 devices, **none reporting firmware via SNMP** — different sysObjectID
or OID for the ePMP firmware version. Worth fixing in towerops monitoring
(profile_device_oids / profile_sensor_oids).

### Per-tower AP layout

Pulling from `inventory.yaml` (authoritative — generated from live
discovery via the `inventory` subcommand):

| Site | APs | Notes |
|--|--|--|
| Verona | 5 (15.1, 15.2, 15.11, 15.12, 15.13) | 2× 5AC Horn (W, NW) + 3× standard sectors |
| Altoga | 3 (95.20, 95.21, 95.22) | All on legacy AirOS 6 — Rocket M5 |
| Climax | 6 (31.11–13, 31.30, 31.31, 31.40) | Mix: 2× ePMP, 1× Rocket Prism 5AC, 2× Rocket M5, 1× Rocket M900 |
| Culleoka | 11 (111.1, .2, .11, .12, .14, .30–35, .50, etc.) | Mix: 6× Rocket Prism 5AC + 5× ePMP |
| 982 | 7 (63.1–6, plus subscribers? `982-1`..`982-6` listed as Linux 4.9.241, model uncertain) | Investigate — these may be ePMP3000 sectors mis-classified |
| Newhope | 4 (143.11–14) | All Cambium ePMP, all "Verona Networks" sysName |
| Lowry | 4 (159.10–13) | All Cambium ePMP |
| 494 | 1 (175.11) | Single ePMP 3000 Omni |

---

## Switches

### Mikrotik mgmt switches

(From `inventory.yaml`):
| Site | Switch | IP | Model |
|--|--|--|--|
| Verona | Verona Server Switch | 10.0.101.10 | MikroTik |
| Newhope | (mgmt switch on tower) | (in `100.64.x`) | MikroTik |
| 380 | Core uplink switch | n/a | MikroTik |
| 380 | "core to office" Mikrotik | 10.250.2.x | MikroTik |

### Netonix tower switches

| Site | Switch | IP | Model |
|--|--|--|--|
| Verona | Verona Netonix | 100.64.15.251 | Netonix |
| Culleoka | Culleoka Netonix | 100.64.27.250 | Netonix WS-26-400-AC |

Netonix devices don't expose useful SNMP — towerops shows 0 of them.
Identification comes from LLDP discovered by adjacent Mikrotiks.

---

## OSPF / Routing

- **OSPFv2** `backbone-v2` (area 0.0.0.0) on every backbone link, SHA-512
  auth (`auth-id=1`). PTP type, BFD enabled where available. All 14 spine
  adjacencies `state=Full`.
- **OSPFv3** `backbone-v3` enabled on some interfaces (mostly disabled in
  templates currently).
- All instances `redistribute=connected` with passthrough filters
  (`/routing filter rule chain=ospf-out rule="accept;"`).
- **Type-5 externals** in the LSDB include the customer-facing /27s
  (e.g. `204.110.191.0/27` from verona, the office /29 from core, etc.).
- Verona keeps a static default `0.0.0.0/0 via 10.250.1.30 distance=1`
  beneath the OSPF default — survives OSPF flap.

## BGP

### Edge → upstream
- **TWC / Charter (AS 11427)** at `71.41.226.117`:
  - IPv4: established, holding **1,034,513 prefixes** (full table).
  - IPv6 (`2605:6000:0:8::f:372`): `state=connect` — **Charter side not
    configured / SYN drops**. Tracked as the IPv6 BGP issue (Charter
    ticket pending).
- Filters: `twc-in`, `twc-out` (manual prefix filtering).

### Edge → preseem / loopbacks
- `edge_core_preseem` peer (AS 393837) configured but **disabled**.
- Two Team-Cymru BOGON server peers configured but **disabled**.

### iBGP across the fleet
- Core has `as=65530 router-id=10.254.254.253 instance=bgp-instance-1`,
  with iBGP peer to climax (`10.250.1.94/32`) currently `disabled=true`.
- iBGP not in active use today; OSPF carries everything.

## MPLS / LDP

All 8 ROS7 routers run LDP. 14 adjacencies, all `operational=true`.

Topology (LDP adjacency map):
```
verona ──── climax ──── core ──── culleoka
              │           │
              ├── 494     ├── newhope ──── lowry
              │           │
              │           └── 982
              │
              └── (climax↔culleoka direct: configured but radio DOWN)
```

`mpls-mtu=1508` everywhere on the labeled spine. FastTrack-bypass `accept`
rules above each `fasttrack-connection` rule for every MPLS-bound
interface (added during this session — without them, MPLS breaks
non-loopback-sourced traffic via the FastTrack/MPLS incompatibility).

LDP advertise-filter is **default** (everything except Type-5 externals).
This means customer-facing /27s (e.g. graham home `204.110.191.0/27`)
ride **plain IP** on the return path, not labeled.

## NAT, hotspots, queues

- **Verona hotspot** on interface `verona`, address-pool `verona-cgnat`,
  HTTPS=false, idle-timeout=5m, profile `hsprof1`. Covers
  `204.110.188.0/22`. Bypass `/ip hotspot ip-binding` entries include
  graham's home `204.110.191.0/27` (added today after a discovery).
  Other bypass entries: `204.110.188.224/27`, `10.250.1.146`,
  one MAC binding.
- **Culleoka hotspot** also active — DHCP server with `culleoka-cgnat`
  pool, 185 leases at snapshot time.
- **Newhope** has 1 NAT rule, 21 PPPoE servers, 72 leases.
- **Edge** has 11 NAT rules, 93 firewall filter rules — heaviest
  firewall stack (border router job).

## Customer-facing public space

Per `inventory.yaml` and live `/ip address`:
- `204.110.188.0/22` block — Charter-allocated; dished out to towers as
  /27s (verona has `.224/27`, etc.).
- `204.110.191.0/27` is graham's home /27 carved out of the same /22
  range, attached to verona's `vlan9_sfpplus1`.
- IPv6 from ARIN: `2606:1c80::/32` advertised to TWC AS 11427 (only IPv4
  is currently established with them; IPv6 BGP broken Charter-side).

## CGNAT pool ownership

From `inventory.yaml`:
- verona: `100.64.0.0/22` + `100.64.12.0/22` (gateway `.3.254`/`.15.254`)
- climax: `100.64.4.0/22` (gw `.7.254`)
- culleoka: `100.64.24.0/22` + `100.64.96.0/20` (gw `.27.254`)
- newhope: `100.64.x.x/22` (separate)
- lowry: `100.64.144.0/20` (gw `.159.254`)

Plus a long tail of `/32` carve-outs assigned to specific subscribers
within each tower.

---

# Suggested improvements (for review)

These are observations from the snapshot — none are urgent unless
flagged. Listed roughly in order of impact / quick-win.

### Operational / monitoring

1. **towerops live monitoring last_checked_at = 2026-03-26**, ~6 weeks
   stale. Either the agent stopped polling or the field isn't being
   updated. Worth checking the agent_token / agent_assignment status and
   SNMP polling jobs in oban_jobs.

2. **Cambium ePMP firmware not exposed via SNMP** in towerops. ePMP uses
   a different OID set (`.1.3.6.1.4.1.17713.21.x`). Add a
   `profile_device_oid` row that points at the ePMP firmware OID so the
   17 ePMPs report their version.

3. **`982-1`..`982-6` are listed as device_role=other Linux 4.9.241**
   with no firmware. These look mis-classified — likely ePMP3000 sectors
   that came back with "Linux ..." in sysDescr but should be tagged as
   APs and have firmware extracted. Investigate `snmp_devices` for these
   six rows.

### Hardware / firmware lifecycle

4. **6 Ubiquiti Rockets on AirOS 6.3.24** (legacy AirOS 6 line):
   - Altoga SW (Rocket M5)
   - Climax South / NE / 900 East (Rocket M5 + M900)
   - Culleoka North UBNT (Rocket M5)
   - Clayton Estates AP (Rocket M5)

   These are AirMax-AC hardware predecessors (Rocket M is the older
   N-only line; AirOS 6 is in maintenance mode). 5 GHz Rocket M radios
   are EOL'd by Ubiquiti — replace with Rocket Prism 5AC (matching what's
   already on the modern towers) or LiteAP AC. Climax 900 East on Rocket
   M900 is harder — the 900 MHz product line is dwindling but UISP P2MP
   alternatives exist.

5. **PowerBeam 5AC 500 at Verona→Altoga on AirOS 8.7.14** (one minor
   release behind 8.7.22). Low priority, but a free 8.7.22 upgrade would
   align with the rest of the AirOS 8 fleet.

6. **Edge on RouterOS 6.49.18.** ROS6 is in extended support only;
   6.49.18 is the last 6.x release. Plan a ROS7 migration:
   - New CCR2004-1G-12S+2XS supports ROS7 fine.
   - OSPF SHA-512 auth IDs need to match across the fleet on cutover.
   - BGP/IPv6 config translates with minor syntax changes.
   - Once on ROS7, MPLS LDP can extend to edge with the same fasttrack-
     bypass + LDP-add pattern used elsewhere.

### Resilience

7. **Climax↔culleoka direct AF11 down** (power injector unplugged). When
   restored, OSPF reconverges to the shorter path; want LDP+bypass
   pre-staged on `climax ether3-culleoka-11ghz` ↔ `culleoka
   ether1-climax-11ghz` so the labeled domain follows.

8. **Charter IPv6 BGP** session in `connect` — pending Charter side
   config. The Charter ticket is open. Re-test
   `/routing/bgp/peer twc-v6` weekly until they action it.

9. **No iBGP/route reflectors in active use.** Today OSPF carries
   everything, which is fine at 9 routers. If the network grows past
   ~15-20 routers or you start carrying public BGP customer prefixes,
   consider standing up iBGP between the spine routers (verona, climax,
   core) with one of them as a route reflector. Already scaffolded on
   core (`bgp-instance-1`) but disabled.

### MPLS specifics

10. **LDP advertise-filter is default.** Type-5 externals (like graham's
    /27, customer-facing /29s) aren't labeled, which means return-path
    traffic to those is plain IP through the labeled domain — works but
    isn't what MPLS people typically expect. If you ever want
    full-label-everywhere (e.g., for VPN-ish features), explicitly add
    these prefixes to LDP advertise-filter.

11. **mpls-mtu=1508** is fine for current single-label use. If you ever
    add stacked labels (BGP-LU, VPN, FRR), bump to 1512+ on every spine
    interface.

12. **FastTrack bypass rules** are on every MPLS-enabled interface today.
    When adding a new tower link, the must-do checklist is:
    1. Set l2mtu correctly on both sides (≥1508 for IP+label).
    2. Add `accept` rules above `fasttrack-connection` for both
       `in-interface=` and `out-interface=` of the new link.
    3. Add `/mpls ldp` instance if greenfield, plus `/mpls ldp interface`
       and `/mpls interface mpls-mtu=1508` on both ends.
    4. Verify LDP `operational=true` and traceroute shows `<MPLS:L=N,E=0>`.

### Configuration consistency

13. **Mixed l2mtu across the spine** (1580/2024/9000) is a function of
    radio/wired/60GHz hardware capability, but worth standardizing for
    operator clarity. Documenting per-link in CLAUDE.md (just done) is
    the practical fix.

14. **Verona's hotspot covers `204.110.188.0/22`** which spans far more
    than the customer realm. Anyone with a public IP in that /22 risks
    being intercepted by the captive portal until they're on the bypass
    list. Specifically: graham's home /27 was added today, but any other
    /27 in the /22 needs the same treatment if it's a non-hotspot user.
    Audit `204.110.188.0/22` allocations and pre-bypass any
    static-IP customers.

15. **OSPF `redistribute=connected` is broad.** Every connected interface
    becomes a Type-5 LSA — including transient/CPE interfaces. Consider
    a tighter route-filter chain on `ospf-out` that limits which
    connected prefixes are advertised (loopbacks, mgmt subnets, /29
    backbone, public /27s — exclude transient CPE).

### Customer service / capacity

16. **Verona at 46 PPPoE customers + hotspot + heavy NAT** is the
    highest-load tower. CCR2004 has plenty of headroom (CPU ≤6% at
    snapshot), but if growth continues, splitting some customer load to
    altoga (which has APs but no router — currently parented to verona)
    would distribute work. Adding a Mikrotik at altoga is a tower-bag
    upgrade; the access points are already there.

17. **Culleoka with 185 DHCP leases** is the second-busiest customer
    site. Monitoring for DHCP pool exhaustion is worth adding (the
    current `100.64.96.0/20` pool fits 4096 leases so plenty of room,
    but watch trend).

---

# Files

- `/Users/graham/dev/network/mikrotik-tool/*.rsc` — full router exports
- `/Users/graham/dev/network/mikrotik-tool/inventory.yaml` — APs, switches, backhauls, CGNAT subnets per tower
- `/Users/graham/dev/network/mikrotik-tool/subnets.yaml` — per-router /20 + /22 mapping
- `/Users/graham/dev/network/mikrotik-tool/routers.yaml` — connection metadata (host, port, creds source)
- `/Users/graham/dev/network/mikrotik-tool/mpls.md` — MPLS deployment runbook (per-router commands + verification)
- `/Users/graham/dev/network/mikrotik-tool/ipv6.md` — IPv6 allocation plan (in NetBox IPAM as of earlier work)
- `/Users/graham/dev/network/CLAUDE.md` — project-wide guidelines including topology + MPLS/FastTrack caveats
- towerops Postgres — `devices`, `snmp_devices`, `snmp_interfaces`,
  `snmp_neighbors`, `alerts`, `monitoring_checks`, etc.
