test: add comprehensive tests for MIB.SnmpTokenizer
Added complete test coverage for MIB.SnmpTokenizer module (32.93% -> target higher):
**Tokenization Tests:**
- Empty input handling
- Simple MIB header (DEFINITIONS, BEGIN, END)
- Reserved word recognition (30+ SMI keywords tested)
- Identifier tokenization (lowercase atoms, uppercase variables)
- Number tokenization (integers, negative numbers, leading zeros, large numbers)
- String tokenization with quotes
- Special characters (::=, {}, (), commas, semicolons, dots)
- OID notation
- Complete OBJECT-TYPE definitions
- Access modifiers (read-only, read-write, not-accessible, read-create)
- SMI types (INTEGER, OCTET STRING, Counter32, Gauge32, TimeTicks, IpAddress)
- Hyphenated keywords (OBJECT-TYPE, MAX-ACCESS, MODULE-IDENTITY, etc.)
**Format and Syntax:**
- Whitespace handling (spaces, tabs, newlines)
- Comment handling (-- comments)
- Multiline input
- Line number tracking
- Token order preservation
- Complex MIB structures (IMPORTS, MODULE-IDENTITY, etc.)
- SIZE constraints
- SEQUENCE OF notation
**Error Handling:**
- format_error/1 for illegal characters
- Unterminated string errors
- Unterminated quote errors
- Unknown error formatting
**Edge Cases:**
- Very long identifiers (up to 200 chars)
- Many tokens (100+ identifiers)
- Deeply nested braces
- Mixed case keywords (case-sensitive)
- Empty strings
- Consecutive operators
- Tabs and mixed whitespace
- Numbers with leading zeros
- Very large numbers
**Reserved Word Coverage:**
- All common SMI/SMIv2 reserved words tested
- Distinction between reserved words and similar identifiers
- Case-sensitive recognition
40 new tests added
Test count: 1,264 -> 1,304 (all passing)
mayor
07977bc16f