export-diff: front-port mappings broken for NetBox >= v4.5.0 (PortTemplateMapping M2M)

[marcinpsk]

Source: PR #74 review comment 3292537728
Files: core/nb_serializer.py (~line 174), core/repo.py, core/netbox_api.py

---

Scope correction (verified May 2026)

The original concern was that _serialize_front_port() drops secondary mappings — this is correct and scoped to export-diff only. The import path is fully correct (see below).

---

What changed in NetBox

NetBox v4.5.0 (released 2026-01-06, issue #20564) introduced Advanced Port Mappings — replacing the old single ForeignKey(rear_port) + rear_port_position integer on FrontPortTemplate with a new M2M through-table (PortTemplateMapping).

Key migrations:

• 0222_port_mappings.py — creates PortTemplateMapping(front_port, front_port_position, rear_port, rear_port_position) and data-migrates all existing rows
• 0223_frontport_positions.py — removes rear_port FK and rear_port_position from FrontPortTemplate; adds positions (how many rows the port can accept)

New through-table:

PortTemplateMapping:
  front_port         → FK → FrontPortTemplate
  front_port_position  (1–1024, unique per front_port)
  rear_port          → FK → RearPortTemplate
  rear_port_position   (1–1024, unique per rear_port)

This enables true M2M assignments — e.g. an MPO breakout cassette where 24 LC front ports map to 2 × 12-position MPO rear ports. It also enables the reverse: one front port at multiple positions mapping to multiple rear ports (crossover modules).

New REST API — FrontPortTemplate no longer has rear_port/rear_port_position scalar fields; instead:

{
  "name": "Port 1", "type": {"value": "8p8c"}, "positions": 1,
  "rear_ports": [
    {"position": 1, "rear_port": 201, "rear_port_position": 1}
  ]
}

New YAML export format from NetBox v4.5.8+ (bug #21704 fixed port-mappings being missing from v4.5.0–v4.5.7 exports):

front-ports:
  - name: Port 1
    type: 8p8c          # no rear_port field
rear-ports:
  - name: Port 1
    type: 8p8c
port-mappings:          # NEW top-level section
  - front_port: Port 1
    front_port_position: 1
    rear_port: Port 1
    rear_port_position: 1

---

✅ Import path — already handled correctly

Our importer fully supports both formats and NetBox >= 4.5. Do NOT let the original issue description mislead: importing old-format YAML (with rear_port: inline on front-port entries) does create mappings on NetBox >= 4.5.

The implementation across two files:

core/repo.py — normalize_port_mappings() (lines ~114–228)

Handles both input formats and unifies them into a _mappings list attached to each front-port dict:

Old inline format:              New stanza format:
  front-ports:                    front-ports:
    - name: FP1                     - name: FP1
      type: 8p8c                      type: 8p8c
      rear_port: RP1           →    port-mappings:
      rear_port_position: 2           - front_port: FP1
                                        rear_port: RP1
                                        rear_port_position: 2

Both produce: fp_dict["_mappings"] = [{"rear_port": "RP1", "front_port_position": 1, "rear_port_position": 2}]

core/netbox_api.py — version detection + _build_link_rear_ports() (lines ~516–554, ~3435–3548)

• Detects NetBox >= 4.5 at startup: self.m2m_front_ports = True
• On NetBox >= 4.5: sends rear_ports: [{position, rear_port_id, rear_port_position}] (M2M API)
• On NetBox < 4.5: sends rear_port + rear_port_position (legacy FK)
• Multiple mappings per front port are fully supported on >= 4.5; on < 4.5, only the first mapping is sent with a warning

---

❌ Export-diff path — _serialize_front_port() is broken

The only unfixed issue is in core/nb_serializer.py used exclusively by export-diff.

Current code (~line 166):

def _serialize_front_port(record):
    ...
    mappings = getattr(record, "mappings", None) or []
    if mappings:
        m = mappings[0]              # ← BUG: drops mappings[1:]
        rear_port = getattr(m, "rear_port", None)
        if rear_port:
            result["rear_port"] = rear_port.name
        rear_pos = getattr(m, "rear_port_position", None)
        ...
    return result

Two problems:

1. Drops all mappings except the first — for any MPO cassette or crossover module with multiple mappings, the exported YAML is incomplete
2. Emits old inline format (rear_port/rear_port_position on the front-port entry) — which is valid for DTL today, but will become invalid once the DTL schema is updated for the new stanza format

The fix for export-diff is:

• Emit front-ports entries with no rear_port/rear_port_position fields
• Collect all mappings from all front ports across the device type
• Emit a top-level port-mappings: stanza with all 4 fields

Additionally, the GraphQL query used by export-diff needs to fetch mappings { rearPort { name } rearPortPosition frontPortPosition } (NetBox >= 4.5) rather than the legacy rear_port { name } scalar — needs investigation whether this is already done.

---

Required DTL schema changes

When DTL PR #3999 moves forward, the JSON schema changes needed are:

schema/components.json — front-port definition

Current:

"front-port": {
  "type": "object",
  "properties": {
    "name":              { "type": "string", "maxLength": 64 },
    "label":             { "type": "string", "maxLength": 64 },
    "type":              { "$ref": "..." },
    "color":             { "type": "string", "pattern": "^[a-f0-9]{6}$" },
    "rear_port":         { "type": "string", "maxLength": 64 },
    "rear_port_position":{ "type": "integer", "minimum": 1 },
    "description":       { "type": "string", "maxLength": 200 }
  },
  "required": ["name", "type", "rear_port"],
  "additionalProperties": false
}

Required change: Make rear_port optional (remove from required):

"front-port": {
  "type": "object",
  "properties": {
    "name":              { "type": "string", "maxLength": 64 },
    "label":             { "type": "string", "maxLength": 64 },
    "type":              { "$ref": "..." },
    "color":             { "type": "string", "pattern": "^[a-f0-9]{6}$" },
    "rear_port":         { "type": "string", "maxLength": 64 },
    "rear_port_position":{ "type": "integer", "minimum": 1 },
    "description":       { "type": "string", "maxLength": 200 }
  },
  "required": ["name", "type"],
  "additionalProperties": false
}

Note: rear_port_position without rear_port should be disallowed — JSON Schema if/then/else or dependentRequired can express this:

"dependentRequired": {
  "rear_port_position": ["rear_port"]
}

schema/devicetype.json (and moduletype.json) — add port-mappings array

Add a top-level port-mappings key alongside front-ports and rear-ports:

"port-mappings": {
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "front_port":          { "type": "string", "maxLength": 64 },
      "front_port_position": { "type": "integer", "minimum": 1, "maximum": 1024 },
      "rear_port":           { "type": "string", "maxLength": 64 },
      "rear_port_position":  { "type": "integer", "minimum": 1, "maximum": 1024 }
    },
    "required": ["front_port", "rear_port"],
    "additionalProperties": false
  }
}

front_port_position and rear_port_position default to 1 when omitted.

tests/definitions_test.py — updated validation logic

The test suite cross-reference checks need updating:

1. rear_port on front-port entry is no longer always required — if port-mappings stanza is present, inline rear_port must be absent (or must match exactly, as a transitional allowance)
2. New cross-reference for stanza: port-mappings[*].front_port must be a name in front-ports; port-mappings[*].rear_port must be a name in rear-ports
3. Uniqueness constraints must be checked in the stanza: (front_port, front_port_position) must be unique; (rear_port, rear_port_position) must be unique across all stanza entries

The existing duplicate-position check:

key = (rear_port_ref, rear_pos)
if key in rear_port_positions:
    pytest.fail(...)

needs to move from inline front-ports entries to the port-mappings stanza.

Backwards compatibility strategy for DTL

Two approaches for the transition period:

Option A — dual-format (recommended for migration): Both rear_port inline and port-mappings stanza are valid. Files with the old format continue to pass validation. Tools (like this importer) already handle both via normalize_port_mappings(). NetBox itself will accept old-format imports as long as someone builds the bridge.

Option B — hard cut: Require port-mappings stanza only, reject inline rear_port. All ~4,000+ device-type files in the library must be mass-updated. Not recommended until NetBox's own import handles both.

Our normalize_port_mappings() already implements Option A semantics — it accepts both formats and errors only on conflict between them.

---

DTL example files with complex mappings (already in library, old format)

The typical multi-position case is MPO cassettes (Commscope, etc.):

# module-types/Commscope/360DM-24LC-LS.yaml
front-ports:
  - name: '{module}:1'
    type: lc-upc
    rear_port: '{module}:MPO1'
    rear_port_position: 1
  ...
  - name: '{module}:12'
    type: lc-upc
    rear_port: '{module}:MPO1'
    rear_port_position: 12
  - name: '{module}:13'
    type: lc-upc
    rear_port: '{module}:MPO2'
    rear_port_position: 1
  ...
rear-ports:
  - name: '{module}:MPO1'
    type: mpo
    positions: 12
  - name: '{module}:MPO2'
    type: mpo
    positions: 12

In the new format this would become:

front-ports:
  - name: '{module}:1'
    type: lc-upc
  ...
rear-ports:
  - name: '{module}:MPO1'
    type: mpo
    positions: 12
  - name: '{module}:MPO2'
    type: mpo
    positions: 12
port-mappings:
  - front_port: '{module}:1'
    front_port_position: 1
    rear_port: '{module}:MPO1'
    rear_port_position: 1
  ...

---

Timeline

Date	Event
Oct 13, 2025	NetBox issue #20564 — M2M mappings proposed
Dec 9, 2025	NetBox v4.5.0 released — FrontPort.rear_port FK removed, PortTemplateMapping M2M introduced
Dec 2025 – Apr 2026	v4.5.0–v4.5.7: YAML export bug — port-mappings section silently absent from exported YAML
Mar 2026	DTL PR #3999 opened — new stanza format proposal, blocked on schema update
Apr 9, 2026	NetBox v4.5.8 — port-mappings section added to DeviceType.to_yaml() (PR #21859)
May 2026	DTL PR #3999 still open; devicetype-library schema not yet updated

---

What still needs to be done (in this repo)

1. Fix core/nb_serializer.py: _serialize_front_port() — emit all mappings, not just first; switch to port-mappings stanza output format
2. Verify GraphQL query used by export-diff fetches mappings { rearPort { name } rearPortPosition frontPortPosition } on NetBox >= 4.5 (not the legacy scalar)
3. Wait for DTL PR #3999 to merge before generating files in the new stanza format (otherwise exported files will fail DTL schema validation)

[marcinpsk]

Update (May 2026) — current export-diff output is valid DTL YAML for all real-world devices

After tracing the full GraphQL → DotDict → serializer pipeline, the current mappings[0] approach already produces valid, DTL-compatible YAML for every device type that was imported from DTL.

Why it's safe today:

The NetBox v4.5.0 migration (0222_port_mappings.py) data-migrated every old (rear_port FK, rear_port_position int) into exactly one PortTemplateMapping row per front port. So for all devices imported from the DTL library, len(mappings) == 1 on NetBox >= 4.5. mappings[0] is therefore lossless for them.

The serialized output for NetBox >= 4.5 is:

front-ports:
  - name: Port 1
    type: 8p8c
    rear_port: RP1           # from mappings[0].rear_port.name  ✓
    rear_port_position: 3    # from mappings[0].rear_port_position (omitted if == 1)

This is the old inline format that DTL schema requires today. ✓

The only lossy case — front ports with 2+ mappings — requires intentionally using the new v4.5 M2M capability after migration (e.g. manually adding a second mapping via the NetBox UI or API). No device type in the current DTL library uses this. A UserWarning is now raised when len(mappings) > 1 during export-diff (cfb2c57), naming the port and referencing this issue.

Remaining gap: NetBox < 4.5 export-diff

The GraphQL tier-2 fallback (for pre-4.5 NetBox) fetches rear_port_position scalar but omits rear_port { name }. This means the serializer cannot emit rear_port: for those records, producing invalid DTL YAML. This is a separate lower-priority bug, only affecting export-diff on old NetBox instances.

---

The main actionable items remain:

1. Wait for DTL PR #3999 to merge before generating files in the new stanza format
2. Once merged: update _serialize_front_port() to emit full port-mappings: stanza for devices with multiple mappings
3. Fix pre-4.5 export-diff to also fetch rear_port { name } in tier-2 query