chore(docs): Add details for network clean-up between tenants

docs/operations/tenant-lifecycle-cleanup.md

@@ -91,6 +91,59 @@ During the flow, NICo:
    platform changes.
 8. Validates inventory before returning the host to `Ready`.
 
+## Network Cleanup (Between Tenants)
+
+Before a released host re-enters the available pool, NICo removes all tenant
+networking from the host path and returns the host to a tenant-free state. This
+runs in the `Assigned/SwitchToAdminNetwork` and
+`Assigned/WaitingForNetworkReconfig` states, ahead of storage and platform
+sanitization.
+
+NICo drives DPU networking declaratively. The DPU agent polls NICo for the
+managed host's desired network configuration roughly every 30 seconds, applies
+it to HBN through NVUE, reconfigures the DPU-side DHCP server, and reports the
+applied configuration version and health back. On release, NICo sets
+`use_admin_network` on the managed-host configuration and increments its
+version; on its next poll the DPU re-renders with the tenant configuration
+removed. NICo does not advance the release until the DPU reports the new
+configuration applied and healthy.
+
+The following table maps each part of network cleanup to the behavior an
+operator can rely on and verify.
+
+| Concern | What NICo does |
+|---|---|
+| Remove prior tenant configuration from the DPU/host path | Sets `use_admin_network` and bumps the managed-host config version. On its next poll the DPU agent re-renders HBN/NVUE and the DPU-side DHCP server with the tenant interface configuration (VPC VRF membership, SVI, VNI, tenant prefixes, and any Network Security Group ACLs) removed. The transition does not complete until the DPU confirms the applied version and reports healthy. |
+| Detach the host from the tenant overlay/VPC | Moves every host interface out of the tenant VPC's VRF and into the admin VRF. The per-VPC VRF, its link-net/SVI addressing, and the per-VPC DPU loopback used as the EVPN next-hop are torn down; the instance's tenant IPs and the per-VPC DPU loopback are released back to their pools. |
+| Revert the host to the admin or isolated network state | With `use_admin_network` set, the DPU places all host interfaces on the admin overlay — a NICo-owned VPC/VRF separate from every tenant VPC — so the host cannot exchange traffic with any tenant instance. If NICo has no configuration for the DPU at all (for example, the host is unknown and the lookup returns `NOT_FOUND`), the DPU enters isolated mode and detaches every interface from any overlay (fail-closed). On transient errors the DPU keeps its last-known configuration. |
+| Reconfigure DHCP behavior on the DPU | The DPU-side DHCP server is reconfigured on the same apply. It serves the admin interface's DHCP parameters (server address, NTP, nameservers, MTU) instead of the tenant's, so the released host — now booting the discovery image — receives an admin/discovery lease. Because the instance is no longer assigned, NICo also serves the host's discovery cloud-init rather than the prior tenant's instance metadata. |
+| Withdraw old routes / BGP configuration | With the tenant VRF removed, the DPU stops advertising the tenant's host routes as BGP EVPN type-5 prefixes (tagged with the VPC's route-target), so they are withdrawn from the fabric and every other DPU importing that route-target drops them from its copy of the VPC VRF. The host-to-DPU BGP session in the tenant VRF, per-interface routing policy, and any routes previously leaked to the underlay are removed. |
+| Prevent unknown or released hosts from using network resources | The release flow is convergence-gated: the state machine blocks until the DPU confirms all tenant interfaces have moved onto the admin overlay (or the machine carries a health alert that prevents reuse), so a released instance cannot linger as a "ghost instance." Tenant IP, VNI, and DPU-loopback allocations return to their pools and the instance record is deleted. Force-delete tears down the same fabric state through the same APIs. |
+| Prevent cross-tenant leakage of IPs, connectivity, or metadata | Termination is gated on every fabric (Ethernet, InfiniBand, NVLink) reporting the host removed from all tenant partitions before the instance is reported deleted. Addresses and VNIs return to their pools only after teardown, and the released instance's DNS records are removed, so the next tenant cannot inherit a live address or reach the prior tenant. The DPU metadata service is reconfigured for the post-tenant phase, so a later occupant cannot read the prior tenant's metadata or identity. |
+
+Verify network cleanup from the per-host cleanup state and DPU/machine network
+status:
+
+```bash
+# Per-host cleanup state (includes the network-reconfig states)
+nico-admin-cli -a <core-api-url> managed-host show <machine-id>
+
+# DPU / machine network status (admin vs tenant, per-fabric sync state)
+nico-admin-cli -a <core-api-url> machine network status --extended
+nico-admin-cli -a <core-api-url> dpu network status --extended
+```
+
+A host stuck in `Assigned/SwitchToAdminNetwork` or
+`Assigned/WaitingForNetworkReconfig` has not yet converged onto the admin
+network; check DPU agent status, DPA status, admin-network config generation,
+and any cleanup-related health reports. See
+[Troubleshooting Stuck Cleanup](#troubleshooting-stuck-cleanup) for the
+per-state checks.
+
+For the isolation model these guarantees rest on, see
+[Default Isolation: The Admin Overlay](../manuals/vpc/vpc_network_virtualization.md#default-isolation-the-admin-overlay)
+and [Network Isolation](../manuals/network_isolation.md).
+
 ## Track Progress
 
 Use two layers of inspection:

docs/overview/lifecycle.md

@@ -58,13 +58,20 @@ NICo continuously monitors hardware health via Redfish polling and DPU agent tel
 NICo schedules UEFI and BMC firmware updates on healthy, unoccupied hosts — entirely out-of-band, without disrupting active tenants. Updates are applied against the site baseline and tracked in the per-machine firmware inventory.
 
 **Tenant transitions (sanitization)**
-When a tenant releases a host, NICo performs a full cleanup sequence before the host re-enters the available pool:
+When a tenant releases a host, NICo performs a full cleanup sequence before the host re-enters the available pool. Host and platform sanitization covers:
 1. Secure erase of all NVMe storage
 2. GPU memory and system memory wipe
 3. TPM reset
 4. Re-attestation via Measured Boot and TPM verification
 5. Firmware integrity re-validation
-6. Network isolation state cleared and re-provisioned for the next tenant
+
+Network cleanup runs in the same transition and returns the host to a tenant-free state before any reuse:
+- **Tenant configuration removed.** NICo removes the prior tenant's configuration from the DPU data path, detaches every host interface from the tenant VPC/overlay, and returns the DPU (and any DPA/Spectrum X interfaces) to the admin overlay — the fail-closed default in which the host carries no tenant traffic.
+- **Routes withdrawn and addresses reclaimed.** As the tenant VRF is torn down, its BGP EVPN routes are withdrawn from the fabric, and tenant IP, VNI, and DPU-loopback allocations are released back to their pools.
+- **DHCP and metadata re-pointed.** The DPU-side DHCP and metadata services are reconfigured for the post-tenant phase, so the released host boots the discovery image rather than the prior tenant's OS or metadata.
+- **Convergence gated.** NICo does not report the release complete until the DPU confirms it has moved off all tenant networks, which prevents a released instance from lingering on the wire or leaking connectivity, addresses, or metadata to the next tenant.
+
+For the step-by-step operator view, see [Tenant Lifecycle Cleanup](../operations/tenant-lifecycle-cleanup.md#network-cleanup-between-tenants).
 
 **Break-fix**
 NICo supports directed provisioning for break-fix workflows: targeted machine provisioning to specific hosts, machine labels for tracking machines under repair, and issue reporting APIs for integration with service management tooling.