Wire proposal

[adracus]

This proposes the wire API for configuring switches.

[hardikdr]

Thanks a lot for the proposal @adracus
I have left a few comments inline.

[hardikdr]

Suggested change

	We currently provision or switches with central templates and lots
	We currently provision our switches with central templates and lots

[hardikdr]

Curious whether anchoring the motivation to the current ironcore-underlay topology is intentional scoping, or whether this is meant to be a more general-purpose network automation API that happens to solve the ironcore use case first.

If the latter, it might be worth framing the current state as an initial use case rather than the baseline, so future use cases feel like natural extensions rather than deviations from the original design.

[hardikdr]

IMO we dont need Server and ServerInterface as a dedicated custom-resource within network-automation, unless we have a Spec in mind that a controller can reconcile. While Server can be very useful for visualisers, or potential validation of the underlay, it might as well be a static input rather than a dedicated CRD.

[hardikdr]

The same argument against Server goes for Link as well. I would recommend avoiding it unless we have a further Spec in mind to reconcile against Link.

In sonic-operator, we used LLDP-based discovery, which helped identify the neighbours serving the same purpose inside the Status of SwitchInterfaces, which could then be used for validation against desired topology, or visualisers.

[hardikdr]

While separating the Device from Switch is a reasonable idea(thanks a lot for that), I find the complete immutability of the Switch inspired more by a very specific use case. Complete swap of the Switch for every change would force users to either not make any incremental changes or accept the potential network disruption with every change, which would be nice to avoid.

In case there is an assumption that every Server is connected with 2 Leaves for redudancy, that is too topology-specific. Even in that case, a naturally degraded Switch would prevent the rollout of the other switch, as it could potentially affect a subset of the Servers.

[hardikdr]

I would vouch for a more hybrid setup, where certain features, such as NTP, DNS, DHCP Relay, Loopback IPs, etc., could remain inline while operational features that can potentially have an independent lifecycle, have dedicated CRDs, such as BGP, etc.

[peanball]

I would argue that some things such as BGP, addressing, etc. could actually be expressed as a "formula" and derived based on the desired topology.

If we know we want a Clos topology (as an example), then we know how we'd want hops to be in a systematic way. Then there is no need to explicitly, manually, configuring them.

[hardikdr]

That sounds interesting. Do you have an example of how the formula could possibly look?

[hardikdr]

I would be critical about a dedicated Host CRD here, this would mean the network-automation to actively reconcile the Host and achieve the desired state. This leads network-automation to cross the structural boundary, I would keep the configuration of the Servers/Hosts separate and have them managed by the MetalAPI.

Instead, some higher-level controller could later prepare the necessary network-related ignition-blob for Servers which then gets applied to the Servers via MetalAPI instead of network-automation handling it.

[hardikdr]

While reviewing this proposal, I ended up drafting an alternative design that builds on some ideas here - particularly the Device/Switch separation - while taking a different stance on immutability, object scope, and extensibility. Would be great to get everyone's thoughts on it as well: https://gist.github.com/hardikdr/5def28b4cb645d374d03f3d7fe65ab35

[peanball]

Some comments.

[peanball]

to add to @hardikdr's point, I think reducing reconfiguration is a process goal, not something to be enforced technically. The single object containing the desired state does not contradict that. Different Versions of the same resource are equally valid.

[hardikdr]

I second this; it's much better to use a process to reduce reconfiguration than to enforce it via immutability.

[peanball]

If this is a proposal for the whole WireAPI, i.e. the "network hardware bootstrapping and configurability", this shouldn't be a delta to whatever we are currently doing, but a desire for configuring a network that is useful for IronCore in general, and specific setups such as ours in particular.

The desire is to modify the configuration of switches as little as possible as they are pretty much "dumb but fast" network elements. Pretty much everything can talk to everything via the switches, i.e. we don't do explicit VLAN segmentation based on user workloads but only for setting the initial topology.

Any segmentation happens on the higher level via SDN, be it dpservice, an IPv6 only CNI or any other means. The WireAPI defines the fundamental network, on top of which other layers build.

The main goal of the WireAPI is then to express the desired topology and provide enough "knobs" to bootstrap the network from pretty much raw uninitialized hardware with minimal manual effort. There is still some manual involvement necessary to get the raw hardware to the point that the WireAPI can talk to it.

There is no desire to continuously "converge" on a dynamically computed state (scheduling, etc.), but there is the desire for zero-downtime maintenance, which means that switches need to be taken out of rotation without interruption. There may be a degradation in high availability during such maintenance activities, when one of the n (for us usually n=2) links goes down for a brief period while one of the switches is maintained.

The topology must be kept in mind for such maintenance activities, as we don't want to turn off all redundant connections to a specific target.

I fully agree with the last paragraph, as is. How we get there (described later in the proposal) is up for debate ;)

[peanball]

Maybe we can highlight that we don't want to have a full general-purpose-every-knob-exposed-via-gNMI-or-SNMP-configurability, but automation of a stable foundation network topology that is mostly stable with the exception of maintenance.

Automation focusses on initial setup and configuration (bootstrapping) to ensure that the desired topology is and remains in place.

[peanball]

This point is a bit confusing or rather too brief.

Imperative relates to when something is set up, but not individual settings. We don't issue a command to set up some VLAN.

Declarative describes what to set holistically, as a big picture. You can see the complete desired state at once.

Together, they mean that we set what we'd like the resource to look like. The imperative leans on the immutable resource, which is explicitly (re)created and then triggers the (re)configuration.

Maybe to consider whether we want a clear overview and traceability of what the system should look like and how it looks now, vs. manual operator control of when things are set.

[peanball]

I agree that this is more a "state" than a CR. The link is a physical property of someone plugging a cable in a data center. We can't will a link into existence by creating a CR, only discover/document the fact that the link exists.

[peanball]

I would argue that some things such as BGP, addressing, etc. could actually be expressed as a "formula" and derived based on the desired topology.

If we know we want a Clos topology (as an example), then we know how we'd want hops to be in a systematic way. Then there is no need to explicitly, manually, configuring them.

[peanball]

This could equally be done with taints and resource versioning. It is still logically the same switch in the same place with the same name, but it is e.g. running an OS update. When it comes back it's still the same switch.

[peanball]

is this something we want to set explicitly or something that could be "derived" from the desire to have DHCP relay?

[peanball]

If we set up the desired topology by hand anyway, we could equally create Switch resources directly. Considering that the existence of a Server is handled by MetalAPI, and the network link configuration of a host is not controlled by WireAPI but instead observed and reported as state, we could also forego the concept of Device as overarching "catch-all" thing.

The things currently represented by Device are wildly different and might not benefit much by a common supertype.