Trust Quorum Tracking

[andrewjstone]

I'm going to be fleshing this issue out over time and adding more details and links. Some of this work is already in progress.

• Protocol described in RFD 238
• TLA+ Spec of Protocol
• Protocol implementation in sans-io style
• Async peer-to-peer layer on top of sprockets and sans-io protocol
  • Networking
  • Sans-io integration
  • bootstore gossip
• Storage
  • Trust quorum persistent state
  • Bootstore data (early networking)
• Sled Agent integration
  • RSS supports both bootstore and trust quorum
  • RSS only uses trust quorum
  • APIs
    • Reconfigure
    • PrepareAndCommit
    • Commit
    • Status
    • ProxyCommit/ProxyPrepareAndCommit
    • LRTQ upgrade
  • Key Manager Support
    • New SecretRetriever for Trust Quorum
    • Key rotation in ZFS
      • Reconciler task for learning about commit/ lrtq upgrade
      • Rotation mechanism
        • Channel Programs (stop gap)
        • ZFS fix for atomic user property updates
• Nexus Support
  • New table for storing trust quorum configuration information
  • APIs for triggering membership changes
    • Add Sleds
    • Get latest membership
    • Get membership by version
    • List memberships [not a ship blocker - not even sure we want this in the external API]
    • Abort configuration
  • Background tasks for reconfiguration.
    • TQ Membership changes (prepare and commit)
    • Allocate sled subnet
    • Start sled agent
• OMDB support
  • Status
    • Latest config
    • By Version
    • List configurations
    • Per sled debug status
  • Expunge Sled
  • LRTQ upgrade
• Early networking migration - Data stored in the bootstore and interaction with it should cutover similar to LRTQ upgrade - [Not a blocker for ship]