How to recover from non-manageable 100 series DCPMM

[DangoNetwork]

I have G2S modules that are never able to initialize in BIOS or in the system. Is there a way to flash firmware on those devices to recover them?

[sscargal]

@DangoNetwork What do you mean by "G2S"? Gen2 (Optane 200)? The title mentions 1st Generation Optane 100 modules.

Regarding your issue of "Unmanageable" modules:

1. What server OEM, Model, and BIOS version do you have?
2. What CPU SKU do you have?
3. What is the Optane model number?
4. Confirm your server and BIOS support the Optane PMem Generation modules. Systems are not backward- or future-compatible, meaning you cannot put Optane 100 series in an Optane 200 series server or vice versa. This extends to Optane 300 also. The Optane generation is locked to the CPU SKU and BIOS.

I wrote PMemchk to help collect information and identify known/common issues. It may help you.

Assuming you have the correct hardware support matrix, you can try to factory reset the Optane Modules. This will depend on your server and BIOS, but here are some general guidelines.

Reset in the BIOS

• Look for an option in the BIOS, usually within the "Security" section, to reset the modules

Reset in UEFI

If your BIOS delivers the ipmctl utility within UEFI, boot the system to the UEFI shell, then:

uefi> ipmctl delete -goal
uefi> ipmctl delete -pcd
uefi> reset -c // Clears CMOS and other options you may have set

Reset in Linux

Within Linux, ensure you have the latest ipmctl and ndctl utilities installed. These utilities support all generations - 100, 200, and 300, and are backward compatible.

# Delete namespaces, if any.
sudo ndctl destroy-namespace all -f

# Delete the Goal (MemoryMode or AppDirect)
sudo ipmctl delete -goal

# Clear Platform Configuration Data / PCD
sudo ipmctl delete -pcd

[DangoNetwork]

Those are Engineering Samples with part numberNMA1XBD512G2S.
So far, nothing can control those dimms. It fails initialization during post. There are no exiting goal configured on the system since I use another set totally fine in memory mode.
Here is how they show up in BIOS.

add 1 more pic from ipmctl

[sscargal]

Thanks @DangoNetwork

I use another set totally fine in memory mode.

Can you provide more clarity, please? Did you use another Optane PMem set in this system, or do you have an identical/similar setup working elsewhere?

What are the differences between the working system and this one?

• Is the Part Number/SKU the same?

• Are the PMem Firmware and BIOS versions the same?

• Have these modules ever worked? (Where did you get them?)

• What CPU SKU (Model number) do you have?

If you have another system that works, do these modules work in that system?

Here are some additional commands you could run that may help explain why the modules are unmanageable.

• ipmctl show -a -dimm
  • Even for unmanageable DIMMs, you should see basic fields like DimmHandle, DimmUID, SubsystemDeviceID, FWVersion, and BootStatus.
• ipmctl show -a -topology
  • If the DIMMs do not appear here, the BIOS has not populated the NFIT table for them, which means the platform is not recognizing the modules at all — a strong indicator of a BIOS/platform incompatibility.
• ipmctl show -a -dimm -display BootStatus
  • The Boot Status Register (BSR) may be readable even on some unmanageable DIMMs and provides the most direct indication of why a DIMM failed to initialize.

[DangoNetwork]

Thanks @DangoNetwork

I use another set totally fine in memory mode.

Can you provide more clarity, please? Did you use another Optane PMem set in this system, or do you have an identical/similar setup working elsewhere?
Using a set with NMAXBD512GQS part number was totally fine.

What are the differences between the working system and this one?
I only have this system

• Is the Part Number/SKU the same?
  No. The ones that work are NMAXBD512GQS. My understanding is that these are Qualifying simples?
• Are the PMem Firmware and BIOS versions the same?
  NMAXBD512GQS is running the latest 1.2.0.5446. The non-functional NMAXBD512G2S is unknown. They never worked.
• Have these modules ever worked? (Where did you get them?)
  Never in my system. (from the 2nd-hand market in China)
• What CPU SKU (Model number) do you have?
  5215L

If you have another system that works, do these modules work in that system? No. I only have this X11SPI-TF to work with.

See the result below

Here are some additional commands you could run that may help explain why the modules are unmanageable.

• ipmctl show -a -dimm
  ---DimmID=0x0001---
  Capacity=0.000 GiB
  LockState=Unknown
  SVNDowngrade=Unknown
  SecureErasePolicy=Unknown
  S3ResumeOptIn=Unknown
  FwActivateOptIn=Unknown
  HealthState=Unmanageable
  HealthStateReason=None
  FWVersion=N/A
  FWAPIVersion=N/A
  FWActiveAPIVersion=N/A
  InterfaceFormatCode=0x0301 (Non-Energy Backed Byte Addressable)
  ManageabilityState=Unmanageable
  PopulationViolation=No
  PhysicalID=0x0025
  DimmHandle=0x0001
  DimmUID=8089-a2-1750-00000463
  SocketID=0x0000
  MemControllerID=0x0000
  ChannelID=0x0000
  ChannelPos=1
  MemoryType=Logical Non-Volatile Device
  Manufacturer=Intel
  VendorID=0x8089
  DeviceID=0x5141
  RevisionID=0x0000
  SubsystemVendorID=0x8089
  SubsystemDeviceID=0x097a
  SubsystemRevisionID=0x0012
  DeviceLocator=DIMMA2
  ManufacturingInfoValid=1
  ManufacturingLocation=0xa2
  ManufacturingDate=17-50
  SerialNumber=0x00000463
  PartNumber=
  BankLabel=P0_Node0_Channel0_Dimm1
  DataWidth=64 b
  TotalWidth=72 b
  Speed=2666 MT/s
  FormFactor=DIMM
  ManufacturerID=N/A
  ControllerRevisionID=N/A
  MemoryCapacity=N/A
  AppDirectCapacity=N/A
  UnconfiguredCapacity=N/A
  InaccessibleCapacity=N/A
  ReservedCapacity=N/A
  PackageSparingCapable=N/A
  PackageSparingEnabled=N/A
  PackageSparesAvailable=N/A
  IsNew=N/A
  AveragePowerReportingTimeConstant=N/A
  ViralPolicy=N/A
  ViralState=N/A
  PeakPowerBudget=N/A
  AvgPowerLimit=N/A
  AvgPowerTimeConstant=N/A
  MemoryBandwidthBoostFeature=N/A
  MemoryBandwidthBoostMaxPowerLimit=N/A
  MemoryBandwidthBoostAveragePowerTimeConstant=N/A
  MaxAveragePowerLimit=N/A
  MaxMemoryBandwidthBoostMaxPowerLimit=N/A
  MaxMemoryBandwidthBoostAveragePowerTimeConstant=N/A
  MemoryBandwidthBoostAveragePowerTimeConstantStep=N/A
  MaxAveragePowerReportingTimeConstant=N/A
  AveragePowerReportingTimeConstantStep=N/A
  LatchedLastShutdownStatus=N/A
  UnlatchedLastShutdownStatus=N/A
  MaxMediaTemperature=N/A
  MaxControllerTemperature=N/A
  ThermalThrottleLossPercent=N/A
  LastShutdownTime=N/A
  ModesSupported=N/A
  SecurityCapabilities=N/A
  MasterPassphraseEnabled=N/A
  ConfigurationStatus=N/A
  SKUViolation=N/A
  ARSStatus=N/A
  OverwriteStatus=N/A
  AitDramEnabled=N/A
  BootStatus=N/A
  ErrorInjectionEnabled=N/A
  MediaTemperatureInjectionEnabled=N/A
  SoftwareTriggersEnabled=N/A
  SoftwareTriggersEnabledDetails=N/A
  PoisonErrorInjectionsCounter=N/A
  PoisonErrorClearCounter=N/A
  MediaTemperatureInjectionsCounter=N/A
  SoftwareTriggersCounter=N/A
  BootStatusRegister=N/A
  AveragePower=N/A
  Average12vPower=N/A
  Average1_2vPower=N/A
  ExtendedAdrEnabled=N/A
  PpcExtendedAdrEnabled=N/A
  LatchSystemShutdownState=N/A
  PreviousPowerCycleLatchSystemShutdownState=N/A
  MixedSKU=N/A
  FIPSModeStatus=N/A
  ---DimmID=0x0101---
  Capacity=0.000 GiB
  LockState=Unknown
  SVNDowngrade=Unknown
  SecureErasePolicy=Unknown
  S3ResumeOptIn=Unknown
  FwActivateOptIn=Unknown
  HealthState=Unmanageable
  HealthStateReason=None
  FWVersion=N/A
  FWAPIVersion=N/A
  FWActiveAPIVersion=N/A
  InterfaceFormatCode=0x0301 (Non-Energy Backed Byte Addressable)
  ManageabilityState=Unmanageable
  PopulationViolation=No
  PhysicalID=0x002c
  DimmHandle=0x0101
  DimmUID=8089-a2-1750-000003d2
  SocketID=0x0000
  MemControllerID=0x0001
  ChannelID=0x0000
  ChannelPos=1
  MemoryType=Logical Non-Volatile Device
  Manufacturer=Intel
  VendorID=0x8089
  DeviceID=0x5141
  RevisionID=0x0000
  SubsystemVendorID=0x8089
  SubsystemDeviceID=0x097a
  SubsystemRevisionID=0x0012
  DeviceLocator=DIMMD2
  ManufacturingInfoValid=1
  ManufacturingLocation=0xa2
  ManufacturingDate=17-50
  SerialNumber=0x000003d2
  PartNumber=
  BankLabel=P0_Node1_Channel0_Dimm1
  DataWidth=64 b
  TotalWidth=72 b
  Speed=2666 MT/s
  FormFactor=DIMM
  ManufacturerID=N/A
  ControllerRevisionID=N/A
  MemoryCapacity=N/A
  AppDirectCapacity=N/A
  UnconfiguredCapacity=N/A
  InaccessibleCapacity=N/A
  ReservedCapacity=N/A
  PackageSparingCapable=N/A
  PackageSparingEnabled=N/A
  PackageSparesAvailable=N/A
  IsNew=N/A
  AveragePowerReportingTimeConstant=N/A
  ViralPolicy=N/A
  ViralState=N/A
  PeakPowerBudget=N/A
  AvgPowerLimit=N/A
  AvgPowerTimeConstant=N/A
  MemoryBandwidthBoostFeature=N/A
  MemoryBandwidthBoostMaxPowerLimit=N/A
  MemoryBandwidthBoostAveragePowerTimeConstant=N/A
  MaxAveragePowerLimit=N/A
  MaxMemoryBandwidthBoostMaxPowerLimit=N/A
  MaxMemoryBandwidthBoostAveragePowerTimeConstant=N/A
  MemoryBandwidthBoostAveragePowerTimeConstantStep=N/A
  MaxAveragePowerReportingTimeConstant=N/A
  AveragePowerReportingTimeConstantStep=N/A
  LatchedLastShutdownStatus=N/A
  UnlatchedLastShutdownStatus=N/A
  MaxMediaTemperature=N/A
  MaxControllerTemperature=N/A
  ThermalThrottleLossPercent=N/A
  LastShutdownTime=N/A
  ModesSupported=N/A
  SecurityCapabilities=N/A
  MasterPassphraseEnabled=N/A
  ConfigurationStatus=N/A
  SKUViolation=N/A
  ARSStatus=N/A
  OverwriteStatus=N/A
  AitDramEnabled=N/A
  BootStatus=N/A
  ErrorInjectionEnabled=N/A
  MediaTemperatureInjectionEnabled=N/A
  SoftwareTriggersEnabled=N/A
  SoftwareTriggersEnabledDetails=N/A
  PoisonErrorInjectionsCounter=N/A
  PoisonErrorClearCounter=N/A
  MediaTemperatureInjectionsCounter=N/A
  SoftwareTriggersCounter=N/A
  BootStatusRegister=N/A
  AveragePower=N/A
  Average12vPower=N/A
  Average1_2vPower=N/A
  ExtendedAdrEnabled=N/A
  PpcExtendedAdrEnabled=N/A
  LatchSystemShutdownState=N/A
  PreviousPowerCycleLatchSystemShutdownState=N/A
  MixedSKU=N/A
  FIPSModeStatus=N/A
  • Even for unmanageable DIMMs, you should see basic fields like DimmHandle, DimmUID, SubsystemDeviceID, FWVersion, and BootStatus.
• ipmctl show -a -topology
  ---DimmID=0x0001---
  MemoryType=Logical Non-Volatile Device
  Capacity=0.000 GiB
  PhysicalID=0x0025
  DeviceLocator=DIMMA2
  SocketID=0x0000
  MemControllerID=0x0000
  ChannelID=0x0000
  ChannelPos=1
  NodeControllerID=0x0000
  BankLabel=P0_Node0_Channel0_Dimm1
  ---DimmID=0x0101---
  MemoryType=Logical Non-Volatile Device
  Capacity=0.000 GiB
  PhysicalID=0x002c
  DeviceLocator=DIMMD2
  SocketID=0x0000
  MemControllerID=0x0001
  ChannelID=0x0000
  ChannelPos=1
  NodeControllerID=0x0000
  BankLabel=P0_Node1_Channel0_Dimm1
  ---DimmID=N/A---
  MemoryType=DDR4
  Capacity=0.000 GiB
  PhysicalID=0x0024
  DeviceLocator=DIMMA1
  SocketID=N/A
  MemControllerID=N/A
  ChannelID=N/A
  ChannelPos=N/A
  NodeControllerID=N/A
  BankLabel=P0_Node0_Channel0_Dimm0
  ---DimmID=N/A---
  MemoryType=DDR4
  Capacity=32.000 GiB
  PhysicalID=0x0026
  DeviceLocator=DIMMB1
  SocketID=N/A
  MemControllerID=N/A
  ChannelID=N/A
  ChannelPos=N/A
  NodeControllerID=N/A
  BankLabel=P0_Node0_Channel1_Dimm0
  ---DimmID=N/A---
  MemoryType=DDR4
  Capacity=32.000 GiB
  PhysicalID=0x0028
  DeviceLocator=DIMMC1
  SocketID=N/A
  MemControllerID=N/A
  ChannelID=N/A
  ChannelPos=N/A
  NodeControllerID=N/A
  BankLabel=P0_Node0_Channel2_Dimm0
  ---DimmID=N/A---
  MemoryType=DDR4
  Capacity=0.000 GiB
  PhysicalID=0x002b
  DeviceLocator=DIMMD1
  SocketID=N/A
  MemControllerID=N/A
  ChannelID=N/A
  ChannelPos=N/A
  NodeControllerID=N/A
  BankLabel=P0_Node1_Channel0_Dimm0
  ---DimmID=N/A---
  MemoryType=DDR4
  Capacity=32.000 GiB
  PhysicalID=0x002d
  DeviceLocator=DIMME1
  SocketID=N/A
  MemControllerID=N/A
  ChannelID=N/A
  ChannelPos=N/A
  NodeControllerID=N/A
  BankLabel=P0_Node1_Channel1_Dimm0
  ---DimmID=N/A---
  MemoryType=DDR4
  Capacity=32.000 GiB
  PhysicalID=0x002f
  DeviceLocator=DIMMF1
  SocketID=N/A
  MemControllerID=N/A
  ChannelID=N/A
  ChannelPos=N/A
  NodeControllerID=N/A
  BankLabel=P0_Node1_Channel2_Dimm0

• If the DIMMs do not appear here, the BIOS has not populated the NFIT table for them, which means the platform is not recognizing the modules at all — a strong indicator of a BIOS/platform incompatibility.
• ipmctl show -a -dimm -display BootStatus
  The command -display does not exist
  Intel(R) Optane(TM) Persistent Memory Command Line Interface Version 03.00.00.538
  Intel(R) Optane(TM) Persistent Memory Driver API Version 10.0.26100.1150
  • The Boot Status Register (BSR) may be readable even on some unmanageable DIMMs and provides the most direct indication of why a DIMM failed to initialize.

[DangoNetwork]

I updated the command output above

[sscargal]

Thanks, @DangoNetwork. This is very helpful.

The ones that work are NMAXBD512GQS. My understanding is that these are Qualifying simples?

You are correct. The "Q" is Qualification Samples.

I strongly suspect the problem is that the firmware/SKU for these unmanageable DIMMs is just too old for the BIOS to support. When Optane reached general availability, the OEMs settled on Optane firmware 1.2.0.5446. To guarantee operational stability, most production BIOSs check the part number and firmware. If the G2S modules aren't on the supported list, the BIOS won't train the PMem, and you can end up in this state.

This is supported by the N/A response we received from the ipmctl commands, especially BootStatus. It looks to me like the BIOS didn't even bother to try training; it failed early, during the BIOS sanity checks.

Sorry, but I don't have any idea how you could make the G2S modules usable on a production system. The engineering samples stopped being produced once the qual-samples were ready. As such, I don't believe the 5446 firmware would even work on them. Given they are 'unmanageable', there's no way I know of to push the updated firmware to them (I always used the UEFI interface for that).

[DangoNetwork]

I could try downgrading the BIOS, and give that a try. But thanks for the help.