VV: Read EDAX EBSD Data (.ang) fully V&V'ed

src/Plugins/OrientationAnalysis/docs/ReadAngDataFilter.md

@@ -6,11 +6,28 @@ IO (Input)
 
 ## Description
 
-This **Filter** will read a single .ang file into a new **Image Geometry**, allowing the immediate use of **Filters** on the data instead of having to generate the intermediate .h5ebsd file. A **Cell Attribute Matrix** and Ensemble Attribute Matrix** will also be created to hold the imported EBSD information. Currently, the user has no control over the names of the created **Attribute Arrays**. The user should be aware that simply reading the file then performing operations that are dependent on the proper crystallographic and sample reference frame will be undefined or simply **wrong**. In order to bring the crystal reference frame and sample reference frame into coincidence, rotations will need to be applied to the data.
+This **Filter** will read a single .ang file into a new **Image Geometry**, allowing the immediate use of **Filters** on the data instead of having to generate the intermediate .h5ebsd file. A **Cell Attribute Matrix** and **Ensemble Attribute Matrix** will also be created to hold the imported EBSD information. Currently, the user has no control over the names of the created **Attribute Arrays**. The user should be aware that simply reading the file then performing operations that are dependent on the proper crystallographic and sample reference frame will be undefined or simply **wrong**. In order to bring the crystal reference frame and sample reference frame into coincidence, rotations will need to be applied to the data.
+
+### Created Data
+
+The **Image Geometry** dimensions come from the `NCOLS_EVEN`/`NROWS` header values, the spacing from `XSTEP`/`YSTEP` (Z spacing is 1.0), the origin is (0, 0, 0), and the length units are always **micrometers**.
+
+| Location | Array | Type | Notes |
+|---|---|---|---|
+| Cell | EulerAngles | float32 (3) | phi1, PHI, phi2 columns interleaved, in radians as stored in the file |
+| Cell | Phases | int32 | Phase values less than 1 (un-indexed points) are remapped to 1 |
+| Cell | Image Quality, Confidence Index, SEM Signal, Fit, X Position, Y Position | float32 | Copied verbatim from the file columns |
+| Ensemble | CrystalStructures | uint32 | From each phase's TSL `Symmetry` code; index 0 is reserved for the invalid phase (999) |
+| Ensemble | MaterialName | string | Whitespace-trimmed material name; index 0 is "Invalid Phase" |
+| Ensemble | LatticeConstants | float32 (6) | a, b, c (Angstroms), alpha, beta, gamma (degrees); index 0 is all zeros |
+
+### Note on Length Units
+
+The created **Image Geometry** is always marked as **micrometers**, matching standard EDAX SEM-based `.ang` files. Legacy DREAM.3D 6.5 detected certain retired TEM/ACOM `.ang` variants and marked them as nanometers; EDAX retired those file formats over a decade ago and this filter does not special-case them. If you are importing such an archival file, set the geometry units manually after import.
 
 ### Default TSL Transformations
 
-If the data has come from a TSL acquisition system and the settings of the acquisition software were in the default modes, he following reference frame transformations may need to be performed based on the version of the OIM Analysis software being used to collect the data:
+If the data has come from a TSL acquisition system and the settings of the acquisition software were in the default modes, the following reference frame transformations may need to be performed based on the version of the OIM Analysis software being used to collect the data:
 
 + Sample Reference Frame: 180<sup>o</sup> about the <010> Axis
 + Crystal Reference Frame: 90<sup>o</sup> about the <001> Axis
@@ -25,7 +42,7 @@ OIMAnalysis can create EBSD data sampled on a hexagonal grid. The user can look
 # GRID: HexGrid
 ```
 
-If the user's .ang files are hexagonal grid files then they will need to run the {ref}`Convert EDAX Hex Grid to Square Grid (.ang)<OrientationAnalysis/ConvertHexGridToSquareGridFilter:Description>` filter to first convert the input files square gridded files.
+If the user's .ang files are hexagonal grid files then they will need to run the {ref}`Convert EDAX Hex Grid to Square Grid (.ang)<OrientationAnalysis/ConvertHexGridToSquareGridFilter:Description>` filter to first convert the input files to square gridded files. This filter rejects hexagonal grid files during preflight with error `-19500`.
 
 
 ### Note on .ang file Data Ordering
@@ -48,10 +65,17 @@ Fit of Solution
 
 ## Example Pipelines
 
-+ INL Export
-+ Export Small IN100 ODF Data (StatsGenerator)
-+ Edax IPF Colors
-+ Confidence Index Histogram
++ Read_EDAX_Ang_File
++ Edax_IPF_Colors
++ CI_Histogram
++ EBSD_Hexagonal_Data_Analysis
++ ReplaceElementAttributesWithNeighbor (SimplnxCore)
++ ExtractVertexGeometry (SimplnxCore)
++ ApplyTransformation_Image (SimplnxCore)
+
+## Differences from DREAM.3D 6.5
+
+Behavioral differences from the legacy `ReadAngData` filter (material-name trimming, length-unit handling for retired TEM/ACOM files, error-code changes, and a legacy crash on non-contiguous phase indices) are documented in the source tree at `src/Plugins/OrientationAnalysis/vv/deviations/ReadAngDataFilter.md`.
 
 ## License & Copyright
 

src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.cpp

@@ -8,9 +8,9 @@
 
 #include <EbsdLib/Core/Orientation.hpp>
 
-using namespace nx::core;
+#include <fmt/format.h>
 
-using FloatVec3Type = std::vector<float>;
+using namespace nx::core;
 
 // -----------------------------------------------------------------------------
 ReadAngData::ReadAngData(DataStructure& dataStructure, const IFilter::MessageHandler& msgHandler, const std::atomic_bool& shouldCancel, ReadAngDataInputValues* inputValues)
@@ -27,97 +27,123 @@ ReadAngData::~ReadAngData() noexcept = default;
 // -----------------------------------------------------------------------------
 Result<> ReadAngData::operator()()
 {
+  m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Reading .ang file '{}'", m_InputValues->InputFile.string()));
   ebsdlib::AngReader reader;
   reader.setFileName(m_InputValues->InputFile.string());
   const int32_t err = reader.readFile();
   if(err < 0)
   {
     return MakeErrorResult(reader.getErrorCode(), reader.getErrorMessage());
   }
-  const auto result = loadMaterialInfo(&reader);
-  if(result.first < 0)
+  Result<> result = loadMaterialInfo(&reader);
+  if(result.invalid())
   {
-    return MakeErrorResult(result.first, result.second);
+    return result;
   }
   if(m_ShouldCancel)
   {
     return {};
   }
 
-  copyRawEbsdData(&reader);
-
-  return {};
+  m_MessageHandler(IFilter::Message::Type::Info, "Copying cell data into the Image Geometry");
+  return copyRawEbsdData(&reader);
 }
 
 // -----------------------------------------------------------------------------
-std::pair<int32, std::string> ReadAngData::loadMaterialInfo(ebsdlib::AngReader* reader) const
+Result<> ReadAngData::loadMaterialInfo(ebsdlib::AngReader* reader) const
 {
-
   const std::vector<ebsdlib::AngPhase::Pointer> phases = reader->getPhaseVector();
   if(phases.empty())
   {
-    return {reader->getErrorCode(), reader->getErrorMessage()};
+    return MakeErrorResult(reader->getErrorCode(), reader->getErrorMessage());
   }
 
-  const DataPath CellEnsembleAttributeMatrixPath = m_InputValues->DataContainerName.createChildPath(m_InputValues->CellEnsembleAttributeMatrixName);
+  const DataPath cellEnsembleAttributeMatrixPath = m_InputValues->DataContainerName.createChildPath(m_InputValues->CellEnsembleAttributeMatrixName);
 
-  auto& crystalStructures = m_DataStructure.getDataRefAs<UInt32Array>(CellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::AngFile::CrystalStructures));
+  auto& crystalStructures = m_DataStructure.getDataRefAs<UInt32Array>(cellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::AngFile::CrystalStructures));
 
-  auto& materialNames = m_DataStructure.getDataRefAs<StringArray>(CellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::AngFile::MaterialName));
+  auto& materialNames = m_DataStructure.getDataRefAs<StringArray>(cellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::AngFile::MaterialName));
 
-  auto& latticeConstants = m_DataStructure.getDataRefAs<Float32Array>(CellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::AngFile::LatticeConstants));
+  auto& latticeConstants = m_DataStructure.getDataRefAs<Float32Array>(cellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::AngFile::LatticeConstants));
 
   const std::string k_InvalidPhase = "Invalid Phase";
+  const usize numTuples = crystalStructures.getNumberOfTuples();
 
-  // Initialize the zero'th element to unknowns. The other elements will
-  // be filled in based on values from the data file
-  crystalStructures[0] = ebsdlib::CrystalStructure::UnknownCrystalStructure;
-  materialNames[0] = k_InvalidPhase;
-
-  for(size_t i = 0; i < 6; i++)
+  // Initialize EVERY slot to the "Invalid Phase" defaults first. Slot 0 is always the
+  // invalid phase, and any slot not covered by a phase section in the file (possible when
+  // the file's phase indices are not contiguous) keeps these defaults instead of
+  // zero-initialized garbage.
+  for(usize tupleIndex = 0; tupleIndex < numTuples; tupleIndex++)
   {
-    latticeConstants.getDataStoreRef().setComponent(0, i, 0.0F);
+    crystalStructures[tupleIndex] = ebsdlib::CrystalStructure::UnknownCrystalStructure;
+    materialNames[tupleIndex] = k_InvalidPhase;
+    for(usize i = 0; i < 6; i++)
+    {
+      latticeConstants.getDataStoreRef().setComponent(tupleIndex, i, 0.0F);
+    }
   }
 
   for(const ebsdlib::AngPhase::Pointer& phase : phases)
   {
     const int32_t phaseID = phase->getPhaseIndex();
+    // .ang phase numbering starts at 1. A phase index < 1 (Phase 0 or negative) is rejected: DREAM3D
+    // 6.5.171 tolerated a Phase 0 section (it skipped only negative indices and wrote Phase 0 into
+    // ensemble slot 0), so this is a documented behavior change vs legacy — see deviation
+    // ReadAngDataFilter-D5. A static "# Phase 0" fixture trips this deterministically.
+    if(phaseID < 1)
+    {
+      return MakeErrorResult(-19502, fmt::format("The .ang file declares phase index {}, but .ang phase numbering starts at 1 (Phase 0 and negative phases are not supported).", phaseID));
+    }
+    // The ensemble arrays were sized in preflight from the same file's largest phase index, so an index
+    // at or above the array count can only mean the file changed between preflight and execute.
+    if(static_cast<usize>(phaseID) >= numTuples)
+    {
+      return MakeErrorResult(
+          -19504, fmt::format("Phase index {} from the .ang file is at or above the Ensemble Attribute Matrix count {}. The input file may have changed since preflight.", phaseID, numTuples));
+    }
     crystalStructures[phaseID] = phase->determineOrientationOpsIndex();
-    std::string materialName = phase->getMaterialName();
-    materialName = nx::core::StringUtilities::replace(materialName, "MaterialName", "");
-    materialName = nx::core::StringUtilities::trimmed(materialName);
-    materialNames[phaseID] = materialName;
+    // EbsdLib's AngPhase::parseMaterialName() rejoins the name tokens with a trailing
+    // space (e.g. "Nickel "), so the name must be trimmed before storing it.
+    materialNames[phaseID] = nx::core::StringUtilities::trimmed(phase->getMaterialName());
 
     std::vector<float> lattConst = phase->getLatticeConstants();
 
-    for(size_t i = 0; i < 6; i++)
+    for(usize i = 0; i < 6; i++)
     {
       latticeConstants.getDataStoreRef().setComponent(phaseID, i, lattConst[i]);
     }
   }
-  return {0, ""};
+  return {};
 }
 
 // -----------------------------------------------------------------------------
-void ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const
+Result<> ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const
 {
-  const DataPath CellAttributeMatrixPath = m_InputValues->DataContainerName.createChildPath(m_InputValues->CellAttributeMatrixName);
-
-  std::vector<size_t> cDims = {1};
+  const DataPath cellAttributeMatrixPath = m_InputValues->DataContainerName.createChildPath(m_InputValues->CellAttributeMatrixName);
 
   const auto& imageGeom = m_DataStructure.getDataRefAs<ImageGeom>(m_InputValues->DataContainerName);
   const size_t totalCells = imageGeom.getNumberOfCells();
 
-  // Prepare the Cell Attribute Matrix with the correct number of tuples based on the total Cells being read from the file.
-  std::vector<size_t> tDims = {imageGeom.getNumXCells(), imageGeom.getNumYCells(), imageGeom.getNumZCells()};
+  // The Image Geometry was sized in preflight from the file's column/row header (NumEvenCols x NumRows),
+  // but the reader allocates its data buffers from NumOddCols x NumRows. Every copy below reads
+  // totalCells elements out of those buffers, so if the reader actually read fewer elements (a file that
+  // changed between preflight and execute, or a malformed header where NCOLS_EVEN > NCOLS_ODD) the copies
+  // would read past the end of the reader's heap buffers. Guard against that out-of-bounds read.
+  if(reader->getNumberOfElements() < totalCells)
+  {
+    return MakeErrorResult(-19503,
+                           fmt::format("The .ang reader produced {} scan points but the Image Geometry created at preflight expects {}. The input file may have changed since preflight, or its "
+                                       "column header (NCOLS_ODD/NCOLS_EVEN) is inconsistent.",
+                                       reader->getNumberOfElements(), totalCells));
+  }
 
   // Adjust the values of the 'phase' data to correct for invalid values and assign the read Phase Data into the actual DataArray
   {
     if(m_ShouldCancel)
     {
-      return;
+      return {};
     }
-    auto& targetArray = m_DataStructure.getDataRefAs<Int32Array>(CellAttributeMatrixPath.createChildPath(ebsdlib::AngFile::Phases));
+    auto& targetArray = m_DataStructure.getDataRefAs<Int32Array>(cellAttributeMatrixPath.createChildPath(ebsdlib::AngFile::Phases));
     int* phasePtr = reinterpret_cast<int32_t*>(reader->getPointerByName(ebsdlib::Ang::PhaseData));
     for(size_t i = 0; i < totalCells; i++)
     {
@@ -133,14 +159,13 @@ void ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const
   {
     if(m_ShouldCancel)
     {
-      return;
+      return {};
     }
     const auto* fComp0 = reinterpret_cast<float*>(reader->getPointerByName(ebsdlib::Ang::Phi1));
     const auto* fComp1 = reinterpret_cast<float*>(reader->getPointerByName(ebsdlib::Ang::Phi));
     const auto* fComp2 = reinterpret_cast<float*>(reader->getPointerByName(ebsdlib::Ang::Phi2));
-    cDims[0] = 3;
 
-    auto& cellEulerAngles = m_DataStructure.getDataRefAs<Float32Array>(CellAttributeMatrixPath.createChildPath(ebsdlib::AngFile::EulerAngles));
+    auto& cellEulerAngles = m_DataStructure.getDataRefAs<Float32Array>(cellAttributeMatrixPath.createChildPath(ebsdlib::AngFile::EulerAngles));
     for(size_t i = 0; i < totalCells; i++)
     {
       cellEulerAngles[3 * i] = fComp0[i];
@@ -151,42 +176,44 @@ void ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const
 
   if(m_ShouldCancel)
   {
-    return;
+    return {};
   }
-  cDims[0] = 1;
+
   {
     auto* fComp0 = reinterpret_cast<float*>(reader->getPointerByName(ebsdlib::Ang::ImageQuality));
-    auto& targetArray = m_DataStructure.getDataRefAs<Float32Array>(CellAttributeMatrixPath.createChildPath(ebsdlib::Ang::ImageQuality));
+    auto& targetArray = m_DataStructure.getDataRefAs<Float32Array>(cellAttributeMatrixPath.createChildPath(ebsdlib::Ang::ImageQuality));
     std::copy(fComp0, fComp0 + totalCells, targetArray.begin());
   }
 
   {
     auto* fComp0 = reinterpret_cast<float*>(reader->getPointerByName(ebsdlib::Ang::ConfidenceIndex));
-    auto& targetArray = m_DataStructure.getDataRefAs<Float32Array>(CellAttributeMatrixPath.createChildPath(ebsdlib::Ang::ConfidenceIndex));
+    auto& targetArray = m_DataStructure.getDataRefAs<Float32Array>(cellAttributeMatrixPath.createChildPath(ebsdlib::Ang::ConfidenceIndex));
     std::copy(fComp0, fComp0 + totalCells, targetArray.begin());
   }
 
   {
     auto* fComp0 = reinterpret_cast<float*>(reader->getPointerByName(ebsdlib::Ang::SEMSignal));
-    auto& targetArray = m_DataStructure.getDataRefAs<Float32Array>(CellAttributeMatrixPath.createChildPath(ebsdlib::Ang::SEMSignal));
+    auto& targetArray = m_DataStructure.getDataRefAs<Float32Array>(cellAttributeMatrixPath.createChildPath(ebsdlib::Ang::SEMSignal));
     std::copy(fComp0, fComp0 + totalCells, targetArray.begin());
   }
 
   {
     auto* fComp0 = reinterpret_cast<float*>(reader->getPointerByName(ebsdlib::Ang::Fit));
-    auto& targetArray = m_DataStructure.getDataRefAs<Float32Array>(CellAttributeMatrixPath.createChildPath(ebsdlib::Ang::Fit));
+    auto& targetArray = m_DataStructure.getDataRefAs<Float32Array>(cellAttributeMatrixPath.createChildPath(ebsdlib::Ang::Fit));
     std::copy(fComp0, fComp0 + totalCells, targetArray.begin());
   }
 
   {
     auto* fComp0 = reinterpret_cast<float*>(reader->getPointerByName(ebsdlib::Ang::XPosition));
-    auto& targetArray = m_DataStructure.getDataRefAs<Float32Array>(CellAttributeMatrixPath.createChildPath(ebsdlib::Ang::XPosition));
+    auto& targetArray = m_DataStructure.getDataRefAs<Float32Array>(cellAttributeMatrixPath.createChildPath(ebsdlib::Ang::XPosition));
     std::copy(fComp0, fComp0 + totalCells, targetArray.begin());
   }
 
   {
     auto* fComp0 = reinterpret_cast<float*>(reader->getPointerByName(ebsdlib::Ang::YPosition));
-    auto& targetArray = m_DataStructure.getDataRefAs<Float32Array>(CellAttributeMatrixPath.createChildPath(ebsdlib::Ang::YPosition));
+    auto& targetArray = m_DataStructure.getDataRefAs<Float32Array>(cellAttributeMatrixPath.createChildPath(ebsdlib::Ang::YPosition));
     std::copy(fComp0, fComp0 + totalCells, targetArray.begin());
   }
+
+  return {};
 }