Skip to content

changes - #1831

Merged
SFJohnson24 merged 1 commit into
mainfrom
ddf_distinct
Aug 19, 2026
Merged

changes#1831
SFJohnson24 merged 1 commit into
mainfrom
ddf_distinct

Conversation

@SFJohnson24

@SFJohnson24 SFJohnson24 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

This pull request improves the robustness and correctness of the Distinct operation, especially in handling empty datasets and grouped operations. It also adds comprehensive unit tests to cover these scenarios. The main changes are as follows:

Enhanced Handling of Empty Datasets in Grouped Operations

  • The Distinct operation now correctly returns an empty grouped result with the appropriate columns when the input dataset is empty or when a filter produces zero rows, instead of failing or returning incorrect results. This is achieved by adding the _build_empty_grouped_result helper method and updating the execution logic to use it as needed. [1] [2] [3]

Improved Reference Column Handling

  • The _check_column_exists_in_dataset function and its usage were updated to allow the referenced domain column name to be configurable (defaulting to "RDOMAIN"), increasing flexibility for different dataset schemas. [1] [2] [3]

Bug Fix in Result Expansion

  • Fixed a bug in _expand_operation_results_in_grouping to handle the case when an operation column is empty, preventing errors during result expansion.

Testing Improvements

  • Added new parameterized unit tests to verify correct behavior of the Distinct operation for both empty datasets and filtered grouped results that yield zero rows, for both Pandas and Dask backends.

Test Infrastructure

  • Added missing pandas import in the test file to support new test cases.

PR addresses attached issue. I updated SCheduleDecisionInstance in the test data for core-000804 to test;

parent_entity,parent_id,parent_rel,rel_type,id,name,description,label,defaultConditionId,epochId,timelineId,timelineExitId,instanceType,conditionAssignments
ScheduleTimeline,ScheduleTimeline_4,instances,definition,ScheduledDecisionInstance_1,Meet requirements for WK6 or end,,,ScheduledActivityInstance_27,,,,ScheduledDecisionInstance,true
ScheduleTimeline,ScheduleTimeline_4,entryId,reference,ScheduledDecisionInstance_1,Meet requirements for WK6 or end,,,ScheduledActivityInstance_27,,,,ScheduledDecisionInstance,true

this produces a scheduledesicioninstance with no timing instance

worth noting: conditionAssignments is cast as a bool and the current data in the repo is not able to be cast as such since it is a string

I also tested CORE-000712 given it's distinct-specific operation logic

@pendingintent pendingintent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good and all tests passing.

@SFJohnson24
SFJohnson24 merged commit 50f165b into main Aug 19, 2026
13 checks passed
@SFJohnson24
SFJohnson24 deleted the ddf_distinct branch August 19, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DDF00046: grouped distinct crashes on empty ScheduledDecisionInstance reference subset

2 participants