Skip to content

Conversation

@janniklinde
Copy link
Contributor

This patch introduces various bugfixes and improvements to stream handling to address synchronization issues leading to deadlocks in specific cases (we now reliably guarantee that the finalizer callback is invoked after the last consumer finished execution). Further, the Tee instruction was modified to be introduced by statement level rewrites to keep track of transient reads / writes which led to issues with the previous DAG rewrite.

Finally, we introduced basic block tracking to remove items from the OOCEvictionManager once they have no pending consumers. To achieve that, we track stream references and once no variable holds the underlying CachingStream we mark it as deletable (because then we know, that no additional stream consumer will be added). Blocks from deletable streams can safely be removed from the cache once their consumption count reaches the final number of registered consumers.

Note that we modified the VariableCPInstruction to access variable remove and copy events.

…ues to avoid Race Conditions and Unclosed Streams
@mboehm7
Copy link
Contributor

mboehm7 commented Dec 29, 2025

LGTM - sorry for the delay, and thanks for the various improvements @janniklinde. These changes are great progress towards full algorithm support. During the merge, I fixed minor issues related to unnecessary imports, missing synchronization of overwritten methods, and duplicated but commented code. For the PCA test, it would be good to add the result comparison with a future PR.

@mboehm7 mboehm7 closed this in c49e25d Dec 29, 2025
@github-project-automation github-project-automation bot moved this from In Progress to Done in SystemDS PR Queue Dec 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants