Skip to content

DM-56033: Remove APDB preload from Daytime pipeline - #291

Open
isullivan wants to merge 3 commits into
mainfrom
tickets/DM-56033
Open

DM-56033: Remove APDB preload from Daytime pipeline#291
isullivan wants to merge 3 commits into
mainfrom
tickets/DM-56033

Conversation

@isullivan

Copy link
Copy Markdown
Contributor

All of the catalogs are now loaded directly in DiaPipelineTask, which is turned on by setting a new config.

@hsinfang hsinfang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@tcjennings @ctslater FYI, with this change, the pipeline yaml for the Daytime AP is going to be a different file. I see #277 is still open. If there is another code we should modify for changes like this, it'd be good to know.

Comment thread pipelines/LSSTCam/ApPipeDaytime.yaml Outdated
subsets:
# Excluding a task deletes every subset that named it, rather than just
# dropping the label, so preload and apPipe have to be redefined. Without
# this, mpSkyEphemerisQuery would belong to no subset at all.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why this preload subset is needed; does anything use it?
And mpSkyEphemerisQuery is in the apPipe subset already.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's because of the pipeline unit tests. Each pipeline is required to define prompt, preload, etc.. and because my import excluded tasks that were part of the preload subset that entire subset was dropped. So, I have to redefine the preload subset here, or carve out a special exception for this pipeline in the tests.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see, thanks for explaining. I now see the test_ap_pipe_subsets test. It feels somewhat unfortunate to me to enforce that for Daytime pipeline, because we intentionally don't want preload inn the Daytime pipeline. I'd prefer to exclude ApPipeDaytime.yaml in the files list of that test, but I don't have the full understanding of previous design thinking of ap_pipe.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree that the preload and other subsets are inappropriate for the Daytime pipeline, and it turned out to be fairly easy to just exclude it from the failing subsets test. The remaining tests still catch a missing task label from the overall apPipe definition, which is the test we care about for this pipeline.

All of the catalogs are instead loaded directly inside DiaPipelineTask for the Daytime pipeline (only)
The Daytime pipeline now loads all of the catalogs within DiaPipelineTask

@ebellm ebellm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing on a couple of Claude suggestions because I certainly wouldn't have thought of them.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude suggests you don' t need to change these contracts:

pipe_base's subset_from_labels prunes any contract whose text mentions an excluded label
(pipelineIR.py:970). That's why the untouched loadDiaCatalogs.apdb_config_url == associateApdb.apdb_config_url contract (line 316) doesn't break Daytime. So the associateApdb.doReloadAllApdbCatalogs or (...) guards only matter if someone runs both loadDiaCatalogs and doReloadAllApdbCatalogs: True — a configuration that wastes the preload and should probably fail loudly rather than be blessed.

with Apdb.containsVisitDetector.
instrument: lsst.obs.lsst.LsstCam
imports:
- location: $AP_PIPE_DIR/pipelines/LSSTCam/ApPipe.yaml

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- location: $AP_PIPE_DIR/pipelines/LSSTCam/ApPipe.yaml
- location: $AP_PIPE_DIR/pipelines/LSSTCam/ApPipe.yaml
labeledSubsetModifyMode: EDIT

Claude suggests that you can avoid redefining the subsets below (which is fragile to future changes):

The comment says excluding a task deletes every subset that named it — true for the default DROP, but the import supports labeledSubsetModifyMode: EDIT, which removes just the missing labels. drp_pipe/pipelines/LSSTCam/DRP.yaml:5 already uses it. This would eliminate the duplicated list, which will otherwise silently drift when tasks are added to the base apPipe subset (nothing in test_pipelines.py checks subset completeness — self.synonyms isn't actually asserted against anything in the visible test code). Worth checking EDIT doesn't leave any other base subset (prompt, etc.) in a misleading state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants