DM-56033: Remove APDB preload from Daytime pipeline - #291
Conversation
hsinfang
left a comment
There was a problem hiding this comment.
LGTM.
@tcjennings @ctslater FYI, with this change, the pipeline yaml for the Daytime AP is going to be a different file. I see #277 is still open. If there is another code we should modify for changes like this, it'd be good to know.
| subsets: | ||
| # Excluding a task deletes every subset that named it, rather than just | ||
| # dropping the label, so preload and apPipe have to be redefined. Without | ||
| # this, mpSkyEphemerisQuery would belong to no subset at all. |
There was a problem hiding this comment.
I don't understand why this preload subset is needed; does anything use it?
And mpSkyEphemerisQuery is in the apPipe subset already.
There was a problem hiding this comment.
It's because of the pipeline unit tests. Each pipeline is required to define prompt, preload, etc.. and because my import excluded tasks that were part of the preload subset that entire subset was dropped. So, I have to redefine the preload subset here, or carve out a special exception for this pipeline in the tests.
There was a problem hiding this comment.
Oh, I see, thanks for explaining. I now see the test_ap_pipe_subsets test. It feels somewhat unfortunate to me to enforce that for Daytime pipeline, because we intentionally don't want preload inn the Daytime pipeline. I'd prefer to exclude ApPipeDaytime.yaml in the files list of that test, but I don't have the full understanding of previous design thinking of ap_pipe.
There was a problem hiding this comment.
agree that the preload and other subsets are inappropriate for the Daytime pipeline, and it turned out to be fairly easy to just exclude it from the failing subsets test. The remaining tests still catch a missing task label from the overall apPipe definition, which is the test we care about for this pipeline.
All of the catalogs are instead loaded directly inside DiaPipelineTask for the Daytime pipeline (only)
The Daytime pipeline now loads all of the catalogs within DiaPipelineTask
5dd0ae4 to
fc21b1c
Compare
ebellm
left a comment
There was a problem hiding this comment.
Passing on a couple of Claude suggestions because I certainly wouldn't have thought of them.
There was a problem hiding this comment.
Claude suggests you don' t need to change these contracts:
pipe_base's subset_from_labels prunes any contract whose text mentions an excluded label
(pipelineIR.py:970). That's why the untouched loadDiaCatalogs.apdb_config_url == associateApdb.apdb_config_url contract (line 316) doesn't break Daytime. So the associateApdb.doReloadAllApdbCatalogs or (...) guards only matter if someone runs both loadDiaCatalogs and doReloadAllApdbCatalogs: True — a configuration that wastes the preload and should probably fail loudly rather than be blessed.
| with Apdb.containsVisitDetector. | ||
| instrument: lsst.obs.lsst.LsstCam | ||
| imports: | ||
| - location: $AP_PIPE_DIR/pipelines/LSSTCam/ApPipe.yaml |
There was a problem hiding this comment.
| - location: $AP_PIPE_DIR/pipelines/LSSTCam/ApPipe.yaml | |
| - location: $AP_PIPE_DIR/pipelines/LSSTCam/ApPipe.yaml | |
| labeledSubsetModifyMode: EDIT |
Claude suggests that you can avoid redefining the subsets below (which is fragile to future changes):
The comment says excluding a task deletes every subset that named it — true for the default DROP, but the import supports labeledSubsetModifyMode: EDIT, which removes just the missing labels. drp_pipe/pipelines/LSSTCam/DRP.yaml:5 already uses it. This would eliminate the duplicated list, which will otherwise silently drift when tasks are added to the base apPipe subset (nothing in test_pipelines.py checks subset completeness — self.synonyms isn't actually asserted against anything in the visible test code). Worth checking EDIT doesn't leave any other base subset (prompt, etc.) in a misleading state.
All of the catalogs are now loaded directly in DiaPipelineTask, which is turned on by setting a new config.