Skip to content

Use file monitoring for pre-processor re-run checks (#11411) - #12021

Open
tdammers wants to merge 2 commits into
haskell:masterfrom
tdammers:wip/tobias-11411-b
Open

Use file monitoring for pre-processor re-run checks (#11411)#12021
tdammers wants to merge 2 commits into
haskell:masterfrom
tdammers:wip/tobias-11411-b

Conversation

@tdammers

@tdammers tdammers commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

This replaces the previous preprocessor re-run logic ("if the
preprocessor output exists and is newer than the input, don't re-run the
preprocessor") with a file monitor, which takes both the input file and
the preprocessor itself into account, solving #11411.

To make this possible, each preprocessor is now tagged with a hash that
uniquely identifies the specific preprocessor version.

The process of finding the actual preprocessor (runPreProcessor) to
run has been split off from the actual running; this was necessary so
that we can get a unique hash of the preprocessor (including version)
without having to actually run it, while also avoiding duplicate work
and a potential mismatch between the hash and the actual preprocessor.
Hence, we now have configurePreProcessor, which returns a pair of the
former runPreProcessor action and the preprocessor hash.

Includes/supersedes #12065, which should be merged first.


@tdammers
tdammers force-pushed the wip/tobias-11411-b branch 2 times, most recently from 3e5f475 to a243a39 Compare June 30, 2026 08:07
@tdammers tdammers changed the title Wip/tobias 11411 b Use file monitoring for pre-processor re-run checks (#11411) Jun 30, 2026
@tdammers
tdammers force-pushed the wip/tobias-11411-b branch 2 times, most recently from 06781e1 to e813cd0 Compare July 8, 2026 08:29
Comment thread Cabal/Cabal.cabal Outdated
@tdammers
tdammers force-pushed the wip/tobias-11411-b branch 3 times, most recently from 7b184b9 to b546d1b Compare July 17, 2026 07:13
@tdammers
tdammers force-pushed the wip/tobias-11411-b branch from 6cdb139 to f9a8eb8 Compare July 28, 2026 09:59
@tdammers
tdammers marked this pull request as ready for review July 28, 2026 15:56
@tdammers
tdammers force-pushed the wip/tobias-11411-b branch from f9a8eb8 to da95570 Compare July 29, 2026 11:18

@sheaf sheaf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very clean and elegant fix on top of the preparatory refactor, fantastic.

It looks like you still need to add some changelog entries:

  1. One very short one for the renaming involving lookupAndConfigureProgramVersion.
  2. One for the actual fix and the changes to the PreProcessor datatype (I think it makes sense to bundle those together in a single changelog entry).

Like with the other MR, I suggest squashing commits and giving the final commit a nice commit message.

Comment thread Cabal/src/Distribution/Simple/PreProcess.hs
Comment thread Cabal/src/Distribution/Simple/PreProcess.hs Outdated
Comment thread Cabal/src/Distribution/Simple/PreProcess/Types.hs
Comment thread Cabal/src/Distribution/Simple/Program/Db.hs
Comment thread Cabal/src/Distribution/Simple/Program/Db.hs
Comment thread Cabal/src/Distribution/Simple/Program/Db.hs Outdated
@tdammers
tdammers force-pushed the wip/tobias-11411-b branch from 0a413f6 to 6a2d989 Compare August 3, 2026 09:02
@tdammers
tdammers requested a review from sheaf August 4, 2026 11:50
This commit includes a series of refactorings that are necessary to
solve haskell#11411:

- Abstract the Rebuild monad into the MonadRebuild typeclass,
  introducing the `withRunRebuildInIO` method of MonadRebuild. This
  allows us to interleave plain `IO` with `MonadRebuild` without
  necessarily having to use `Rebuild` itself. This is especially
  important because `Rebuild` also encodes the "current working
  directory" concern, which is relevant within `cabal-install`, but
  something we don't want to require within the `Cabal` library. Since
  the `MonadRebuild` typeclass allows for arbitrary custom monadic state
  per instance, it can live in the `Cabal` subproject, together with
  operations defined in terms of the typeclass, while `Rebuild` itself
  remains in `cabal-install`, using the custom monadic state to cover
  the "current working directory" concern.
- Change the `StateT` part of `Rebuild` into `WriterT`, since we only
  ever emit additional files from within a Rebuild context; consuming
  them happens on the outside, so Writer is sufficient.
- Provide `MonadRebuild` instances for the existing `Rebuild` type, and
  for the most minimal type that can satisfy the typeclass, `WriterT
  [MonitorFilePath] IO`.
- Implement most file monitoring / Rebuild actions in terms of
  `MonadRebuild`, so they can live in `Cabal`
  (`Distribution.Simple.RebuildMonad`), leaving only
  `cabal-install` specific things in the existing `cabal-install`
  module (`Distribution.Client.RebuildMonad`). Specifically, we
  implement `rerunIfChanged'` in terms of `MonadRebuild` and a custom
  function for chaining `IO` actions (serially or in parallel, as
  needed); the existing `rerunIfChanged` and
  `rerunConcurrentlyIfChanged` functions, which live in `cabal-install`,
  are now implemented in terms of `rerunIfChanged'`, which lives in
  `Cabal`).
- Move large parts of the `Distribution.Client.FileMonitor` module from
  `cabal-install` to `Cabal` (as `Distribution.Simple.FileMonitor`);
  we will be needing these parts to solve haskell#11411
  (`Distribution.Client.FileMonitor` still re-exports all of them
  though).
- Move most of the `Distribution.Client.HashValue` module over to
  `Cabal` (with the exception of TUF-related functions, which are
  specific to `cabal-install`; again we re-export everything in
  `cabal-install` to maintain API compatibility).
- Move `getFilePathRootDirectory` from `Distribution.Client.Glob` into
  `Distribution.Simple.FileMonitor.Types` - it has little to do with
  actual globs, and is only used for file monitoring purposes.
- Shuffle some utility functions around to make them accessible where we
  need them.

The big picture with all this is that we want to reuse the existing file
monitoring machinery to keep track of build-tool versions alongside
generated source files. However, the tracking needs to happen in the
`Cabal` library, while file monitoring has so far only been available in
`cabal-install`; hence we need to move the required file monitoring code
and its dependencies into the `Cabal` library, but without introducing
any `cabal-install` specific concerns into `Cabal`, and without breaking
API compatibility on the `cabal-install` side.
This replaces the previous preprocessor re-run logic ("if the
preprocessor output exists and is newer than the input, don't re-run the
preprocessor") with a file monitor, which takes both the input file and
the preprocessor itself into account, solving haskell#11411.

To make this possible, each preprocessor is now tagged with a hash that
uniquely identifies the specific preprocessor version.

The process of finding the actual preprocessor (`runPreProcessor`) to
run has been split off from the actual running; this was necessary so
that we can get a unique hash of the preprocessor (including version)
without having to actually run it, while also avoiding duplicate work
and a potential mismatch between the hash and the actual preprocessor.
Hence, we now have `configurePreProcessor`, which returns a pair of the
former `runPreProcessor` action and the preprocessor hash.

Other changes:

- Use GHC.Fingerprint instead of SHA256 in FileMonitor (avoiding a
  dependency on cryptohash-sha256, which is not a boot library).
- Clean up the ProgramDb API, so that we can be explicit about whether
  we want to configure preprocessors on the fly as needed or not.
@tdammers
tdammers force-pushed the wip/tobias-11411-b branch from 202ea4f to d2f0c48 Compare August 4, 2026 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants