From 12fb01c970d39f50ccb382347d3e14a21e3d6291 Mon Sep 17 00:00:00 2001 From: Peter Josef Scheibel Date: Tue, 11 May 2021 14:13:56 -0700 Subject: [PATCH 1/3] add initial proposal --- seps/sep-0002.md | 102 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 102 insertions(+) create mode 100644 seps/sep-0002.md diff --git a/seps/sep-0002.md b/seps/sep-0002.md new file mode 100644 index 0000000..c70d173 --- /dev/null +++ b/seps/sep-0002.md @@ -0,0 +1,102 @@ +# Use-variants: for packages with conflicting libraries + +## The problem + +The `intel-oneapi-mkl` package is installed with many libraries. Some +of these libraries define the same symbols but differ in their +implementation. For example there is a set of libraries built with +threading support, and another set which offers the same +functionality but are built without threading support. + +Right now, the only mechanism Spack offers to choose between these +libraries is a variant: you could add a `threading` variant and then +choose which libraries to supply to dependents based on the value +of that variant. The problem with a variant is that different +variant values generate distinct installations of a package: we would +install `intel-oneapi-mkl` once for each choice of `threading`, even +though the installations would be the same. + +See also: https://github.com/spack/spack/discussions/22749 + +## Proposed Changes + +* Modify variant declarations with a flag to indicate that the variant + determines how the package is used rather than how it is installed +* During concretization, this variant can be treated like any other + variant. +* After concretization, the variant is stripped from the node and + stored on the edge connecting the node to its parent. + +## Example Use Cases + +### Intel oneAPI MKL + +Described in the introduction. This would be resolved by adding a +`threading` variant: + +``` +variant('threading', values=['openmp', 'sequential'], + interface=True) +``` + +The new logic is the specification of `interface=True`: this +indicates that this variant describes how the package is used rather +than how it is installed. + +Dependents do not have to specify a value for `threading`: if they +don't choose a value, then all dependents still need to "agree" on +using the same libraries. Both the new and old concretizer already +enforce consistency of variants amongst dependents. + +### Boost + +If users want to create a boost installation with many components, +then dependents may only need a subset of the Boost libraries; this +is a bigger issue if the unneeded Boost libraries define conflicting +symbols (not with one another, but with symbols defined in the +dependent or some sibling dependency of Boost). + +Because Boost component libraries don't conflict with each other, +they do not need to be constrained like the dependents of +`intel-oneapi-mkl`. However, when we call +`root_spec['boost'].libs`, we want to know which libraries the +dependent actually needs: to handle this, the dependent can +recompute its dependencies based on its concrete spec, then pass the +dependency variants it requires as parameters to the dependency when +requesting `.libs`. + +We could avoid doing this universally by marking variants where this +is a concern: + +``` +variant('regex', request=True) +``` + +This is not essential though: the dependent could universally +recompute dependency information without any hints from dependents. + +### Intel Parallel Studio + +This provides multiple virtual packages: `mkl`, `blas`, `lapack`, +`mpi`, and more. Each implementation has multiple sets of conflicting +libraries. If a dependent is only using `intel-parallel-studio` for +`mpi`, then we don't want use variants related to `mkl` to be +recorded. + +This is murky pending a completed implementation of finer selection +of virtual providers: Currently we cannot say that we depend on +`intel-parallel-studio` *only* for `mpi` - if you ask for +`intel-parallel-studio` it is assumed that you are using it for +everything it provides (and there is no alternative `blas` +implementation for example). + +Once there is a means of recording which virtuals are being used +by a dependent, we will want to only propagate an interface variant +to the dependent edge if the dependent actually uses the virtual +associated with that variant. For that purpose, I think it would +be suitable to name the associated virtual in the variant +declaration: + +``` +variant('threading', type=VariantType.Interface, provider='mkl') +``` From d53592fccaa364eb8a21452e573b3ffcbc3bab5b Mon Sep 17 00:00:00 2001 From: Peter Josef Scheibel Date: Tue, 11 May 2021 14:15:03 -0700 Subject: [PATCH 2/3] tweak example for consistency --- seps/sep-0002.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/seps/sep-0002.md b/seps/sep-0002.md index c70d173..654c096 100644 --- a/seps/sep-0002.md +++ b/seps/sep-0002.md @@ -98,5 +98,5 @@ be suitable to name the associated virtual in the variant declaration: ``` -variant('threading', type=VariantType.Interface, provider='mkl') +variant('threading', interface=True, provider='mkl') ``` From 41565f3c8b9bbe5ac41d9067db68c25b4137dc8a Mon Sep 17 00:00:00 2001 From: Peter Josef Scheibel Date: Thu, 27 May 2021 14:30:29 -0700 Subject: [PATCH 3/3] add clarifications about concretization semantics --- seps/sep-0002.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/seps/sep-0002.md b/seps/sep-0002.md index 654c096..49877ef 100644 --- a/seps/sep-0002.md +++ b/seps/sep-0002.md @@ -21,9 +21,9 @@ See also: https://github.com/spack/spack/discussions/22749 ## Proposed Changes * Modify variant declarations with a flag to indicate that the variant - determines how the package is used rather than how it is installed -* During concretization, this variant can be treated like any other - variant. + determines how the package is used rather than how it is installed. +* During concretization, this variant is treated like any other + variant (no changes to the concretization algorithm are required). * After concretization, the variant is stripped from the node and stored on the edge connecting the node to its parent. @@ -74,6 +74,10 @@ variant('regex', request=True) This is not essential though: the dependent could universally recompute dependency information without any hints from dependents. +For this reason, the Boost use case can be handled completely +separately, since any variant modifications would only be used +to speed up computations and are not required to achieve the +semantics desired for the use case. ### Intel Parallel Studio @@ -100,3 +104,11 @@ declaration: ``` variant('threading', interface=True, provider='mkl') ``` + +In other words, this has the same concretization semantics discussed +for the Intel oneAPI MKL use case (i.e. that this is treated +exactly the same as other variants during concretization), but +adds controls on when the variant influences the hash of dependents: +it only affects the dependent's hash when the dependent is actually +using the package for the virtual named in the variant's `provider=` +option.