Skip to content

[SPARK-59458][YARN] Fix 'propogate' typos in YarnAllocator comments #58757

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
uros-b wants to merge 1 commit into
base: master
Choose a base branch
from
+3 −3
Open

[SPARK-59458][YARN] Fix 'propogate' typos in YarnAllocator comments #58757

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,21 +329,21 @@ private[yarn] class YarnAllocator(
// to request YARN containers with extra resources without Spark scheduling on
// them, the user can specify resources via the <code>spark.yarn.executor.resource.</code>
// config. Those configs are only used in the base default profile though and do
// not get propogated into any other custom ResourceProfiles. This is because
// not get propagated into any other custom ResourceProfiles. This is because
// there would be no way to remove them if you wanted a stage to not have them.
// This results in your default profile getting custom resources defined in
// <code>spark.yarn.executor.resource.</code> plus spark defined resources of
// GPU or FPGA. Spark converts GPU and FPGA resources into the YARN built in
// types <code>yarn.io/gpu</code>) and <code>yarn.io/fpga</code>, but does not
// know the mapping of any other resources. Any other Spark custom resources
// are not propogated to YARN for the default profile. So if you want Spark
// are not propagated to YARN for the default profile. So if you want Spark
// to schedule based off a custom resource and have it requested from YARN, you
// must specify it in both YARN (<code>spark.yarn.{driver/executor}.resource.</code>)
// and Spark (<code>spark.{driver/executor}.resource.</code>) configs. Leave the Spark
// config off if you only want YARN containers with the extra resources but Spark not to
// schedule using them. Now for custom ResourceProfiles, it doesn't currently have a way
// to only specify YARN resources without Spark scheduling off of them. This means for
// custom ResourceProfiles we propogate all the resources defined in the ResourceProfile
// custom ResourceProfiles we propagate all the resources defined in the ResourceProfile
// to YARN. We still convert GPU and FPGA to the YARN build in types as well. This requires
// that the name of any custom resources you specify match what they are defined as in YARN.
val customResources = if (rp.id == DEFAULT_RESOURCE_PROFILE_ID) {
Expand Down