how to load a finetuned model especially during validation phase #11977

Activity

1. DN6 commented on Jul 23, 2025

   

   DN6

   on Jul 23, 2025

   Last edited by DN6

   Collaborator

   More actions

   cc: @sayakpaul @linoytsaban
2. sayakpaul commented on Jul 23, 2025

   

   sayakpaul

   on Jul 23, 2025

   Member

   More actions

   During validation, we send a pipeline:

   diffusers/examples/dreambooth/train_dreambooth_sd3.py

   Line 162 in 1c50a5f

   pipeline,

   The pipeline there is first initialized like this:

   diffusers/examples/dreambooth/train_dreambooth_sd3.py

   Lines 1724 to 1734 in 1c50a5f

   	pipeline = StableDiffusion3Pipeline.from_pretrained(
   	args.pretrained_model_name_or_path,
   	vae=vae,
   	text_encoder=accelerator.unwrap_model(text_encoder_one),
   	text_encoder_2=accelerator.unwrap_model(text_encoder_two),
   	text_encoder_3=accelerator.unwrap_model(text_encoder_three),
   	transformer=accelerator.unwrap_model(transformer),
   	revision=args.revision,
   	variant=args.variant,
   	torch_dtype=weight_dtype,
   	)

   This is to ensure that the models being requested for fine-tuning are present in pipeline. We're using args.pretrained_model_name_or_path, with from_pretrained() to load the other components of the pipeline such as tokenizers and the scheduler.
3. micklexqg commented on Jul 24, 2025

   

   micklexqg

   on Jul 24, 2025

   Author

   More actions

   During validation, we send a pipeline:

   diffusers/examples/dreambooth/train_dreambooth_sd3.py

   Line 162 in 1c50a5f
   pipeline,

   The pipeline there is first initialized like this:

   diffusers/examples/dreambooth/train_dreambooth_sd3.py

   Lines 1724 to 1734 in 1c50a5f
   pipeline = StableDiffusion3Pipeline.from_pretrained(
   args.pretrained_model_name_or_path,
   vae=vae,
   text_encoder=accelerator.unwrap_model(text_encoder_one),
   text_encoder_2=accelerator.unwrap_model(text_encoder_two),
   text_encoder_3=accelerator.unwrap_model(text_encoder_three),
   transformer=accelerator.unwrap_model(transformer),
   revision=args.revision,
   variant=args.variant,
   torch_dtype=weight_dtype,
   )

   This is to ensure that the models being requested for fine-tuning are present in pipeline. We're using args.pretrained_model_name_or_path, with from_pretrained() to load the other components of the pipeline such as tokenizers and the scheduler.

   Thanks, I understand it, from_pretrained() with args.pretrained_model_name_or_path is to load the other components and the finetuned model can be directly loaded with corresponding argument.
   for example,

   pipeline = StableDiffusion3Pipeline.from_pretrained(
   args.pretrained_model_name_or_path,
   vae=vae,
   text_encoder=accelerator.unwrap_model(text_encoder_one),
   text_encoder_2=accelerator.unwrap_model(text_encoder_two),
   text_encoder_3=accelerator.unwrap_model(text_encoder_three),
   transformer=accelerator.unwrap_model(transformer),
   revision=args.revision,
   variant=args.variant,
   torch_dtype=weight_dtype,
   )
   in the above code, vae maybe the non fine-tuned components which is "other components" stated above, text_encoder and transformer maybe the fine-tuned components and can be loaded with trained model argument that maybe fine-tuned" accelerator.unwrap_model(text_encoder_one)" and "accelerator.unwrap_model(transformer)" here?
   I do not know if is it right?
   Another problem, if I load the pretrained components with the given pipline that maybe StableDiffusion3Pipeline here, but one component is modified (such as the input channel is changed) and I saved components in a new folder. In this case, should I create a new pipline as the modified component is not appropriate with the previous pipline?
4. sayakpaul commented on Jul 24, 2025

   

   sayakpaul

   on Jul 24, 2025

   Member

   More actions

   accelerator.unwrap_model is a utility that we use to ensure the corresponding model isn't wrapped with any distributed utilities. So, it's safe.

   In this case, should I create a new pipline as the modified component is not appropriate with the previous pipline?

   You can just save the component and use the pipeline like so:

   finetuned_component = AutoModel.from_pretrained("your-finetuned-directory")
   pipe = DiffusionPipeline(pretrained_ckpt_id, transformer=finetuned_component)
5. micklexqg commented on Jul 24, 2025

   

   micklexqg

   on Jul 24, 2025

   Author

   More actions

   accelerator.unwrap_model is a utility that we use to ensure the corresponding model isn't wrapped with any distributed utilities. So, it's safe.

   In this case, should I create a new pipline as the modified component is not appropriate with the previous pipline?

   You can just save the component and use the pipeline like so:

   finetuned_component = AutoModel.from_pretrained("your-finetuned-directory")
   pipe = DiffusionPipeline(pretrained_ckpt_id, transformer=finetuned_component)

   Thanks, as some core processings need to be modified, I copied the previous pipeline which maybe DiffusionPipline here and renamed it, but i can not be imported. I also try to import it manually with the relative path, but still failed, so how to import a newly custom pipeline？
6. github-actions commented on Jan 9, 2026

   

   github-actionsBot

   on Jan 9, 2026 – with GitHub Actions

   Contributor

   More actions

   This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

   Please note that issues that do not follow the contributing guidelines are likely to be ignored.
7. github-actions

   added

   staleIssues that haven't received updatesIssues that haven't received updates

   on Jan 9, 2026
8. huggingface

   locked and limited conversation to collaborators on Jan 9, 2026
9. yiyixuxu

   converted this issue into a discussion #12940 on Jan 9, 2026