From 37e47fb024d2aa063edb747768ddc07e63bfbecc Mon Sep 17 00:00:00 2001 From: stephen37 Date: Fri, 16 Oct 2026 01:05:35 +0200 Subject: [PATCH 1/4] docs(flux3): the prompt upsampler is required, ours or any LLM --- docs/source/en/api/pipelines/flux3.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index b33b86c479ed..f8c3244a84fc 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -16,7 +16,7 @@ FLUX.3 is an image generation model from Black Forest Labs: a single transformer ([`Flux3Transformer2DModel`]) generates images from a text prompt, alone or together with one or more reference images. Text conditioning comes from Qwen3-VL, and images use the FLUX.2 packed autoencoder ([`AutoencoderKLFlux2`]). -FLUX.3 ships as a modular pipeline. Prompting matters: the model responds best to full scene captions that describe the subject, composition, lighting, and atmosphere. Black Forest Labs provides a [prompt upsampler](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup) that expands short prompts automatically; see [Prompt upsampling](#prompt-upsampling). Get started with a text-to-image generation: +FLUX.3 ships as a modular pipeline. It takes full scene captions, not short prompts: a request sent as typed gives generic output. Run the [prompt upsampler](#prompt-upsampling) on every request, text-to-image and editing alike, or write the caption with any LLM using the system prompts in [`system_prompts.json`](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup/blob/main/system_prompts.json). The pipeline does not run an upsampler on its own; the example below passes a full caption: ```python import torch @@ -44,7 +44,7 @@ image.save("fox.png") ## Prompt upsampling -The prompt upsampler, [black-forest-labs/flux-3-image-klein-pup](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup), turns the user prompt, together with the reference images when editing, into a full scene caption. It is a separate model; run it with Transformers and pass its output as `prompt`: +The prompt upsampler, [black-forest-labs/flux-3-image-klein-pup](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup), turns a short request, together with the reference images when editing, into the full scene caption the model takes. It is a separate model; run it with Transformers and pass its output as `prompt`: ```python import json @@ -88,9 +88,11 @@ image = pipe(prompt=upsample("An astronaut riding a horse"), output="images")[0] For an edit, pass the same reference images to both: `pipe(prompt=upsample(instruction, [image]), image=image, ...)`. The upsampler also writes the [layout](#layout-control) captions from a list of boxes; see its model card. +Any other LLM can take its place: give it `system_prompts["t2i"]` (or `"i2i"` with the references) as the system prompt and the request as the user message, and pass the caption it returns as `prompt`. + ## Image-to-image -Pass one or more reference images with `image`. Every reference keeps its own resolution (center-cropped to the 16-pixel grid; references above 16 megapixels are downscaled first), and the generation size defaults to one megapixel at the first reference's aspect ratio. Describe the *target* image as a complete caption of the reference with your edit folded in, rather than a one-line instruction, or let the [prompt upsampler](#prompt-upsampling) write it from the instruction and the references: +Pass one or more reference images with `image`. Every reference keeps its own resolution (center-cropped to the 16-pixel grid; references above 16 megapixels are downscaled first), and the generation size defaults to one megapixel at the first reference's aspect ratio. The prompt is the *target* image as a complete caption of the reference with the edit folded in, not a one-line instruction; the [prompt upsampler](#prompt-upsampling) writes it from the instruction and the references: ```python night = pipe( From ae8f61ba8496ad9c1d98c7453e3cfc39f689dad8 Mon Sep 17 00:00:00 2001 From: stephen37 Date: Fri, 16 Oct 2026 01:09:29 +0200 Subject: [PATCH 2/4] docs(flux3): every example goes through upsample() --- docs/source/en/api/pipelines/flux3.md | 70 +++++++++------------------ 1 file changed, 24 insertions(+), 46 deletions(-) diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index f8c3244a84fc..3e362fc5b999 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -16,48 +16,24 @@ FLUX.3 is an image generation model from Black Forest Labs: a single transformer ([`Flux3Transformer2DModel`]) generates images from a text prompt, alone or together with one or more reference images. Text conditioning comes from Qwen3-VL, and images use the FLUX.2 packed autoencoder ([`AutoencoderKLFlux2`]). -FLUX.3 ships as a modular pipeline. It takes full scene captions, not short prompts: a request sent as typed gives generic output. Run the [prompt upsampler](#prompt-upsampling) on every request, text-to-image and editing alike, or write the caption with any LLM using the system prompts in [`system_prompts.json`](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup/blob/main/system_prompts.json). The pipeline does not run an upsampler on its own; the example below passes a full caption: - -```python -import torch -from diffusers import ModularPipeline - -pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein-base") -pipe.load_components(dtype={"vae": torch.float32, "default": torch.bfloat16}) -pipe.to("cuda") - -image = pipe( - prompt=( - "A wide-angle wildlife photograph of a red fox standing alert in a snow-covered forest " - "clearing at golden hour. The fox is centered in the frame, its thick orange winter coat " - "contrasting with the fresh white snow, breath visible in the cold air. Snow-laden pine " - "trees fill the softly blurred background, and low warm sunlight rakes across the scene " - "from the left, casting long blue shadows." - ), - generator=torch.Generator().manual_seed(0), - output="images", -)[0] -image.save("fox.png") -``` - -`prompt` is the only required input: the pipeline carries the recommended sampling settings (50 steps, guidance scale 3.5, scheduler shift 6.93) and generates 1024 x 1024 images by default. It takes one prompt per call; use `num_images_per_prompt` for several images of the same prompt. - -## Prompt upsampling - -The prompt upsampler, [black-forest-labs/flux-3-image-klein-pup](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup), turns a short request, together with the reference images when editing, into the full scene caption the model takes. It is a separate model; run it with Transformers and pass its output as `prompt`: +FLUX.3 ships as a modular pipeline. It takes full scene captions, not short prompts: a request sent as typed gives generic output. Every request on this page goes through the [prompt upsampler](#prompt-upsampling) first, a separate 9B model ([black-forest-labs/flux-3-image-klein-pup](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup)) that the pipeline does not run on its own. Load it beside the pipeline, then generate: ```python import json import torch +from diffusers import ModularPipeline from huggingface_hub import hf_hub_download from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration +pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein-base") +pipe.load_components(dtype={"vae": torch.float32, "default": torch.bfloat16}) +pipe.to("cuda") -model_id = "black-forest-labs/flux-3-image-klein-pup" -processor = AutoProcessor.from_pretrained(model_id) -upsampler = Qwen3_5ForConditionalGeneration.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto").eval() -system_prompts = json.load(open(hf_hub_download(model_id, "system_prompts.json"))) +pup_id = "black-forest-labs/flux-3-image-klein-pup" +processor = AutoProcessor.from_pretrained(pup_id) +upsampler = Qwen3_5ForConditionalGeneration.from_pretrained(pup_id, dtype=torch.bfloat16, device_map="auto").eval() +system_prompts = json.load(open(hf_hub_download(pup_id, "system_prompts.json"))) def upsample(prompt, images=()): @@ -83,10 +59,16 @@ def upsample(prompt, images=()): return processor.decode(output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True).strip() -image = pipe(prompt=upsample("An astronaut riding a horse"), output="images")[0] +caption = upsample("A red fox in a snow-covered forest clearing at golden hour") +image = pipe(prompt=caption, generator=torch.Generator().manual_seed(0), output="images")[0] +image.save("fox.png") ``` -For an edit, pass the same reference images to both: `pipe(prompt=upsample(instruction, [image]), image=image, ...)`. The upsampler also writes the [layout](#layout-control) captions from a list of boxes; see its model card. +`prompt` is the only required input of the pipeline: it carries the recommended sampling settings (50 steps, guidance scale 3.5, scheduler shift 6.93) and generates 1024 x 1024 images by default. It takes one prompt per call; use `num_images_per_prompt` for several images of the same prompt. + +## Prompt upsampling + +The prompt upsampler turns a short request, together with the reference images when editing, into the full scene caption the model takes: medium and style, scene, lighting and camera, each subject with its colors, materials and position in the frame. `upsample` above runs it with Transformers: `system_prompts["t2i"]` for text-to-image, `system_prompts["i2i"]` with the references first in the user message for an edit, greedy decoding, thinking off. The [model card](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup) has the decoding settings and the layout form (`t2i_bbox`, `i2i_bbox`), which keeps a list of boxes you pass as a box table after the caption. Any other LLM can take its place: give it `system_prompts["t2i"]` (or `"i2i"` with the references) as the system prompt and the request as the user message, and pass the caption it returns as `prompt`. @@ -96,22 +78,18 @@ Pass one or more reference images with `image`. Every reference keeps its own re ```python night = pipe( - prompt=( - "The same red fox in the same snow-covered forest clearing, now at night under a full moon: " - "cool blue moonlight, deep shadows, stars visible through the pine branches, the fox's eyes " - "catching the light." - ), + prompt=upsample("the same scene at night under a full moon", [image]), image=image, generator=torch.Generator().manual_seed(0), output="images", )[0] ``` -With several references, refer to them in the prompt as ``, ``, ... in the order they are passed to `image`: +With several references, name them by position in the request, the first image, the second image, ... in the order they are passed to both `upsample` and `image`: ```python combined = pipe( - prompt="The fox from sitting on the wooden bench from , in the same snowy clearing.", + prompt=upsample("put the fox from the first image on the wooden bench from the second image", [image, bench]), image=[image, bench], height=768, width=1360, @@ -123,7 +101,7 @@ Pass `height` and/or `width` to override the derived size. ## Layout control -To place content explicitly, tag regions in the caption with `` and append a JSON list of regions to the prompt. Each region has an `id`, a `bbox` of `[y0, x0, y1, x1]` (top-left and bottom-right corners, origin top-left) normalized to 0-1000 on both axes so it is independent of the output size, and a `desc`: +To place content explicitly, the caption tags regions with `` and ends with a JSON list of regions. Each region has an `id`, a `bbox` of `[y0, x0, y1, x1]` (top-left and bottom-right corners, origin top-left) normalized to 0-1000 on both axes so it is independent of the output size, and a `desc`. This is the form the upsampler writes with its `t2i_bbox` and `i2i_bbox` system prompts from a request and a list of boxes (scaffold on its [model card](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup)); the prompts below are that output, passed as `prompt`: ```python prompt = ( @@ -167,7 +145,7 @@ pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein") pipe.load_components(dtype=torch.bfloat16) pipe.to("cuda") -image = pipe(prompt=prompt, generator=torch.Generator().manual_seed(0), output="images")[0] +image = pipe(prompt=upsample("An astronaut riding a horse"), generator=torch.Generator().manual_seed(0), output="images")[0] ``` It serves the same two workflows as the base model. `num_inference_steps` defaults to 16; the model is trained for 4 to 16 steps. @@ -183,7 +161,7 @@ h, w = height // 16, width // 16 noise = torch.randn((h * w, 128), generator=torch.Generator().manual_seed(0)).to(torch.bfloat16) latents = noise.T.reshape(1, 128, h, w) -image = pipe(prompt=prompt, latents=latents, height=height, width=width, output="images")[0] +image = pipe(prompt=caption, latents=latents, height=height, width=width, output="images")[0] ``` ## Workflows @@ -202,7 +180,7 @@ print(t2i.blocks.doc) ## Memory -Loading everything in bf16 takes about 20 GB of weights (the transformer is 5.5B parameters, the Qwen3-VL-4B text encoder 4B), with the image VAE in fp32. For GPUs with less memory, enable component offloading: +Loading everything in bf16 takes about 20 GB of weights (the transformer is 5.5B parameters, the Qwen3-VL-4B text encoder 4B), with the image VAE in fp32; the prompt upsampler is another 18.8 GB, placed by `device_map="auto"`. For GPUs with less memory, enable component offloading: ```python pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein-base") From c77762162d9843dfa44b1093abafb89fb07d779d Mon Sep 17 00:00:00 2001 From: stephen37 Date: Fri, 16 Oct 2026 02:07:48 +0200 Subject: [PATCH 3/4] docs(flux3): strongly advise the prompt upsampler --- docs/source/en/api/pipelines/flux3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index 3e362fc5b999..a9625fd6393f 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -16,7 +16,7 @@ FLUX.3 is an image generation model from Black Forest Labs: a single transformer ([`Flux3Transformer2DModel`]) generates images from a text prompt, alone or together with one or more reference images. Text conditioning comes from Qwen3-VL, and images use the FLUX.2 packed autoencoder ([`AutoencoderKLFlux2`]). -FLUX.3 ships as a modular pipeline. It takes full scene captions, not short prompts: a request sent as typed gives generic output. Every request on this page goes through the [prompt upsampler](#prompt-upsampling) first, a separate 9B model ([black-forest-labs/flux-3-image-klein-pup](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup)) that the pipeline does not run on its own. Load it beside the pipeline, then generate: +FLUX.3 ships as a modular pipeline. It takes full scene captions, not short prompts: a request sent as typed gives generic output. We strongly advise running the [prompt upsampler](#prompt-upsampling) on every request, text-to-image and editing alike, or any LLM with the same system prompts; every example on this page does. The upsampler is a separate 9B model ([black-forest-labs/flux-3-image-klein-pup](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup)) that the pipeline does not run on its own. Load it beside the pipeline, then generate: ```python import json From 2020b5cf1c44a8f48bfbd9df445e29fa83fc8e16 Mon Sep 17 00:00:00 2001 From: stephen37 Date: Fri, 16 Oct 2026 02:10:56 +0200 Subject: [PATCH 4/4] docs(flux3): the model is FLUX 3 Image Klein --- docs/source/en/api/pipelines/flux3.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index a9625fd6393f..2a99edd44fa1 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -12,11 +12,11 @@ # See the License for the specific language governing permissions and # limitations under the License. --> -# FLUX.3 +# FLUX 3 Image Klein -FLUX.3 is an image generation model from Black Forest Labs: a single transformer ([`Flux3Transformer2DModel`]) generates images from a text prompt, alone or together with one or more reference images. Text conditioning comes from Qwen3-VL, and images use the FLUX.2 packed autoencoder ([`AutoencoderKLFlux2`]). +FLUX 3 Image Klein is a 5.6B image generation model from Black Forest Labs: a single transformer ([`Flux3Transformer2DModel`]) generates images from a text prompt, alone or together with one or more reference images. Text conditioning comes from Qwen3-VL, and images use the FLUX.2 packed autoencoder ([`AutoencoderKLFlux2`]). -FLUX.3 ships as a modular pipeline. It takes full scene captions, not short prompts: a request sent as typed gives generic output. We strongly advise running the [prompt upsampler](#prompt-upsampling) on every request, text-to-image and editing alike, or any LLM with the same system prompts; every example on this page does. The upsampler is a separate 9B model ([black-forest-labs/flux-3-image-klein-pup](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup)) that the pipeline does not run on its own. Load it beside the pipeline, then generate: +FLUX 3 Image Klein ships as a modular pipeline. It takes full scene captions, not short prompts: a request sent as typed gives generic output. We strongly advise running the [prompt upsampler](#prompt-upsampling) on every request, text-to-image and editing alike, or any LLM with the same system prompts; every example on this page does. The upsampler is a separate 9B model ([black-forest-labs/flux-3-image-klein-pup](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup)) that the pipeline does not run on its own. Load it beside the pipeline, then generate: ```python import json