About Batch size set

[zyandtom]

hi, looks like this method does not support batch_size > 1? because if we use position_id to compute pe, there will have a shape not match error in torch.outer

[OpenJarvisAI]

If not support batchsize>1, how does InternVL3 trained?

[zyandtom]

If not support batchsize>1, how does InternVL3 trained?

probably they have adapted it, the code here seems hard coded...

[Lechatelia]

This behavior is due to the use of a packing strategy for input data in this codebase, where the traditional batch_size dimension is effectively reduced to 1.

If you're aiming to use InternVL3, it's worth noting that in most tasks—especially those with context lengths not exceeding 32K—there is typically no need to use V2PE. In such cases, you can simply use the InternVL2.5 codebase for training without modification.

[zyandtom]

This behavior is due to the use of a packing strategy for input data in this codebase, where the traditional batch_size dimension is effectively reduced to 1.

If you're aiming to use InternVL3, it's worth noting that in most tasks—especially those with context lengths not exceeding 32K—there is typically no need to use V2PE. In such cases, you can simply use the InternVL2.5 codebase for training without modification.

thank you for reply!