[Question] image_features not matched to input text

[sibosutd]

LLaVA-UHD/llava_uhd/train/llava-uhd/adapt_llava.py

Lines 169 to 173 in 69e75d0

	if i < num_images:
	for j in range(5):
	cur_image_features = image_features[cur_image_idx+j*16]
	cur_new_input_embeds.append(cur_image_features)
	cur_new_labels.append(torch.full((cur_image_features.shape[0],), IGNORE_INDEX, device=cur_labels.device, dtype=cur_labels.dtype))

1. In the code snippet above, I notice that the value of cur_image_idx doesn't change within a single batch. This implies that cur_image_features remain identical for images within the same batch, which seems unusual. Could you confirm if this is the intended behavior?

2. Another point of confusion I have pertains to the line for j in range(5): and the expression j*16. Based on the settings used in the Resampler, I would expect the image_features to have dimensions [batch_size*8, 64, 5120]. Can you clarify why the image features are selected using for j in range(5): and j*16?