Can Paged Attention be used in training to reduce memory usage?

[CACACAZ]

During the training of LLM, sequences in a batch are typically padded to the same length. Additional memory and computational waste are introduced by these paddings. If used in training, could paged attention reduce memory usage and thus allow for a bigger batch size?