When used for instance segmentation image rotation, points beyond the image boundary are not handled accurately

[sunstudio]

Describe the bug

I am doing instance segmentation image enhancement, and there is a polygon object. For the convenience of discussion, it is assumed to be a 4-sided polygon. After rotation, if one of the points exceeds the image boundary, the polygon in the target image becomes 3 points, which will reduce a large part of the target object area.

Screenshots

In the left picture, the blue is the mask and the black rectangle is the canvas. In the right picture, after the augmentation rotation, the lower right corner of the mask rotates outside the canvas and is lost. In this case, the correct mask should be all the blue areas in the canvas, but the algorithm simply deletes the points outside the canvas and obtains the triangular area in the picture.

[ternaus]

Could you please provide the minimal code that reproduces the issue?

[ternaus]

Does using mask to represent the polygon will get the desired bahavior?