Can't improve rotation error when training with my custom data

[CongDuy038]

I'm having a problem with training on my custom dataset. I use the weight pretrained file efficientdet-d2.h5
It seems that the translation error has been improved very well. However, the rotation error is quite high and does not decrease. Here is the output of the epochs I train:

mAP: 1.0000
ADD: 0.7526
ADD-S: 0.7555
5cm_5degree: 0.0000
TranslationErrorMean_in_mm: 9.8703
TranslationErrorStd_in_mm: 5.0323
RotationErrorMean_in_degree: 111.7360
RotationErrorStd_in_degree: 36.6118
2D-Projection: 0.1885
Summed_Translation_Rotation_Error: 163.2504
ADD(-S): 0.7526
AveragePointDistanceMean_in_mm: 9.8673
AveragePointDistanceStd_in_mm: 5.0325
AverageSymmetricPointDistanceMean_in_mm: 9.8284
AverageSymmetricPointDistanceStd_in_mm: 5.0331
MixedAveragePointDistanceMean_in_mm: 9.8673
MixedAveragePointDistanceStd_in_mm: 5.0325
Epoch 00001: ADD improved from -inf to 0.75258, saving model to checkpoints/02_10_2022_12_17_12/object_2/phi_0_linemod_best_ADD.h5
1000/1000 [==============================] - 1124s 1s/step - loss: 1.5798 - classification_loss: 0.6198 - regression_loss: 0.0557 - transformation_loss: 45.2144
Epoch 2/4
Running network: 100% (679 of 679) |######| Elapsed Time: 0:00:36 Time: 0:00:36
Parsing annotations: 100% (679 of 679) |##| Elapsed Time: 0:00:00 Time: 0:00:00
100% 679/679 [00:08<00:00, 77.82it/s]
679 instances of class object with average precision: 1.0000
679 instances of class object with ADD accuracy: 0.1532
679 instances of class object with ADD-S-Accuracy: 0.1561
679 instances of class object with 5cm-5degree-Accuracy: 0.0000
class object with Translation Differences in mm: Mean: 19.1808 and Std: 6.8850
class object with Rotation Differences in degree: Mean: 127.1700 and Std: 33.4669
679 instances of class object with 2d-projection-Accuracy: 0.1915
679 instances of class object with ADD(-S)-Accuracy: 0.1532
class object with Transformed Point Distances in mm: Mean: 19.1779 and Std: 6.8854
class object with Transformed Symmetric Point Distances in mm: Mean: 19.1396 and Std: 6.8854
class object with Mixed Transformed Point Distances in mm: Mean: 19.1779 and Std: 6.8854
mAP: 1.0000
ADD: 0.1532
ADD-S: 0.1561
5cm_5degree: 0.0000
TranslationErrorMean_in_mm: 19.1808
TranslationErrorStd_in_mm: 6.8850
RotationErrorMean_in_degree: 127.1700
RotationErrorStd_in_degree: 33.4669
2D-Projection: 0.1915
Summed_Translation_Rotation_Error: 186.7028
ADD(-S): 0.1532
AveragePointDistanceMean_in_mm: 19.1779
AveragePointDistanceStd_in_mm: 6.8854
AverageSymmetricPointDistanceMean_in_mm: 19.1396
AverageSymmetricPointDistanceStd_in_mm: 6.8854
MixedAveragePointDistanceMean_in_mm: 19.1779
MixedAveragePointDistanceStd_in_mm: 6.8854

Epoch 00002: ADD did not improve from 0.75258
1000/1000 [==============================] - 318s 318ms/step - loss: 0.5754 - classification_loss: 0.0534 - regression_loss: 0.0288 - transformation_loss: 24.6600
Epoch 3/4
Running network: 100% (679 of 679) |######| Elapsed Time: 0:00:36 Time: 0:00:36
Parsing annotations: 100% (679 of 679) |##| Elapsed Time: 0:00:00 Time: 0:00:00
100% 679/679 [00:08<00:00, 80.08it/s]
679 instances of class object with average precision: 1.0000
679 instances of class object with ADD accuracy: 0.3888
679 instances of class object with ADD-S-Accuracy: 0.3932
679 instances of class object with 5cm-5degree-Accuracy: 0.0000
class object with Translation Differences in mm: Mean: 13.7990 and Std: 4.9412
class object with Rotation Differences in degree: Mean: 115.2414 and Std: 40.1505
679 instances of class object with 2d-projection-Accuracy: 0.2018
679 instances of class object with ADD(-S)-Accuracy: 0.3888
class object with Transformed Point Distances in mm: Mean: 13.7954 and Std: 4.9409
class object with Transformed Symmetric Point Distances in mm: Mean: 13.7568 and Std: 4.9408
class object with Mixed Transformed Point Distances in mm: Mean: 13.7954 and Std: 4.9409
mAP: 1.0000
ADD: 0.3888
ADD-S: 0.3932
5cm_5degree: 0.0000
TranslationErrorMean_in_mm: 13.7990
TranslationErrorStd_in_mm: 4.9412
RotationErrorMean_in_degree: 115.2414
RotationErrorStd_in_degree: 40.1505
2D-Projection: 0.2018
Summed_Translation_Rotation_Error: 174.1321
ADD(-S): 0.3888
AveragePointDistanceMean_in_mm: 13.7954
AveragePointDistanceStd_in_mm: 4.9409
AverageSymmetricPointDistanceMean_in_mm: 13.7568
AverageSymmetricPointDistanceStd_in_mm: 4.9408
MixedAveragePointDistanceMean_in_mm: 13.7954
MixedAveragePointDistanceStd_in_mm: 4.9409

Epoch 00003: ADD did not improve from 0.75258
1000/1000 [==============================] - 313s 313ms/step - loss: 0.5011 - classification_loss: 0.0364 - regression_loss: 0.0232 - transformation_loss: 22.0763
Epoch 4/4
Running network: 100% (679 of 679) |######| Elapsed Time: 0:00:37 Time: 0:00:37
Parsing annotations: 100% (679 of 679) |##| Elapsed Time: 0:00:00 Time: 0:00:00
100% 679/679 [00:08<00:00, 77.52it/s]
679 instances of class object with average precision: 1.0000
679 instances of class object with ADD accuracy: 0.5626
679 instances of class object with ADD-S-Accuracy: 0.5700
679 instances of class object with 5cm-5degree-Accuracy: 0.0000
class object with Translation Differences in mm: Mean: 12.2080 and Std: 6.1116
class object with Rotation Differences in degree: Mean: 130.5616 and Std: 35.5283
679 instances of class object with 2d-projection-Accuracy: 0.4227
679 instances of class object with ADD(-S)-Accuracy: 0.5626
class object with Transformed Point Distances in mm: Mean: 12.2090 and Std: 6.1138
class object with Transformed Symmetric Point Distances in mm: Mean: 12.1643 and Std: 6.1119
class object with Mixed Transformed Point Distances in mm: Mean: 12.2090 and Std: 6.1138
mAP: 1.0000
ADD: 0.5626
ADD-S: 0.5700
5cm_5degree: 0.0000
TranslationErrorMean_in_mm: 12.2080
TranslationErrorStd_in_mm: 6.1116
RotationErrorMean_in_degree: 130.5616
RotationErrorStd_in_degree: 35.5283
2D-Projection: 0.4227
Summed_Translation_Rotation_Error: 184.4095
ADD(-S): 0.5626
AveragePointDistanceMean_in_mm: 12.2090
AveragePointDistanceStd_in_mm: 6.1138
AverageSymmetricPointDistanceMean_in_mm: 12.1643
AverageSymmetricPointDistanceStd_in_mm: 6.1119
MixedAveragePointDistanceMean_in_mm: 12.2090
MixedAveragePointDistanceStd_in_mm: 6.1138

Epoch 00004: ADD did not improve from 0.75258
1000/1000 [==============================] - 317s 317ms/step - loss: 0.4195 - classification_loss: 0.0321 - regression_loss: 0.0201 - transformation_loss: 18.3656

==================================================================================

This is the image when I run the evaluate. It detects objects well, but the rotation doesn't seem to match the ground truth box.

Tell me how I can reduce the rotation error!

[8umpk1n]

hello! I want to ask some questions about the production of datasets, can you give me some help?

[Hijsm]

Are you sure your annotation data is alright? The ground truth in your images does not seem well fitted around your object, nor is it consistently off. I think it would help a lot if you had the right dataset.

[madhanuman]

Were you able to solve this @CongDuy038? I am standing infront of the exact same problem and i can not see where my mistake is...

[SMhang]

May I ask if you have resolved it? I have encountered the same problem as you