Skip to content

Latest commit

 

History

History
2687 lines (2600 loc) · 267 KB

askubuntu.md

File metadata and controls

2687 lines (2600 loc) · 267 KB
Raw

Results

Train/Test with Complete • Test with Incomplete • Train/Test with Incomplete

Complete

  • Epoch 3
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 71.56 73.39 65.14 70.64 58.72 61.47 63.30 64.22 60.55 66.97 65.60 4.70
BERT-bs4+Dropout0.1 71.56 60.55 72.48 53.21 60.55 66.97 61.47 65.14 65.14 66.06 64.31 5.39
EmbraceBERT-bs4-WRONG 68.81 40.37 68.81 71.56 57.80 72.48 72.48 63.30 63.30 51.38 63.03 9.97
EmbraceBERT-bs4+Dropout0.1-WRONG
EmbraceBERT-bs4+Dropout0.3-WRONG
CondensedEmbraceBERT-bs4 70.64 49.54 73.39 61.47 60.55 72.48 76.15 65.14 68.81 58.72 65.69 7.77
CondensedEmbraceBERT-bs4+Dropout0.1 68.81 47.71 66.06 48.62 48.62 67.89 64.22 67.89 55.96 45.87 58.17 9.21
CondensedEmbraceBERT-bs4+Dropout0.3
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 87.16 88.99 84.40 86.24 88.99 88.99 81.65 84.40 90.83 86.24 86.79 2.63
FrozenEBERT-bs4-ep100-WRONG 81.65 75.23 81.65 88.99 81.65 86.24 86.24 77.98 84.40 86.24 83.03 4.00
FrozenEBERT-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 64.22 77.06 56.88 56.88 38.53 52.29 45.87 61.47 55.05 62.39 57.06 9.97
BERT-bs16+Dropout0.1 60.55 51.38 56.88 52.29 32.11 44.95 33.94 55.96 58.72 53.21 50.00 9.44
EmbraceBERT-bs16-WRONG 67.89 46.79 56.88 41.28 43.12 57.80 57.80 41.28 40.37 36.70 48.99 9.79
EmbraceBERT-bs16+Dropout0.1-WRONG 66.97 42.20 54.13 44.04 39.45 53.21 61.47 44.95 39.45 38.53 48.44 9.48
EmbraceBERT-bs16+Dropout0.3-WRONG 67.89 43.12 56.88 43.12 38.53 52.29 55.96 43.12 41.28 37.61 47.98 9.33
CondensedEmbraceBERT-bs16 63.30 44.95 55.96 44.04 39.45 58.72 47.71 47.71 40.37 39.45 48.17 8.01
CondensedEmbraceBERT-bs16+Dropout0.1 66.06 45.87 50.46 33.94 40.37 59.63 51.38 45.87 37.61 36.70 46.79 9.82
CondensedEmbraceBERT-bs16+Dropout0.3
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 78.90 80.73 83.49 69.72 62.39 62.39 68.81 70.64 71.56 74.31 72.29 6.80
FrozenEBERT-bs16-ep100-WRONG 76.15 63.30 66.97 68.81 70.64 68.81 77.98 70.64 49.54 64.22 67.71 7.47
FrozenEBERT-bs16-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 82.57 87.16 36.70 74.31 48.62 72.48 68.81 68.81 42.20 36.70 61.83 18.06
RoBERTa-bs4+Dropout0.1 61.47 12.84 36.70 74.31 36.70 78.90 56.88 46.79 36.70 36.70 47.80 19.12
EmbraceRoBERTa-bs4-WRONG 36.70 36.70 36.70 26.61 36.70 36.70 77.06 36.70 36.70 36.70 39.72 12.80
EmbraceRoBERTa-bs4+Dropout0.1-WRONG 36.70 36.70 36.70 35.78 36.70 36.70 49.54 36.70 36.70 36.70 37.89 3.89
EmbraceRoBERTa-bs4+Dropout0.3-WRONG 36.70 36.70 36.70 12.84 36.70 36.70 44.95 36.70 36.70 36.70 35.14 7.83
CondensedEmbraceRoBERTa-bs4 36.70 39.45 36.70 40.37 36.70 36.70 73.39 43.12 36.70 36.70 41.65 10.79
CondensedEmbraceRoBERTa-bs4+Dropout0.1 36.70 36.70 36.70 40.37 36.70 36.70 39.45 37.61 36.70 37.61 37.52 1.26
CondensedEmbraceRoBERTa-bs4+Dropout0.3
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 89.91 88.99 90.83 88.99 87.16 91.74 88.07 88.07 88.99 90.83 89.36 1.37
FrozenERoBERTa-bs4-ep100-WRONG 86.24 91.74 89.91 80.73 92.66 75.23 88.99 88.99 90.83 65.14 85.05 8.39
FrozenERoBERTa-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 37.61 22.94 36.70 53.21 36.70 52.29 36.70 12.84 37.61 36.70 36.33 11.29
RoBERTa-bs16+Dropout0.1 36.70 17.43 36.70 58.72 36.70 61.47 37.61 12.84 36.70 36.70 37.16 14.27
EmbraceRoBERTa-bs16-WRONG 35.78 36.70 36.70 35.78 36.70 36.70 36.70 36.70 36.70 36.70 36.51 0.37
EmbraceRoBERTa-bs16+Dropout0.1-WRONG 35.78 36.70 36.70 35.78 36.70 36.70 36.70 36.70 36.70 34.86 36.33 0.61
EmbraceRoBERTa-bs16+Dropout0.3-WRONG 34.86 36.70 36.70 35.78 36.70 36.70 36.70 36.70 36.70 34.86 36.24 0.74
CondensedEmbraceRoBERTa-bs16 36.70 41.28 36.70 26.61 36.70 36.70 37.61 36.70 34.86 48.62 37.25 5.16
CondensedEmbraceRoBERTa-bs16+Dropout0.1 35.78 39.45 36.70 36.70 35.78 36.70 37.61 36.70 34.86 37.61 36.79 1.19
CondensedEmbraceRoBERTa-bs16+Dropout0.3
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 81.65 77.06 36.70 86.24 77.06 87.16 76.15 66.97 57.80 74.31 72.11 14.40
FrozenERoBERTa-bs16-ep100-WRONG 36.70 37.61 36.70 37.61 36.70 57.80 37.61 66.06 36.70 36.70 42.02 10.13
FrozenERoBERTa-bs16-ep100+Dropout0.1
  • Epoch 30
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 88.99 89.91 92.66 91.74 89.91 89.91 89.91 88.99 87.16 90.83 90.00 1.45
BERT-bs4+Dropout0.1 89.91 88.99 91.74 91.74 90.83 89.91 90.83 89.91 85.32 89.91 89.91 1.74
BERT-bs4+Dropout0.3 89.91 88.07 92.66 91.74 90.83 91.74 90.83 89.91 88.99 88.99 90.37 1.38
EmbraceBERT-bs4 89.91 89.91 88.99 91.74 88.99 85.32 89.91 88.99 90.83 88.07 89.27 1.64
EmbraceBERT-bs4+Dropout0.1 90.83 87.16 90.83 89.91 91.74 88.07 88.99 88.99 89.91 91.74 89.82 1.45
EmbraceBERT-bs4+Dropout0.3 90.83 87.16 87.16 89.91 89.91 90.83 88.07 90.83 88.99 88.07 89.17 1.41
EmbraceBERT-bs4+Branches
CondensedEmbraceBERT-bs4 88.99 88.99 90.83 90.83 88.07 88.99 91.74 89.91 89.91 88.07 89.63 1.16
CondensedEmbraceBERT-bs4+Dropout0.1 90.83 87.16 90.83 91.74 89.91 88.99 88.07 91.74 89.91 88.99 89.82 1.45
CondensedEmbraceBERT-bs4+Dropout0.3 89.91 88.07 88.07 89.91 89.91 88.07 90.83 90.83 90.83 88.07 89.45 1.18
CondensedEmbraceBERT-bs4+Branches
CondensedEmbraceBERT-bs4+Branches+Dropout0.1
CondensedEmbraceBERT-bs4+Branches+Dropout0.3
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 88.07 88.99 91.74 92.66 88.99 89.91 91.74 90.83 87.16 91.74 90.18 1.74
FrozenEBERT-bs4-ep100 91.74 90.83 89.91 91.74 88.07 87.16 88.99 88.99 89.91 87.16 89.45 1.60
FrozenEBERT-bs4-ep100+Dropout0.1 90.83 86.24 88.07 91.74 88.07 88.07 89.91 89.91 89.91 87.16 88.99 1.64
FrozenEBERT-bs4-ep100+Dropout0.3 90.83 87.16 88.99 89.91 88.99 88.99 87.16 91.74 88.99 85.32 88.81 1.78
FrozenEBERT-bs4-ep100+Branches
FrozenCEBERT-bs4-ep100 89.91 93.58 89.91 93.58 88.99 88.99 91.74 91.74 90.83 89.91 90.92 1.61
FrozenCEBERT-bs4-ep100+Dropout0.1 88.07 88.07 89.91 89.91 89.91 88.07 89.91 89.91 89.91 88.99 89.27 0.83
FrozenCEBERT-bs4-ep100+Dropout0.3 89.91 88.07 86.24 91.74 89.91 88.99 90.83 91.74 89.91 88.07 89.54 1.65
FrozenCEBERT-bs4-ep100+Branches 90.83 89.91 87.16 90.83 92.66 86.24 90.83 90.83 90.83 89.91 90.00 1.81
FrozenCEBERT-bs4-ep100+Branches+Dropout0.1
FrozenCEBERT-bs4-ep100+Branches+Dropout0.3
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 88.99 89.91 92.66 90.83 90.83 88.99 89.91 89.91 87.16 88.07 89.72 1.47
BERT-bs16+Dropout0.1 89.91 89.91 91.74 90.83 91.74 89.91 91.74 90.83 88.07 91.74 90.64 1.15
BERT-bs16+Dropout0.3 86.24 89.91 91.74 90.83 91.74 89.91 91.74 88.07 88.99 88.07 89.72 1.78
EmbraceBERT-bs16 90.83 88.07 83.49 89.91 88.99 88.99 90.83 89.91 89.91 88.07 88.90 2.03
EmbraceBERT-bs16+Dropout0.1 89.91 85.32 87.16 88.07 87.16 88.99 90.83 88.07 88.99 88.99 88.35 1.48
EmbraceBERT-bs16+Dropout0.3 90.83 86.24 87.16 87.16 88.99 89.91 88.99 86.24 89.91 88.99 88.44 1.55
EmbraceBERT-bs16+Branches 89.91 87.16 85.32 91.74 87.16
CondensedEmbraceBERT-bs16 89.91 88.99 90.83 88.07 87.16 87.16 88.99 87.16 88.99 88.07 88.53 1.18
CondensedEmbraceBERT-bs16+Dropout0.1 91.74 88.07 88.07 86.24 86.24 88.99 91.74 87.16 88.99 88.07 88.53 1.85
CondensedEmbraceBERT-bs16+Dropout0.3 92.66 88.07 86.24 85.32 88.07 88.07 89.91 90.83 82.57 87.16 87.89 2.72
CondensedEmbraceBERT-bs16+Branches 89.91 84.40 85.32 89.91
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 89.91 89.91 91.74 89.91 90.83 89.91 90.83 90.83 87.16 89.91 90.09 1.15
FrozenEBERT-bs16-ep100 90.83 88.07 88.07 88.07 89.91 89.91 90.83 88.99 89.91 88.07 89.27 1.09
FrozenEBERT-bs16-ep100+Dropout0.1 90.83 86.24 89.91 88.07 88.07 88.99 88.07 87.16 87.16 88.07 88.26 1.28
FrozenEBERT-bs16-ep100+Dropout0.3 91.74 88.07 90.83 87.16 88.99 90.83 88.99 88.99 88.07 86.24 88.99 1.64
FrozenEBERT-bs16-ep100+Branches 89.91 88.99 88.07
FrozenCEBERT-bs16-ep100 91.74 88.07 90.83 88.07 88.99 86.24 88.07 88.07 88.99 88.07 88.72 1.48
FrozenCEBERT-bs16-ep100+Dropout0.1 91.74 87.16 89.91 87.16 88.99 88.07 90.83 87.16 89.91 88.07 88.90 1.56
FrozenCEBERT-bs16-ep100+Dropout0.3 91.74 88.07 90.83 88.07 88.99 89.91 91.74 91.74 84.40 87.16 89.27 2.29
FrozenCEBERT-bs16-ep100+Branches 89.91 89.91 90.83 89.91 88.07 87.16 91.74 88.99 88.07 86.24 89.08 1.61
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 90.83 91.74 90.83 92.66 90.83 88.99 88.99 90.83 91.74 90.83 90.83 1.09
RoBERTa-bs4+Dropout0.1 90.83 88.07 90.83 90.83 89.91 89.91 88.99 92.66 89.91 91.74 90.37 1.25
RoBERTa-bs4+Dropout0.3 91.74 85.32 89.91 91.74 89.91 90.83 89.91 90.83 91.74 89.91 90.18 1.79
EmbraceRoBERTa-bs4 90.83 89.91 92.66 89.91 92.66 88.99 91.74 91.74 91.74 88.99 90.92 1.33
EmbraceRoBERTa-bs4+Dropout0.1 90.83 89.91 91.74 90.83 92.66 92.66 90.83 90.83 91.74 90.83 91.28 0.85
EmbraceRoBERTa-bs4+Dropout0.3 88.07 87.16 92.66 90.83 88.99 88.99 90.83 88.07 91.74 92.66 90.00 1.90
EmbraceRoBERTa-bs4+Branches
CondensedEmbraceRoBERTa-bs4 91.74 88.99 92.66 90.83 91.74 89.91 91.74 90.83 90.83 88.07 90.73 1.33
CondensedEmbraceRoBERTa-bs4+Dropout0.1 88.99 90.83 88.99 90.83 90.83 90.83 93.58 89.91 90.83 89.91 90.55 1.23
CondensedEmbraceRoBERTa-bs4+Dropout0.3 89.91 90.83 92.66 91.74 91.74 87.16 88.99 91.74 90.83 91.74 90.73 1.56
CondensedEmbraceRoBERTa-bs4+Branches
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 88.99 91.74 88.99 93.58 90.83 86.24 88.99 88.99 91.74 88.99 89.91 1.97
FrozenERoBERTa-bs4-ep100 88.99 89.91 90.83 90.83 91.74 90.83 90.83 91.74 91.74 88.99 90.64 0.99
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenERoBERTa-bs4-ep100+Branches
FrozenCERoBERTa-bs4-ep100 92.66 91.74 92.66 90.83 91.74 91.74 94.50 91.74 91.74 88.07 91.74 1.54
FrozenCERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100+Dropout0.3
FrozenCERoBERTa-bs4-ep100+Branches
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 89.91 89.91 90.83 90.83 89.91 90.83 88.99 88.07 88.07 88.07 89.54 1.10
RoBERTa-bs16+Dropout0.1 91.74 90.83 88.07 91.74 90.83 90.83 88.99 90.83 90.83 91.74 90.64 1.15
RoBERTa-bs16+Dropout0.3 91.74 90.83 90.83 91.74 88.07 90.83 88.99 91.74 90.83 88.07 90.37 1.38
EmbraceRoBERTa-bs16 86.24 88.99 89.91 86.24 89.91 90.83 89.91 90.83 91.74 90.83 89.54 1.80
EmbraceRoBERTa-bs16+Dropout0.1 87.16 89.91 92.66 89.91 93.58 90.83 91.74 89.91 91.74 91.74 90.92 1.71
EmbraceRoBERTa-bs16+Dropout0.3 88.07 89.91 92.66 87.16 88.99 90.83 88.99 91.74 88.99 89.91 89.72 1.58
EmbraceRoBERTa-bs16+Branches
CondensedEmbraceRoBERTa-bs16 87.16 89.91 89.91 88.99 90.83 90.83 88.99 89.91 91.74 91.74 90.00 1.33
CondensedEmbraceRoBERTa-bs16+Dropout0.1 88.07 91.74 89.91 88.99 91.74 89.91 90.83 90.83 91.74 88.99 90.28 1.24
CondensedEmbraceRoBERTa-bs16+Dropout0.3 91.74 90.83 88.99 89.91 89.91 88.99 91.74 89.91 91.74 90.83 90.46 1.02
CondensedEmbraceRoBERTa-bs16+Branches
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 90.83 89.91 91.74 90.83 91.74 88.99 88.07 88.99 90.83 88.99 90.09 1.22
FrozenERoBERTa-bs16-ep100 90.83 90.83 92.66 88.99 88.99 90.83 89.91 91.74 89.91 90.83 90.55 1.09
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenERoBERTa-bs16-ep100+Branches
FrozenCERoBERTa-bs16-ep100 89.91 91.74 92.66 91.74 90.83 88.99 83.49 90.83 89.91 90.83 90.09 2.42
FrozenCERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100+Dropout0.3
FrozenCERoBERTa-bs16-ep100+Branches
  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 87.16 90.83 92.66 89.91 88.07 91.74 88.99 89.91 88.99 90.83 89.91 1.59
BERT-bs4+Dropout0.1 89.91 88.07 91.74 90.83 90.83 89.91 90.83 89.91 86.24 88.99 89.72 1.52
BERT-bs4+Dropout0.3 90.83 88.99 92.66 91.74 90.83 91.74 90.83 89.91 88.99 88.99 90.55 1.23
EmbraceBERT-bs4 89.91 90.83 88.99 92.66 88.99 86.24 90.83 88.99 89.91 87.16 89.45 1.75
EmbraceBERT-bs4+Dropout0.1 90.83 88.07 90.83 90.83 90.83 88.07 89.91 89.91 89.91 91.74 90.09 1.15
EmbraceBERT-bs4+Dropout0.3 90.83 89.91 86.24 89.91 89.91 90.83 88.07 90.83 89.91 88.07 89.45 1.44
EmbraceBERT-bs4+Branches
CondensedEmbraceBERT-bs4 89.91 88.07 90.83 92.66 89.91 89.91 90.83 90.83 90.83 89.91 90.37 1.10
CondensedEmbraceBERT-bs4+Dropout0.1 90.83 87.16 90.83 91.74 89.91 88.99 90.83 92.66 90.83 88.99 90.28 1.49
CondensedEmbraceBERT-bs4+Dropout0.3 90.83 87.16 88.07 89.91 89.91 88.99 91.74 91.74 90.83 88.07 89.72 1.52
CondensedEmbraceBERT-bs4+Branches
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERTwithAttClsProjection-bs4 91.74 90.83 90.83 90.83 90.83 91.74 88.99 90.83 89.91 88.99 90.55 0.92
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 88.07 88.99 92.66 92.66 88.99 91.74 89.91 90.83 88.99 88.99 90.18 1.59
FrozenEBERT-bs4-ep100 89.91 90.83 92.66 92.66 88.99 88.07 90.83 88.99 88.99 87.16 89.91 1.74
FrozenEBERT-bs4-ep100+Dropout0.1 90.83 88.99 91.74 89.91 89.91 90.83 91.74 90.83 89.91 88.99 90.37 0.94
FrozenEBERT-bs4-ep100+Dropout0.3 88.07 90.83 87.16 89.91 88.07 90.83 88.99 92.66 88.07 89.91 89.45 1.60
FrozenEBERT-bs4+Branches
FrozenCEBERT-bs4-ep100 89.91 89.91 92.66 93.58 89.91 91.74 91.74 91.74 90.83 89.91 91.19 1.24
FrozenCEBERT-bs4-ep100+Dropout0.1 90.83 92.66 87.16 92.66 88.07 88.99 91.74 91.74 90.83 86.24 90.09 2.20
FrozenCEBERT-bs4-ep100+Dropout0.3 90.83 89.91 87.16 88.07 88.99 90.83 90.83 90.83 89.91 87.16 89.45 1.44
FrozenCEBERT-bs4-ep100+Branches 89.91 89.91 90.83 89.91 92.66 91.74 90.83 90.83 89.91 88.07 90.46 1.17
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 88.99 89.91 91.74 90.83 91.74 91.74 90.83 89.91 88.07 90.83 90.46 1.17
BERT-bs16+Dropout0.1 89.91 89.91 92.66 90.83 91.74 90.83 91.74 90.83 88.99 90.83 90.83 1.00
BERT-bs16+Dropout0.3 88.07 89.91 91.74 89.91 91.74 91.74 91.74 88.99 89.91 88.99 90.28 1.31
EmbraceBERT-bs16 89.91 88.99 88.07 91.74 88.99 89.91 89.91 90.83 88.99 88.07 89.54 1.10
EmbraceBERT-bs16+Branches
EmbraceBERT-bs16+Dropout0.1 90.83 88.07 90.83 88.99 88.99 88.99 90.83 87.16 88.99 88.99 89.27 1.16
EmbraceBERT-bs16+Dropout0.3 90.83 86.24 88.99 88.07 88.99 90.83 89.91 86.24 89.91 89.91 88.99 1.59
CondensedEmbraceBERT-bs16 91.74 88.07 88.07 90.83 88.99 89.91 89.91 88.99 88.99 88.99 89.45 1.10
CondensedEmbraceBERT-bs16+Dropout0.1 91.74 88.99 89.91 87.16 88.07 90.83 91.74 87.16 88.99 88.99 89.36 1.60
CondensedEmbraceBERT-bs16+Dropout0.3 92.66 88.99 88.99 87.16 89.91 90.83 91.74 90.83 82.57 88.99 89.27 2.69
CondensedEmbraceBERT-bs16+Branches
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 89.91 90.83 91.74 90.83 90.83 90.83 88.99 89.91 88.07 90.83 90.28 1.02
FrozenEBERT-bs16-ep100 92.66 89.91 87.16 91.74 89.91 88.99 91.74 90.83 90.83 88.07 90.18 1.64
FrozenEBERT-bs16-ep100+Dropout0.1 91.74 89.91 88.99 88.07 89.91 90.83 89.91 87.16 89.91 89.91 89.63 1.23
FrozenEBERT-bs16-ep100+Dropout0.3 89.91 89.91 89.91 88.99 89.91 90.83 91.74 88.07 89.91 88.99 89.82 0.96
FrozenEBERT-bs16+Branches
FrozenCEBERT-bs16-ep100 90.83 88.99 87.16 89.91 88.99 91.74 89.91 88.99 89.91 86.24 89.27 1.54
FrozenCEBERT-bs16-ep100+Dropout0.1 91.74 90.83 89.91 86.24 87.16 92.66 91.74 88.07 89.91 88.07 89.63 2.05
FrozenCEBERT-bs16-ep100+Dropout0.3 92.66 91.74 91.74 87.16 89.91 88.99 92.66 89.91 86.24 88.07 89.91 2.17
FrozenCEBERT-bs16-ep100+Branches 92.66 90.83 91.74 84.40 90.83 88.07 91.74 89.91 88.07 87.16 89.54 2.43
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 90.83 91.74 88.07 90.83 90.83 88.99 90.83 91.74 90.83 89.91 90.46 1.10
RoBERTa-bs4+Dropout0.1 91.74 88.07 90.83 91.74 89.91 88.99 88.99 92.66 89.91 91.74 90.46 1.43
RoBERTa-bs4+Dropout0.3 91.74 86.24 89.91 91.74 89.91 89.91 89.91 90.83 91.74 88.99 90.09 1.58
EmbraceRoBERTa-bs4 89.91 89.91 91.74 89.91 92.66 88.99 91.74 91.74 91.74 87.16 90.55 1.59
EmbraceRoBERTa-bs4+Dropout0.1 90.83 89.91 91.74 90.83 92.66 92.66 90.83 90.83 91.74 91.74 91.38 0.84
EmbraceRoBERTa-bs4+Dropout0.3 86.24 88.07 93.58 90.83 89.91 88.99 90.83 88.07 91.74 92.66 90.09 2.16
CondensedEmbraceRoBERTa-bs4 92.66 90.83 90.83 90.83 92.66 88.99 91.74 91.74 91.74 89.91 91.19 1.10
CondensedEmbraceRoBERTa-bs4+Dropout0.1 88.99 90.83 88.99 90.83 90.83 90.83 92.66 89.91 90.83 90.83 90.55 1.01
CondensedEmbraceRoBERTa-bs4+Dropout0.3 88.99 90.83 92.66 91.74 91.74 88.07 88.99 91.74 90.83 91.74 90.73 1.45
CondensedEmbraceRoBERTa-bs4+Branches
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 90.83 90.83 88.99 90.83 88.99 88.99 85.32 88.99 92.66 89.91 89.63 1.84
FrozenERoBERTa-bs4-ep100 92.66 89.91 89.91 89.91 91.74 88.99 90.83 91.74 90.83 90.83 90.73 1.04
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenERoBERTa-bs4-ep100+Branches
FrozenCERoBERTa-bs4-ep100 91.74 90.83 92.66 91.74 92.66 88.07 91.74 91.74 91.74 90.83 91.38 1.24
FrozenCERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100+Dropout0.3
FrozenCERoBERTa-bs4-ep100+Branches
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 88.99 89.91 89.91 89.91 90.83 91.74 88.99 90.83 88.07 89.91 89.91 1.00
RoBERTa-bs16+Dropout0.1 91.74 89.91 88.07 91.74 89.91 88.99 88.99 90.83 90.83 91.74 90.28 1.24
RoBERTa-bs16+Dropout0.3 91.74 90.83 91.74 91.74 88.07 90.83 88.99 91.74 90.83 91.74 90.83 1.23
EmbraceRoBERTa-bs16 88.07 88.99 88.99 88.07 90.83 90.83 89.91 90.83 91.74 90.83 89.91 1.23
EmbraceRoBERTa-bs16+Dropout0.1 88.99 89.91 92.66 88.99 92.66 90.83 91.74 89.91 91.74 91.74 90.92 1.33
EmbraceRoBERTa-bs16+Dropout0.3 88.07 89.91 92.66 88.07 89.91 90.83 88.99 91.74 89.91 89.91 90.00 1.39
EmbraceRoBERTa-bs16+Branches
CondensedEmbraceRoBERTa-bs16 88.07 88.99 91.74 91.74 89.91 90.83 89.91 90.83 90.83 90.83 90.37 1.10
CondensedEmbraceRoBERTa-bs16+Dropout0.1 86.24 91.74 89.91 90.83 91.74 92.66 90.83 90.83 91.74 90.83 90.73 1.66
CondensedEmbraceRoBERTa-bs16+Dropout0.3 92.66 90.83 90.83 90.83 89.91 88.99 92.66 92.66 91.74 91.74 91.28 1.18
CondensedEmbraceRoBERTa-bs16+Branches
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 88.99 88.99 90.83 89.91 90.83 89.91 88.99 88.99 89.91 90.83 89.82 0.76
FrozenERoBERTa-bs16-ep100 88.07 89.91 88.99 88.99 90.83 90.83 91.74 91.74 91.74 90.83 90.37 1.25
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenERoBERTa-bs16-ep100+Branches
FrozenCERoBERTa-bs16-ep100 88.07 89.91 90.83 90.83 90.83 89.91 88.99 91.74 92.66 90.83 90.46 1.24
FrozenCERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100+Dropout0.3
FrozenCERoBERTa-bs16-ep100+Branches

  • Epoch 100 - bs4 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |

  • Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |89.91|89.91|91.74|91.74|89.91|92.66|91.74|90.83|88.07|91.74|90.83|1.30| | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERTwithAtt-bs8 |89.91|88.99|90.83|90.83|90.83|90.83|87.16|90.83|91.74|91.74|90.37|1.31| | BERTwithProjection-bs8 |89.91|88.99|90.83|88.99|88.07|88.99|89.91|89.91|90.83|88.99|89.54|0.84| | BERTwithProjectionAtt-bs8 |90.83|88.99|88.99|91.74|88.99|88.99|88.99|90.83|87.16|89.91|89.54|1.24| | BERTwithAttClsProjection-bs8 |91.74|90.83|90.83|91.74|91.74|92.66|90.83|91.74|90.83|84.40|90.73|2.19| | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |90.83|87.16|86.24|88.07|88.07|88.99|86.24|91.74|89.91|88.07|88.53|1.75| | EmbraceBERT-bs8-p_att_clsquery_weights |88.99|87.16|91.74|89.91|85.32|88.99|88.99|89.91|89.91|90.83|89.17|1.73| | EmbraceBERTwithProj-bs8 |92.66|91.74|89.91|91.74|91.74|90.83|91.74|90.83|90.83|91.74|91.38|0.73| | EmbraceBERTwithProj-bs8-p_attclsqw |90.83|90.83|92.66|89.91|91.74|90.83|90.83|91.74|91.74|92.66|91.38|0.84| | EBERTconcatAtt-bs8 |91.74|90.83|89.91|89.91|90.83|91.74|89.91|90.83|88.99|88.99|90.37|0.94| | EBERTconcatAtt-bs8_attclsqw |90.83|89.91|88.99|83.49|89.91|91.74|88.07|89.91|89.91|88.99|89.17|2.12| | EBERTconcatAttwithProj-bs8 |89.91|88.99|91.74|89.91|90.83|90.83|90.83|93.58|90.83|87.16|90.46|1.60| | EBERTconcatAttwithProj-bs8_attclsqw |88.99|90.83|88.99|91.74|89.91|92.66|89.91|90.83|92.66|91.74|90.83|1.30| | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |88.07|90.83|88.07|85.32|88.07|88.07|88.07|88.99|88.07|90.83|88.44|1.49| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |89.91|88.99|91.74|91.74|87.16|89.91|91.74|89.91|88.99|88.99|89.91|1.42| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |90.83|92.66|91.74|90.83|91.74|90.83|92.66|91.74|88.99|89.91|91.19|1.10| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |90.83|92.66|91.74|91.74|91.74|91.74|88.99|92.66|91.74|92.66|91.65|1.04| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |89.91|90.83|88.99|89.91|88.99|89.91|88.99|89.91|88.99|91.74|89.82|0.87| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |89.91|90.83|91.74|91.74|89.91|89.91|91.74|90.83|89.91|91.74|90.83|0.82| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |91.74|91.74|90.83|91.74|90.83|88.99|88.99|89.91|88.99|89.91|90.37|1.10| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |87.16|90.83|87.16|89.91|92.66|87.16|88.07|91.74|91.74|90.83|89.72|2.04|

STT Error: trained with COMPLETE data and tested with INCOMPLETE data

gtts-google - Good only with bs=16 RoBERTa

  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 75.23 78.90 79.82 77.06 75.23 75.23 76.15 73.39 77.98 78.90 76.79 1.97
BERT-bs4+Dropout0.1 75.23 78.90 78.90 76.15 80.73 76.15 80.73 73.39 72.48 78.90 77.16 2.77
BERT-bs4+Dropout0.3 77.98 78.90 77.98 78.90 81.65 72.48 80.73 71.56 78.90 81.65 78.07 3.29
FrozenBERT-bs4-ep100 75.23 76.15 79.82 73.39 72.48 75.23 76.15 73.39 77.98 73.39 75.32 2.19
FrozenBERT-bs4-ep100+Dropout0.1 75.23 78.90 77.06 77.98 77.98 73.39 80.73 72.48 77.98 77.06 76.88 2.39
FrozenBERT-bs4-ep100+Dropout0.3 77.06 74.31 75.23 77.98 77.98 74.31 80.73 73.39 74.31 75.23 76.06 2.19
EmbraceBERT-bs4 75.23 77.98 74.31 76.15 76.15 73.39 80.73 77.06 74.31 76.15 76.15 2.01
EmbraceBERT-bs4+Dropout0.1 77.06 76.15 75.23 77.98 77.98 76.15 77.98 70.64 78.90 79.82 76.79 2.43
EmbraceBERT-bs4+Dropout0.3 76.15 75.23 76.15 77.06 77.98 76.15 75.23 75.23 77.98 71.56 75.87 1.74
CondensedEmbraceBERT-bs4 26.61 26.61 11.93 34.86 25.69 21.10 11.01 38.53 31.19 11.01 23.85 9.43
CondensedEmbraceBERT-bs4+Dropout0.1 81.65 80.73 77.98 80.73 76.15 77.98 76.15 75.23 76.15 77.06 77.98 2.17
CondensedEmbraceBERT-bs4+Dropout0.3 78.90 77.98 77.98 77.06 74.31 77.06 77.06 76.15 78.90 74.31 76.97 1.56
FrozenEBERT-bs4-ep100 77.98 77.06 74.31 74.31 76.15 74.31 78.90 77.06 71.56 76.15 75.78 2.06
FrozenEBERT-bs4-ep100+Dropout0.1 77.06 76.15 73.39 77.06 77.06 82.57 74.31 76.15 75.23 75.23 76.42 2.36
FrozenEBERT-bs4-ep100+Dropout0.3 76.15 74.31 70.64 76.15 77.98 75.23 72.48 73.39 72.48 76.15 74.50 2.12
FrozenCEBERT-bs4-ep100 31.19 22.02 11.93 37.61 14.68 26.61 7.34 40.37 14.68 10.09 21.65 11.19
FrozenCEBERT-bs4-ep100+Dropout0.1 81.65 80.73 74.31 79.82 71.56 75.23 78.90 75.23 76.15 76.15 76.97 3.03
FrozenCEBERT-bs4-ep100+Dropout0.3 78.90 79.82 76.15 77.06 70.64 77.06 76.15 72.48 73.39 76.15 75.78 2.70
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERTwithAtt-bs4 77.98 78.90 76.15 76.15 74.31 80.73 78.90 75.23 77.98 69.72 76.61 2.94
BERTwithAttClsProjection-bs4 77.98 78.90 77.98 77.98 77.98 78.90 79.82 80.73 78.90 79.82 78.90 0.92
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4+Branches 77.06 77.98 77.06 78.90 73.39 76.15 79.82 74.31 77.98 79.82 77.25 2.04
EmbraceBERT-bs4+Branches+Dropout0.1 76.15 77.06 77.06 76.15 80.73 77.06 78.90 80.73 79.82 77.06 78.07 1.71
EmbraceBERT-bs4+Branches+Dropout0.3 78.90 77.98 80.73 77.06 78.90 76.15 78.90 78.90 79.82 77.06 78.44 1.31
CondensedEmbraceBERT-bs4+Branches 80.73 79.82 77.06 77.06 72.48 77.06 80.73 76.15 77.06 77.06 77.52 2.33
CondensedEmbraceBERT-bs4+Branches+Dropout0.1 77.06 77.06 77.98 74.31 77.06 74.31 79.82 78.90 79.82 75.23 77.16 1.94
CondensedEmbraceBERT-bs4+Branches+Dropout0.3 81.65 77.06 81.65 77.06 75.23 76.15 78.90 78.90 78.90 75.23 78.07 2.23
FrozenEmbraceBERT-bs4+Branches 77.98 74.31 77.98 77.98 74.31 77.06 79.82 75.23 77.06 74.31 76.61 1.85
FrozenEmbraceBERT-bs4+Branches+Dropout0.1 76.15 73.39 75.23 77.06 80.73 77.06 77.98 80.73 77.06 77.98 77.34 2.13
FrozenEmbraceBERT-bs4+Branches+Dropout0.3 77.98 74.31 78.90 78.90 76.15 77.98 79.82 79.82 79.82 77.06 78.07 1.71
FrozenCEBERT-bs4+Branches 73.39 74.31 81.65 77.98 77.06 77.06 69.72 81.65 76.15 77.06 76.61 3.41
FrozenCEBERT-bs4+Branches+Dropout0.1 77.98 81.65 78.90 71.56 78.90 73.39 79.82 79.82 78.90 75.23 77.61 3.02
FrozenCEBERT-bs4+Branches+Dropout0.3 77.98 76.15 74.31 77.98 77.98 78.90 80.73 78.90 77.06 70.64 77.06 2.69
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 77.98 75.23 78.90 74.31 75.23 80.73 78.90 77.06 75.23 79.82 77.34 2.13
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery_attclsqw 77.98 78.90 78.90 76.15 74.31 76.15 78.90 73.39 79.82 74.31 76.88 2.20
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 77.98 78.90 80.73 77.06 77.06 77.06 81.65 73.39 76.15 77.06 77.71 2.21
BERT-bs16+Dropout0.1 77.06 77.98 82.57 78.90 78.90 75.23 78.90 74.31 74.31 75.23 77.34 2.50
BERT-bs16+Dropout0.3 75.23 76.15 79.82 77.98 76.15 76.15 79.82 71.56 73.39 78.90 76.51 2.57
FrozenBERT-bs16-ep100 77.06 77.98 77.06 76.15 77.06 76.15 77.98 73.39 70.64 77.98 76.15 2.25
FrozenBERT-bs16-ep100+Dropout0.1 78.90 73.39 77.06 77.06 77.06 74.31 78.90 72.48 75.23 77.98 76.24 2.15
FrozenBERT-bs16-ep100+Dropout0.3 79.82 76.15 80.73 76.15 77.06 77.06 80.73 75.23 73.39 76.15 77.25 2.31
EmbraceBERT-bs16 75.23 77.06 78.90 76.15 74.31 77.98 77.06 77.06 77.98 76.15 76.79 1.30
EmbraceBERT-bs16+Dropout0.1 74.31 77.06 77.98 77.06 73.39 77.98 77.98 74.31 77.06 79.82 76.70 1.93
EmbraceBERT-bs16+Dropout0.3 74.31 76.15 75.23 77.06 74.31 77.06 75.23 75.23 78.90 77.98 76.15 1.48
CondensedEmbraceBERT-bs16 45.87 18.35 11.93 22.02 16.51 14.68 11.01 15.60 3.67 16.51 17.61 10.51
CondensedEmbraceBERT-bs16+Dropout0.1 76.15 76.15 74.31 73.39 77.98 80.73 80.73 75.23 76.15 79.82 77.06 2.50
CondensedEmbraceBERT-bs16+Dropout0.3 76.15 77.06 78.90 77.06 75.23 79.82 80.73 78.90 73.39 77.98 77.52 2.10
FrozenEBERT-bs16-ep100 75.23 75.23 75.23 77.98 74.31 78.90 77.98 76.15 77.06 71.56 75.96 2.04
FrozenEBERT-bs16-ep100+Dropout0.1 77.98 77.06 77.98 76.15 74.31 78.90 77.98 74.31 76.15 77.06 76.79 1.48
FrozenEBERT-bs16-ep100+Dropout0.3 74.31 76.15 77.06 79.82 76.15 76.15 79.82 77.98 77.06 77.98 77.25 1.63
FrozenCEBERT-bs16-ep100 47.71 15.60 13.76 39.45 14.68 8.26 11.01 11.93 5.50 6.42 17.43 13.58
FrozenCEBERT-bs16-ep100+Dropout0.1 76.15 78.90 74.31 78.90 75.23 78.90 80.73 76.15 77.06 78.90 77.52 1.94
FrozenCEBERT-bs16-ep100+Dropout0.3 77.06 77.98 76.15 74.31 74.31 77.98 80.73 74.31 72.48 77.98 76.33 2.35
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16+Branches 74.31 75.23 71.56 77.98 77.06 78.90 80.73 75.23 76.15 78.90 76.61 2.54
EmbraceBERT-bs16+Branches+Dropout0.1 76.15 77.06 81.65 75.23 77.98 75.23 77.98 75.23 77.06 78.90 77.25 1.92
EmbraceBERT-bs16+Branches+Dropout0.3 75.23 77.06 79.82 73.39 81.65 74.31 78.90 75.23 74.31 77.98 76.79 2.60
CondensedEmbraceBERT-bs16+Branches 72.48 77.98 74.31 75.23 78.90 77.06 80.73 77.98 75.23 79.82 76.97 2.48
CondensedEmbraceBERT-bs16+Branches+Dropout0.1 77.06 76.15 79.82 76.15 74.31 74.31 77.98 76.15 76.15 77.06 76.51 1.55
CondensedEmbraceBERT-bs16+Branches+Dropout0.3 79.82 78.90 78.90 75.23 77.06 77.98 77.06 74.31 79.82 75.23 77.43 1.89
FrozenEmbraceBERT-bs16+Branches 74.31 75.23 76.15 77.06 76.15 77.98 78.90 77.06 77.06 79.82 76.97 1.56
FrozenEmbraceBERT-bs16+Branches+Dropout0.1 80.73 75.23 77.06 77.98 79.82 75.23 75.23 76.15 77.98 75.23 77.06 1.92
FrozenEmbraceBERT-bs16+Branches+Dropout0.3 73.39 75.23 78.90 74.31 77.98 73.39 77.98 75.23 75.23 77.98 75.96 1.96
FrozenCEBERT-bs16+Branches 76.15 77.98 83.49 71.56 77.98 76.15 77.98 76.15 77.98 75.23 77.06 2.84
FrozenCEBERT-bs16+Branches+Dropout0.1 78.90 76.15 78.90 74.31 77.06 76.15 78.90 74.31 77.98 78.90 77.16 1.76
FrozenCEBERT-bs16+Branches+Dropout0.3 79.82 75.23 78.90 77.06 76.15 77.98 78.90 73.39 77.98 74.31 76.97 2.03
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery 75.23 78.90 76.15 76.15 76.15 77.06 78.90 77.06 72.48 78.90 76.70 1.89
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 77.06 77.06 77.98 74.31 73.39 73.39 75.23 74.31 75.23 74.31 75.23 1.54
RoBERTa-bs4+Dropout0.1 77.98 76.15 77.06 74.31 78.90 77.98 81.65 70.64 79.82 77.06 77.16 2.89
RoBERTa-bs4+Dropout0.3 76.15 74.31 75.23 77.06 80.73 74.31 80.73 74.31 79.82 76.15 76.88 2.49
FrozenRoBERTa-bs4-ep100 76.15 77.06 77.06 72.48 70.64 74.31 79.82 72.48 74.31 75.23 74.95 2.56
FrozenRoBERTa-bs4-ep100+Dropout0.1 78.90 77.06 76.15 75.23 75.23 75.23 75.23 76.15 77.98 75.23 76.24 1.26
FrozenRoBERTa-bs4-ep100+Dropout0.3 75.23 77.06 77.98 78.90 72.48 77.06 78.90 77.06 69.72 73.39 75.78 2.88
EmbraceRoBERTa-bs4 78.90 77.98 77.06 73.39 79.82 73.39 73.39 76.15 75.23 75.23 76.06 2.23
EmbraceRoBERTa-bs4+Dropout0.1 76.15 74.31 77.98 73.39 77.06 77.98 77.06 77.06 75.23 77.98 76.42 1.54
EmbraceRoBERTa-bs4+Dropout0.3 74.31 76.15 79.82 74.31 72.48 76.15 77.06 76.15 76.15 77.06 75.96 1.87
CondensedEmbraceRoBERTa-bs4 41.28 37.61 7.34 41.28 11.93 32.11 5.50 11.01 11.93 13.76 21.38 14.02
CondensedEmbraceRoBERTa-bs4+Dropout0.1 67.89 75.23 72.48 73.39 77.06 75.23 77.06 77.98 74.31 77.98 74.86 2.94
CondensedEmbraceRoBERTa-bs4+Dropout0.3 67.89 72.48 77.98 76.15 74.31 75.23 76.15 77.98 74.31 77.06 74.95 2.87
FrozenERoBERTa-bs4-ep100 77.06 76.15 77.06 72.48 80.73 73.39 74.31 77.06 75.23 74.31 75.78 2.25
FrozenERoBERTa-bs4-ep100+Dropout0.1 73.39 77.98 73.39 77.98 78.90 76.15 78.90 77.06 71.56 77.06 76.24 2.44
FrozenERoBERTa-bs4-ep100+Dropout0.3 73.39 79.82 72.48 77.98 74.31 75.23 77.06 77.98 71.56 77.06 75.69 2.57
FrozenCERoBERTa-bs4-ep100 41.28 40.37 5.50 44.04 11.01 33.03 5.50 12.84 7.34 9.17 21.01 15.60
FrozenCERoBERTa-bs4-ep100+Dropout0.1 77.06 73.39 76.15 74.31 79.82 76.15 77.06 77.06 75.23 77.06 76.33 1.68
FrozenCERoBERTa-bs4-ep100+Dropout0.3 77.98 77.06 75.23 75.23 79.82 73.39 77.06 78.90 72.48 77.98 76.51 2.25
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceRoBERTa-bs4+Branches 75.23 76.15 76.15 76.15 77.06 77.98 75.23 77.98 73.39 76.15 76.15 1.30
EmbraceRoBERTa-bs4+Branches+Dropout0.1 77.98 77.98 76.15 77.06 77.06 76.15 77.06 77.98 75.23 75.23 76.79 1.01
EmbraceRoBERTa-bs4+Branches+Dropout0.3 78.90 77.06 74.31 75.23 77.06 75.23 73.39 77.98 71.56 77.06 75.78 2.14
CondensedEmbraceRoBERTa-bs4+Branches 75.23 74.31 75.23 75.23 77.98 76.15 76.15 77.06 72.48 77.06 75.69 1.49
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.1 77.06 77.06 77.06 77.98 77.98 74.31 78.90 78.90 77.06 77.98 77.43 1.24
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.3 78.90 78.90 77.98 78.90 77.98 76.15 76.15 77.06 77.98 81.65 78.17 1.52
FrozenEmbraceRoBERTa-bs4+Branches 77.06 74.31 77.06 77.06 74.31 77.98 77.06 77.06 69.72 76.15 75.78 2.33
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.1 77.98 77.98 73.39 76.15 76.15 76.15 77.06 76.15 74.31 76.15 76.15 1.36
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.3 77.98 77.06 74.31 74.31 77.06 75.23 74.31 77.06 71.56 77.06 75.60 1.89
FrozenCERoBERTa-bs4+Branches 75.23 75.23 77.06 77.98 77.98 76.15 73.39 77.98 70.64 76.15 75.78 2.22
FrozenCERoBERTa-bs4+Branches+Dropout0.1 75.23 76.15 75.23 78.90 76.15 74.31 77.06 79.82 77.98 77.06 76.79 1.64
FrozenCERoBERTa-bs4+Branches+Dropout0.3 75.23 77.98 74.31 75.23 77.06 73.39 76.15 77.06 79.82 81.65 76.79 2.39
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 77.98 77.06 77.98 77.06 73.39 77.06 75.23 76.15 76.15 77.98 76.61 1.38
RoBERTa-bs16+Dropout0.1 77.06 76.15 77.98 77.06 76.15 77.98 80.73 76.15 76.15 77.06 77.25 1.35
RoBERTa-bs16+Dropout0.3 78.90 78.90 77.98 74.31 77.06 77.06 77.98 76.15 76.15 77.98 77.25 1.35
FrozenRoBERTa-bs16-ep100 76.15 77.98 77.06 77.06 70.64 76.15 76.15 72.48 75.23 77.98 75.69 2.26
FrozenRoBERTa-bs16-ep100+Dropout0.1 76.15 73.39 79.82 76.15 77.06 77.06 76.15 77.06 76.15 76.15 76.51 1.49
FrozenRoBERTa-bs16-ep100+Dropout0.3 77.06 77.98 77.98 76.15 76.15 77.98 78.90 76.15 76.15 77.06 77.16 0.96
EmbraceRoBERTa-bs16 74.31 76.15 75.23 77.98 69.72 76.15 74.31 74.31 73.39 77.06 74.86 2.18
EmbraceRoBERTa-bs16+Dropout0.1 77.98 77.98 76.15 76.15 73.39 75.23 77.06 77.98 75.23 77.06 76.42 1.42
EmbraceRoBERTa-bs16+Dropout0.3 77.06 76.15 77.98 77.06 71.56 73.39 76.15 77.06 75.23 75.23 75.69 1.85
CondensedEmbraceRoBERTa-bs16 38.53 36.70 11.93 31.19 12.84 36.70 7.34 16.51 11.01 26.61 22.94 11.63
CondensedEmbraceRoBERTa-bs16+Dropout0.1 76.15 75.23 76.15 77.06 70.64 76.15 75.23 77.06 76.15 77.06 75.69 1.80
CondensedEmbraceRoBERTa-bs16+Dropout0.3 76.15 76.15 76.15 77.06 75.23 77.06 76.15 77.06 77.06 77.06 76.51 0.61
FrozenERoBERTa-bs16-ep100 72.48 74.31 73.39 77.06 70.64 74.31 74.31 73.39 77.06 78.90 74.59 2.32
FrozenERoBERTa-bs16-ep100+Dropout0.1 73.39 76.15 76.15 77.98 81.65 76.15 77.06 77.98 78.90 78.90 77.43 2.10
FrozenERoBERTa-bs16-ep100+Dropout0.3 72.48 76.15 76.15 77.98 77.06 77.98 74.31 77.06 74.31 77.98 76.15 1.79
FrozenCERoBERTa-bs16-ep100 39.45 32.11 9.17 33.94 11.93 31.19 6.42 13.76 11.01 11.93 20.09 11.82
FrozenCERoBERTa-bs16-ep100+Dropout0.1 75.23 78.90 77.98 77.98 77.98 77.06 77.98 77.06 74.31 78.90 77.34 1.42
FrozenCERoBERTa-bs16-ep100+Dropout0.3 75.23 76.15 77.06 77.98 77.98 75.23 78.90 77.98 73.39 77.06 76.70 1.60
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceRoBERTa-bs16+Branches 73.39 77.06 75.23 77.06 77.98 76.15 75.23 74.31 74.31 77.06 75.78 1.43
EmbraceRoBERTa-bs16+Branches+Dropout0.1 76.15 77.98 75.23 77.06 77.98 75.23 76.15 79.82 74.31 77.06 76.70 1.55
EmbraceRoBERTa-bs16+Branches+Dropout0.3 77.98 77.06 76.15 77.98 78.90 75.23 76.15 77.98 73.39 76.15 76.70 1.55
CondensedEmbraceRoBERTa-bs16+Branches 76.15 77.06 76.15 77.06 76.15 77.06 77.06 76.15 74.31 76.15 76.33 0.80
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1 76.15 77.98 76.15 77.06 77.98 77.98 73.39 77.98 77.98 77.06 76.97 1.39
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3 77.98 77.06 75.23 77.98 76.15 73.39 77.98 77.98 76.15 77.06 76.70 1.43
FrozenEmbraceRoBERTa-bs16+Branches 73.39 76.15 75.23 77.98 77.98 73.39 74.31 76.15 74.31 77.06 75.60 1.65
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1 76.15 77.98 75.23 77.06 78.90 76.15 77.06 78.90 72.48 77.06 76.70 1.80
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3 79.82 77.06 77.06 76.15 78.90 75.23 76.15 76.15 74.31 76.15 76.70 1.55
FrozenCERoBERTa-bs16+Branches 75.23 76.15 77.06 77.06 77.98 77.98 76.15 76.15 73.39 77.06 76.42 1.30
FrozenCERoBERTa-bs16+Branches+Dropout0.1 74.31 77.06 77.98 78.90 78.90 76.15 72.48 77.98 77.06 77.06 76.79 1.93
FrozenCERoBERTa-bs16+Branches+Dropout0.3 75.23 77.98 75.23 77.98 76.15 75.23 77.06 79.82 75.23 76.15 76.61 1.49

  • Epoch 100 - bs4 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |

  • Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |77.06|78.90|77.98|78.90|77.06|76.15|82.57|73.39|76.15|76.15|77.43|2.29| | BERTwithAtt-bs8 |80.73|80.73|77.06|77.98|77.98|80.73|77.06|78.90|80.73|79.82|79.17|1.48| | BERTwithAttClsProjection-bs8 |74.31|77.98|78.90|77.98|77.98|79.82|79.82|78.90|77.98|77.06|78.07|1.50| | BERTwithProjection-bs8 |74.31|75.23|77.06|69.72|76.15|75.23|73.39|74.31|77.06|76.15|74.86|2.06| | BERTwithProjectionAtt-bs8 |76.15|77.98|77.06|77.06|77.06|74.31|78.90|77.98|76.15|78.90|77.16|1.33| | EmbraceBERT-bs8 |77.98|77.06|73.39|77.98|73.39|77.98|76.15|76.15|75.23|80.73|76.61|2.14| | EmbraceBERT-bs8-p_att_clsquery_weights |77.06|75.23|77.06|77.98|76.15|75.23|79.82|77.06|76.15|76.15|76.79|1.30| | EmbraceBERTwithProj-bs8 |77.98|79.82|74.31|76.15|77.06|80.73|77.98|76.15|80.73|73.39|77.43|2.40| | EmbraceBERTwithProj-bs8-p_attclsqw |78.90|77.98|76.15|77.98|74.31|78.90|77.98|77.98|78.90|77.98|77.71|1.36| | EBERTconcatAtt-bs8 |78.90|80.73|76.15|75.23|77.06|77.98|78.90|77.98|77.06|72.48|77.25|2.16| | EBERTconcatAtt-bs8_attclsqw |77.06|77.06|78.90|77.98|80.73|78.90|75.23|79.82|80.73|77.98|78.44|1.65| | EBERTconcatAttwithProj-bs8 |77.06|78.90|77.98|77.98|76.15|77.06|77.06|79.82|77.06|78.90|77.80|1.07| | EBERTconcatAttwithProj-bs8_attclsqw |72.48|75.23|76.15|78.90|76.15|77.98|73.39|77.06|79.82|79.82|76.70|2.40| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |72.48|75.23|77.06|67.89|74.31|75.23|78.90|79.82|73.39|77.06|75.14|3.27| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |78.90|78.90|79.82|76.15|74.31|77.98|77.98|73.39|77.98|73.39|76.88|2.28| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |76.15|78.90|77.98|79.82|76.15|77.98|78.90|78.90|70.64|77.06|77.25|2.49| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |77.98|76.15|76.15|78.90|79.82|77.06|73.39|76.15|76.15|77.98|76.97|1.71| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |75.23|77.98|73.39|76.15|79.82|79.82|79.82|77.06|75.23|78.90|77.34|2.17| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |77.06|78.90|78.90|77.98|79.82|78.90|77.06|76.15|75.23|77.98|77.80|1.35| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |77.06|79.82|77.98|77.06|76.15|72.48|76.15|74.31|75.23|73.39|75.96|2.08| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |74.31|74.31|74.31|77.06|78.90|76.15|78.90|74.31|77.06|77.06|76.24|1.76|

gtts-sphinx - Not good with BERT, good with RoBERTa all

  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 64.22 77.98 77.98 74.31 77.98 69.72 67.89 72.48 70.64 66.97 72.02 4.72
BERT-bs4+Dropout0.1 65.14 77.06 77.06 72.48 68.81 66.06 72.48 73.39 66.06 69.72 70.83 4.16
BERT-bs4+Dropout0.3 71.56 77.98 74.31 73.39 75.23 64.22 71.56 62.39 74.31 64.22 70.92 5.11
FrozenBERT-bs4-ep100 69.72 76.15 76.15 73.39 74.31 73.39 66.97 70.64 67.89 66.06 71.47 3.54
FrozenBERT-bs4-ep100+Dropout0.1 58.72 74.31 75.23 75.23 73.39 61.47 71.56 64.22 67.89 73.39 69.54 5.78
FrozenBERT-bs4-ep100+Dropout0.3 65.14 75.23 66.06 76.15 73.39 67.89 74.31 69.72 68.81 69.72 70.64 3.69
EmbraceBERT-bs4 61.47 73.39 69.72 68.81 71.56 65.14 72.48 68.81 66.06 63.30 68.07 3.78
EmbraceBERT-bs4+Dropout0.1 62.39 71.56 66.06 70.64 66.97 69.72 67.89 66.97 64.22 78.90 68.53 4.36
EmbraceBERT-bs4+Dropout0.3 55.05 72.48 66.97 65.14 73.39 77.98 61.47 65.14 70.64 66.97 67.52 6.18
CondensedEmbraceBERT-bs4 33.03 21.10 12.84 41.28 29.36 14.68 26.61 40.37 41.28 17.43 27.80 10.49
CondensedEmbraceBERT-bs4+Dropout0.1 80.73 76.15 67.89 63.30 68.81 76.15 71.56 60.55 65.14 73.39 70.37 6.06
CondensedEmbraceBERT-bs4+Dropout0.3 68.81 69.72 72.48 72.48 70.64 68.81 75.23 69.72 72.48 63.30 70.37 3.04
FrozenEBERT-bs4-ep100 55.96 64.22 71.56 68.81 72.48 66.97 68.81 70.64 65.14 66.97 67.16 4.51
FrozenEBERT-bs4-ep100+Dropout0.1 72.48 70.64 58.72 72.48 67.89 77.98 66.97 66.97 64.22 69.72 68.81 4.96
FrozenEBERT-bs4-ep100+Dropout0.3 69.72 69.72 64.22 66.06 77.98 77.06 59.63 67.89 66.97 69.72 68.90 5.21
FrozenCEBERT-bs4-ep100 37.61 16.51 11.01 40.37 18.35 12.84 24.77 43.12 44.95 12.84 26.24 13.09
FrozenCEBERT-bs4-ep100+Dropout0.1 77.06 77.06 65.14 67.89 65.14 79.82 70.64 64.22 62.39 62.39 69.17 6.25
FrozenCEBERT-bs4-ep100+Dropout0.3 67.89 78.90 72.48 59.63 69.72 72.48 78.90 67.89 70.64 57.80 69.63 6.60
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERTwithAtt-bs4 75.23 62.39 67.89 74.31 61.47 75.23 77.06 66.97 67.89 68.81 69.72 5.22
BERTwithAttClsProjection-bs4 80.73 70.64 75.23 60.55 63.30 77.06 69.72 71.56 72.48 66.97 70.83 5.81
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4+Branches 76.15 72.48 71.56 66.06 57.80 74.31 71.56 68.81 71.56 69.72 70.00 4.86
EmbraceBERT-bs4+Branches+Dropout0.1 78.90 65.14 65.14 69.72 73.39 68.81 77.06 75.23 56.88 68.81 69.91 6.25
EmbraceBERT-bs4+Branches+Dropout0.3 68.81 66.06 68.81 66.06 66.06 76.15 72.48 71.56 64.22 74.31 69.45 3.81
CondensedEmbraceBERT-bs4+Branches 75.23 68.81 72.48 60.55 65.14 74.31 70.64 73.39 69.72 66.97 69.72 4.32
CondensedEmbraceBERT-bs4+Branches+Dropout0.1 80.73 65.14 73.39 70.64 76.15 69.72 74.31 73.39 66.97 68.81 71.93 4.38
CondensedEmbraceBERT-bs4+Branches+Dropout0.3 63.30 64.22 69.72 69.72 66.06 70.64 72.48 64.22 59.63 69.72 66.97 3.87
FrozenEmbraceBERT-bs4+Branches 80.73 69.72 75.23 66.06 60.55 76.15 74.31 70.64 70.64 64.22 70.83 5.76
FrozenEmbraceBERT-bs4+Branches+Dropout0.1 78.90 66.97 71.56 71.56 70.64 70.64 84.40 79.82 53.21 66.97 71.47 8.17
FrozenEmbraceBERT-bs4+Branches+Dropout0.3 69.72 67.89 66.06 70.64 58.72 74.31 68.81 67.89 66.06 72.48 68.26 4.05
FrozenCEBERT-bs4+Branches 74.31 65.14 74.31 73.39 77.06 66.97 55.05 75.23 70.64 57.80 68.99 7.22
FrozenCEBERT-bs4+Branches+Dropout0.1 78.90 68.81 70.64 67.89 75.23 77.98 78.90 78.90 65.14 69.72 73.21 5.06
FrozenCEBERT-bs4+Branches+Dropout0.3 68.81 70.64 63.30 73.39 66.97 81.65 72.48 65.14 58.72 66.97 68.81 5.96
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 75.23 70.64 77.98 69.72 66.97 67.89 66.06 73.39 67.89 80.73 71.65 4.74
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery_attclsqw 68.81 74.31 73.39 68.81 66.97 59.63 71.56 71.56 71.56 59.63 68.62 4.95
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 70.64 77.98 78.90 74.31 74.31 58.72 69.72 66.97 70.64 73.39 71.56 5.52
BERT-bs16+Dropout0.1 69.72 75.23 74.31 77.06 70.64 60.55 71.56 65.14 73.39 67.89 70.55 4.74
BERT-bs16+Dropout0.3 64.22 77.98 77.06 70.64 71.56 66.06 72.48 66.97 68.81 64.22 70.00 4.64
FrozenBERT-bs16-ep100 71.56 80.73 76.15 75.23 75.23 65.14 67.89 70.64 67.89 74.31 72.48 4.48
FrozenBERT-bs16-ep100+Dropout0.1 68.81 74.31 75.23 78.90 73.39 66.06 75.23 71.56 70.64 68.81 72.29 3.64
FrozenBERT-bs16-ep100+Dropout0.3 71.56 75.23 76.15 75.23 71.56 66.06 73.39 72.48 67.89 66.06 71.56 3.55
EmbraceBERT-bs16 70.64 66.06 73.39 71.56 62.39 70.64 63.30 71.56 65.14 67.89 68.26 3.65
EmbraceBERT-bs16+Dropout0.1 65.14 66.06 73.39 68.81 66.06 69.72 61.47 60.55 71.56 68.81 67.16 3.91
EmbraceBERT-bs16+Dropout0.3 67.89 70.64 68.81 64.22 66.06 70.64 54.13 62.39 71.56 69.72 66.61 5.03
CondensedEmbraceBERT-bs16 40.37 17.43 22.02 29.36 20.18 6.42 19.27 2.75 18.35 9.17 18.53 10.46
CondensedEmbraceBERT-bs16+Dropout0.1 64.22 66.06 70.64 67.89 62.39 72.48 62.39 70.64 70.64 67.89 67.52 3.46
CondensedEmbraceBERT-bs16+Dropout0.3 66.06 66.06 66.97 65.14 63.30 75.23 56.88 70.64 64.22 69.72 66.42 4.63
FrozenEBERT-bs16-ep100 72.48 66.97 66.06 67.89 66.06 75.23 67.89 69.72 66.97 62.39 68.17 3.41
FrozenEBERT-bs16-ep100+Dropout0.1 73.39 64.22 70.64 61.47 76.15 73.39 77.98 63.30 66.97 63.30 69.08 5.67
FrozenEBERT-bs16-ep100+Dropout0.3 71.56 70.64 69.72 63.30 71.56 72.48 64.22 68.81 70.64 66.06 68.90 3.08
FrozenCEBERT-bs16-ep100 44.95 15.60 15.60 33.94 19.27 8.26 22.94 3.67 19.27 7.34 19.08 11.93
FrozenCEBERT-bs16-ep100+Dropout0.1 72.48 69.72 64.22 69.72 65.14 76.15 76.15 72.48 67.89 63.30 69.72 4.40
FrozenCEBERT-bs16-ep100+Dropout0.3 72.48 72.48 69.72 70.64 66.06 66.97 66.97 66.06 64.22 70.64 68.62 2.78
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16+Branches 56.88 66.97 69.72 71.56 60.55 69.72 57.80 73.39 67.89 74.31 66.88 5.99
EmbraceBERT-bs16+Branches+Dropout0.1 69.72 54.13 69.72 68.81 68.81 59.63 75.23 66.06 68.81 58.72 65.96 6.09
EmbraceBERT-bs16+Branches+Dropout0.3 68.81 57.80 69.72 61.47 76.15 55.96 77.06 71.56 64.22 71.56 67.43 6.93
CondensedEmbraceBERT-bs16+Branches 58.72 71.56 70.64 67.89 58.72 71.56 57.80 67.89 68.81 70.64 66.42 5.40
CondensedEmbraceBERT-bs16+Branches+Dropout0.1 65.14 67.89 69.72 68.81 73.39 64.22 66.97 73.39 64.22 59.63 67.34 4.07
CondensedEmbraceBERT-bs16+Branches+Dropout0.3 68.81 61.47 71.56 66.97 74.31 57.80 71.56 69.72 65.14 67.89 67.52 4.72
FrozenEmbraceBERT-bs16+Branches 57.80 67.89 71.56 66.06 63.30 70.64 71.56 75.23 69.72 68.81 68.26 4.66
FrozenEmbraceBERT-bs16+Branches+Dropout0.1 72.48 59.63 70.64 68.81 67.89 58.72 72.48 64.22 70.64 59.63 66.51 5.23
FrozenEmbraceBERT-bs16+Branches+Dropout0.3 68.81 60.55 72.48 66.06 76.15 60.55 75.23 64.22 62.39 68.81 67.52 5.46
FrozenCEBERT-bs16+Branches 72.48 66.97 72.48 53.21 68.81 62.39 80.73 66.97 66.97 64.22 67.52 6.84
FrozenCEBERT-bs16+Branches+Dropout0.1 65.14 64.22 69.72 66.06 73.39 66.97 77.06 68.81 70.64 58.72 68.07 4.85
FrozenCEBERT-bs16+Branches+Dropout0.3 66.97 60.55 70.64 67.89 72.48 70.64 74.31 66.06 60.55 68.81 67.89 4.36
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery 67.89 64.22 69.72 61.47 78.90 59.63 80.73 70.64 69.72 77.98 70.09 6.90
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 68.81 85.32 62.39 67.89 73.39 55.05 69.72 50.46 82.57 66.97 68.26 10.26
RoBERTa-bs4+Dropout0.1 61.47 65.14 74.31 54.13 75.23 60.55 65.14 54.13 82.57 76.15 66.88 9.26
RoBERTa-bs4+Dropout0.3 55.96 54.13 70.64 64.22 69.72 64.22 67.89 73.39 81.65 71.56 67.34 7.74
FrozenRoBERTa-bs4-ep100 69.72 84.40 62.39 77.06 72.48 58.72 64.22 51.38 78.90 69.72 68.90 9.47
FrozenRoBERTa-bs4-ep100+Dropout0.1 56.88 65.14 70.64 62.39 72.48 67.89 76.15 74.31 77.98 78.90 70.28 6.81
FrozenRoBERTa-bs4-ep100+Dropout0.3 45.87 72.48 66.06 72.48 75.23 78.90 78.90 51.38 78.90 80.73 70.09 11.56
EmbraceRoBERTa-bs4 82.57 71.56 80.73 82.57 80.73 70.64 75.23 66.97 73.39 76.15 76.06 5.18
EmbraceRoBERTa-bs4+Dropout0.1 84.40 54.13 74.31 72.48 78.90 69.72 64.22 71.56 79.82 76.15 72.57 8.17
EmbraceRoBERTa-bs4+Dropout0.3 80.73 53.21 62.39 77.98 82.57 62.39 73.39 68.81 79.82 75.23 71.65 9.19
CondensedEmbraceRoBERTa-bs4 34.86 40.37 4.59 48.62 12.84 46.79 9.17 15.60 5.50 18.35 23.67 16.36
CondensedEmbraceRoBERTa-bs4+Dropout0.1 80.73 57.80 68.81 80.73 83.49 57.80 70.64 63.30 78.90 75.23 71.74 9.13
CondensedEmbraceRoBERTa-bs4+Dropout0.3 77.06 60.55 65.14 78.90 77.06 62.39 58.72 75.23 77.06 71.56 70.37 7.45
FrozenERoBERTa-bs4-ep100 87.16 70.64 78.90 81.65 78.90 66.97 73.39 67.89 76.15 81.65 76.33 6.22
FrozenERoBERTa-bs4-ep100+Dropout0.1 76.15 67.89 57.80 76.15 83.49 58.72 55.96 69.72 66.97 73.39 68.62 8.58
FrozenERoBERTa-bs4-ep100+Dropout0.3 77.06 70.64 67.89 75.23 82.57 68.81 65.14 75.23 63.30 80.73 72.66 6.18
FrozenCERoBERTa-bs4-ep100 37.61 45.87 7.34 45.87 16.51 44.04 11.01 14.68 4.59 13.76 24.13 16.17
FrozenCERoBERTa-bs4-ep100+Dropout0.1 80.73 59.63 61.47 71.56 82.57 67.89 55.96 74.31 75.23 74.31 *70.37 8.49
FrozenCERoBERTa-bs4-ep100+Dropout0.3 84.40 64.22 58.72 63.30 83.49 67.89 60.55 79.82 76.15 75.23 *71.38 9.12
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceRoBERTa-bs4+Branches 77.06 68.81 68.81 75.23 72.48 62.39 62.39 66.97 57.80 69.72 68.17 5.72
EmbraceRoBERTa-bs4+Branches+Dropout0.1 77.98 70.64 70.64 78.90 83.49 74.31 58.72 66.97 72.48 66.06 72.02 6.79
EmbraceRoBERTa-bs4+Branches+Dropout0.3 66.97 76.15 72.48 80.73 75.23 75.23 59.63 70.64 61.47 72.48 71.10 6.31
CondensedEmbraceRoBERTa-bs4+Branches 76.15 55.05 66.06 78.90 71.56 65.14 55.96 59.63 71.56 69.72 66.97 7.73
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.1 71.56 56.88 61.47 78.90 82.57 59.63 70.64 60.55 63.30 70.64 67.61 8.19
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.3 66.97 72.48 62.39 75.23 78.90 61.47 57.80 69.72 75.23 66.06 68.62 6.52
FrozenEmbraceRoBERTa-bs4+Branches 83.49 66.97 69.72 77.98 71.56 58.72 67.89 60.55 54.13 70.64 68.17 8.35
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.1 77.06 71.56 72.48 78.90 81.65 76.15 58.72 66.06 71.56 69.72 72.39 6.32
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.3 66.97 72.48 72.48 79.82 76.15 71.56 60.55 66.06 62.39 72.48 70.09 5.72
FrozenCERoBERTa-bs4+Branches 74.31 54.13 71.56 82.57 70.64 64.22 57.80 60.55 74.31 70.64 68.07 8.28
FrozenCERoBERTa-bs4+Branches+Dropout0.1 77.98 60.55 57.80 82.57 82.57 63.30 66.06 65.14 63.30 69.72 68.90 8.57
FrozenCERoBERTa-bs4+Branches+Dropout0.3 67.89 72.48 61.47 74.31 76.15 61.47 61.47 67.89 71.56 69.72 68.44 5.18
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 52.29 54.13 61.47 55.05 73.39 68.81 77.98 69.72 72.48 67.89 65.32 8.55
RoBERTa-bs16+Dropout0.1 62.39 62.39 69.72 60.55 66.97 64.22 71.56 67.89 66.06 63.30 65.50 3.36
RoBERTa-bs16+Dropout0.3 60.55 59.63 66.06 56.88 68.81 70.64 69.72 66.06 66.97 71.56 65.69 4.77
FrozenRoBERTa-bs16-ep100 55.96 55.05 63.30 57.80 69.72 68.81 78.90 67.89 74.31 69.72 66.15 7.55
FrozenRoBERTa-bs16-ep100+Dropout0.1 57.80 58.72 58.72 63.30 72.48 77.98 77.06 58.72 71.56 65.14 66.15 7.57
FrozenRoBERTa-bs16-ep100+Dropout0.3 55.96 78.90 62.39 64.22 72.48 74.31 79.82 67.89 65.14 68.81 68.99 7.13
EmbraceRoBERTa-bs16 76.15 64.22 69.72 71.56 73.39 72.48 62.39 64.22 68.81 72.48 69.54 4.34
EmbraceRoBERTa-bs16+Dropout0.1 78.90 70.64 60.55 72.48 77.06 67.89 65.14 69.72 64.22 78.90 70.55 6.02
EmbraceRoBERTa-bs16+Dropout0.3 70.64 61.47 55.05 74.31 77.98 73.39 58.72 72.48 65.14 76.15 68.53 7.51
CondensedEmbraceRoBERTa-bs16 32.11 36.70 11.01 25.69 16.51 48.62 6.42 13.76 16.51 21.10 22.84 12.35
CondensedEmbraceRoBERTa-bs16+Dropout0.1 73.39 66.06 61.47 81.65 77.98 66.06 60.55 77.06 62.39 80.73 70.73 7.89
CondensedEmbraceRoBERTa-bs16+Dropout0.3 79.82 64.22 55.05 76.15 77.06 62.39 59.63 76.15 66.06 78.90 69.54 8.58
FrozenERoBERTa-bs16-ep100 81.65 68.81 66.97 68.81 75.23 67.89 71.56 66.06 71.56 74.31 71.28 4.50
FrozenERoBERTa-bs16-ep100+Dropout0.1 81.65 59.63 55.05 77.98 78.90 66.97 63.30 69.72 65.14 71.56 68.99 8.24
FrozenERoBERTa-bs16-ep100+Dropout0.3 75.23 66.06 55.05 73.39 77.98 66.97 62.39 69.72 72.48 70.64 68.99 6.37
FrozenCERoBERTa-bs16-ep100 30.28 33.94 14.68 23.85 12.84 41.28 6.42 13.76 14.68 11.01 20.28 10.86
FrozenCERoBERTa-bs16-ep100+Dropout0.1 82.57 68.81 66.06 80.73 80.73 67.89 61.47 73.39 59.63 74.31 71.56 7.72
FrozenCERoBERTa-bs16-ep100+Dropout0.3 77.06 66.06 59.63 83.49 79.82 67.89 56.88 69.72 59.63 68.81 68.90 8.51
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceRoBERTa-bs16+Branches 77.06 61.47 64.22 75.23 79.82 69.72 57.80 66.97 60.55 69.72 68.26 7.05
EmbraceRoBERTa-bs16+Branches+Dropout0.1 70.64 66.06 72.48 72.48 77.98 67.89 56.88 67.89 55.05 74.31 68.17 6.94
EmbraceRoBERTa-bs16+Branches+Dropout0.3 71.56 67.89 69.72 72.48 79.82 67.89 60.55 66.06 70.64 62.39 68.90 5.15
CondensedEmbraceRoBERTa-bs16+Branches 72.48 57.80 66.06 77.98 77.98 70.64 57.80 61.47 59.63 65.14 66.70 7.36
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1 74.31 66.97 67.89 74.31 76.15 66.06 55.96 70.64 61.47 71.56 68.53 5.96
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3 74.31 65.14 63.30 67.89 77.98 65.14 55.05 74.31 62.39 65.14 67.06 6.45
FrozenEmbraceRoBERTa-bs16+Branches 77.98 64.22 63.30 77.06 80.73 47.71 59.63 62.39 62.39 67.89 66.33 9.47
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1 72.48 66.97 69.72 73.39 76.15 67.89 61.47 65.14 54.13 68.81 67.61 6.02
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3 71.56 68.81 67.89 67.89 81.65 66.97 62.39 62.39 70.64 62.39 68.26 5.49
FrozenCERoBERTa-bs16+Branches 72.48 56.88 66.06 75.23 76.15 69.72 55.05 66.97 60.55 66.06 66.51 6.89
FrozenCERoBERTa-bs16+Branches+Dropout0.1 75.23 67.89 64.22 74.31 77.98 70.64 55.05 72.48 64.22 72.48 69.45 6.44
FrozenCERoBERTa-bs16+Branches+Dropout0.3 73.39 67.89 60.55 73.39 76.15 63.30 55.05 75.23 61.47 66.06 67.25 6.80

  • Epoch 100 - bs4 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |

  • Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |73.39|71.56|77.06|77.06|71.56|66.97|73.39|71.56|73.39|73.39|72.94|2.76| | BERTwithAtt-bs8 |72.48|66.06|74.31|68.81|64.22|72.48|70.64|71.56|77.98|73.39|71.19|3.81| | BERTwithAttClsProjection-bs8 |78.90|63.30|71.56|68.81|69.72|75.23|72.48|74.31|67.89|67.89|71.01|4.23| | BERTwithProjection-bs8 |67.89|60.55|72.48|68.81|67.89|61.47|73.39|72.48|65.14|69.72|67.98|4.23| | BERTwithProjectionAtt-bs8 |68.81|66.97|61.47|66.06|67.89|78.90|70.64|67.89|67.89|74.31|69.08|4.50| | EmbraceBERT-bs8 |78.90|65.14|67.89|72.48|58.72|75.23|70.64|72.48|66.97|68.81|69.72|5.33| | EmbraceBERT-bs8-p_att_clsquery_weights |55.05|58.72|71.56|66.06|60.55|72.48|70.64|57.80|66.97|66.06|64.59|5.88| | EmbraceBERTwithProj-bs8 |73.39|83.49|72.48|79.82|66.06|66.06|76.15|60.55|63.30|66.06|70.73|7.14| | EmbraceBERTwithProj-bs8-p_attclsqw |69.72|66.06|76.15|58.72|66.06|70.64|69.72|77.06|77.98|78.90|71.10|6.14| | EBERTconcatAtt-bs8 |74.31|77.98|78.90|75.23|64.22|73.39|71.56|74.31|66.06|66.06|72.20|4.87| | EBERTconcatAtt-bs8_attclsqw |68.81|68.81|65.14|64.22|61.47|71.56|68.81|71.56|74.31|69.72|68.44|3.65| | EBERTconcatAttwithProj-bs8 |76.15|66.97|74.31|66.97|74.31|59.63|70.64|70.64|79.82|55.05|69.45|7.20| | EBERTconcatAttwithProj-bs8_attclsqw |62.39|77.06|64.22|76.15|58.72|76.15|66.06|74.31|77.06|73.39|70.55|6.61| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |69.72|66.06|66.97|57.80|74.31|51.38|69.72|66.97|67.89|73.39|66.42|6.61| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |73.39|75.23|77.06|68.81|58.72|68.81|65.14|69.72|70.64|77.06|70.46|5.39| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |70.64|77.98|72.48|81.65|61.47|72.48|70.64|66.06|71.56|70.64|71.56|5.29| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |75.23|68.81|71.56|73.39|75.23|66.97|66.97|74.31|69.72|75.23|71.74|3.23| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |65.14|70.64|74.31|69.72|55.96|75.23|73.39|70.64|52.29|66.97|67.43|7.32| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |58.72|73.39|68.81|68.81|77.06|70.64|69.72|75.23|66.06|72.48|70.09|4.91| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |67.89|77.06|66.06|71.56|74.31|59.63|74.31|70.64|79.82|68.81|71.01|5.52| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |61.47|69.72|66.06|70.64|71.56|62.39|62.39|74.31|77.06|58.72|67.43|5.81|

gtts-witai - GOOD with BERT and RoBERTa all

  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 71.56 86.24 84.40 87.16 79.82 83.49 79.82 80.73 74.31 78.90 80.64 4.72
BERT-bs4+Dropout0.1 71.56 85.32 88.07 84.40 75.23 79.82 82.57 78.90 71.56 77.06 79.45 5.40
BERT-bs4+Dropout0.3 79.82 82.57 81.65 79.82 83.49 82.57 80.73 77.98 83.49 73.39 80.55 2.92
FrozenBERT-bs4-ep100 73.39 83.49 83.49 89.91 78.90 83.49 79.82 77.98 70.64 76.15 79.72 5.34
FrozenBERT-bs4-ep100+Dropout0.1 68.81 83.49 80.73 82.57 81.65 78.90 79.82 76.15 74.31 85.32 79.17 4.66
FrozenBERT-bs4-ep100+Dropout0.3 68.81 79.82 80.73 81.65 77.98 82.57 77.06 77.98 74.31 83.49 78.44 4.15
EmbraceBERT-bs4 77.06 84.40 82.57 84.40 84.40 78.90 86.24 85.32 79.82 80.73 82.39 2.92
EmbraceBERT-bs4+Dropout0.1 74.31 84.40 79.82 84.40 88.07 80.73 84.40 77.06 79.82 78.90 81.19 3.90
EmbraceBERT-bs4+Dropout0.3 73.39 81.65 77.06 79.82 85.32 86.24 79.82 85.32 83.49 78.90 81.10 3.90
CondensedEmbraceBERT-bs4 37.61 12.84 9.17 30.28 26.61 16.51 18.35 42.20 33.94 11.93 23.94 11.13
CondensedEmbraceBERT-bs4+Dropout0.1 87.16 82.57 81.65 78.90 83.49 87.16 83.49 79.82 80.73 85.32 83.03 2.73
CondensedEmbraceBERT-bs4+Dropout0.3 82.57 81.65 79.82 83.49 84.40 84.40 86.24 87.16 90.83 71.56 83.21 4.86
FrozenEBERT-bs4-ep100 77.06 83.49 81.65 85.32 86.24 81.65 85.32 83.49 79.82 79.82 82.39 2.78
FrozenEBERT-bs4-ep100+Dropout0.1 79.82 83.49 71.56 79.82 79.82 84.40 82.57 79.82 77.06 72.48 79.08 4.08
FrozenEBERT-bs4-ep100+Dropout0.3 81.65 79.82 70.64 82.57 87.16 84.40 76.15 86.24 77.06 80.73 80.64 4.76
FrozenCEBERT-bs4-ep100 44.04 11.93 11.01 29.36 16.51 16.51 19.27 50.46 30.28 5.50 23.49 13.99
FrozenCEBERT-bs4-ep100+Dropout0.1 83.49 88.07 75.23 88.07 77.06 87.16 83.49 82.57 79.82 68.81 81.38 5.92
FrozenCEBERT-bs4-ep100+Dropout0.3 79.82 88.99 78.90 73.39 82.57 86.24 90.83 87.16 80.73 68.81 81.74 6.60
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERTwithAtt-bs4 85.32 76.15 77.06 84.40 74.31 88.07 85.32 81.65 77.06 76.15 80.55 4.69
BERTwithAttClsProjection-bs4 84.40 77.06 81.65 69.72 77.06 84.40 76.15 74.31 85.32 74.31 78.44 4.98
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4+Branches 88.07 85.32 88.07 79.82 81.65 87.16 82.57 83.49 91.74 84.40 85.23 3.39
EmbraceBERT-bs4+Branches+Dropout0.1 90.83 80.73 79.82 78.90 81.65 79.82 85.32 86.24 66.97 82.57 81.28 5.91
EmbraceBERT-bs4+Branches+Dropout0.3 85.32 83.49 77.06 74.31 78.90 82.57 84.40 76.15 74.31 81.65 79.82 3.98
CondensedEmbraceBERT-bs4+Branches 86.24 82.57 86.24 74.31 84.40 86.24 82.57 78.90 87.16 79.82 82.84 3.91
CondensedEmbraceBERT-bs4+Branches+Dropout0.1 88.99 77.06 80.73 81.65 84.40 79.82 82.57 84.40 79.82 77.98 81.74 3.34
CondensedEmbraceBERT-bs4+Branches+Dropout0.3 78.90 79.82 78.90 79.82 78.90 82.57 82.57 72.48 75.23 80.73 78.99 2.94
FrozenEmbraceBERT-bs4+Branches 90.83 81.65 87.16 79.82 81.65 85.32 81.65 79.82 90.83 80.73 83.94 4.09
FrozenEmbraceBERT-bs4+Branches+Dropout0.1 90.83 77.06 80.73 79.82 79.82 84.40 88.07 86.24 64.22 83.49 81.47 7.00
FrozenEmbraceBERT-bs4+Branches+Dropout0.3 84.40 81.65 76.15 77.98 76.15 84.40 79.82 77.98 77.98 81.65 79.82 2.93
FrozenCEBERT-bs4+Branches 85.32 83.49 82.57 81.65 83.49 80.73 67.89 84.40 80.73 66.97 79.72 6.31
FrozenCEBERT-bs4+Branches+Dropout0.1 85.32 79.82 76.15 77.06 82.57 80.73 85.32 88.07 81.65 78.90 81.56 3.63
FrozenCEBERT-bs4+Branches+Dropout0.3 80.73 80.73 67.89 82.57 82.57 87.16 85.32 78.90 73.39 80.73 80.00 5.35
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 83.49 74.31 84.40 81.65 80.73 78.90 77.06 82.57 77.98 88.07 80.92 3.80
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery_attclsqw 79.82 82.57 79.82 83.49 78.90 71.56 85.32 83.49 78.90 79.82 80.37 3.63
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 77.98 88.07 87.16 81.65 84.40 77.06 78.90 80.73 77.06 82.57 81.56 3.79
BERT-bs16+Dropout0.1 76.15 88.07 85.32 78.90 82.57 77.06 78.90 80.73 76.15 78.90 80.28 3.77
BERT-bs16+Dropout0.3 73.39 88.07 85.32 77.06 83.49 79.82 79.82 81.65 76.15 76.15 80.09 4.36
FrozenBERT-bs16-ep100 77.98 88.99 83.49 79.82 83.49 76.15 76.15 80.73 71.56 84.40 80.28 4.79
FrozenBERT-bs16-ep100+Dropout0.1 74.31 84.40 81.65 78.90 82.57 77.98 78.90 78.90 71.56 80.73 78.99 3.61
FrozenBERT-bs16-ep100+Dropout0.3 77.98 83.49 83.49 81.65 85.32 77.98 80.73 78.90 70.64 77.98 79.82 3.96
EmbraceBERT-bs16 80.73 82.57 81.65 85.32 83.49 81.65 77.98 80.73 81.65 83.49 81.93 1.88
EmbraceBERT-bs16+Dropout0.1 82.57 80.73 87.16 81.65 85.32 80.73 70.64 77.06 83.49 79.82 80.92 4.36
EmbraceBERT-bs16+Dropout0.3 83.49 85.32 82.57 78.90 85.32 85.32 65.14 79.82 80.73 80.73 80.73 5.67
CondensedEmbraceBERT-bs16 42.20 18.35 17.43 21.10 13.76 11.01 15.60 8.26 14.68 10.09 17.25 9.12
CondensedEmbraceBERT-bs16+Dropout0.1 83.49 76.15 81.65 82.57 83.49 82.57 78.90 79.82 84.40 77.98 81.10 2.60
CondensedEmbraceBERT-bs16+Dropout0.3 84.40 82.57 85.32 81.65 82.57 85.32 74.31 78.90 74.31 80.73 81.01 3.85
FrozenEBERT-bs16-ep100 79.82 83.49 76.15 84.40 84.40 84.40 78.90 81.65 79.82 79.82 81.28 2.70
FrozenEBERT-bs16-ep100+Dropout0.1 88.07 78.90 81.65 77.06 88.07 85.32 82.57 73.39 84.40 77.06 81.65 4.71
FrozenEBERT-bs16-ep100+Dropout0.3 86.24 88.07 76.15 79.82 85.32 88.99 76.15 81.65 79.82 79.82 82.20 4.44
FrozenCEBERT-bs16-ep100 51.38 18.35 14.68 26.61 17.43 9.17 19.27 10.09 14.68 4.59 18.62 12.36
FrozenCEBERT-bs16-ep100+Dropout0.1 87.16 83.49 79.82 81.65 82.57 88.07 84.40 80.73 81.65 76.15 82.57 3.31
FrozenCEBERT-bs16-ep100+Dropout0.3 86.24 85.32 84.40 79.82 81.65 83.49 85.32 76.15 75.23 80.73 81.83 3.67
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16+Branches 77.98 80.73 84.40 81.65 82.57 81.65 77.98 82.57 83.49 77.98 81.10 2.25
EmbraceBERT-bs16+Branches+Dropout0.1 82.57 70.64 77.06 80.73 85.32 77.98 88.07 77.98 77.06 73.39 79.08 4.99
EmbraceBERT-bs16+Branches+Dropout0.3 74.31 66.06 74.31 74.31 85.32 77.06 86.24 80.73 71.56 76.15 76.61 5.82
CondensedEmbraceBERT-bs16+Branches 80.73 81.65 82.57 80.73 84.40 77.06 76.15 80.73 83.49 79.82 80.73 2.46
CondensedEmbraceBERT-bs16+Branches+Dropout0.1 82.57 70.64 75.23 80.73 87.16 78.90 83.49 76.15 75.23 73.39 78.35 4.86
CondensedEmbraceBERT-bs16+Branches+Dropout0.3 79.82 73.39 74.31 83.49 88.07 77.06 87.16 72.48 75.23 78.90 78.99 5.33
FrozenEmbraceBERT-bs16+Branches 79.82 78.90 82.57 80.73 84.40 81.65 85.32 83.49 85.32 79.82 82.20 2.25
FrozenEmbraceBERT-bs16+Branches+Dropout0.1 85.32 70.64 79.82 81.65 84.40 77.06 82.57 76.15 79.82 74.31 79.17 4.40
FrozenEmbraceBERT-bs16+Branches+Dropout0.3 72.48 69.72 81.65 77.98 85.32 78.90 87.16 74.31 71.56 74.31 77.34 5.61
FrozenCEBERT-bs16+Branches 82.57 80.73 81.65 67.89 80.73 75.23 89.91 77.98 76.15 77.06 78.99 5.43
FrozenCEBERT-bs16+Branches+Dropout0.1 77.06 73.39 80.73 79.82 87.16 78.90 86.24 75.23 74.31 72.48 78.53 4.84
FrozenCEBERT-bs16+Branches+Dropout0.3 81.65 75.23 77.98 83.49 80.73 81.65 85.32 75.23 66.06 79.82 78.72 5.23
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery 83.49 72.48 82.57 77.98 83.49 75.23 87.16 83.49 80.73 81.65 80.83 4.17
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 80.73 88.07 75.23 80.73 86.24 73.39 78.90 72.48 86.24 81.65 80.37 5.19
RoBERTa-bs4+Dropout0.1 72.48 77.98 82.57 70.64 83.49 75.23 76.15 70.64 83.49 85.32 77.80 5.33
RoBERTa-bs4+Dropout0.3 68.81 71.56 77.98 77.98 75.23 74.31 74.31 88.07 85.32 82.57 77.61 5.79
FrozenRoBERTa-bs4-ep100 80.73 88.07 75.23 84.40 85.32 76.15 77.98 71.56 88.07 83.49 81.10 5.40
FrozenRoBERTa-bs4-ep100+Dropout0.1 68.81 81.65 81.65 77.06 80.73 81.65 84.40 84.40 80.73 83.49 80.46 4.38
FrozenRoBERTa-bs4-ep100+Dropout0.3 65.14 84.40 77.06 87.16 85.32 86.24 87.16 66.06 87.16 84.40 81.01 8.21
EmbraceRoBERTa-bs4 84.40 88.07 89.91 88.99 88.07 77.06 82.57 78.90 82.57 84.40 84.50 4.11
EmbraceRoBERTa-bs4+Dropout0.1 86.24 68.81 84.40 85.32 88.99 81.65 81.65 78.90 86.24 84.40 82.66 5.36
EmbraceRoBERTa-bs4+Dropout0.3 88.99 73.39 76.15 87.16 88.99 74.31 82.57 82.57 87.16 82.57 82.39 5.64
CondensedEmbraceRoBERTa-bs4 38.53 41.28 1.83 44.95 9.17 48.62 3.67 15.60 8.26 12.84 22.48 17.61
CondensedEmbraceRoBERTa-bs4+Dropout0.1 87.16 73.39 85.32 87.16 87.16 73.39 85.32 74.31 83.49 82.57 81.93 5.58
CondensedEmbraceRoBERTa-bs4+Dropout0.3 87.16 76.15 84.40 87.16 86.24 76.15 77.06 84.40 87.16 84.40 83.03 4.44
FrozenERoBERTa-bs4-ep100 88.07 86.24 88.99 88.99 86.24 75.23 80.73 79.82 82.57 87.16 84.40 4.38
FrozenERoBERTa-bs4-ep100+Dropout0.1 83.49 76.15 70.64 87.16 88.07 75.23 73.39 83.49 83.49 81.65 80.28 5.69
FrozenERoBERTa-bs4-ep100+Dropout0.3 86.24 84.40 77.98 85.32 87.16 77.06 74.31 85.32 78.90 81.65 81.83 4.26
FrozenCERoBERTa-bs4-ep100 41.28 44.95 3.67 49.54 14.68 44.04 3.67 15.60 3.67 8.26 22.94 18.51
FrozenCERoBERTa-bs4-ep100+Dropout0.1 88.99 73.39 72.48 85.32 90.83 82.57 62.39 82.57 83.49 82.57 80.46 8.17
FrozenCERoBERTa-bs4-ep100+Dropout0.3 85.32 76.15 77.06 81.65 88.99 83.49 67.89 87.16 87.16 83.49 81.83 6.12
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceRoBERTa-bs4+Branches 80.73 80.73 78.90 84.40 85.32 81.65 76.15 81.65 74.31 78.90 80.28 3.21
EmbraceRoBERTa-bs4+Branches+Dropout0.1 88.99 83.49 81.65 88.99 88.99 86.24 69.72 81.65 79.82 83.49 83.30 5.55
EmbraceRoBERTa-bs4+Branches+Dropout0.3 77.98 83.49 86.24 88.07 85.32 81.65 71.56 79.82 79.82 79.82 81.38 4.51
CondensedEmbraceRoBERTa-bs4+Branches 82.57 67.89 78.90 87.16 82.57 78.90 63.30 76.15 86.24 80.73 78.44 7.23
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.1 82.57 71.56 76.15 90.83 88.99 73.39 86.24 81.65 80.73 84.40 81.65 6.07
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.3 79.82 88.07 75.23 84.40 88.99 71.56 74.31 82.57 83.49 77.06 80.55 5.62
FrozenEmbraceRoBERTa-bs4+Branches 85.32 79.82 77.06 85.32 84.40 75.23 79.82 76.15 66.97 78.90 78.90 5.30
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.1 88.99 82.57 84.40 88.99 87.16 85.32 66.97 78.90 77.06 82.57 82.29 6.33
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.3 77.06 83.49 83.49 88.07 86.24 81.65 72.48 75.23 78.90 80.73 80.73 4.64
FrozenCERoBERTa-bs4+Branches 78.90 65.14 81.65 88.99 84.40 77.06 67.89 78.90 83.49 79.82 78.62 6.89
FrozenCERoBERTa-bs4+Branches+Dropout0.1 82.57 74.31 72.48 91.74 88.07 76.15 77.06 82.57 77.98 85.32 80.83 5.95
FrozenCERoBERTa-bs4+Branches+Dropout0.3 77.98 85.32 73.39 86.24 88.07 70.64 77.06 81.65 82.57 79.82 80.28 5.34
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 67.89 70.64 73.39 72.48 85.32 77.06 83.49 82.57 85.32 78.90 77.71 6.06
RoBERTa-bs16+Dropout0.1 74.31 77.98 80.73 77.06 79.82 76.15 83.49 79.82 80.73 75.23 78.53 2.73
RoBERTa-bs16+Dropout0.3 69.72 77.98 76.15 71.56 81.65 79.82 82.57 75.23 82.57 84.40 78.17 4.69
FrozenRoBERTa-bs16-ep100 71.56 72.48 73.39 77.06 84.40 76.15 83.49 81.65 85.32 78.90 78.44 4.84
FrozenRoBERTa-bs16-ep100+Dropout0.1 66.97 77.98 71.56 78.90 84.40 84.40 85.32 76.15 81.65 74.31 78.17 5.74
FrozenRoBERTa-bs16-ep100+Dropout0.3 64.22 90.83 74.31 77.98 83.49 79.82 85.32 82.57 80.73 76.15 79.54 6.80
EmbraceRoBERTa-bs16 83.49 75.23 80.73 85.32 85.32 82.57 74.31 78.90 80.73 78.90 80.55 3.62
EmbraceRoBERTa-bs16+Dropout0.1 87.16 82.57 72.48 86.24 86.24 82.57 78.90 81.65 82.57 85.32 82.57 4.14
EmbraceRoBERTa-bs16+Dropout0.3 77.98 77.06 65.14 85.32 84.40 83.49 73.39 81.65 81.65 83.49 79.36 5.92
CondensedEmbraceRoBERTa-bs16 31.19 34.86 8.26 27.52 9.17 55.96 5.50 15.60 15.60 24.77 22.84 14.64
CondensedEmbraceRoBERTa-bs16+Dropout0.1 83.49 76.15 78.90 88.99 87.16 80.73 70.64 81.65 77.98 85.32 81.10 5.21
CondensedEmbraceRoBERTa-bs16+Dropout0.3 85.32 77.06 71.56 87.16 85.32 71.56 71.56 82.57 84.40 83.49 80.00 6.07
FrozenERoBERTa-bs16-ep100 85.32 77.98 80.73 84.40 85.32 80.73 79.82 81.65 82.57 80.73 81.93 2.32
FrozenERoBERTa-bs16-ep100+Dropout0.1 84.40 75.23 69.72 88.07 87.16 80.73 75.23 82.57 82.57 82.57 80.83 5.48
FrozenERoBERTa-bs16-ep100+Dropout0.3 82.57 77.06 73.39 85.32 86.24 78.90 71.56 81.65 84.40 79.82 80.09 4.70
FrozenCERoBERTa-bs16-ep100 29.36 37.61 8.26 23.85 14.68 49.54 3.67 14.68 13.76 9.17 20.46 13.79
FrozenCERoBERTa-bs16-ep100+Dropout0.1 85.32 78.90 81.65 88.07 84.40 81.65 73.39 82.57 77.06 84.40 81.74 4.09
FrozenCERoBERTa-bs16-ep100+Dropout0.3 79.82 75.23 73.39 88.07 87.16 77.98 70.64 86.24 75.23 77.98 79.17 5.77
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceRoBERTa-bs16+Branches 81.65 77.06 73.39 85.32 84.40 82.57 65.14 77.06 80.73 79.82 78.72 5.68
EmbraceRoBERTa-bs16+Branches+Dropout0.1 77.06 75.23 82.57 85.32 82.57 80.73 68.81 82.57 73.39 82.57 79.08 4.97
EmbraceRoBERTa-bs16+Branches+Dropout0.3 81.65 77.06 79.82 84.40 85.32 79.82 70.64 84.40 84.40 77.06 80.46 4.38
CondensedEmbraceRoBERTa-bs16+Branches 81.65 72.48 74.31 88.99 84.40 80.73 68.81 76.15 77.06 75.23 77.98 5.69
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1 82.57 75.23 83.49 85.32 86.24 77.98 68.81 82.57 77.06 82.57 80.18 5.09
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3 82.57 75.23 78.90 82.57 88.99 80.73 68.81 82.57 81.65 77.06 79.91 5.12
FrozenEmbraceRoBERTa-bs16+Branches 83.49 76.15 73.39 88.07 82.57 58.72 66.97 76.15 77.98 77.98 76.15 8.00
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1 79.82 75.23 81.65 88.07 83.49 80.73 70.64 83.49 69.72 79.82 79.27 5.51
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3 80.73 75.23 79.82 82.57 86.24 78.90 70.64 83.49 84.40 76.15 79.82 4.51
FrozenCERoBERTa-bs16+Branches 82.57 68.81 76.15 86.24 83.49 79.82 66.97 79.82 76.15 75.23 77.52 5.86
FrozenCERoBERTa-bs16+Branches+Dropout0.1 82.57 74.31 81.65 85.32 86.24 82.57 68.81 80.73 77.98 81.65 80.18 4.98
FrozenCERoBERTa-bs16+Branches+Dropout0.3 77.98 76.15 76.15 84.40 88.07 83.49 70.64 81.65 80.73 79.82 79.91 4.70

  • Epoch 100 - bs4 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |

  • Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |79.82|86.24|82.57|80.73|76.15|80.73|81.65|83.49|77.06|85.32|81.38|3.07| | BERTwithAtt-bs8 |77.06|76.15|86.24|80.73|75.23|82.57|80.73|80.73|88.99|86.24|81.47|4.38| | BERTwithAttClsProjection-bs8 |87.16|72.48|76.15|83.49|84.40|83.49|83.49|82.57|84.40|77.06|81.47|4.38| | BERTwithProjection-bs8 |82.57|73.39|84.40|78.90|74.31|68.81|80.73|83.49|77.98|77.98|78.26|4.68| | BERTwithProjectionAtt-bs8 |82.57|77.06|79.82|77.98|77.98|84.40|79.82|77.06|76.15|81.65|79.45|2.57| | EmbraceBERT-bs8 |86.24|83.49|77.98|83.49|79.82|83.49|80.73|84.40|87.16|84.40|83.12|2.70| | EmbraceBERT-bs8-p_att_clsquery_weights |69.72|77.06|85.32|80.73|73.39|86.24|74.31|72.48|79.82|80.73|77.98|5.24| | EmbraceBERTwithProj-bs8 |76.15|89.91|82.57|86.24|67.89|72.48|78.90|80.73|76.15|72.48|78.35|6.39| | EmbraceBERTwithProj-bs8-p_attclsqw |71.56|80.73|85.32|65.14|76.15|80.73|82.57|80.73|86.24|87.16|79.63|6.57| | EBERTconcatAtt-bs8 |85.32|88.07|87.16|80.73|76.15|82.57|83.49|80.73|79.82|77.98|82.20|3.67| | EBERTconcatAtt-bs8_attclsqw |77.06|74.31|75.23|75.23|75.23|82.57|77.98|81.65|83.49|79.82|78.26|3.23| | EBERTconcatAttwithProj-bs8 |80.73|76.15|85.32|81.65|86.24|77.98|82.57|84.40|86.24|69.72|81.10|4.98| | EBERTconcatAttwithProj-bs8_attclsqw |77.06|84.40|77.06|85.32|73.39|85.32|79.82|84.40|88.99|84.40|82.02|4.65| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |77.98|73.39|75.23|74.31|81.65|71.56|77.98|79.82|79.82|82.57|77.43|3.49| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |84.40|83.49|85.32|80.73|75.23|84.40|83.49|82.57|80.73|85.32|82.57|2.90| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |74.31|87.16|84.40|89.91|68.81|86.24|73.39|74.31|82.57|75.23|79.63|6.85| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |86.24|80.73|85.32|82.57|87.16|81.65|76.15|86.24|79.82|86.24|83.21|3.43| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |83.49|83.49|86.24|82.57|71.56|86.24|85.32|85.32|65.14|76.15|80.55|6.85| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |68.81|85.32|88.99|78.90|87.16|71.56|83.49|88.99|77.98|85.32|81.65|6.75| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |80.73|85.32|76.15|77.06|83.49|71.56|85.32|77.06|83.49|79.82|80.00|4.30| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |73.39|82.57|69.72|79.82|80.73|76.15|72.48|80.73|79.82|82.57|77.80|4.32|

macsay-google - NOT GOOD with BERT, good with RoBERTa all

  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 80.73 81.65 83.49 83.49 83.49 78.90 82.57 84.40 82.57 81.65 82.29 1.54
BERT-bs4+Dropout0.1 81.65 81.65 81.65 82.57 86.24 79.82 86.24 83.49 80.73 82.57 82.66 2.03
BERT-bs4+Dropout0.3 82.57 82.57 82.57 84.40 88.99 82.57 86.24 82.57 85.32 82.57 84.04 2.10
FrozenBERT-bs4-ep100 79.82 81.65 83.49 83.49 80.73 79.82 83.49 83.49 81.65 80.73 81.83 1.47
FrozenBERT-bs4-ep100+Dropout0.1 83.49 79.82 81.65 82.57 86.24 79.82 87.16 81.65 80.73 81.65 82.48 2.37
FrozenBERT-bs4-ep100+Dropout0.3 80.73 80.73 81.65 83.49 85.32 81.65 88.07 84.40 82.57 78.90 82.75 2.52
EmbraceBERT-bs4 77.06 82.57 78.90 80.73 83.49 75.23 79.82 85.32 72.48 78.90 79.45 3.67
EmbraceBERT-bs4+Dropout0.1 82.57 81.65 81.65 84.40 80.73 79.82 85.32 77.06 80.73 85.32 81.93 2.46
EmbraceBERT-bs4+Dropout0.3 82.57 82.57 81.65 80.73 81.65 77.98 74.31 81.65 80.73 79.82 80.37 2.40
CondensedEmbraceBERT-bs4 32.11 23.85 7.34 25.69 32.11 26.61 9.17 55.05 36.70 13.76 26.24 13.48
CondensedEmbraceBERT-bs4+Dropout0.1 86.24 85.32 77.98 83.49 79.82 78.90 83.49 74.31 79.82 83.49 81.28 3.53
CondensedEmbraceBERT-bs4+Dropout0.3 81.65 84.40 80.73 81.65 80.73 79.82 82.57 85.32 81.65 78.90 81.74 1.86
FrozenEBERT-bs4-ep100 80.73 78.90 77.98 82.57 81.65 76.15 78.90 80.73 72.48 75.23 78.53 2.99
FrozenEBERT-bs4-ep100+Dropout0.1 83.49 81.65 79.82 80.73 75.23 80.73 84.40 77.98 78.90 78.90 80.18 2.54
FrozenEBERT-bs4-ep100+Dropout0.3 78.90 81.65 79.82 82.57 78.90 77.06 66.06 81.65 77.98 80.73 78.53 4.48
FrozenCEBERT-bs4-ep100 31.19 13.76 6.42 25.69 23.85 29.36 11.01 51.38 24.77 10.09 22.75 12.62
FrozenCEBERT-bs4-ep100+Dropout0.1 82.57 84.40 77.98 83.49 72.48 77.06 86.24 73.39 77.98 79.82 79.54 4.38
FrozenCEBERT-bs4-ep100+Dropout0.3 78.90 84.40 80.73 80.73 79.82 78.90 85.32 78.90 78.90 78.90 80.55 2.28
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERTwithAtt-bs4 80.73 83.49 84.40 87.16 82.57 85.32 84.40 83.49 80.73 79.82 83.21 2.17
BERTwithAttClsProjection-bs4 84.40 84.40 83.49 84.40 86.24 81.65 85.32 85.32 83.49 78.90 83.76 2.01
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4+Branches 84.40 86.24 81.65 83.49 78.90 79.82 82.57 82.57 77.98 79.82 81.74 2.48
EmbraceBERT-bs4+Branches+Dropout0.1 82.57 85.32 82.57 82.57 83.49 87.16 80.73 84.40 85.32 80.73 83.49 1.97
EmbraceBERT-bs4+Branches+Dropout0.3 77.98 83.49 81.65 81.65 81.65 85.32 82.57 81.65 86.24 88.07 83.03 2.73
CondensedEmbraceBERT-bs4+Branches 84.40 84.40 79.82 83.49 79.82 80.73 83.49 82.57 81.65 80.73 82.11 1.70
CondensedEmbraceBERT-bs4+Branches+Dropout0.1 82.57 85.32 83.49 80.73 80.73 82.57 81.65 82.57 85.32 82.57 82.75 1.52
CondensedEmbraceBERT-bs4+Branches+Dropout0.3 80.73 81.65 83.49 82.57 81.65 83.49 83.49 78.90 83.49 83.49 82.29 1.48
FrozenEmbraceBERT-bs4+Branches 82.57 80.73 80.73 85.32 77.98 81.65 85.32 83.49 81.65 82.57 82.20 2.10
FrozenEmbraceBERT-bs4+Branches+Dropout0.1 83.49 82.57 80.73 82.57 83.49 84.40 85.32 86.24 80.73 81.65 83.12 1.75
FrozenEmbraceBERT-bs4+Branches+Dropout0.3 76.15 83.49 79.82 82.57 80.73 86.24 83.49 84.40 85.32 85.32 82.75 2.92
FrozenCEBERT-bs4+Branches 77.06 86.24 84.40 84.40 79.82 87.16 68.81 84.40 81.65 87.16 82.11 5.40
FrozenCEBERT-bs4+Branches+Dropout0.1 80.73 87.16 82.57 82.57 79.82 82.57 78.90 85.32 84.40 80.73 82.48 2.44
FrozenCEBERT-bs4+Branches+Dropout0.3 80.73 84.40 78.90 86.24 82.57 87.16 80.73 82.57 82.57 83.49 82.94 2.40
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 83.49 82.57 83.49 80.73 80.73 84.40 82.57 82.57 79.82 83.49 82.39 1.41
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery_attclsqw 81.65 81.65 83.49 83.49 77.98 79.82 82.57 82.57 86.24 79.82 81.93 2.21
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 80.73 83.49 84.40 81.65 84.40 79.82 85.32 80.73 82.57 80.73 82.39 1.83
BERT-bs16+Dropout0.1 82.57 80.73 86.24 85.32 86.24 81.65 85.32 84.40 80.73 83.49 83.67 2.04
BERT-bs16+Dropout0.3 81.65 80.73 83.49 82.57 84.40 82.57 86.24 83.49 82.57 83.49 83.12 1.43
FrozenBERT-bs16-ep100 81.65 82.57 84.40 82.57 83.49 78.90 81.65 81.65 81.65 80.73 81.93 1.42
FrozenBERT-bs16-ep100+Dropout0.1 81.65 80.73 84.40 82.57 84.40 81.65 84.40 83.49 84.40 85.32 83.30 1.47
FrozenBERT-bs16-ep100+Dropout0.3 80.73 81.65 86.24 82.57 82.57 83.49 83.49 84.40 83.49 84.40 83.30 1.47
EmbraceBERT-bs16 82.57 83.49 82.57 84.40 81.65 83.49 77.06 83.49 82.57 81.65 82.29 1.93
EmbraceBERT-bs16+Dropout0.1 78.90 85.32 85.32 82.57 81.65 80.73 79.82 80.73 83.49 81.65 82.02 2.06
EmbraceBERT-bs16+Dropout0.3 79.82 83.49 84.40 82.57 81.65 82.57 74.31 80.73 84.40 79.82 81.38 2.84
CondensedEmbraceBERT-bs16 38.53 18.35 11.93 21.10 22.94 17.43 7.34 22.94 6.42 13.76 18.07 8.85
CondensedEmbraceBERT-bs16+Dropout0.1 82.57 80.73 83.49 82.57 81.65 82.57 79.82 82.57 83.49 84.40 82.39 1.28
CondensedEmbraceBERT-bs16+Dropout0.3 80.73 86.24 84.40 81.65 82.57 81.65 77.06 83.49 82.57 82.57 82.29 2.29
FrozenEBERT-bs16-ep100 79.82 81.65 80.73 85.32 82.57 84.40 77.98 81.65 81.65 77.06 81.28 2.43
FrozenEBERT-bs16-ep100+Dropout0.1 81.65 84.40 82.57 79.82 83.49 81.65 81.65 76.15 82.57 80.73 81.47 2.16
FrozenEBERT-bs16-ep100+Dropout0.3 81.65 81.65 79.82 79.82 83.49 80.73 80.73 81.65 83.49 79.82 81.28 1.31
FrozenCEBERT-bs16-ep100 44.95 16.51 12.84 40.37 22.02 14.68 10.09 13.76 6.42 7.34 18.90 12.66
FrozenCEBERT-bs16-ep100+Dropout0.1 82.57 84.40 80.73 84.40 79.82 79.82 81.65 83.49 80.73 81.65 81.93 1.64
FrozenCEBERT-bs16-ep100+Dropout0.3 80.73 85.32 81.65 81.65 77.98 77.06 81.65 80.73 84.40 81.65 81.28 2.36
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16+Branches 77.06 81.65 80.73 83.49 79.82 82.57 78.90 82.57 77.98 82.57 80.73 2.09
EmbraceBERT-bs16+Branches+Dropout0.1 78.90 79.82 82.57 81.65 79.82 82.57 83.49 80.73 83.49 82.57 81.56 1.56
EmbraceBERT-bs16+Branches+Dropout0.3 79.82 79.82 82.57 82.57 82.57 83.49 78.90 80.73 79.82 80.73 81.10 1.49
CondensedEmbraceBERT-bs16+Branches 75.23 81.65 82.57 82.57 77.98 81.65 77.98 82.57 79.82 83.49 80.55 2.56
CondensedEmbraceBERT-bs16+Branches+Dropout0.1 81.65 79.82 82.57 82.57 82.57 82.57 79.82 81.65 82.57 82.57 81.83 1.07
CondensedEmbraceBERT-bs16+Branches+Dropout0.3 80.73 80.73 82.57 80.73 81.65 84.40 83.49 80.73 83.49 82.57 82.11 1.31
FrozenEmbraceBERT-bs16+Branches 77.98 84.40 80.73 85.32 79.82 81.65 79.82 82.57 82.57 84.40 81.93 2.25
FrozenEmbraceBERT-bs16+Branches+Dropout0.1 83.49 78.90 87.16 81.65 78.90 82.57 80.73 75.23 83.49 82.57 81.47 3.09
FrozenEmbraceBERT-bs16+Branches+Dropout0.3 77.06 80.73 87.16 83.49 79.82 84.40 78.90 76.15 78.90 80.73 80.73 3.23
FrozenCEBERT-bs16+Branches 83.49 86.24 85.32 80.73 84.40 79.82 80.73 85.32 82.57 81.65 83.03 2.14
FrozenCEBERT-bs16+Branches+Dropout0.1 77.98 79.82 83.49 78.90 86.24 84.40 82.57 77.98 81.65 79.82 81.28 2.70
FrozenCEBERT-bs16+Branches+Dropout0.3 77.98 81.65 84.40 81.65 77.06 84.40 80.73 78.90 79.82 79.82 80.64 2.34
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery 79.82 82.57 82.57 79.82 82.57 80.73 87.16 79.82 80.73 88.99 82.48 3.03
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 87.16 81.65 79.82 80.73 83.49 81.65 79.82 78.90 83.49 79.82 81.65 2.36
RoBERTa-bs4+Dropout0.1 82.57 80.73 81.65 81.65 83.49 84.40 77.06 80.73 83.49 84.40 82.02 2.10
RoBERTa-bs4+Dropout0.3 83.49 77.98 82.57 82.57 79.82 82.57 80.73 80.73 84.40 82.57 81.74 1.81
FrozenRoBERTa-bs4-ep100 84.40 78.90 81.65 80.73 82.57 83.49 77.06 78.90 87.16 77.06 81.19 3.13
FrozenRoBERTa-bs4-ep100+Dropout0.1 83.49 85.32 81.65 79.82 82.57 84.40 77.06 81.65 79.82 79.82 81.56 2.37
FrozenRoBERTa-bs4-ep100+Dropout0.3 81.65 82.57 81.65 84.40 82.57 84.40 81.65 77.06 85.32 80.73 82.20 2.22
EmbraceRoBERTa-bs4 83.49 84.40 83.49 80.73 81.65 80.73 81.65 80.73 80.73 79.82 81.74 1.45
EmbraceRoBERTa-bs4+Dropout0.1 85.32 79.82 85.32 81.65 80.73 79.82 84.40 84.40 85.32 80.73 82.75 2.28
EmbraceRoBERTa-bs4+Dropout0.3 79.82 78.90 84.40 82.57 83.49 81.65 80.73 85.32 88.07 80.73 82.57 2.66
CondensedEmbraceRoBERTa-bs4 43.12 38.53 5.50 44.04 11.93 39.45 4.59 17.43 12.84 8.26 22.57 15.74
CondensedEmbraceRoBERTa-bs4+Dropout0.1 83.49 83.49 79.82 83.49 82.57 81.65 80.73 81.65 82.57 83.49 82.29 1.23
CondensedEmbraceRoBERTa-bs4+Dropout0.3 79.82 83.49 82.57 83.49 79.82 77.98 81.65 86.24 85.32 80.73 82.11 2.47
FrozenERoBERTa-bs4-ep100 87.16 81.65 81.65 82.57 80.73 82.57 80.73 80.73 80.73 79.82 81.83 1.96
FrozenERoBERTa-bs4-ep100+Dropout0.1 82.57 84.40 80.73 83.49 82.57 78.90 81.65 80.73 84.40 82.57 82.20 1.65
FrozenERoBERTa-bs4-ep100+Dropout0.3 84.40 81.65 82.57 83.49 82.57 83.49 84.40 83.49 86.24 86.24 83.85 1.43
FrozenCERoBERTa-bs4-ep100 44.95 39.45 4.59 50.46 12.84 38.53 3.67 15.60 7.34 7.34 22.48 17.62
FrozenCERoBERTa-bs4-ep100+Dropout0.1 82.57 84.40 79.82 84.40 83.49 80.73 85.32 81.65 87.16 86.24 83.58 2.26
FrozenCERoBERTa-bs4-ep100+Dropout0.3 85.32 82.57 82.57 83.49 81.65 81.65 84.40 81.65 84.40 81.65 82.94 1.31
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceRoBERTa-bs4+Branches 81.65 81.65 80.73 78.90 85.32 86.24 83.49 83.49 84.40 85.32 83.12 2.22
EmbraceRoBERTa-bs4+Branches+Dropout0.1 81.65 88.07 80.73 81.65 83.49 86.24 81.65 83.49 84.40 84.40 83.58 2.19
EmbraceRoBERTa-bs4+Branches+Dropout0.3 82.57 85.32 85.32 79.82 84.40 86.24 80.73 80.73 83.49 81.65 83.03 2.14
CondensedEmbraceRoBERTa-bs4+Branches 82.57 83.49 86.24 80.73 81.65 80.73 81.65 82.57 82.57 80.73 82.29 1.59
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.1 86.24 85.32 81.65 84.40 83.49 81.65 83.49 84.40 82.57 85.32 83.85 1.49
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.3 84.40 86.24 80.73 82.57 83.49 85.32 81.65 82.57 85.32 83.49 83.58 1.66
FrozenEmbraceRoBERTa-bs4+Branches 82.57 80.73 80.73 80.73 81.65 86.24 83.49 79.82 80.73 85.32 82.20 2.06
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.1 81.65 87.16 81.65 80.73 83.49 84.40 80.73 87.16 82.57 83.49 83.30 2.24
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.3 81.65 82.57 83.49 80.73 86.24 87.16 80.73 79.82 83.49 83.49 82.94 2.25
FrozenCERoBERTa-bs4+Branches 78.90 84.40 83.49 82.57 83.49 82.57 82.57 81.65 81.65 82.57 82.39 1.41
FrozenCERoBERTa-bs4+Branches+Dropout0.1 82.57 85.32 81.65 82.57 83.49 83.49 83.49 85.32 80.73 86.24 83.49 1.64
FrozenCERoBERTa-bs4+Branches+Dropout0.3 80.73 87.16 81.65 79.82 81.65 84.40 80.73 83.49 84.40 83.49 82.75 2.12
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 86.24 82.57 80.73 81.65 79.82 87.16 77.98 82.57 86.24 81.65 82.66 2.86
RoBERTa-bs16+Dropout0.1 83.49 83.49 77.98 82.57 79.82 84.40 80.73 82.57 86.24 82.57 82.39 2.24
RoBERTa-bs16+Dropout0.3 81.65 85.32 77.98 82.57 80.73 83.49 77.98 82.57 85.32 84.40 82.20 2.54
FrozenRoBERTa-bs16-ep100 85.32 85.32 80.73 82.57 81.65 84.40 79.82 79.82 85.32 81.65 82.66 2.15
FrozenRoBERTa-bs16-ep100+Dropout0.1 83.49 83.49 78.90 80.73 79.82 82.57 77.06 78.90 82.57 77.98 80.55 2.24
FrozenRoBERTa-bs16-ep100+Dropout0.3 85.32 84.40 78.90 80.73 77.98 84.40 81.65 79.82 83.49 79.82 81.65 2.46
EmbraceRoBERTa-bs16 77.98 82.57 81.65 85.32 79.82 81.65 82.57 81.65 82.57 82.57 81.83 1.83
EmbraceRoBERTa-bs16+Dropout0.1 84.40 85.32 81.65 82.57 81.65 83.49 85.32 85.32 85.32 82.57 83.76 1.48
EmbraceRoBERTa-bs16+Dropout0.3 82.57 81.65 84.40 84.40 76.15 83.49 80.73 86.24 79.82 80.73 82.02 2.73
CondensedEmbraceRoBERTa-bs16 37.61 32.11 15.60 32.11 12.84 43.12 4.59 17.43 12.84 23.85 23.21 11.90
CondensedEmbraceRoBERTa-bs16+Dropout0.1 78.90 83.49 82.57 83.49 79.82 81.65 83.49 82.57 84.40 83.49 82.39 1.68
CondensedEmbraceRoBERTa-bs16+Dropout0.3 80.73 83.49 79.82 84.40 80.73 82.57 78.90 81.65 86.24 84.40 82.29 2.21
FrozenERoBERTa-bs16-ep100 77.06 82.57 81.65 84.40 83.49 79.82 78.90 81.65 83.49 81.65 81.47 2.16
FrozenERoBERTa-bs16-ep100+Dropout0.1 80.73 81.65 83.49 81.65 86.24 83.49 80.73 81.65 88.99 84.40 83.30 2.52
FrozenERoBERTa-bs16-ep100+Dropout0.3 77.06 79.82 83.49 82.57 84.40 85.32 80.73 83.49 87.16 82.57 82.66 2.74
FrozenCERoBERTa-bs16-ep100 36.70 30.28 11.93 34.86 12.84 38.53 4.59 16.51 14.68 15.60 21.65 11.56
FrozenCERoBERTa-bs16-ep100+Dropout0.1 85.32 81.65 84.40 83.49 82.57 82.57 83.49 84.40 84.40 85.32 83.76 1.16
FrozenCERoBERTa-bs16-ep100+Dropout0.3 81.65 81.65 84.40 79.82 83.49 82.57 84.40 83.49 85.32 84.40 83.12 1.60
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceRoBERTa-bs16+Branches 80.73 83.49 82.57 83.49 79.82 83.49 80.73 85.32 86.24 84.40 83.03 1.98
EmbraceRoBERTa-bs16+Branches+Dropout0.1 80.73 85.32 84.40 80.73 82.57 81.65 84.40 84.40 81.65 84.40 83.03 1.65
EmbraceRoBERTa-bs16+Branches+Dropout0.3 83.49 82.57 81.65 85.32 83.49 81.65 83.49 83.49 85.32 81.65 83.21 1.30
CondensedEmbraceRoBERTa-bs16+Branches 80.73 86.24 84.40 82.57 81.65 82.57 82.57 82.57 84.40 84.40 83.21 1.54
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1 81.65 81.65 83.49 82.57 81.65 84.40 80.73 84.40 83.49 80.73 82.48 1.33
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3 78.90 83.49 79.82 83.49 82.57 85.32 83.49 82.57 83.49 81.65 82.48 1.81
FrozenEmbraceRoBERTa-bs16+Branches 80.73 84.40 79.82 82.57 84.40 77.06 80.73 81.65 85.32 81.65 81.83 2.35
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1 83.49 81.65 83.49 81.65 82.57 81.65 86.24 82.57 82.57 84.40 83.03 1.38
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3 83.49 82.57 81.65 82.57 83.49 81.65 82.57 84.40 85.32 80.73 82.84 1.30
FrozenCERoBERTa-bs16+Branches 81.65 85.32 84.40 80.73 79.82 80.73 82.57 82.57 82.57 82.57 82.29 1.59
FrozenCERoBERTa-bs16+Branches+Dropout0.1 79.82 81.65 82.57 83.49 82.57 87.16 80.73 84.40 84.40 82.57 82.94 1.98
FrozenCERoBERTa-bs16+Branches+Dropout0.3 79.82 81.65 80.73 82.57 79.82 85.32 82.57 81.65 84.40 81.65 82.02 1.70

  • Epoch 100 - bs4 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |

  • Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |81.65|84.40|83.49|84.40|82.57|81.65|86.24|83.49|79.82|81.65|82.94|1.75| | BERTwithAtt-bs8 |83.49|84.40|83.49|86.24|80.73|84.40|81.65|83.49|82.57|80.73|83.12|1.65| | BERTwithAttClsProjection-bs8 |86.24|84.40|80.73|85.32|82.57|82.57|85.32|85.32|83.49|73.39|82.94|3.56| | BERTwithProjection-bs8 |75.23|79.82|82.57|80.73|80.73|83.49|84.40|82.57|82.57|80.73|81.28|2.43| | BERTwithProjectionAtt-bs8 |85.32|80.73|77.98|86.24|84.40|82.57|86.24|81.65|78.90|83.49|82.75|2.78| | EmbraceBERT-bs8 |85.32|83.49|78.90|78.90|77.98|81.65|78.90|85.32|83.49|82.57|81.65|2.66| | EmbraceBERT-bs8-p_att_clsquery_weights |81.65|78.90|82.57|83.49|76.15|79.82|84.40|71.56|77.06|81.65|79.72|3.73| | EmbraceBERTwithProj-bs8 |85.32|86.24|79.82|84.40|83.49|88.07|85.32|79.82|86.24|84.40|84.31|2.54| | EmbraceBERTwithProj-bs8-p_attclsqw |88.07|81.65|82.57|82.57|81.65|81.65|84.40|80.73|86.24|84.40|83.39|2.23| | EBERTconcatAtt-bs8 |84.40|84.40|84.40|84.40|83.49|84.40|84.40|78.90|81.65|76.15|82.66|2.77| | EBERTconcatAtt-bs8_attclsqw |83.49|81.65|81.65|80.73|82.57|82.57|82.57|84.40|86.24|82.57|82.84|1.48| | EBERTconcatAttwithProj-bs8 |81.65|83.49|81.65|83.49|85.32|78.90|85.32|86.24|84.40|81.65|83.21|2.13| | EBERTconcatAttwithProj-bs8_attclsqw |79.82|80.73|84.40|83.49|80.73|84.40|79.82|81.65|85.32|80.73|82.11|1.98| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |80.73|80.73|82.57|77.06|78.90|77.98|85.32|79.82|77.06|82.57|80.28|2.54| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |83.49|83.49|87.16|81.65|80.73|79.82|83.49|81.65|80.73|82.57|82.48|1.99| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |85.32|84.40|80.73|82.57|82.57|80.73|87.16|83.49|76.15|83.49|82.66|2.86| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |79.82|83.49|84.40|87.16|82.57|81.65|82.57|83.49|85.32|85.32|83.58|1.99| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |82.57|83.49|82.57|80.73|87.16|83.49|85.32|83.49|82.57|86.24|83.76|1.84| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |82.57|87.16|82.57|85.32|83.49|83.49|82.57|81.65|74.31|83.49|82.66|3.16| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |85.32|84.40|82.57|85.32|83.49|83.49|84.40|83.49|83.49|78.90|83.49|1.74| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |82.57|85.32|82.57|78.90|84.40|77.98|82.57|83.49|82.57|81.65|82.20|2.14|

macsay-sphinx - Not good with BERT, good with RoBERTa all

  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 41.28 44.04 44.95 42.20 47.71 42.20 49.54 37.61 40.37 43.12 43.30 3.30
BERT-bs4+Dropout0.1 37.61 49.54 46.79 41.28 33.03 32.11 49.54 44.95 29.36 40.37 40.46 6.94
BERT-bs4+Dropout0.3 44.95 47.71 43.12 44.95 45.87 25.69 52.29 38.53 39.45 36.70 41.93 6.96
FrozenBERT-bs4-ep100 42.20 39.45 44.04 32.11 31.19 44.04 37.61 39.45 40.37 34.86 38.53 4.34
FrozenBERT-bs4-ep100+Dropout0.1 37.61 48.62 41.28 50.46 40.37 29.36 52.29 31.19 37.61 38.53 40.73 7.32
FrozenBERT-bs4-ep100+Dropout0.3 44.95 43.12 27.52 44.95 51.38 38.53 52.29 37.61 37.61 40.37 41.83 6.89
EmbraceBERT-bs4 38.53 36.70 35.78 38.53 39.45 30.28 44.04 35.78 28.44 33.94 36.15 4.29
EmbraceBERT-bs4+Dropout0.1 34.86 34.86 38.53 36.70 33.03 41.28 31.19 36.70 37.61 39.45 36.42 2.87
EmbraceBERT-bs4+Dropout0.3 31.19 33.03 44.95 34.86 38.53 40.37 30.28 29.36 35.78 41.28 35.96 4.94
CondensedEmbraceBERT-bs4 24.77 20.18 20.18 42.20 20.18 14.68 23.85 31.19 30.28 18.35 24.59 7.63
CondensedEmbraceBERT-bs4+Dropout0.1 53.21 40.37 38.53 26.61 40.37 40.37 39.45 25.69 36.70 50.46 39.17 8.22
CondensedEmbraceBERT-bs4+Dropout0.3 42.20 36.70 43.12 38.53 33.94 34.86 43.12 25.69 41.28 37.61 37.71 5.10
FrozenEBERT-bs4-ep100 27.52 28.44 33.03 31.19 40.37 33.03 46.79 37.61 22.94 39.45 34.04 6.72
FrozenEBERT-bs4-ep100+Dropout0.1 39.45 35.78 29.36 34.86 35.78 50.46 32.11 30.28 31.19 38.53 35.78 5.86
FrozenEBERT-bs4-ep100+Dropout0.3 43.12 34.86 33.94 33.94 40.37 38.53 26.61 23.85 31.19 44.04 35.05 6.33
FrozenCEBERT-bs4-ep100 24.77 19.27 21.10 43.12 18.35 12.84 33.03 34.86 27.52 14.68 24.95 9.18
FrozenCEBERT-bs4-ep100+Dropout0.1 48.62 47.71 23.85 26.61 34.86 41.28 44.04 21.10 33.03 44.04 36.51 9.59
FrozenCEBERT-bs4-ep100+Dropout0.3 42.20 46.79 43.12 31.19 31.19 36.70 45.87 22.94 35.78 29.36 36.51 7.50
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERTwithAtt-bs4 35.78 32.11 33.94 45.87 35.78 40.37 45.87 33.03 44.04 50.46 39.72 6.14
BERTwithAttClsProjection-bs4 60.55 47.71 56.88 42.20 36.70 46.79 44.95 48.62 39.45 37.61 46.15 7.45
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4+Branches 43.12 49.54 37.61 35.78 30.28 48.62 48.62 33.03 31.19 30.28 38.81 7.58
EmbraceBERT-bs4+Branches+Dropout0.1 46.79 46.79 30.28 44.04 44.04 38.53 43.12 36.70 34.86 34.86 40.00 5.43
EmbraceBERT-bs4+Branches+Dropout0.3 36.70 40.37 41.28 35.78 42.20 46.79 44.04 35.78 38.53 39.45 40.09 3.43
CondensedEmbraceBERT-bs4+Branches 41.28 37.61 33.94 26.61 32.11 37.61 52.29 44.95 29.36 34.86 37.06 7.21
CondensedEmbraceBERT-bs4+Branches+Dropout0.1 44.95 44.04 46.79 32.11 46.79 41.28 44.95 48.62 39.45 33.03 42.20 5.44
CondensedEmbraceBERT-bs4+Branches+Dropout0.3 39.45 38.53 46.79 44.95 40.37 49.54 45.87 41.28 37.61 36.70 42.11 4.15
FrozenEmbraceBERT-bs4+Branches 49.54 34.86 37.61 33.94 29.36 41.28 49.54 35.78 30.28 31.19 37.34 6.96
FrozenEmbraceBERT-bs4+Branches+Dropout0.1 49.54 44.04 33.94 44.04 39.45 42.20 54.13 41.28 32.11 37.61 41.83 6.33
FrozenEmbraceBERT-bs4+Branches+Dropout0.3 36.70 39.45 37.61 43.12 33.94 43.12 41.28 29.36 44.04 40.37 38.90 4.38
FrozenCEBERT-bs4+Branches 38.53 33.03 49.54 45.87 47.71 35.78 29.36 48.62 37.61 33.94 40.00 6.95
FrozenCEBERT-bs4+Branches+Dropout0.1 46.79 42.20 46.79 33.94 44.95 44.04 49.54 52.29 35.78 33.03 42.94 6.30
FrozenCEBERT-bs4+Branches+Dropout0.3 47.71 38.53 26.61 49.54 43.12 55.05 45.87 28.44 33.03 24.77 39.27 10.06
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 43.12 43.12 39.45 21.10 31.19 35.78 43.12 47.71 41.28 40.37 38.62 7.24
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery_attclsqw 33.03 39.45 50.46 38.53 28.44 24.77 43.12 40.37 44.95 37.61 38.07 7.27
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 34.86 44.04 52.29 46.79 35.78 29.36 51.38 36.70 38.53 43.12 41.28 7.12
BERT-bs16+Dropout0.1 32.11 40.37 50.46 53.21 29.36 34.86 55.96 35.78 40.37 41.28 41.38 8.61
BERT-bs16+Dropout0.3 34.86 43.12 49.54 45.87 29.36 33.03 59.63 44.04 36.70 40.37 41.65 8.42
FrozenBERT-bs16-ep100 42.20 44.95 40.37 45.87 33.03 33.03 48.62 34.86 33.94 44.04 40.09 5.61
FrozenBERT-bs16-ep100+Dropout0.1 42.20 46.79 49.54 45.87 26.61 28.44 50.46 41.28 42.20 39.45 41.28 7.68
FrozenBERT-bs16-ep100+Dropout0.3 44.95 44.95 46.79 46.79 32.11 37.61 52.29 45.87 36.70 44.95 43.30 5.67
EmbraceBERT-bs16 40.37 33.03 41.28 49.54 30.28 41.28 41.28 32.11 36.70 35.78 38.17 5.45
EmbraceBERT-bs16+Dropout0.1 31.19 35.78 42.20 30.28 31.19 44.95 36.70 35.78 41.28 44.04 37.34 5.22
EmbraceBERT-bs16+Dropout0.3 38.53 34.86 36.70 30.28 29.36 36.70 33.94 33.94 35.78 44.04 35.41 3.94
CondensedEmbraceBERT-bs16 33.94 17.43 27.52 26.61 22.02 11.01 19.27 9.17 14.68 11.93 19.36 7.70
CondensedEmbraceBERT-bs16+Dropout0.1 29.36 28.44 35.78 39.45 30.28 44.04 39.45 42.20 40.37 44.04 37.34 5.72
CondensedEmbraceBERT-bs16+Dropout0.3 34.86 38.53 36.70 33.03 29.36 45.87 33.94 35.78 40.37 39.45 36.79 4.33
FrozenEBERT-bs16-ep100 40.37 34.86 38.53 44.95 30.28 46.79 44.04 34.86 34.86 29.36 37.89 5.76
FrozenEBERT-bs16-ep100+Dropout0.1 34.86 33.94 38.53 23.85 50.46 44.95 44.04 32.11 37.61 38.53 37.89 7.08
FrozenEBERT-bs16-ep100+Dropout0.3 36.70 39.45 45.87 29.36 32.11 41.28 42.20 36.70 35.78 35.78 37.52 4.61
FrozenCEBERT-bs16-ep100 38.53 21.10 23.85 29.36 20.18 10.09 31.19 8.26 12.84 11.93 20.73 9.59
FrozenCEBERT-bs16-ep100+Dropout0.1 35.78 40.37 29.36 37.61 28.44 40.37 41.28 43.12 41.28 42.20 37.98 4.98
FrozenCEBERT-bs16-ep100+Dropout0.3 36.70 43.12 33.94 31.19 27.52 36.70 40.37 29.36 37.61 38.53 35.50 4.70
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16+Branches 24.77 34.86 35.78 43.12 29.36 38.53 35.78 40.37 33.94 44.95 36.15 5.76
EmbraceBERT-bs16+Branches+Dropout0.1 37.61 35.78 44.04 37.61 44.04 33.03 44.04 33.03 42.20 32.11 38.35 4.64
EmbraceBERT-bs16+Branches+Dropout0.3 36.70 36.70 40.37 40.37 45.87 34.86 38.53 44.04 39.45 41.28 39.82 3.21
CondensedEmbraceBERT-bs16+Branches 31.19 39.45 36.70 40.37 27.52 42.20 35.78 40.37 37.61 51.38 38.26 6.10
CondensedEmbraceBERT-bs16+Branches+Dropout0.1 30.28 37.61 39.45 44.04 46.79 30.28 35.78 40.37 40.37 31.19 37.61 5.44
CondensedEmbraceBERT-bs16+Branches+Dropout0.3 39.45 34.86 41.28 43.12 50.46 33.03 39.45 43.12 36.70 32.11 39.36 5.25
FrozenEmbraceBERT-bs16+Branches 26.61 38.53 36.70 37.61 30.28 41.28 40.37 40.37 33.03 43.12 36.79 5.00
FrozenEmbraceBERT-bs16+Branches+Dropout0.1 41.28 37.61 43.12 38.53 44.95 32.11 34.86 30.28 46.79 31.19 38.07 5.60
FrozenEmbraceBERT-bs16+Branches+Dropout0.3 38.53 40.37 48.62 41.28 42.20 34.86 38.53 42.20 43.12 40.37 41.01 3.41
FrozenCEBERT-bs16+Branches 41.28 34.86 51.38 28.44 38.53 30.28 48.62 38.53 37.61 31.19 38.07 7.14
FrozenCEBERT-bs16+Branches+Dropout0.1 35.78 40.37 51.38 41.28 46.79 31.19 42.20 37.61 45.87 31.19 40.37 6.28
FrozenCEBERT-bs16+Branches+Dropout0.3 41.28 38.53 44.95 41.28 48.62 39.45 41.28 32.11 30.28 33.03 39.08 5.51
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery 42.20 33.94 44.04 23.85 42.20 28.44 52.29 35.78 34.86 42.20 37.98 7.85
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 43.12 56.88 37.61 44.95 43.12 38.53 47.71 30.28 53.21 39.45 43.49 7.38
RoBERTa-bs4+Dropout0.1 40.37 43.12 49.54 35.78 45.87 35.78 35.78 28.44 57.80 42.20 41.47 7.89
RoBERTa-bs4+Dropout0.3 38.53 33.03 42.20 37.61 38.53 37.61 34.86 39.45 56.88 44.95 40.37 6.36
FrozenRoBERTa-bs4-ep100 44.04 56.88 39.45 51.38 39.45 38.53 42.20 22.02 55.05 39.45 42.84 9.53
FrozenRoBERTa-bs4-ep100+Dropout0.1 40.37 34.86 40.37 34.86 48.62 39.45 50.46 37.61 59.63 45.87 43.21 7.48
FrozenRoBERTa-bs4-ep100+Dropout0.3 35.78 39.45 43.12 38.53 44.95 55.96 54.13 35.78 54.13 51.38 45.32 7.57
EmbraceRoBERTa-bs4 63.30 38.53 49.54 53.21 51.38 40.37 37.61 33.03 48.62 48.62 46.42 8.55
EmbraceRoBERTa-bs4+Dropout0.1 66.06 30.28 44.04 38.53 55.96 44.04 32.11 39.45 51.38 40.37 44.22 10.38
EmbraceRoBERTa-bs4+Dropout0.3 55.05 30.28 33.03 46.79 62.39 42.20 33.03 44.04 56.88 38.53 44.22 10.47
CondensedEmbraceRoBERTa-bs4 33.94 52.29 11.93 41.28 17.43 36.70 6.42 17.43 10.09 24.77 25.23 14.42
CondensedEmbraceRoBERTa-bs4+Dropout0.1 58.72 34.86 36.70 46.79 64.22 38.53 33.94 39.45 50.46 39.45 44.31 9.92
CondensedEmbraceRoBERTa-bs4+Dropout0.3 57.80 31.19 34.86 46.79 52.29 41.28 32.11 41.28 49.54 37.61 42.48 8.48
FrozenERoBERTa-bs4-ep100 62.39 39.45 49.54 56.88 51.38 40.37 38.53 34.86 49.54 55.96 47.89 8.71
FrozenERoBERTa-bs4-ep100+Dropout0.1 49.54 36.70 35.78 39.45 55.05 39.45 36.70 33.03 45.87 39.45 41.10 6.55
FrozenERoBERTa-bs4-ep100+Dropout0.3 55.96 33.94 41.28 40.37 56.88 44.04 33.94 55.05 38.53 43.12 44.31 8.27
FrozenCERoBERTa-bs4-ep100 36.70 52.29 12.84 43.12 13.76 34.86 4.59 13.76 5.50 18.35 23.58 15.90
FrozenCERoBERTa-bs4-ep100+Dropout0.1 62.39 32.11 33.94 35.78 55.96 39.45 35.78 39.45 49.54 41.28 42.57 9.58
FrozenCERoBERTa-bs4-ep100+Dropout0.3 57.80 34.86 38.53 35.78 57.80 38.53 35.78 54.13 46.79 36.70 43.67 9.07
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceRoBERTa-bs4+Branches 46.79 36.70 40.37 45.87 46.79 40.37 35.78 36.70 38.53 38.53 40.64 4.08
EmbraceRoBERTa-bs4+Branches+Dropout0.1 48.62 40.37 39.45 55.05 55.05 39.45 34.86 37.61 42.20 33.94 42.66 7.28
EmbraceRoBERTa-bs4+Branches+Dropout0.3 43.12 38.53 44.04 56.88 55.96 44.95 33.03 37.61 40.37 38.53 43.30 7.35
CondensedEmbraceRoBERTa-bs4+Branches 44.04 40.37 35.78 54.13 46.79 40.37 34.86 33.94 47.71 36.70 41.47 6.26
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.1 44.95 34.86 35.78 59.63 47.71 37.61 34.86 36.70 38.53 43.12 41.38 7.41
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.3 38.53 37.61 34.86 43.12 55.05 40.37 36.70 43.12 44.04 41.28 41.47 5.36
FrozenEmbraceRoBERTa-bs4+Branches 51.38 38.53 37.61 48.62 50.46 39.45 31.19 35.78 38.53 39.45 41.10 6.38
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.1 46.79 41.28 37.61 58.72 55.96 43.12 33.94 38.53 38.53 34.86 42.94 8.06
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.3 43.12 41.28 47.71 55.05 53.21 44.04 33.03 35.78 41.28 39.45 43.39 6.64
FrozenCERoBERTa-bs4+Branches 44.04 39.45 36.70 57.80 42.20 40.37 34.86 33.94 44.04 35.78 40.92 6.62
FrozenCERoBERTa-bs4+Branches+Dropout0.1 44.04 33.94 33.94 60.55 47.71 36.70 33.94 35.78 39.45 39.45 40.55 7.96
FrozenCERoBERTa-bs4+Branches+Dropout0.3 38.53 33.03 33.03 35.78 51.38 38.53 35.78 42.20 43.12 41.28 39.27 5.27
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 38.53 35.78 43.12 38.53 44.04 42.20 53.21 34.86 47.71 37.61 41.56 5.43
RoBERTa-bs16+Dropout0.1 39.45 37.61 43.12 41.28 42.20 40.37 47.71 36.70 47.71 40.37 41.65 3.53
RoBERTa-bs16+Dropout0.3 39.45 36.70 41.28 37.61 43.12 39.45 43.12 34.86 43.12 43.12 40.18 2.90
FrozenRoBERTa-bs16-ep100 37.61 39.45 39.45 37.61 38.53 43.12 54.13 31.19 47.71 40.37 40.92 5.93
FrozenRoBERTa-bs16-ep100+Dropout0.1 39.45 36.70 38.53 36.70 46.79 44.95 44.95 36.70 47.71 36.70 40.92 4.38
FrozenRoBERTa-bs16-ep100+Dropout0.3 37.61 45.87 40.37 36.70 44.95 39.45 45.87 35.78 46.79 41.28 41.47 3.93
EmbraceRoBERTa-bs16 48.62 35.78 40.37 37.61 56.88 46.79 36.70 34.86 39.45 41.28 41.83 6.59
EmbraceRoBERTa-bs16+Dropout0.1 51.38 36.70 35.78 44.04 55.05 41.28 36.70 39.45 39.45 44.04 42.39 6.11
EmbraceRoBERTa-bs16+Dropout0.3 43.12 37.61 37.61 48.62 57.80 45.87 36.70 43.12 43.12 47.71 44.13 6.05
CondensedEmbraceRoBERTa-bs16 19.27 37.61 11.01 36.70 18.35 37.61 14.68 10.09 7.34 19.27 21.19 11.21
CondensedEmbraceRoBERTa-bs16+Dropout0.1 40.37 38.53 36.70 54.13 53.21 38.53 31.19 44.95 39.45 44.04 42.11 6.81
CondensedEmbraceRoBERTa-bs16+Dropout0.3 48.62 39.45 33.94 45.87 54.13 40.37 33.03 42.20 40.37 44.95 42.29 6.09
FrozenERoBERTa-bs16-ep100 49.54 32.11 42.20 34.86 54.13 44.04 37.61 38.53 39.45 38.53 41.10 6.31
FrozenERoBERTa-bs16-ep100+Dropout0.1 64.22 34.86 35.78 40.37 54.13 42.20 38.53 37.61 44.04 38.53 43.03 8.78
FrozenERoBERTa-bs16-ep100+Dropout0.3 44.04 33.03 35.78 36.70 47.71 41.28 37.61 38.53 44.04 38.53 39.72 4.25
FrozenCERoBERTa-bs16-ep100 18.35 38.53 12.84 33.03 9.17 33.03 17.43 13.76 8.26 9.17 19.36 10.73
FrozenCERoBERTa-bs16-ep100+Dropout0.1 58.72 39.45 44.04 46.79 58.72 42.20 32.11 38.53 37.61 43.12 44.13 8.23
FrozenCERoBERTa-bs16-ep100+Dropout0.3 44.04 37.61 33.94 44.95 53.21 42.20 39.45 36.70 38.53 33.94 40.46 5.57
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceRoBERTa-bs16+Branches 44.95 38.53 37.61 41.28 46.79 38.53 33.03 37.61 42.20 36.70 39.72 3.89
EmbraceRoBERTa-bs16+Branches+Dropout0.1 43.12 36.70 44.95 42.20 50.46 43.12 35.78 35.78 41.28 40.37 41.38 4.33
EmbraceRoBERTa-bs16+Branches+Dropout0.3 44.04 37.61 41.28 40.37 53.21 43.12 37.61 36.70 44.95 38.53 41.74 4.70
CondensedEmbraceRoBERTa-bs16+Branches 47.71 36.70 36.70 46.79 47.71 40.37 34.86 36.70 36.70 39.45 40.37 4.84
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1 46.79 37.61 39.45 44.95 51.38 42.20 33.03 38.53 42.20 35.78 41.19 5.20
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3 41.28 38.53 37.61 40.37 55.96 34.86 35.78 39.45 46.79 35.78 40.64 6.07
FrozenEmbraceRoBERTa-bs16+Branches 48.62 38.53 37.61 41.28 44.04 34.86 33.03 36.70 43.12 36.70 39.45 4.51
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1 45.87 36.70 38.53 42.20 50.46 43.12 35.78 33.94 38.53 40.37 40.55 4.75
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3 44.04 38.53 37.61 39.45 55.05 44.04 37.61 34.86 44.95 33.03 40.92 6.03
FrozenCERoBERTa-bs16+Branches 48.62 36.70 34.86 46.79 47.71 40.37 37.61 37.61 35.78 36.70 40.28 5.07
FrozenCERoBERTa-bs16+Branches+Dropout0.1 46.79 37.61 40.37 45.87 48.62 41.28 32.11 39.45 43.12 36.70 41.19 4.81
FrozenCERoBERTa-bs16+Branches+Dropout0.3 40.37 38.53 35.78 45.87 54.13 36.70 36.70 36.70 44.04 37.61 40.64 5.52

  • Epoch 100 - bs4 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |

  • Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |39.45|46.79|47.71|46.79|36.70|33.03|55.05|46.79|39.45|45.87|43.76|6.14| | BERTwithAtt-bs8 |44.04|35.78|47.71|40.37|36.70|41.28|44.95|39.45|41.28|39.45|41.10|3.48| | BERTwithAttClsProjection-bs8 |55.05|35.78|42.20|39.45|39.45|42.20|42.20|50.46|38.53|29.36|41.47|6.80| | BERTwithProjection-bs8 |39.45|31.19|49.54|33.03|36.70|44.04|45.87|30.28|35.78|38.53|38.44|6.08| | BERTwithProjectionAtt-bs8 |38.53|34.86|32.11|41.28|42.20|55.05|35.78|35.78|34.86|51.38|40.18|7.17| | EmbraceBERT-bs8 |50.46|36.70|30.28|41.28|27.52|40.37|41.28|42.20|40.37|43.12|39.36|6.20| | EmbraceBERT-bs8-p_att_clsquery_weights |25.69|25.69|48.62|32.11|33.94|38.53|45.87|34.86|38.53|37.61|36.15|7.11| | EmbraceBERTwithProj-bs8 |45.87|46.79|46.79|50.46|50.46|49.54|55.96|33.94|31.19|43.12|45.41|7.23| | EmbraceBERTwithProj-bs8-p_attclsqw |58.72|32.11|52.29|33.94|42.20|44.04|36.70|53.21|41.28|45.87|44.04|8.25| | EBERTconcatAtt-bs8 |45.87|45.87|42.20|45.87|30.28|41.28|39.45|41.28|32.11|32.11|39.63|5.74| | EBERTconcatAtt-bs8_attclsqw |41.28|41.28|39.45|35.78|40.37|38.53|36.70|40.37|53.21|43.12|41.01|4.57| | EBERTconcatAttwithProj-bs8 |39.45|38.53|44.04|31.19|45.87|28.44|43.12|44.04|54.13|33.03|40.18|7.34| | EBERTconcatAttwithProj-bs8_attclsqw |26.61|46.79|35.78|39.45|38.53|41.28|33.94|47.71|40.37|44.04|39.45|5.97| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |47.71|33.03|36.70|14.68|34.86|24.77|36.70|33.94|26.61|41.28|33.03|8.72| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |35.78|44.04|47.71|37.61|30.28|31.19|32.11|43.12|41.28|44.95|38.81|5.96| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |43.12|44.04|33.03|46.79|50.46|35.78|45.87|47.71|45.87|41.28|43.39|5.13| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |44.95|44.04|42.20|49.54|40.37|37.61|41.28|44.95|50.46|47.71|44.31|3.89| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |31.19|45.87|41.28|38.53|27.52|44.04|39.45|37.61|29.36|40.37|37.52|5.88| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |35.78|39.45|32.11|39.45|44.04|43.12|36.70|45.87|40.37|42.20|39.91|3.96| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |31.19|52.29|46.79|44.95|40.37|34.86|41.28|53.21|46.79|35.78|42.75|7.00| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |30.28|38.53|49.54|37.61|44.95|31.19|43.12|41.28|44.04|32.11|39.27|6.17|

macsay-witai - GOOD all

  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 66.97 75.23 74.31 78.90 75.23 66.97 72.48 72.48 69.72 74.31 72.66 3.62
BERT-bs4+Dropout0.1 66.97 76.15 77.06 74.31 70.64 72.48 75.23 71.56 64.22 68.81 71.74 3.95
BERT-bs4+Dropout0.3 71.56 82.57 71.56 71.56 78.90 66.97 72.48 66.97 77.98 64.22 72.48 5.52
FrozenBERT-bs4-ep100 71.56 72.48 71.56 82.57 70.64 70.64 71.56 67.89 65.14 66.97 71.10 4.44
FrozenBERT-bs4-ep100+Dropout0.1 64.22 77.06 70.64 74.31 74.31 64.22 74.31 64.22 73.39 71.56 70.83 4.62
FrozenBERT-bs4-ep100+Dropout0.3 66.97 74.31 67.89 69.72 75.23 72.48 75.23 70.64 66.06 69.72 70.83 3.20
EmbraceBERT-bs4 68.81 76.15 76.15 78.90 80.73 72.48 70.64 79.82 67.89 71.56 74.31 4.42
EmbraceBERT-bs4+Dropout0.1 65.14 78.90 71.56 74.31 75.23 74.31 74.31 67.89 72.48 77.98 73.21 3.99
EmbraceBERT-bs4+Dropout0.3 60.55 76.15 76.15 71.56 82.57 79.82 64.22 72.48 77.06 72.48 73.30 6.39
CondensedEmbraceBERT-bs4 43.12 11.93 8.26 31.19 31.19 14.68 24.77 38.53 35.78 16.51 25.60 11.54
CondensedEmbraceBERT-bs4+Dropout0.1 82.57 81.65 76.15 66.97 79.82 77.98 74.31 66.97 72.48 80.73 75.96 5.42
CondensedEmbraceBERT-bs4+Dropout0.3 76.15 78.90 75.23 77.98 75.23 72.48 72.48 79.82 81.65 67.89 75.78 3.87
FrozenEBERT-bs4-ep100 66.97 71.56 66.06 77.98 77.06 72.48 71.56 80.73 72.48 66.06 72.29 4.83
FrozenEBERT-bs4-ep100+Dropout0.1 67.89 78.90 63.30 72.48 69.72 76.15 74.31 70.64 70.64 69.72 71.38 4.16
FrozenEBERT-bs4-ep100+Dropout0.3 72.48 73.39 64.22 72.48 81.65 77.98 61.47 73.39 73.39 72.48 72.29 5.53
FrozenCEBERT-bs4-ep100 45.87 14.68 9.17 26.61 21.10 17.43 23.85 46.79 27.52 11.93 24.50 12.32
FrozenCEBERT-bs4-ep100+Dropout0.1 77.98 77.98 70.64 76.15 66.06 79.82 76.15 65.14 71.56 69.72 73.12 4.94
FrozenCEBERT-bs4-ep100+Dropout0.3 73.39 77.06 79.82 64.22 71.56 77.06 84.40 77.98 73.39 59.63 73.85 6.97
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERTwithAtt-bs4 79.82 67.89 66.06 73.39 67.89 77.98 79.82 71.56 75.23 75.23 73.49 4.78
BERTwithAttClsProjection-bs4 79.82 72.48 78.90 57.80 68.81 77.98 69.72 65.14 75.23 68.81 71.47 6.53
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4+Branches 82.57 78.90 80.73 73.39 66.97 82.57 75.23 77.98 77.06 75.23 77.06 4.48
EmbraceBERT-bs4+Branches+Dropout0.1 83.49 70.64 69.72 65.14 73.39 73.39 81.65 77.98 60.55 79.82 73.58 6.97
EmbraceBERT-bs4+Branches+Dropout0.3 70.64 67.89 74.31 69.72 70.64 77.06 78.90 68.81 66.97 77.06 72.20 4.06
CondensedEmbraceBERT-bs4+Branches 77.98 77.98 75.23 69.72 72.48 74.31 75.23 75.23 80.73 72.48 75.14 3.03
CondensedEmbraceBERT-bs4+Branches+Dropout0.1 75.23 68.81 75.23 66.97 77.98 69.72 78.90 78.90 74.31 71.56 73.76 4.09
CondensedEmbraceBERT-bs4+Branches+Dropout0.3 66.97 67.89 70.64 77.98 73.39 74.31 80.73 69.72 65.14 71.56 71.83 4.64
FrozenEmbraceBERT-bs4+Branches 82.57 75.23 79.82 73.39 67.89 80.73 73.39 76.15 77.98 70.64 75.78 4.38
FrozenEmbraceBERT-bs4+Branches+Dropout0.1 82.57 63.30 66.06 66.97 68.81 72.48 84.40 80.73 56.88 77.98 72.02 8.68
FrozenEmbraceBERT-bs4+Branches+Dropout0.3 70.64 66.97 67.89 72.48 64.22 77.06 74.31 71.56 66.06 77.06 70.83 4.28
FrozenCEBERT-bs4+Branches 76.15 72.48 75.23 75.23 75.23 65.14 59.63 77.98 72.48 61.47 71.10 6.23
FrozenCEBERT-bs4+Branches+Dropout0.1 73.39 70.64 67.89 67.89 73.39 72.48 82.57 80.73 71.56 72.48 73.30 4.60
FrozenCEBERT-bs4+Branches+Dropout0.3 66.97 70.64 60.55 80.73 76.15 79.82 77.06 73.39 63.30 65.14 71.38 6.78
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 76.15 74.31 72.48 67.89 70.64 70.64 75.23 75.23 65.14 83.49 73.12 4.80
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery_attclsqw 71.56 82.57 68.81 71.56 66.97 65.14 77.98 77.98 71.56 79.82 73.39 5.55
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 68.81 79.82 79.82 74.31 78.90 64.22 73.39 74.31 73.39 72.48 73.94 4.66
BERT-bs16+Dropout0.1 68.81 76.15 76.15 76.15 68.81 63.30 77.98 73.39 74.31 68.81 72.39 4.47
BERT-bs16+Dropout0.3 66.97 77.98 77.06 72.48 71.56 68.81 77.06 72.48 75.23 65.14 72.48 4.22
FrozenBERT-bs16-ep100 67.89 79.82 76.15 71.56 77.06 65.14 71.56 72.48 68.81 74.31 72.48 4.26
FrozenBERT-bs16-ep100+Dropout0.1 71.56 73.39 77.06 72.48 73.39 65.14 75.23 67.89 69.72 70.64 71.65 3.32
FrozenBERT-bs16-ep100+Dropout0.3 73.39 76.15 77.98 72.48 72.48 65.14 75.23 69.72 70.64 69.72 72.29 3.52
EmbraceBERT-bs16 75.23 74.31 80.73 79.82 77.98 71.56 61.47 72.48 73.39 72.48 73.94 5.14
EmbraceBERT-bs16+Dropout0.1 76.15 69.72 83.49 70.64 77.06 71.56 61.47 76.15 76.15 73.39 73.58 5.52
EmbraceBERT-bs16+Dropout0.3 77.06 76.15 75.23 71.56 76.15 77.06 54.13 75.23 69.72 74.31 72.66 6.57
CondensedEmbraceBERT-bs16 42.20 18.35 21.10 25.69 17.43 10.09 20.18 8.26 8.26 15.60 18.72 9.55
CondensedEmbraceBERT-bs16+Dropout0.1 72.48 67.89 76.15 77.98 72.48 68.81 65.14 80.73 75.23 69.72 72.66 4.64
CondensedEmbraceBERT-bs16+Dropout0.3 76.15 74.31 80.73 77.06 73.39 77.98 63.30 74.31 63.30 71.56 73.21 5.52
FrozenEBERT-bs16-ep100 72.48 67.89 66.06 76.15 77.06 77.98 64.22 71.56 66.97 65.14 70.55 4.93
FrozenEBERT-bs16-ep100+Dropout0.1 80.73 67.89 77.06 66.97 89.91 77.98 77.06 72.48 78.90 69.72 75.87 6.55
FrozenEBERT-bs16-ep100+Dropout0.3 79.82 75.23 74.31 72.48 78.90 86.24 71.56 75.23 74.31 69.72 75.78 4.54
FrozenCEBERT-bs16-ep100 44.04 18.35 20.18 24.77 20.18 9.17 27.52 10.09 11.93 10.09 19.63 10.18
FrozenCEBERT-bs16-ep100+Dropout0.1 76.15 71.56 64.22 79.82 74.31 76.15 71.56 77.98 76.15 66.06 73.39 4.80
FrozenCEBERT-bs16-ep100+Dropout0.3 77.98 72.48 77.06 76.15 73.39 74.31 69.72 70.64 61.47 69.72 72.29 4.57
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16+Branches 66.97 69.72 79.82 77.06 71.56 70.64 63.30 77.06 79.82 69.72 72.57 5.33
EmbraceBERT-bs16+Branches+Dropout0.1 76.15 58.72 72.48 77.98 77.06 65.14 81.65 68.81 69.72 63.30 71.10 6.92
EmbraceBERT-bs16+Branches+Dropout0.3 67.89 60.55 73.39 72.48 80.73 66.06 71.56 77.98 62.39 65.14 69.82 6.24
CondensedEmbraceBERT-bs16+Branches 71.56 73.39 78.90 77.06 72.48 66.06 62.39 75.23 79.82 73.39 73.03 5.16
CondensedEmbraceBERT-bs16+Branches+Dropout0.1 72.48 60.55 68.81 75.23 81.65 66.97 74.31 70.64 66.06 61.47 69.82 6.13
CondensedEmbraceBERT-bs16+Branches+Dropout0.3 68.81 63.30 69.72 80.73 80.73 63.30 74.31 69.72 66.06 72.48 70.92 5.95
FrozenEmbraceBERT-bs16+Branches 67.89 66.06 77.98 73.39 72.48 72.48 68.81 76.15 80.73 70.64 72.66 4.38
FrozenEmbraceBERT-bs16+Branches+Dropout0.1 75.23 62.39 76.15 73.39 74.31 65.14 74.31 63.30 68.81 65.14 69.82 5.15
FrozenEmbraceBERT-bs16+Branches+Dropout0.3 66.97 61.47 79.82 77.06 77.98 66.97 75.23 73.39 61.47 65.14 70.55 6.58
FrozenCEBERT-bs16+Branches 70.64 74.31 77.98 56.88 68.81 64.22 83.49 66.97 65.14 70.64 69.91 7.10
FrozenCEBERT-bs16+Branches+Dropout0.1 64.22 61.47 77.06 75.23 79.82 67.89 76.15 66.97 62.39 61.47 69.27 6.76
FrozenCEBERT-bs16+Branches+Dropout0.3 68.81 65.14 75.23 80.73 74.31 68.81 75.23 67.89 58.72 70.64 70.55 5.87
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery 77.06 59.63 70.64 65.14 74.31 62.39 82.57 66.97 70.64 69.72 69.91 6.52
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 70.64 83.49 66.97 76.15 79.82 68.81 70.64 63.30 80.73 73.39 73.39 6.20
RoBERTa-bs4+Dropout0.1 66.97 66.97 76.15 63.30 77.98 66.97 68.81 61.47 82.57 77.06 70.83 6.70
RoBERTa-bs4+Dropout0.3 63.30 62.39 69.72 65.14 68.81 66.06 67.89 79.82 83.49 74.31 70.09 6.67
FrozenRoBERTa-bs4-ep100 70.64 84.40 67.89 78.90 77.98 70.64 70.64 61.47 83.49 72.48 73.85 6.84
FrozenRoBERTa-bs4-ep100+Dropout0.1 62.39 66.97 69.72 69.72 76.15 76.15 78.90 77.06 77.06 77.98 73.21 5.31
FrozenRoBERTa-bs4-ep100+Dropout0.3 59.63 73.39 68.81 79.82 78.90 84.40 79.82 59.63 82.57 79.82 74.68 8.62
EmbraceRoBERTa-bs4 84.40 76.15 81.65 82.57 80.73 72.48 72.48 69.72 75.23 79.82 77.52 4.74
EmbraceRoBERTa-bs4+Dropout0.1 84.40 63.30 79.82 80.73 86.24 72.48 70.64 71.56 83.49 78.90 77.16 6.97
EmbraceRoBERTa-bs4+Dropout0.3 85.32 68.81 63.30 83.49 86.24 69.72 75.23 73.39 80.73 74.31 76.06 7.30
CondensedEmbraceRoBERTa-bs4 34.86 40.37 4.59 39.45 8.26 45.87 7.34 14.68 11.93 19.27 22.66 14.97
CondensedEmbraceRoBERTa-bs4+Dropout0.1 81.65 65.14 78.90 82.57 86.24 63.30 73.39 71.56 78.90 79.82 76.15 7.19
CondensedEmbraceRoBERTa-bs4+Dropout0.3 83.49 64.22 72.48 78.90 84.40 73.39 69.72 78.90 84.40 75.23 76.51 6.40
FrozenERoBERTa-bs4-ep100 86.24 75.23 83.49 82.57 80.73 70.64 72.48 69.72 77.98 82.57 78.17 5.55
FrozenERoBERTa-bs4-ep100+Dropout0.1 77.98 66.97 61.47 76.15 85.32 64.22 69.72 74.31 74.31 72.48 72.29 6.65
FrozenERoBERTa-bs4-ep100+Dropout0.3 77.98 70.64 74.31 80.73 83.49 67.89 67.89 82.57 75.23 76.15 75.69 5.37
FrozenCERoBERTa-bs4-ep100 38.53 49.54 6.42 44.04 13.76 42.20 6.42 15.60 8.26 12.84 23.76 16.62
FrozenCERoBERTa-bs4-ep100+Dropout0.1 85.32 64.22 66.06 74.31 83.49 75.23 58.72 77.06 77.06 75.23 73.67 7.95
FrozenCERoBERTa-bs4-ep100+Dropout0.3 83.49 65.14 72.48 76.15 84.40 76.15 62.39 83.49 79.82 75.23 75.87 7.17
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceRoBERTa-bs4+Branches 77.06 67.89 69.72 79.82 82.57 69.72 66.06 71.56 66.97 71.56 72.29 5.35
EmbraceRoBERTa-bs4+Branches+Dropout0.1 78.90 73.39 74.31 85.32 80.73 83.49 62.39 72.48 74.31 72.48 75.78 6.29
EmbraceRoBERTa-bs4+Branches+Dropout0.3 73.39 72.48 77.98 84.40 83.49 75.23 65.14 68.81 73.39 70.64 74.50 5.77
CondensedEmbraceRoBERTa-bs4+Branches 77.06 61.47 69.72 85.32 77.98 71.56 61.47 66.06 78.90 74.31 72.39 7.44
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.1 77.06 66.97 68.81 86.24 84.40 66.06 73.39 70.64 68.81 74.31 73.67 6.67
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.3 73.39 75.23 61.47 74.31 84.40 64.22 68.81 76.15 78.90 72.48 72.94 6.40
FrozenEmbraceRoBERTa-bs4+Branches 78.90 68.81 68.81 81.65 78.90 67.89 67.89 67.89 61.47 73.39 71.56 6.09
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.1 78.90 72.48 75.23 83.49 81.65 82.57 61.47 70.64 73.39 74.31 75.41 6.31
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.3 72.48 74.31 76.15 84.40 82.57 75.23 66.97 61.47 70.64 76.15 74.04 6.42
FrozenCERoBERTa-bs4+Branches 74.31 58.72 74.31 86.24 80.73 70.64 63.30 71.56 75.23 72.48 72.75 7.41
FrozenCERoBERTa-bs4+Branches+Dropout0.1 75.23 66.06 65.14 87.16 81.65 67.89 68.81 71.56 66.97 73.39 72.39 6.85
FrozenCERoBERTa-bs4+Branches+Dropout0.3 72.48 73.39 62.39 77.06 84.40 63.30 69.72 74.31 75.23 73.39 72.57 6.08
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 60.55 62.39 68.81 65.14 75.23 71.56 78.90 74.31 79.82 69.72 70.64 6.26
RoBERTa-bs16+Dropout0.1 65.14 68.81 74.31 66.06 69.72 67.89 80.73 72.48 75.23 70.64 71.10 4.48
RoBERTa-bs16+Dropout0.3 61.47 65.14 67.89 59.63 77.98 73.39 74.31 67.89 76.15 77.06 70.09 6.28
FrozenRoBERTa-bs16-ep100 61.47 60.55 63.30 66.97 77.06 72.48 78.90 70.64 79.82 69.72 70.09 6.69
FrozenRoBERTa-bs16-ep100+Dropout0.1 57.80 66.06 59.63 68.81 78.90 75.23 79.82 65.14 77.06 68.81 69.72 7.42
FrozenRoBERTa-bs16-ep100+Dropout0.3 61.47 77.98 62.39 66.06 74.31 70.64 82.57 71.56 77.06 68.81 71.28 6.51
EmbraceRoBERTa-bs16 79.82 66.06 75.23 76.15 84.40 81.65 66.06 69.72 76.15 74.31 74.95 5.89
EmbraceRoBERTa-bs16+Dropout0.1 82.57 73.39 64.22 79.82 84.40 73.39 70.64 77.06 71.56 78.90 75.60 5.81
EmbraceRoBERTa-bs16+Dropout0.3 77.98 68.81 61.47 78.90 84.40 77.06 63.30 77.06 72.48 79.82 74.13 7.08
CondensedEmbraceRoBERTa-bs16 26.61 33.94 11.93 28.44 9.17 49.54 7.34 12.84 16.51 25.69 22.20 12.54
CondensedEmbraceRoBERTa-bs16+Dropout0.1 81.65 68.81 72.48 86.24 85.32 74.31 66.06 78.90 70.64 79.82 76.42 6.63
CondensedEmbraceRoBERTa-bs16+Dropout0.3 81.65 66.06 63.30 80.73 84.40 63.30 63.30 77.98 73.39 77.98 73.21 8.03
FrozenERoBERTa-bs16-ep100 78.90 70.64 71.56 73.39 84.40 73.39 71.56 72.48 75.23 76.15 74.77 3.98
FrozenERoBERTa-bs16-ep100+Dropout0.1 84.40 66.06 62.39 76.15 85.32 75.23 66.06 77.06 77.98 76.15 74.68 7.27
FrozenERoBERTa-bs16-ep100+Dropout0.3 76.15 66.06 64.22 75.23 82.57 71.56 63.30 76.15 78.90 75.23 72.94 6.14
FrozenCERoBERTa-bs16-ep100 23.85 34.86 8.26 25.69 14.68 44.04 5.50 13.76 13.76 14.68 19.91 11.52
FrozenCERoBERTa-bs16-ep100+Dropout0.1 83.49 72.48 76.15 84.40 84.40 73.39 63.30 77.06 67.89 77.98 76.06 6.70
FrozenCERoBERTa-bs16-ep100+Dropout0.3 77.98 66.06 67.89 83.49 83.49 72.48 63.30 79.82 67.89 72.48 73.49 6.96
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceRoBERTa-bs16+Branches 80.73 65.14 64.22 76.15 80.73 75.23 58.72 70.64 73.39 68.81 71.38 6.89
EmbraceRoBERTa-bs16+Branches+Dropout0.1 76.15 66.97 76.15 75.23 78.90 73.39 64.22 76.15 65.14 74.31 72.66 4.95
EmbraceRoBERTa-bs16+Branches+Dropout0.3 78.90 69.72 71.56 75.23 82.57 73.39 65.14 75.23 80.73 66.06 73.85 5.58
CondensedEmbraceRoBERTa-bs16+Branches 77.98 66.06 66.06 82.57 78.90 73.39 63.30 66.97 66.06 66.97 70.83 6.45
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1 79.82 66.06 74.31 76.15 79.82 74.31 60.55 75.23 72.48 77.06 73.58 5.73
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3 77.06 64.22 71.56 73.39 83.49 72.48 58.72 74.31 73.39 71.56 72.02 6.36
FrozenEmbraceRoBERTa-bs16+Branches 77.98 66.06 64.22 74.31 77.06 55.96 59.63 69.72 71.56 70.64 68.72 6.87
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1 77.06 66.97 71.56 77.06 77.06 71.56 66.97 73.39 62.39 74.31 71.83 4.77
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3 80.73 66.97 70.64 72.48 82.57 72.48 64.22 74.31 80.73 64.22 72.94 6.40
FrozenCERoBERTa-bs16+Branches 77.06 61.47 66.06 78.90 81.65 72.48 59.63 72.48 66.06 70.64 70.64 6.96
FrozenCERoBERTa-bs16+Branches+Dropout0.1 78.90 64.22 71.56 76.15 78.90 73.39 61.47 74.31 72.48 77.06 72.84 5.57
FrozenCERoBERTa-bs16+Branches+Dropout0.3 71.56 66.97 66.97 73.39 83.49 75.23 60.55 73.39 73.39 71.56 71.65 5.74

  • Epoch 100 - bs4 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |

  • Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |72.48|77.98|77.98|74.31|71.56|66.97|77.06|77.98|67.89|75.23|73.94|3.92| | BERTwithAtt-bs8 |66.06|76.15|86.24|77.06|61.47|75.23|75.23|73.39|75.23|77.06|74.31|6.30| | BERTwithAttClsProjection-bs8 |77.98|58.72|74.31|67.89|71.56|73.39|74.31|77.98|70.64|65.14|71.19|5.66| | BERTwithProjection-bs8 |68.81|67.89|77.06|65.14|65.14|63.30|77.06|75.23|69.72|66.06|69.54|4.89| | BERTwithProjectionAtt-bs8 |76.15|70.64|66.06|63.30|67.89|80.73|75.23|68.81|62.39|71.56|70.28|5.54| | EmbraceBERT-bs8 |83.49|78.90|68.81|79.82|67.89|71.56|74.31|77.98|78.90|73.39|75.50|4.86| | EmbraceBERT-bs8-p_att_clsquery_weights |60.55|66.06|77.98|70.64|62.39|69.72|66.97|57.80|66.97|66.06|66.51|5.37| | EmbraceBERTwithProj-bs8 |72.48|77.06|71.56|77.06|63.30|63.30|75.23|65.14|67.89|66.97|70.00|5.13| | EmbraceBERTwithProj-bs8-p_attclsqw |72.48|66.06|74.31|57.80|66.06|67.89|66.97|76.15|77.98|71.56|69.72|5.67| | EBERTconcatAtt-bs8 |82.57|87.16|77.06|71.56|62.39|71.56|77.98|70.64|71.56|69.72|74.22|6.73| | EBERTconcatAtt-bs8_attclsqw |65.14|72.48|60.55|71.56|61.47|68.81|70.64|76.15|77.98|65.14|68.99|5.58| | EBERTconcatAttwithProj-bs8 |72.48|67.89|77.98|70.64|78.90|63.30|72.48|66.97|78.90|66.97|71.65|5.25| | EBERTconcatAttwithProj-bs8_attclsqw |64.22|74.31|66.06|71.56|62.39|72.48|66.06|76.15|74.31|72.48|70.00|4.61| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |78.90|66.06|68.81|59.63|78.90|62.39|72.48|67.89|68.81|66.06|68.99|5.99| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |71.56|78.90|77.98|65.14|61.47|72.48|71.56|76.15|64.22|80.73|72.02|6.29| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |69.72|77.98|69.72|74.31|61.47|77.98|69.72|66.06|75.23|66.06|70.83|5.20| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |73.39|68.81|74.31|80.73|75.23|67.89|68.81|69.72|75.23|77.98|73.21|4.10| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |71.56|77.06|78.90|77.98|62.39|80.73|77.06|74.31|59.63|67.89|72.75|6.88| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |60.55|77.06|79.82|67.89|78.90|68.81|79.82|83.49|64.22|76.15|73.67|7.32| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |66.97|81.65|65.14|69.72|78.90|58.72|80.73|70.64|77.98|63.30|71.38|7.62| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |65.14|67.89|65.14|67.89|66.97|66.97|66.97|71.56|73.39|70.64|68.26|2.60|

STT Error: trained and tested with INCOMPLETE data

Note: 100, 110 epochs all result in the same accuracy (baseline models)

gtts-google

  • Epoch 30
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 74.31 75.23 77.06 72.48 73.39 74.31 71.56 73.39 77.98 77.98 74.77 2.14
BERT-bs4+Dropout0.1 74.31 73.39 77.98 75.23 74.31 69.72 76.15 73.39 74.31 77.98 74.68 2.29
EmbraceBERT-bs4-WRONG 72.48 72.48 72.48 73.39 76.15 74.31 74.31 74.31 75.23 66.97 73.21 2.39
EmbraceBERT-bs4+Dropout0.1-WRONG 72.48 72.48 75.23 72.48 77.98 77.06 73.39 74.31 71.56 71.56 73.85 2.14
EmbraceBERT-bs4+Dropout0.3-WRONG
CondensedEmbraceBERT-bs4 77.06 67.89 73.39 72.48 77.06 71.56 67.89 76.15 75.23 68.81 72.75 3.46
CondensedEmbraceBERT-bs4+Dropout0.1 74.31 73.39 76.15 75.23 76.15 75.23 73.39 71.56 72.48 66.97 73.49 2.61
CondensedEmbraceBERT-bs4+Dropout0.3 76.15 73.39 71.56 76.15 77.06 75.23 74.31 73.39 77.06 70.64 74.50 2.12
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 73.39 76.15 77.06 72.48 73.39 73.39 76.15 73.39 74.31 77.06 74.68 1.65
FrozenEBERT-bs4-ep100-WRONG 74.31 73.39 70.64 72.48 73.39 77.06 68.81 73.39 76.15 68.81 72.84 2.63
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 74.31 73.39 75.23 71.56 71.56 68.81 77.06 72.48 72.48 78.90 73.58 2.78
BERT-bs16+Dropout0.1 75.23 75.23 76.15 72.48 66.97 64.22 76.15 69.72 69.72 77.98 72.39 4.31
EmbraceBERT-bs16-WRONG 73.39 71.56 70.64 70.64 72.48 76.15 71.56 73.39 71.56 67.89 71.93 2.06
EmbraceBERT-bs16+Dropout0.1-WRONG 74.31 74.31 74.31 71.56 75.23 72.48 70.64 73.39 70.64 69.72 72.66 1.83
EmbraceBERT-bs16+Dropout0.3-WRONG
CondensedEmbraceBERT-bs16 73.39 74.31 73.39 70.64 73.39 77.98 71.56 71.56 71.56 66.97 72.48 2.69
CondensedEmbraceBERT-bs16+Dropout0.1 73.39 69.72 76.15 71.56 77.98 77.06 75.23 69.72 67.89 69.72 72.84 3.41
CondensedEmbraceBERT-bs16+Dropout0.3 74.31 66.97 71.56 69.72 78.90 76.15 74.31 68.81 70.64 70.64 72.20 3.46
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 75.23 74.31 77.98 74.31 73.39 70.64 76.15 71.56 72.48 75.23 74.13 2.08
FrozenEBERT-bs16-ep100-WRONG 74.31 68.81 72.48 70.64 70.64 72.48 73.39 73.39 68.81 67.89 71.28 2.13
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 75.23 75.23 75.23 77.06 76.15 78.90 79.82 77.98 74.31 70.64 76.06 2.48
RoBERTa-bs4+Dropout0.1 78.90 73.39 74.31 77.06 77.06 79.82 75.23 72.48 75.23 77.06 76.06 2.23
EmbraceRoBERTa-bs4-WRONG 68.81 79.82 79.82 72.48 77.06 72.48 75.23 80.73 69.72 74.31 75.05 4.04
EmbraceRoBERTa-bs4+Dropout0.1-WRONG 77.98 77.98 75.23 78.90 78.90 75.23 78.90 77.06 69.72 79.82 76.97 2.83
EmbraceRoBERTa-bs4+Dropout0.3-WRONG 72.48 77.98 76.15 77.98 75.23 74.31 74.31 80.73 72.48 78.90 76.06 2.64
CondensedEmbraceRoBERTa-bs4 71.56 74.31 78.90 74.31 78.90 74.31 81.65 77.06 73.39 75.23 75.96 2.92
CondensedEmbraceRoBERTa-bs4+Dropout0.1 76.15 79.82 78.90 76.15 79.82 73.39 79.82 77.98 68.81 76.15 76.70 3.31
CondensedEmbraceRoBERTa-bs4+Dropout0.3 76.15 77.98 78.90 69.72 78.90 78.90 73.39 70.64 65.14 72.48 74.22 4.50
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 76.15 70.64 68.81 75.23 76.15 79.82 76.15 76.15 76.15 75.23 75.05 2.95
FrozenERoBERTa-bs4-ep100-WRONG 69.72 78.90 77.98 72.48 71.56 71.56 77.06 79.82 72.48 75.23 74.68 3.39
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 77.98 77.06 74.31 76.15 70.64 76.15 74.31 75.23 68.81 67.89 73.85 3.34
RoBERTa-bs16+Dropout0.1 78.90 74.31 69.72 74.31 76.15 72.48 70.64 69.72 74.31 72.48 73.30 2.77
EmbraceRoBERTa-bs16-WRONG 72.48 71.56 74.31 74.31 77.06 74.31 74.31 73.39 73.39 73.39 73.85 1.38
EmbraceRoBERTa-bs16+Dropout0.1-WRONG 75.23 73.39 73.39 74.31 72.48 76.15 80.73 72.48 77.98 74.31 75.05 2.49
EmbraceRoBERTa-bs16+Dropout0.3-WRONG 74.31 74.31 77.98 74.31 74.31 75.23 78.90 74.31 73.39 75.23 75.23 1.69
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1 77.06 73.39 77.06 75.23 78.90 74.31 77.98 70.64 71.56 71.56 74.77 2.79
CondensedEmbraceRoBERTa-bs16+Dropout0.3 77.06 76.15 78.90 72.48 69.72 77.06 76.15 75.23 72.48 74.31 74.95 2.60
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 77.98 77.98 76.15 73.39 76.15 77.98 69.72 77.98 77.98 67.89 75.32 3.56
FrozenERoBERTa-bs16-ep100 73.39 75.23 77.06 66.97 78.90 70.64 77.98 75.23 74.31 72.48 74.22 3.39
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 73.39 76.15 77.06 71.56 76.15 74.31 74.31 73.39 77.06 77.06 75.05 1.83
BERT-bs4+Dropout0.1 74.31 73.39 78.90 75.23 73.39 74.31 74.31 69.72 72.48 76.15 74.22 2.26
EmbraceBERT-bs4 76.15 70.64 73.39 72.48 73.39 74.31 69.72 74.31 77.06 69.72 73.12 2.39
EmbraceBERT-bs4+Dropout0.1 76.15 73.39 74.31 70.64 73.39 76.15 74.31 67.89 69.72 68.81 72.48 2.84
EmbraceBERT-bs4-p_selfatt 71.56 72.48 74.31 73.39 77.06 73.39 74.31 73.39 76.15 77.06 74.31 1.79
EmbraceBERT-bs4-p_att_clsquery_weights 77.06 73.39 75.23 69.72 72.48 72.48 73.39 74.31 76.15 72.48 73.67 2.01
CondensedEmbraceBERT-bs4 76.15 69.72 66.97 75.23 77.98 72.48 71.56 76.15 74.31 67.89 72.84 3.56
CondensedEmbraceBERT-bs4+Dropout0.1 77.98 72.48 75.23 76.15 74.31 76.15 74.31 68.81 72.48 68.81 73.67 2.90
CondensedEmbraceBERT-bs4+Dropout0.3 75.23 70.64 70.64 77.06 76.15 72.48 75.23 74.31 77.06 67.89 73.67 2.96
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 70.64 77.06 77.06 74.31 72.48 75.23 76.15 74.31 75.23 76.15 74.86 1.93
FrozenEBERT-bs4-ep100-WRONG 76.15 71.56 73.39 76.15 74.31 72.48 73.39 76.15 77.06 66.06 73.67 3.07
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3 67.89 77.06 77.06 69.72 73.39 69.72 77.06 72.48 72.48 77.06 73.39 3.36
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1 69.72 76.15 77.06 74.31 71.56 77.06 73.39 73.39 71.56 76.15 74.04 2.43
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10 72.48 75.23 77.06 77.06 77.06 73.39 80.73 72.48 72.48 76.15 75.41 2.59
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3 76.15 75.23 76.15 73.39 73.39 76.15 74.31 76.15 77.06 77.06 75.50 1.30
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10 74.31 77.06 78.90 76.15 75.23 76.15 77.06 74.31 74.31 77.06 76.06 1.45
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 75.23 73.39 77.98 71.56 73.39 74.31 75.23 76.15 71.56 74.31 74.31 1.88
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 72.48 74.31 77.98 73.39 72.48 73.39 77.06 71.56 74.31 77.98 74.50 2.24
BERT-bs16+Dropout0.1 72.48 76.15 77.98 75.23 76.15 69.72 77.06 72.48 75.23 77.06 74.95 2.46
EmbraceBERT-bs16 77.06 72.48 71.56 70.64 73.39 77.06 73.39 74.31 69.72 66.97 72.66 2.98
EmbraceBERT-bs16+Dropout0.1 74.31 72.48 74.31 72.48 75.23 75.23 77.98 70.64 69.72 69.72 73.21 2.56
CondensedEmbraceBERT-bs16 76.15 71.56 70.64 67.89 75.23 76.15 76.15 75.23 68.81 68.81 72.66 3.28
CondensedEmbraceBERT-bs16+Dropout0.1 75.23 68.81 73.39 72.48 75.23 75.23 76.15 68.81 67.89 69.72 72.29 3.04
CondensedEmbraceBERT-bs16+Dropout0.3 75.23 68.81 75.23 71.56 77.98 74.31 75.23 67.89 71.56 71.56 72.94 3.02
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 72.48 74.31 77.98 75.23 73.39 75.23 77.98 71.56 73.39 77.98 74.95 2.25
FrozenEBERT-bs16-ep100-WRONG 75.23 71.56 72.48 70.64 77.06 73.39 75.23 70.64 71.56 67.89 72.57 2.58
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3 71.56 76.15 81.65 74.31 69.72 74.31 73.39 70.64 70.64 77.06 73.94 3.46
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1 73.39 77.06 77.98 71.56 73.39 74.31 68.81 72.48 72.48 75.23 73.67 2.53
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10 77.06 75.23 76.15 75.23 70.64 71.56 71.56 74.31 73.39 77.06 74.22 2.23
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3 70.64 76.15 80.73 74.31 74.31 72.48 71.56 73.39 71.56 75.23 74.04 2.78
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10 78.90 73.39 76.15 77.06 71.56 72.48 70.64 70.64 73.39 76.15 74.04 2.72
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery 70.64 75.23 74.31 71.56 72.48 77.06 72.48 71.56 72.48 69.72 72.75 2.09
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 74.31 70.64 75.23 79.82 76.15 78.90 77.06 72.48 72.48 73.39 75.05 2.81
RoBERTa-bs4+Dropout0.1 77.98 73.39 77.06 76.15 77.98 79.82 77.98 73.39 75.23 77.98 76.70 2.02
EmbraceRoBERTa-bs4-WRONG 71.56 79.82 78.90 78.90 72.48 75.23 77.06 77.98 71.56 72.48 75.60 3.16
EmbraceRoBERTa-bs4+Dropout0.1-WRONG 76.15 73.39 77.06 75.23 77.98 70.64 79.82 77.06 70.64 69.72 74.77 3.31
EmbraceRoBERTa-bs4+Dropout0.3-WRONG 74.31 77.98 76.15 77.06 73.39 77.06 78.90 78.90 68.81 72.48 75.50 3.07
CondensedEmbraceRoBERTa-bs4 74.31 73.39 77.98 68.81 76.15 68.81 81.65 76.15 72.48 71.56 74.13 3.82
CondensedEmbraceRoBERTa-bs4+Dropout0.1 75.23 77.06 77.06 76.15 80.73 74.31 77.98 75.23 71.56 72.48 75.78 2.54
CondensedEmbraceRoBERTa-bs4+Dropout0.3 73.39 74.31 77.06 69.72 78.90 73.39 75.23 74.31 70.64 74.31 74.13 2.56
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 73.39 77.98 76.15 80.73 74.31 78.90 75.23 71.56 73.39 77.98 75.96 2.75
FrozenERoBERTa-bs4-ep100-WRONG 72.48 79.82 78.90 74.31 74.31 77.06 77.98 79.82 76.15 73.39 76.42 2.56
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 77.06 76.15 77.98 77.06 74.31 78.90 76.15 75.23 77.06 68.81 75.87 2.66
RoBERTa-bs16+Dropout0.1 78.90 72.48 71.56 77.06 72.48 75.23 72.48 71.56 71.56 73.39 73.67 2.43
EmbraceRoBERTa-bs16-WRONG 75.23 77.06 72.48 70.64 72.48 73.39 77.98 77.98 74.31 72.48 74.40 2.44
EmbraceRoBERTa-bs16+Dropout0.1-WRONG 74.31 74.31 76.15 71.56 72.48 72.48 77.98 70.64 70.64 75.23 73.58 2.31
EmbraceRoBERTa-bs16+Dropout0.3-WRONG 76.15 73.39 75.23 69.72 75.23 75.23 79.82 75.23 70.64 73.39 74.40 2.70
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1 77.06 72.48 75.23 75.23 77.98 73.39 77.06 72.48 67.89 76.15 74.50 2.87
CondensedEmbraceRoBERTa-bs16+Dropout0.3 77.98 76.15 78.90 71.56 74.31 73.39 74.31 73.39 67.89 75.23 74.31 2.99
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 79.82 77.06 75.23 77.06 76.15 78.90 74.31 76.15 77.06 67.89 75.96 3.09
FrozenERoBERTa-bs16-ep100 74.31 73.39 71.56 67.89 77.06 76.15 77.98 79.82 73.39 67.89 73.94 3.81
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
  • Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |75.23|73.39|76.15|71.56|73.39|73.39|77.06|74.31|74.31|77.98|74.68|1.84| | BERTwithAtt-bs8 |73.39|71.56|75.23|75.23|69.72|74.31|72.48|76.15|74.31|75.23|73.76|1.89| | BERTwithAttClsProjection-bs8 |74.31|77.98|74.31|77.98|75.23|73.39|76.15|74.31|75.23|76.15|75.50|1.48| | BERTwithProjection-bs8 |74.31|71.56|70.64|74.31|71.56|71.56|77.06|75.23|76.15|74.31|73.67|2.09| | BERTwithProjectionAtt-bs8 |73.39|76.15|77.98|73.39|74.31|77.06|77.06|74.31|72.48|73.39|74.95|1.84| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |78.90|71.56|70.64|67.89|74.31|77.06|73.39|71.56|74.31|69.72|72.94|3.18| | EmbraceBERT-bs8-p_att_clsquery_weights |69.72|71.56|70.64|73.39|67.89|69.72|77.06|73.39|75.23|69.72|71.83|2.72| | EmbraceBERTwithProj-bs8 |78.90|77.06|75.23|71.56|72.48|76.15|76.15|76.15|73.39|74.31|75.14|2.11| | EmbraceBERTwithProj-bs8-p_attclsqw |73.39|77.06|77.06|76.15|73.39|75.23|78.90|77.06|74.31|76.15|75.87|1.69| | EBERTconcatAtt-bs8 |75.23|71.56|71.56|68.81|68.81|70.64|69.72|76.15|76.15|71.56|72.02|2.70| | EBERTconcatAtt-bs8_attclsqw |68.81|67.89|70.64|69.72|77.06|72.48|71.56|70.64|74.31|72.48|71.56|2.56| | EBERTconcatAttwithProj-bs8 |77.98|73.39|75.23|77.98|71.56|75.23|73.39|77.98|74.31|73.39|75.05|2.16| | EBERTconcatAttwithProj-bs8_attclsqw |75.23|70.64|77.06|74.31|77.98|79.82|74.31|75.23|72.48|74.31|75.14|2.51| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |76.15|75.23|77.06|75.23|75.23|76.15|70.64|75.23|74.31|73.39|74.86|1.70| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |68.81|77.06|74.31|72.48|69.72|76.15|72.48|69.72|71.56|70.64|72.29|2.65| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |74.31|76.15|75.23|76.15|73.39|75.23|75.23|77.06|74.31|76.15|75.32|1.04| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |76.15|77.98|76.15|75.23|77.06|75.23|73.39|76.15|74.31|75.23|75.69|1.25| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |77.98|73.39|71.56|73.39|69.72|68.81|73.39|73.39|73.39|74.31|72.94|2.40| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |75.23|71.56|77.98|77.98|72.48|71.56|70.64|73.39|69.72|72.48|73.30|2.74| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |76.15|76.15|76.15|74.31|75.23|72.48|77.06|77.98|76.15|73.39|75.50|1.59| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |75.23|77.98|74.31|76.15|77.06|74.31|74.31|76.15|76.15|75.23|75.69|1.18|

gtts-sphinx

  • Epoch 30
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 83.49 81.65 81.65 85.32 78.90 85.32 87.16 88.07 88.07 87.16 84.68 2.99
BERT-bs4+Dropout0.1 88.99 81.65 82.57 86.24 83.49 84.40 87.16 87.16 84.40 87.16 85.32 2.25
EmbraceBERT-bs4-WRONG 79.82 79.82 82.57 79.82 81.65 83.49 84.40 82.57 81.65 78.90 81.47 1.73
EmbraceBERT-bs4+Dropout0.1-WRONG 80.73 81.65 81.65 76.15 81.65 80.73 83.49 83.49 84.40 78.90 81.28 2.29
EmbraceBERT-bs4+Dropout0.3-WRONG
CondensedEmbraceBERT-bs4 79.82 80.73 83.49 77.06 84.40 80.73 85.32 80.73 84.40 81.65 81.83 2.42
CondensedEmbraceBERT-bs4+Dropout0.1 77.98 81.65 81.65 85.32 77.98 79.82 81.65 80.73 84.40 80.73 81.19 2.26
CondensedEmbraceBERT-bs4+Dropout0.3 82.57 83.49 78.90 86.24 78.90 78.90 86.24 80.73 83.49 78.90 81.83 2.84
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 87.16 81.65 83.49 85.32 79.82 85.32 86.24 83.49 90.83 84.40 84.77 2.88
FrozenEBERT-bs4-ep100-WRONG 77.06 73.39 81.65 76.15 82.57 81.65 80.73 80.73 83.49 81.65 79.91 3.08
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 84.40 82.57 84.40 83.49 74.31 76.15 86.24 81.65 80.73 81.65 81.56 3.54
BERT-bs16+Dropout0.1 82.57 83.49 87.16 79.82 73.39 72.48 87.16 78.90 83.49 82.57 81.10 4.79
EmbraceBERT-bs16-WRONG 78.90 79.82 82.57 81.65 77.98 77.98 80.73 82.57 80.73 77.06 80.00 1.87
EmbraceBERT-bs16+Dropout0.1-WRONG 77.06 81.65 82.57 80.73 80.73 78.90 80.73 80.73 83.49 80.73 80.73 1.69
EmbraceBERT-bs16+Dropout0.3-WRONG
CondensedEmbraceBERT-bs16 79.82 81.65 83.49 80.73 77.98 78.90 79.82 85.32 82.57 81.65 81.19 2.10
CondensedEmbraceBERT-bs16+Dropout0.1 76.15 82.57 81.65 78.90 82.57 79.82 78.90 79.82 86.24 75.23 80.18 3.08
CondensedEmbraceBERT-bs16+Dropout0.3 72.48 81.65 81.65 80.73 79.82 78.90 76.15 77.06 84.40 77.06 78.99 3.24
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 88.07 82.57 84.40 83.49 82.57 84.40 90.83 81.65 85.32 86.24 84.95 2.67
FrozenEBERT-bs16-ep100-WRONG 77.98 76.15 83.49 85.32 82.57 80.73 83.49 79.82 79.82 80.73 81.01 2.63
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 88.07 88.07 81.65 81.65 86.24 84.40 85.32 81.65 83.49 84.40 84.50 2.34
RoBERTa-bs4+Dropout0.1 88.07 88.07 88.07 84.40 85.32 84.40 85.32 81.65 83.49 78.90 84.77 2.82
EmbraceRoBERTa-bs4-WRONG 86.24 87.16 79.82 86.24 86.24 84.40 86.24 82.57 85.32 86.24 85.05 2.13
EmbraceRoBERTa-bs4+Dropout0.1-WRONG 84.40 87.16 88.99 85.32 85.32 85.32 81.65 77.98 87.16 84.40 84.77 2.94
EmbraceRoBERTa-bs4+Dropout0.3-WRONG 86.24 84.40 88.07 83.49 83.49 88.99 85.32 75.23 86.24 82.57 84.40 3.62
CondensedEmbraceRoBERTa-bs4 86.24 85.32 85.32 82.57 84.40 85.32 87.16 82.57 87.16 83.49 84.95 1.60
CondensedEmbraceRoBERTa-bs4+Dropout0.1 85.32 85.32 83.49 85.32 86.24 84.40 88.07 79.82 88.07 86.24 85.23 2.26
CondensedEmbraceRoBERTa-bs4+Dropout0.3 86.24 86.24 85.32 81.65 84.40 86.24 85.32 75.23 88.99 84.40 84.40 3.53
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 87.16 87.16 79.82 82.57 85.32 84.40 83.49 81.65 82.57 85.32 83.94 2.26
FrozenERoBERTa-bs4-ep100-WRONG 87.16 86.24 83.49 88.99 85.32 84.40 80.73 83.49 82.57 84.40 84.68 2.25
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 87.16 88.99 79.82 83.49 85.32 86.24 86.24 82.57 85.32 86.24 85.14 2.45
RoBERTa-bs16+Dropout0.1 86.24 86.24 77.06 84.40 85.32 85.32 88.07 83.49 84.40 84.40 84.50 2.77
EmbraceRoBERTa-bs16-WRONG 82.57 85.32 85.32 88.07 90.83 88.07 84.40 81.65 86.24 86.24 85.87 2.57
EmbraceRoBERTa-bs16+Dropout0.1-WRONG 84.40 86.24 81.65 85.32 86.24 83.49 87.16 82.57 83.49 86.24 84.68 1.74
EmbraceRoBERTa-bs16+Dropout0.3-WRONG 86.24 86.24 81.65 84.40 85.32 81.65 84.40 83.49 85.32 85.32 84.40 1.59
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1 84.40 84.40 87.16 85.32 85.32 85.32 84.40 83.49 86.24 83.49 84.95 1.10
CondensedEmbraceRoBERTa-bs16+Dropout0.3 82.57 84.40 86.24 85.32 88.07 87.16 85.32 84.40 83.49 81.65 84.86 1.89
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 86.24 87.16 77.06 84.40 84.40 84.40 86.24 83.49 86.24 87.16 84.68 2.81
FrozenERoBERTa-bs16-ep100 87.16 85.32 84.40 81.65 86.24 85.32 80.73 80.73 82.57 84.40 83.85 2.18
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 88.07 82.57 83.49 85.32 82.57 85.32 86.24 86.24 85.32 87.16 85.23 1.76
BERT-bs4+Dropout0.1 88.07 81.65 82.57 88.07 83.49 84.40 84.40 87.16 86.24 89.91 85.60 2.56
EmbraceBERT-bs4 80.73 81.65 86.24 78.90 83.49 81.65 85.32 81.65 82.57 80.73 82.29 2.09
EmbraceBERT-bs4+Dropout0.1 81.65 74.31 81.65 82.57 80.73 79.82 80.73 79.82 82.57 81.65 80.55 2.28
EmbraceBERT-bs4-p_selfatt 86.24 86.24 83.49 78.90 85.32 82.57 86.24 79.82 85.32 81.65 83.58 2.61
EmbraceBERT-bs4-p_att_clsquery_weights 81.65 83.49 84.40 86.24 78.90 81.65 83.49 80.73 82.57 84.40 82.75 2.00
CondensedEmbraceBERT-bs4 81.65 86.24 81.65 75.23 82.57 79.82 83.49 82.57 83.49 78.90 81.56 2.86
CondensedEmbraceBERT-bs4+Dropout0.1 81.65 81.65 82.57 83.49 79.82 83.49 80.73 80.73 86.24 79.82 82.02 1.89
CondensedEmbraceBERT-bs4+Dropout0.3 82.57 87.16 82.57 83.49 78.90 79.82 84.40 80.73 84.40 77.98 82.20 2.70
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 88.99 81.65 85.32 88.07 83.49 85.32 87.16 85.32 88.07 87.16 86.06 2.16
FrozenEBERT-bs4-ep100-WRONG 78.90 78.90 83.49 75.23 82.57 78.90 84.40 88.07 82.57 80.73 81.38 3.43
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3 77.06 86.24 88.07 86.24 82.57 77.98 85.32 84.40 79.82 83.49 83.12 3.53
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1 83.49 78.90 84.40 86.24 79.82 79.82 82.57 78.90 78.90 82.57 81.56 2.51
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10 88.99 83.49 84.40 87.16 86.24 80.73 89.91 85.32 82.57 84.40 85.32 2.69
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3 79.82 80.73 79.82 80.73 77.06 80.73 82.57 80.73 78.90 81.65 80.28 1.44
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10 85.32 82.57 86.24 84.40 83.49 83.49 87.16 83.49 84.40 88.07 84.86 1.70
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 79.82 82.57 85.32 83.49 84.40 83.49 82.57 81.65 86.24 83.49 83.30 1.73
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 88.07 82.57 83.49 82.57 80.73 81.65 87.16 82.57 86.24 86.24 84.13 2.43
BERT-bs16+Dropout0.1 84.40 81.65 86.24 79.82 81.65 80.73 91.74 88.07 86.24 85.32 84.59 3.52
EmbraceBERT-bs16 77.06 80.73 81.65 82.57 82.57 81.65 82.57 83.49 81.65 79.82 81.38 1.74
EmbraceBERT-bs16+Dropout0.1 79.82 80.73 84.40 77.98 82.57 81.65 83.49 80.73 82.57 76.15 81.01 2.39
CondensedEmbraceBERT-bs16 80.73 80.73 82.57 83.49 82.57 81.65 85.32 86.24 81.65 80.73 82.57 1.83
CondensedEmbraceBERT-bs16+Dropout0.1 81.65 82.57 82.57 79.82 80.73 80.73 80.73 84.40 82.57 76.15 81.19 2.10
CondensedEmbraceBERT-bs16+Dropout0.3 81.65 80.73 85.32 82.57 81.65 77.98 81.65 82.57 80.73 77.06 81.19 2.22
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 90.83 83.49 83.49 83.49 83.49 82.57 88.07 86.24 88.07 86.24 85.60 2.60
FrozenEBERT-bs16-ep100-WRONG 82.57 80.73 83.49 83.49 82.57 83.49 82.57 82.57 81.65 84.40 82.75 0.99
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3 78.90 80.73 78.90 80.73 87.16 80.73 83.49 77.06 86.24 81.65 81.56 3.06
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1 79.82 79.82 81.65 77.98 84.40 81.65 75.23 74.31 88.07 81.65 80.46 3.87
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10 81.65 86.24 82.57 81.65 77.98 77.98 84.40 76.15 79.82 85.32 81.38 3.21
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3 78.90 84.40 81.65 81.65 87.16 81.65 82.57 74.31 86.24 81.65 82.02 3.46
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10 84.40 85.32 80.73 82.57 77.98 79.82 84.40 76.15 80.73 84.40 81.65 2.93
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery 77.98 76.15 77.98 81.65 81.65 79.82 83.49 78.90 82.57 81.65 80.18 2.25
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 87.16 88.07 82.57 81.65 85.32 84.40 85.32 81.65 79.82 80.73 83.67 2.65
RoBERTa-bs4+Dropout0.1 88.99 87.16 87.16 83.49 84.40 84.40 85.32 82.57 80.73 82.57 84.68 2.39
EmbraceRoBERTa-bs4-WRONG 86.24 87.16 83.49 88.99 87.16 83.49 85.32 81.65 86.24 87.16 85.69 2.10
EmbraceRoBERTa-bs4+Dropout0.1-WRONG 84.40 88.07 88.07 85.32 86.24 85.32 81.65 79.82 88.07 86.24 85.32 2.63
EmbraceRoBERTa-bs4+Dropout0.3-WRONG 84.40 87.16 87.16 86.24 88.99 87.16 85.32 76.15 88.99 83.49 85.50 3.55
CondensedEmbraceRoBERTa-bs4 85.32 86.24 85.32 84.40 82.57 86.24 88.07 79.82 88.07 84.40 85.05 2.36
CondensedEmbraceRoBERTa-bs4+Dropout0.1 85.32 86.24 87.16 85.32 87.16 87.16 86.24 80.73 87.16 85.32 85.78 1.85
CondensedEmbraceRoBERTa-bs4+Dropout0.3 84.40 85.32 83.49 82.57 84.40 84.40 84.40 78.90 87.16 84.40 83.94 2.02
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 88.07 86.24 84.40 83.49 85.32 84.40 84.40 82.57 82.57 82.57 84.40 1.69
FrozenERoBERTa-bs4-ep100-WRONG 86.24 88.99 86.24 88.99 86.24 84.40 84.40 80.73 87.16 86.24 85.96 2.29
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 86.24 86.24 78.90 87.16 82.57 85.32 86.24 82.57 86.24 86.24 84.77 2.47
RoBERTa-bs16+Dropout0.1 88.07 87.16 79.82 83.49 83.49 87.16 87.16 82.57 81.65 83.49 84.40 2.66
EmbraceRoBERTa-bs16-WRONG 86.24 86.24 82.57 87.16 90.83 88.99 83.49 80.73 87.16 86.24 85.96 2.84
EmbraceRoBERTa-bs16+Dropout0.1-WRONG 79.82 86.24 79.82 84.40 88.07 83.49 85.32 82.57 80.73 82.57 83.30 2.62
EmbraceRoBERTa-bs16+Dropout0.3-WRONG 81.65 85.32 79.82 80.73 88.07 81.65 82.57 84.40 84.40 84.40 83.30 2.35
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1 84.40 84.40 83.49 81.65 87.16 84.40 85.32 84.40 85.32 81.65 84.22 1.58
CondensedEmbraceRoBERTa-bs16+Dropout0.3 82.57 85.32 81.65 81.65 86.24 87.16 86.24 83.49 83.49 80.73 83.85 2.14
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 88.07 87.16 79.82 88.07 87.16 83.49 85.32 82.57 83.49 86.24 85.14 2.59
FrozenERoBERTa-bs16-ep100 82.57 85.32 80.73 83.49 88.07 83.49 80.73 84.40 87.16 85.32 84.13 2.32
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
  • Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |85.32|81.65|88.07|83.49|85.32|81.65|86.24|85.32|85.32|86.24|84.86|1.94| | BERTwithAtt-bs8 |80.73|82.57|81.65|82.57|85.32|83.49|85.32|87.16|81.65|87.16|83.76|2.21| | BERTwithAttClsProjection-bs8 |85.32|89.91|86.24|86.24|85.32|82.57|86.24|84.40|84.40|84.40|85.50|1.83| | BERTwithProjection-bs8 |85.32|82.57|82.57|81.65|78.90|82.57|78.90|83.49|81.65|82.57|82.02|1.84| | BERTwithProjectionAtt-bs8 |84.40|83.49|83.49|82.57|82.57|84.40|87.16|83.49|82.57|84.40|83.85|1.31| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |80.73|81.65|84.40|81.65|81.65|82.57|82.57|84.40|80.73|77.06|81.74|1.99| | EmbraceBERT-bs8-p_att_clsquery_weights |80.73|85.32|87.16|85.32|78.90|88.07|82.57|81.65|83.49|80.73|83.39|2.86| | EmbraceBERTwithProj-bs8 |79.82|84.40|84.40|88.07|88.07|88.99|86.24|87.16|80.73|88.99|85.69|3.13| | EmbraceBERTwithProj-bs8-p_attclsqw |87.16|84.40|86.24|82.57|86.24|84.40|87.16|86.24|84.40|83.49|85.23|1.50| | EBERTconcatAtt-bs8 |82.57|83.49|82.57|86.24|80.73|85.32|79.82|78.90|80.73|84.40|82.48|2.30| | EBERTconcatAtt-bs8_attclsqw |88.99|77.98|82.57|81.65|84.40|87.16|84.40|81.65|83.49|77.98|83.03|3.34| | EBERTconcatAttwithProj-bs8 |86.24|79.82|85.32|81.65|83.49|80.73|82.57|86.24|84.40|82.57|83.30|2.12| | EBERTconcatAttwithProj-bs8_attclsqw |84.40|88.07|83.49|81.65|83.49|85.32|86.24|83.49|83.49|84.40|84.40|1.69| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |77.06|83.49|83.49|79.82|83.49|79.82|77.98|80.73|81.65|82.57|81.01|2.21| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |84.40|87.16|82.57|80.73|81.65|82.57|77.06|82.57|80.73|83.49|82.29|2.50| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |86.24|86.24|83.49|85.32|84.40|84.40|80.73|86.24|83.49|83.49|84.40|1.64| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |87.16|83.49|87.16|85.32|85.32|83.49|83.49|82.57|84.40|82.57|84.50|1.61| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |82.57|81.65|77.98|81.65|82.57|80.73|77.98|81.65|77.98|87.16|81.19|2.67| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |79.82|84.40|74.31|84.40|82.57|78.90|82.57|77.06|78.90|82.57|80.55|3.15| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |84.40|85.32|81.65|81.65|81.65|84.40|84.40|86.24|88.07|84.40|84.22|2.00| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |86.24|85.32|88.07|82.57|86.24|83.49|83.49|84.40|86.24|82.57|84.86|1.75|

gtts-witai

  • Epoch 30
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 88.99 90.83 94.50 88.99 91.74 90.83 93.58 91.74 90.83 90.83 91.28 1.65
BERT-bs4+Dropout0.1 89.91 90.83 91.74 89.91 90.83 88.07 92.66 90.83 89.91 88.99 90.37 1.25
EmbraceBERT-bs4-WRONG 88.99 88.99 89.91 88.07 89.91 89.91 89.91 91.74 84.40 88.07 88.99 1.83
EmbraceBERT-bs4+Dropout0.1-WRONG 88.99 89.91 88.07 91.74 88.99 88.99 88.07 88.07 88.99 89.91 89.17 1.07
EmbraceBERT-bs4+Dropout0.3-WRONG
CondensedEmbraceBERT-bs4 86.24 88.07 88.99 87.16 88.07 87.16 89.91 92.66 85.32 84.40 87.80 2.25
CondensedEmbraceBERT-bs4+Dropout0.1 88.07 88.07 88.99 91.74 91.74 89.91 89.91 89.91 87.16 89.91 89.54 1.43
CondensedEmbraceBERT-bs4+Dropout0.3 87.16 88.07 87.16 83.49 88.99 88.07 88.07 92.66 88.99 88.07 88.07 2.13
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 92.66 91.74 93.58 88.99 90.83 90.83 91.74 92.66 92.66 89.91 91.56 1.35
FrozenEBERT-bs4-ep100-WRONG 90.83 90.83 86.24 87.16 90.83 88.99 89.91 90.83 84.40 87.16 88.72 2.21
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 89.91 89.91 90.83 86.24 87.16 81.65 90.83 87.16 90.83 87.16 88.17 2.77
BERT-bs16+Dropout0.1 85.32 90.83 89.91 86.24 84.40 82.57 89.91 89.91 88.99 88.99 87.71 2.70
EmbraceBERT-bs16-WRONG 88.99 80.73 88.99 82.57 87.16 88.99 83.49 92.66 87.16 83.49 86.42 3.52
EmbraceBERT-bs16+Dropout0.1-WRONG 88.07 90.83 84.40 86.24 88.99 88.99 87.16 83.49 85.32 84.40 86.79 2.29
EmbraceBERT-bs16+Dropout0.3-WRONG
CondensedEmbraceBERT-bs16 88.07 86.24 85.32 83.49 88.07 85.32 88.07 88.99 86.24 84.40 86.42 1.73
CondensedEmbraceBERT-bs16+Dropout0.1 88.07 89.91 87.16 85.32 87.16 87.16 87.16 83.49 87.16 84.40 86.70 1.75
CondensedEmbraceBERT-bs16+Dropout0.3 84.40 88.99 86.24 83.49 87.16 88.07 88.99 81.65 88.07 83.49 86.06 2.49
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 90.83 88.99 91.74 88.99 89.91 88.99 93.58 90.83 89.91 89.91 90.37 1.38
FrozenEBERT-bs16-ep100-WRONG 90.83 88.07 86.24 88.99 88.99 86.24 88.07 91.74 88.07 87.16 88.44 1.70
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 89.91 88.99 88.99 91.74 88.99 88.07 87.16 87.16 87.16 88.99 88.72 1.36
RoBERTa-bs4+Dropout0.1 90.83 86.24 87.16 87.16 88.07 86.24 89.91 87.16 88.99 89.91 88.17 1.56
EmbraceRoBERTa-bs4-WRONG 89.91 89.91 86.24 87.16 89.91 86.24 88.99 89.91 88.07 87.16 88.35 1.48
EmbraceRoBERTa-bs4+Dropout0.1-WRONG 91.74 90.83 88.99 88.99 88.99 88.07 88.99 88.07 87.16 88.07 88.99 1.30
EmbraceRoBERTa-bs4+Dropout0.3-WRONG 87.16 89.91 88.07 88.99 88.99 85.32 88.99 90.83 87.16 88.07 88.35 1.48
CondensedEmbraceRoBERTa-bs4 90.83 88.07 88.07 88.07 89.91 82.57 89.91 90.83 87.16 84.40 87.98 2.58
CondensedEmbraceRoBERTa-bs4+Dropout0.1 85.32 85.32 87.16 88.99 91.74 88.07 87.16 87.16 87.16 88.99 87.71 1.80
CondensedEmbraceRoBERTa-bs4+Dropout0.3 90.83 87.16 90.83 88.07 90.83 88.07 88.99 92.66 88.99 88.99 89.54 1.60
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 88.99 88.99 89.91 89.91 88.99 86.24 90.83 86.24 88.07 88.99 88.72 1.42
FrozenERoBERTa-bs4-ep100-WRONG 88.99 88.07 86.24 86.24 90.83 84.40 89.91 88.99 87.16 88.07 87.89 1.83
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 88.07 88.99 88.99 88.99 87.16 87.16 86.24 88.07 90.83 88.07 88.26 1.22
RoBERTa-bs16+Dropout0.1 88.99 88.99 88.07 86.24 88.07 88.07 87.16 89.91 89.91 86.24 88.17 1.26
EmbraceRoBERTa-bs16-WRONG 89.91 89.91 88.07 88.07 88.99 89.91 86.24 86.24 88.99 87.16 88.35 1.36
EmbraceRoBERTa-bs16+Dropout0.1-WRONG 88.99 88.99 88.99 88.07 89.91 87.16 88.07 88.07 88.07 88.99 88.53 0.74
EmbraceRoBERTa-bs16+Dropout0.3-WRONG 88.99 88.99 89.91 89.91 92.66 88.07 88.07 88.99 91.74 89.91 89.72 1.41
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1 82.57 89.91 88.99 87.16 89.91 88.07 87.16 88.99 88.07 85.32 87.61 2.14
CondensedEmbraceRoBERTa-bs16+Dropout0.3 86.24 89.91 87.16 88.07 91.74 88.07 87.16 88.07 89.91 87.16 88.35 1.59
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 88.99 88.07 88.07 88.07 88.07 88.07 86.24 89.91 88.99 89.91 88.44 1.02
FrozenERoBERTa-bs16-ep100 90.83 90.83 88.07 88.99 88.07 87.16 88.99 88.99 88.99 88.07 88.90 1.12
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 88.99 91.74 92.66 88.99 91.74 90.83 93.58 91.74 92.66 90.83 91.38 1.43
BERT-bs4+Dropout0.1 89.91 92.66 92.66 88.99 91.74 88.07 92.66 90.83 89.91 90.83 90.83 1.54
EmbraceBERT-bs4 88.07 90.83 88.07 88.99 87.16 87.16 89.91 93.58 88.99 83.49 88.62 2.50
EmbraceBERT-bs4+Dropout0.1 89.91 90.83 92.66 86.24 88.99 87.16 87.16 92.66 88.99 89.91 89.45 2.10
EmbraceBERT-bs4-p_selfatt 89.91 88.99 91.74 90.83 92.66 88.99 93.58 85.32 88.07 92.66 90.28 2.40
EmbraceBERT-bs4-p_att_clsquery_weights 87.16 87.16 90.83 91.74 88.07 88.99 90.83 89.91 82.57 89.91 88.72 2.53
CondensedEmbraceBERT-bs4 87.16 91.74 88.99 88.99 88.99 88.07 89.91 91.74 88.07 87.16 89.08 1.56
CondensedEmbraceBERT-bs4+Dropout0.1 89.91 91.74 91.74 89.91 90.83 90.83 89.91 91.74 88.99 88.07 90.37 1.18
CondensedEmbraceBERT-bs4+Dropout0.3 88.07 92.66 88.99 88.99 89.91 88.07 88.99 92.66 90.83 91.74 90.09 1.68
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 91.74 93.58 92.66 88.99 91.74 92.66 94.50 92.66 92.66 88.99 92.02 1.69
FrozenEBERT-bs4-ep100-WRONG 90.83 92.66 89.91 88.07 91.74 90.83 90.83 90.83 88.07 87.16 90.09 1.68
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3 88.07 91.74 91.74 89.91 84.40 88.07 89.91 89.91 89.91 91.74 89.54 2.14
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1 89.91 91.74 91.74 91.74 88.99 91.74 89.91 91.74 88.07 90.83 90.64 1.28
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10 91.74 90.83 92.66 93.58 90.83 89.91 92.66 92.66 88.99 91.74 91.56 1.35
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3 83.49 93.58 91.74 91.74 88.07 88.07 90.83 91.74 90.83 89.91 90.00 2.70
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10 91.74 92.66 92.66 89.91 91.74 86.24 89.91 91.74 90.83 92.66 91.01 1.87
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 90.83 87.16 91.74 88.07 91.74 90.83 89.91 88.99 88.99 91.74 90.00 1.56
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 89.91 91.74 92.66 89.91 89.91 88.07 93.58 88.99 89.91 92.66 90.73 1.71
BERT-bs16+Dropout0.1 88.99 90.83 92.66 87.16 90.83 88.07 92.66 91.74 88.99 90.83 90.28 1.80
EmbraceBERT-bs16 88.99 87.16 89.91 89.91 89.91 89.91 89.91 91.74 88.07 86.24 89.17 1.52
EmbraceBERT-bs16+Dropout0.1 89.91 90.83 89.91 88.99 88.99 89.91 88.07 89.91 87.16 87.16 89.08 1.19
CondensedEmbraceBERT-bs16 88.99 88.99 88.99 87.16 89.91 89.91 92.66 93.58 88.07 89.91 89.82 1.86
CondensedEmbraceBERT-bs16+Dropout0.1 88.99 92.66 90.83 87.16 88.07 88.99 88.07 91.74 88.07 83.49 88.81 2.45
CondensedEmbraceBERT-bs16+Dropout0.3 88.07 90.83 90.83 88.07 89.91 88.99 91.74 90.83 88.07 86.24 89.36 1.65
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 91.74 91.74 92.66 89.91 91.74 88.07 94.50 91.74 88.99 92.66 91.38 1.80
FrozenEBERT-bs16-ep100-WRONG 89.91 88.07 88.99 88.07 92.66 89.91 90.83 90.83 88.99 90.83 89.91 1.36
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3 89.91 89.91 90.83 90.83 88.07 91.74 87.16 87.16 90.83 88.07 89.45 1.60
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1 89.91 88.99 89.91 87.16 89.91 92.66 88.99 88.07 88.07 88.07 89.17 1.47
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10 88.99 92.66 91.74 91.74 89.91 89.91 87.16 90.83 89.91 91.74 90.46 1.55
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3 88.99 92.66 89.91 90.83 88.99 91.74 86.24 88.99 89.91 88.99 89.72 1.68
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10 88.99 92.66 92.66 91.74 91.74 89.91 87.16 89.91 89.91 89.91 90.46 1.65
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery 90.83 88.99 87.16 88.99 87.16 88.07 88.99 88.99 86.24 89.91 88.53 1.31
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 88.99 89.91 88.99 91.74 89.91 87.16 86.24 87.16 88.07 88.99 88.72 1.54
RoBERTa-bs4+Dropout0.1 88.07 86.24 87.16 89.91 88.07 88.07 89.91 87.16 87.16 89.91 88.17 1.26
EmbraceRoBERTa-bs4-WRONG 88.99 88.07 88.99 88.99 89.91 87.16 90.83 89.91 85.32 87.16 88.53 1.55
EmbraceRoBERTa-bs4+Dropout0.1-WRONG 89.91 88.99 89.91 88.99 88.99 87.16 89.91 88.07 89.91 87.16 88.90 1.04
EmbraceRoBERTa-bs4+Dropout0.3-WRONG 88.99 89.91 90.83 89.91 88.99 86.24 88.07 88.99 87.16 88.07 88.72 1.30
CondensedEmbraceRoBERTa-bs4 90.83 88.07 88.07 87.16 90.83 84.40 89.91 90.83 88.07 87.16 88.53 1.98
CondensedEmbraceRoBERTa-bs4+Dropout0.1 86.24 89.91 87.16 89.91 90.83 88.07 87.16 88.07 89.91 88.99 88.62 1.43
CondensedEmbraceRoBERTa-bs4+Dropout0.3 88.07 88.07 90.83 88.99 88.99 87.16 88.99 89.91 86.24 86.24 88.35 1.42
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 91.74 88.99 89.91 91.74 88.99 86.24 88.99 87.16 89.91 88.07 89.17 1.68
FrozenERoBERTa-bs4-ep100-WRONG 88.99 88.99 88.99 86.24 89.91 85.32 88.99 88.99 88.07 85.32 87.98 1.61
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 89.91 87.16 88.07 86.24 88.07 88.99 88.07 88.99 91.74 88.99 88.62 1.43
RoBERTa-bs16+Dropout0.1 88.07 88.07 89.91 89.91 88.99 88.99 87.16 87.16 89.91 87.16 88.53 1.10
EmbraceRoBERTa-bs16-WRONG 88.99 90.83 88.07 89.91 88.07 89.91 89.91 86.24 89.91 87.16 88.90 1.39
EmbraceRoBERTa-bs16+Dropout0.1-WRONG 86.24 88.99 86.24 88.07 89.91 88.07 88.99 88.99 88.99 88.99 88.35 1.16
EmbraceRoBERTa-bs16+Dropout0.3-WRONG 88.07 89.91 87.16 90.83 91.74 88.99 89.91 89.91 88.99 88.07 89.36 1.31
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1 87.16 89.91 87.16 88.99 88.07 90.83 88.99 88.99 88.07 88.07 88.62 1.10
CondensedEmbraceRoBERTa-bs16+Dropout0.3 87.16 88.99 86.24 89.91 92.66 87.16 88.07 88.07 88.07 88.99 88.53 1.70
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 88.99 88.99 88.07 88.99 88.07 87.16 88.07 89.91 90.83 89.91 88.90 1.04
FrozenERoBERTa-bs16-ep100 87.16 89.91 89.91 89.91 88.07 88.07 89.91 88.07 89.91 87.16 88.81 1.15
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
  • Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |90.83|89.91|91.74|88.07|91.74|88.99|92.66|89.91|89.91|94.50|90.83|1.79| | BERTwithAtt-bs8 |87.16|91.74|88.99|89.91|90.83|91.74|89.91|92.66|88.99|92.66|90.46|1.70| | BERTwithAttClsProjection-bs8 |91.74|88.07|90.83|90.83|90.83|91.74|88.07|92.66|91.74|88.99|90.55|1.54| | BERTwithProjection-bs8 |84.40|93.58|91.74|85.32|87.16|89.91|84.40|92.66|87.16|90.83|88.72|3.28| | BERTwithProjectionAtt-bs8 |89.91|87.16|91.74|86.24|87.16|89.91|89.91|90.83|89.91|88.99|89.17|1.68| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |88.99|91.74|92.66|89.91|88.07|89.91|87.16|93.58|89.91|87.16|89.91|2.09| | EmbraceBERT-bs8-p_att_clsquery_weights |87.16|90.83|89.91|90.83|88.99|89.91|91.74|90.83|91.74|88.07|90.00|1.45| | EmbraceBERTwithProj-bs8 |91.74|92.66|89.91|92.66|88.99|90.83|90.83|92.66|90.83|91.74|91.28|1.18| | EmbraceBERTwithProj-bs8-p_attclsqw |91.74|90.83|88.99|84.40|92.66|91.74|91.74|88.07|91.74|88.99|90.09|2.39| | EBERTconcatAtt-bs8 |88.99|89.91|88.07|89.91|85.32|88.99|88.99|91.74|89.91|91.74|89.36|1.75| | EBERTconcatAtt-bs8_attclsqw |90.83|88.07|88.99|86.24|85.32|88.99|91.74|89.91|88.99|90.83|88.99|1.92| | EBERTconcatAttwithProj-bs8 |89.91|93.58|91.74|83.49|88.99|88.99|89.91|90.83|90.83|90.83|89.91|2.50| | EBERTconcatAttwithProj-bs8_attclsqw |88.99|89.91|87.16|88.99|89.91|89.91|89.91|89.91|90.83|90.83|89.63|1.01| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |92.66|90.83|88.99|86.24|87.16|88.99|92.66|90.83|88.99|87.16|89.45|2.14| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |90.83|90.83|90.83|88.99|84.40|85.32|89.91|90.83|89.91|88.99|89.08|2.23| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |89.91|90.83|90.83|89.91|90.83|90.83|90.83|90.83|88.99|89.91|90.37|0.62| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |91.74|89.91|91.74|89.91|89.91|89.91|89.91|91.74|91.74|90.83|90.73|0.87| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |88.99|90.83|89.91|90.83|90.83|88.99|87.16|90.83|86.24|88.07|89.27|1.59| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |85.32|90.83|89.91|90.83|89.91|88.99|91.74|90.83|89.91|88.99|89.72|1.68| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |91.74|91.74|89.91|90.83|89.91|91.74|91.74|91.74|91.74|88.99|91.01|0.99| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |91.74|92.66|92.66|88.99|89.91|88.99|88.99|89.91|88.99|90.83|90.37|1.44|

macsay-google

  • Epoch 30
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 85.32 86.24 88.07 82.57 80.73 85.32 87.16 83.49 85.32 88.99 85.32 2.39
BERT-bs4+Dropout0.1 83.49 86.24 89.91 85.32 80.73 80.73 86.24 83.49 84.40 88.07 84.86 2.79
EmbraceBERT-bs4-WRONG 80.73 87.16 80.73 84.40 84.40 81.65 85.32 86.24 80.73 86.24 83.76 2.43
EmbraceBERT-bs4+Dropout0.1-WRONG 9.82 86.24 86.24 81.65 88.99 77.98 83.49 84.40 86.24 80.73 83.58 3.29
EmbraceBERT-bs4+Dropout0.3-WRONG
CondensedEmbraceBERT-bs4 77.98 88.99 78.90 82.57 81.65 81.65 85.32 88.99 78.90 69.72 81.47 5.41
CondensedEmbraceBERT-bs4+Dropout0.1 80.73 85.32 86.24 86.24 87.16 83.49 85.32 85.32 83.49 77.06 84.04 2.91
CondensedEmbraceBERT-bs4+Dropout0.3 82.57 86.24 82.57 84.40 86.24 85.32 85.32 79.82 85.32 78.90 83.67 2.49
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 86.24 86.24 86.24 83.49 79.82 81.65 85.32 82.57 87.16 85.32 84.40 2.28
FrozenEBERT-bs4-ep100-WRONG 77.06 87.16 81.65 80.73 79.82 80.73 86.24 86.24 79.82 85.32 82.48 3.29
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 84.40 84.40 87.16 80.73 77.98 78.90 83.49 84.40 84.40 87.16 83.30 2.98
BERT-bs16+Dropout0.1 81.65 83.49 88.07 79.82 75.23 76.15 88.99 82.57 82.57 84.40 82.29 4.23
EmbraceBERT-bs16-WRONG 81.65 82.57 77.06 84.40 75.23 80.73 81.65 82.57 78.90 73.39 79.82 3.38
EmbraceBERT-bs16+Dropout0.1-WRONG 84.40 86.24 81.65 82.57 83.49 82.57 76.15 81.65 78.90 76.15 81.38 3.18
EmbraceBERT-bs16+Dropout0.3-WRONG
CondensedEmbraceBERT-bs16 80.73 87.16 79.82 81.65 75.23 83.49 76.15 79.82 84.40 75.23 80.37 3.81
CondensedEmbraceBERT-bs16+Dropout0.1 81.65 82.57 78.90 81.65 69.72 84.40 82.57 80.73 82.57 77.06 80.18 4.00
CondensedEmbraceBERT-bs16+Dropout0.3 83.49 81.65 78.90 81.65 80.73 81.65 76.15 80.73 84.40 79.82 80.92 2.20
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 84.40 84.40 88.99 79.82 81.65 82.57 85.32 82.57 85.32 88.99 84.40 2.81
FrozenEBERT-bs16-ep100-WRONG 80.73 82.57 77.98 86.24 84.40 83.49 80.73 86.24 82.57 79.82 82.48 2.58
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 88.99 84.40 82.57 81.65 85.32 80.73 84.40 81.65 86.24 85.32 84.13 2.39
RoBERTa-bs4+Dropout0.1 82.57 79.82 82.57 88.07 84.40 84.40 84.40 81.65 83.49 69.72 82.11 4.61
EmbraceRoBERTa-bs4-WRONG 84.40 85.32 81.65 80.73 82.57 83.49 83.49 84.40 77.06 88.99 83.21 2.96
EmbraceRoBERTa-bs4+Dropout0.1-WRONG 80.73 82.57 79.82 83.49 79.82 82.57 84.40 79.82 81.65 85.32 82.02 1.89
EmbraceRoBERTa-bs4+Dropout0.3-WRONG 77.98 84.40 77.06 83.49 83.49 84.40 86.24 81.65 81.65 85.32 82.57 2.87
CondensedEmbraceRoBERTa-bs4 75.23 78.90 77.98 85.32 81.65 82.57 85.32 83.49 81.65 81.65 81.38 3.04
CondensedEmbraceRoBERTa-bs4+Dropout0.1 77.06 83.49 80.73 83.49 80.73 81.65 82.57 83.49 76.15 85.32 81.47 2.78
CondensedEmbraceRoBERTa-bs4+Dropout0.3 73.39 82.57 83.49 82.57 81.65 83.49 81.65 86.24 82.57 87.16 82.48 3.49
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 87.16 85.32 82.57 79.82 86.24 77.98 84.40 83.49 77.98 80.73 82.57 3.15
FrozenERoBERTa-bs4-ep100-WRONG 82.57 79.82 83.49 84.40 81.65 77.06 83.49 77.06 79.82 85.32 81.47 2.78
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 84.40 79.82 80.73 86.24 82.57 78.90 81.65 85.32 86.24 83.49 82.94 2.50
RoBERTa-bs16+Dropout0.1 80.73 82.57 83.49 80.73 77.98 82.57 81.65 84.40 84.40 76.15 81.47 2.56
EmbraceRoBERTa-bs16-WRONG 76.15 83.49 81.65 84.40 83.49 86.24 79.82 83.49 87.16 82.57 82.84 2.99
EmbraceRoBERTa-bs16+Dropout0.1-WRONG 78.90 84.40 78.90 79.82 81.65 82.57 82.57 80.73 86.24 87.16 82.29 2.75
EmbraceRoBERTa-bs16+Dropout0.3-WRONG 72.48 83.49 83.49 81.65 80.73 83.49 77.98 83.49 86.24 87.16 82.02 4.02
CondensedEmbraceRoBERTa-bs16 71.56 81.65 83.49 82.57 86.24 84.40 78.90 85.32 85.32 85.32 82.48 4.19
CondensedEmbraceRoBERTa-bs16+Dropout0.1 75.23 81.65 82.57 83.49 82.57 82.57 84.40 84.40 81.65 85.32 82.39 2.65
CondensedEmbraceRoBERTa-bs16+Dropout0.3 78.90 85.32 79.82 78.90 80.73 84.40 78.90 85.32 84.40 87.16 82.39 3.06
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 85.32 81.65 81.65 85.32 83.49 82.57 82.57 84.40 85.32 81.65 83.39 1.50
FrozenERoBERTa-bs16-ep100 73.39 84.40 84.40 84.40 82.57 83.49 79.82 84.40 87.16 84.40 82.84 3.60
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 85.32 86.24 88.07 85.32 82.57 86.24 86.24 83.49 86.24 85.32 85.50 1.47
BERT-bs4+Dropout0.1 79.82 87.16 88.99 84.40 83.49 84.40 85.32 84.40 85.32 87.16 85.05 2.36
EmbraceBERT-bs4 80.73 87.16 78.90 80.73 86.24 80.73 83.49 85.32 81.65 86.24 83.12 2.79
EmbraceBERT-bs4+Dropout0.1 79.82 84.40 86.24 82.57 84.40 81.65 84.40 80.73 85.32 76.15 82.57 2.90
EmbraceBERT-bs4-p_selfatt 87.16 86.24 82.57 86.24 83.49 79.82 88.99 81.65 84.40 86.24 84.68 2.66
EmbraceBERT-bs4-p_att_clsquery_weights 86.24 82.57 83.49 85.32 79.82 79.82 88.07 77.06 88.99 85.32 83.67 3.67
CondensedEmbraceBERT-bs4 80.73 88.07 84.40 80.73 77.06 81.65 85.32 86.24 85.32 77.06 82.66 3.61
CondensedEmbraceBERT-bs4+Dropout0.1 81.65 88.07 83.49 82.57 86.24 85.32 88.07 85.32 85.32 79.82 84.59 2.56
CondensedEmbraceBERT-bs4+Dropout0.3 82.57 86.24 81.65 83.49 83.49 83.49 84.40 81.65 87.16 77.06 83.12 2.63
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 86.24 86.24 88.07 81.65 79.82 83.49 82.57 84.40 86.24 85.32 84.40 2.39
FrozenEBERT-bs4-ep100-WRONG 70.64 84.40 78.90 80.73 79.82 85.32 84.40 86.24 85.32 83.49 81.93 4.46
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3 78.90 86.24 87.16 86.24 84.40 87.16 78.90 84.40 88.99 82.57 84.50 3.27
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1 87.16 84.40 88.99 81.65 87.16 77.06 86.24 87.16 88.07 86.24 85.41 3.39
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10 86.24 85.32 87.16 89.91 87.16 84.40 89.91 83.49 87.16 88.07 86.88 2.01
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3 85.32 87.16 86.24 86.24 86.24 88.99 84.40 84.40 88.07 79.82 85.69 2.40
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10 87.16 84.40 86.24 87.16 87.16 85.32 90.83 88.07 87.16 86.24 86.97 1.63
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 82.57 82.57 86.24 81.65 84.40 84.40 87.16 78.90 84.40 82.57 83.49 2.25
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 85.32 85.32 88.07 81.65 82.57 83.49 86.24 81.65 84.40 87.16 84.59 2.12
BERT-bs16+Dropout0.1 81.65 83.49 88.07 80.73 79.82 84.40 85.32 83.49 87.16 88.99 84.31 2.94
EmbraceBERT-bs16 82.57 86.24 78.90 85.32 83.49 84.40 84.40 83.49 88.07 79.82 83.67 2.62
EmbraceBERT-bs16+Dropout0.1 81.65 88.07 83.49 82.57 85.32 82.57 82.57 85.32 84.40 82.57 83.85 1.84
CondensedEmbraceBERT-bs16 81.65 88.07 81.65 83.49 83.49 83.49 82.57 82.57 86.24 81.65 83.49 2.01
CondensedEmbraceBERT-bs16+Dropout0.1 84.40 85.32 88.99 82.57 85.32 82.57 81.65 86.24 85.32 78.90 84.13 2.66
CondensedEmbraceBERT-bs16+Dropout0.3 85.32 83.49 77.98 81.65 81.65 79.82 81.65 86.24 84.40 79.82 82.20 2.50
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 86.24 84.40 88.07 78.90 83.49 85.32 85.32 79.82 85.32 87.16 84.40 2.81
FrozenEBERT-bs16-ep100-WRONG 85.32 82.57 81.65 85.32 84.40 84.40 85.32 84.40 82.57 85.32 84.13 1.30
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3 84.40 85.32 85.32 87.16 85.32 86.24 83.49 82.57 86.24 81.65 84.77 1.65
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1 81.65 85.32 84.40 82.57 83.49 82.57 81.65 80.73 85.32 83.49 83.12 1.49
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10 85.32 83.49 87.16 88.99 85.32 88.07 86.24 85.32 88.99 85.32 86.42 1.73
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3 81.65 85.32 86.24 86.24 85.32 88.07 85.32 81.65 86.24 81.65 84.77 2.18
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10 85.32 84.40 88.07 88.99 85.32 87.16 86.24 83.49 88.07 85.32 86.24 1.69
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery 80.73 85.32 84.40 83.49 83.49 84.40 83.49 79.82 86.24 88.07 83.94 2.29
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 88.07 85.32 79.82 82.57 85.32 79.82 84.40 83.49 82.57 87.16 83.85 2.63
RoBERTa-bs4+Dropout0.1 83.49 79.82 84.40 85.32 87.16 82.57 80.73 85.32 80.73 73.39 82.29 3.72
EmbraceRoBERTa-bs4-WRONG 84.40 83.49 81.65 77.06 82.57 81.65 83.49 85.32 78.90 83.49 82.20 2.40
EmbraceRoBERTa-bs4+Dropout0.1-WRONG 84.40 85.32 87.16 85.32 82.57 84.40 84.40 83.49 86.24 83.49 84.68 1.30
EmbraceRoBERTa-bs4+Dropout0.3-WRONG 78.90 82.57 84.40 79.82 84.40 82.57 86.24 86.24 77.06 84.40 82.66 2.97
CondensedEmbraceRoBERTa-bs4 82.57 83.49 79.82 77.06 82.57 82.57 86.24 84.40 83.49 80.73 82.29 2.43
CondensedEmbraceRoBERTa-bs4+Dropout0.1 72.48 84.40 84.40 87.16 83.49 81.65 83.49 84.40 77.98 83.49 82.29 3.96
CondensedEmbraceRoBERTa-bs4+Dropout0.3 74.31 86.24 87.16 82.57 81.65 86.24 83.49 85.32 79.82 83.49 83.03 3.63
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 87.16 86.24 81.65 78.90 85.32 80.73 83.49 84.40 78.90 84.40 83.12 2.79
FrozenERoBERTa-bs4-ep100-WRONG 86.24 79.82 84.40 80.73 83.49 77.98 82.57 81.65 81.65 85.32 82.39 2.42
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 83.49 77.98 80.73 83.49 81.65 80.73 83.49 84.40 85.32 82.57 82.39 2.04
RoBERTa-bs16+Dropout0.1 84.40 79.82 82.57 80.73 80.73 83.49 81.65 84.40 84.40 79.82 82.20 1.80
EmbraceRoBERTa-bs16-WRONG 76.15 83.49 82.57 83.49 81.65 83.49 79.82 83.49 85.32 80.73 82.02 2.47
EmbraceRoBERTa-bs16+Dropout0.1-WRONG 77.06 83.49 80.73 80.73 81.65 83.49 80.73 83.49 82.57 82.57 81.65 1.88
EmbraceRoBERTa-bs16+Dropout0.3-WRONG 72.48 81.65 81.65 79.82 83.49 84.40 80.73 82.57 82.57 84.40 81.38 3.28
CondensedEmbraceRoBERTa-bs16 72.48 83.49 83.49 77.06 84.40 83.49 77.98 86.24 81.65 83.49 81.38 4.00
CondensedEmbraceRoBERTa-bs16+Dropout0.1 74.31 84.40 80.73 85.32 83.49 82.57 81.65 82.57 85.32 87.16 82.75 3.35
CondensedEmbraceRoBERTa-bs16+Dropout0.3 82.57 80.73 78.90 85.32 82.57 83.49 78.90 82.57 82.57 83.49 82.11 1.94
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 86.24 77.98 78.90 81.65 81.65 83.49 83.49 86.24 84.40 82.57 82.66 2.61
FrozenERoBERTa-bs16-ep100 77.06 82.57 79.82 78.90 78.90 84.40 80.73 81.65 84.40 80.73 80.92 2.28
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
  • Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |83.49|83.49|86.24|80.73|82.57|85.32|83.49|84.40|84.40|88.99|84.31|2.11| | BERTwithAtt-bs8 |79.82|83.49|84.40|79.82|84.40|83.49|85.32|84.40|84.40|84.40|83.39|1.86| | BERTwithAttClsProjection-bs8 |83.49|87.16|84.40|81.65|84.40|88.07|85.32|83.49|87.16|79.82|84.50|2.44| | BERTwithProjection-bs8 |81.65|83.49|85.32|80.73|86.24|85.32|79.82|85.32|85.32|85.32|83.85|2.18| | BERTwithProjectionAtt-bs8 |83.49|87.16|88.07|84.40|81.65|87.16|83.49|77.98|84.40|83.49|84.13|2.81| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |82.57|85.32|86.24|83.49|85.32|84.40|87.16|86.24|85.32|77.98|84.40|2.50| | EmbraceBERT-bs8-p_att_clsquery_weights |86.24|83.49|84.40|85.32|79.82|84.40|86.24|83.49|81.65|82.57|83.76|1.93| | EmbraceBERTwithProj-bs8 |85.32|84.40|84.40|88.07|84.40|83.49|83.49|85.32|87.16|88.99|85.50|1.83| | EmbraceBERTwithProj-bs8-p_attclsqw |85.32|84.40|79.82|82.57|85.32|88.07|85.32|83.49|85.32|86.24|84.59|2.12| | EBERTconcatAtt-bs8 |84.40|84.40|83.49|84.40|81.65|78.90|84.40|86.24|82.57|85.32|83.58|1.99| | EBERTconcatAtt-bs8_attclsqw |84.40|82.57|86.24|84.40|85.32|86.24|87.16|87.16|86.24|88.07|85.78|1.55| | EBERTconcatAttwithProj-bs8 |83.49|85.32|83.49|82.57|83.49|83.49|82.57|83.49|82.57|84.40|83.49|0.82| | EBERTconcatAttwithProj-bs8_attclsqw |77.98|82.57|86.24|81.65|86.24|87.16|85.32|81.65|83.49|84.40|83.67|2.65| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |80.73|87.16|87.16|86.24|85.32|85.32|88.99|83.49|84.40|87.16|85.60|2.21| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |84.40|77.98|84.40|82.57|88.07|81.65|84.40|84.40|88.99|83.49|84.04|2.94| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |82.57|87.16|84.40|78.90|86.24|84.40|86.24|84.40|78.90|87.16|84.04|2.91| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |82.57|83.49|87.16|86.24|86.24|83.49|87.16|81.65|86.24|83.49|84.77|1.93| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |85.32|86.24|83.49|83.49|85.32|86.24|87.16|86.24|82.57|80.73|84.68|1.93| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |88.07|85.32|82.57|86.24|81.65|85.32|83.49|79.82|82.57|79.82|83.49|2.59| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |85.32|86.24|80.73|86.24|87.16|85.32|84.40|86.24|85.32|83.49|85.05|1.74| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |85.32|83.49|84.40|80.73|84.40|82.57|79.82|84.40|81.65|88.07|83.49|2.28|

macsay-sphinx

  • Epoch 30
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 70.64 75.23 66.97 72.48 72.48 67.89 69.72 66.06 72.48 70.64 70.46 2.72
BERT-bs4+Dropout0.1 72.48 75.23 66.97 71.56 73.39 64.22 70.64 68.81 72.48 70.64 70.64 3.07
EmbraceBERT-bs4-WRONG 66.97 67.89 70.64 67.89 66.06 59.63 68.81 68.81 68.81 68.81 67.43 2.85
EmbraceBERT-bs4+Dropout0.1-WRONG 64.22 68.81 67.89 67.89 68.81 65.14 69.72 66.06 71.56 73.39 68.35 2.67
EmbraceBERT-bs4+Dropout0.3-WRONG
CondensedEmbraceBERT-bs4 67.89 69.72 69.72 66.06 70.64 62.39 70.64 64.22 72.48 70.64 68.44 3.08
CondensedEmbraceBERT-bs4+Dropout0.1 63.30 70.64 65.14 59.63 66.06 65.14 66.06 64.22 70.64 64.22 65.50 3.10
CondensedEmbraceBERT-bs4+Dropout0.3 65.14 69.72 62.39 60.55 68.81 62.39 67.89 66.06 75.23 66.97 66.51 4.07
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 72.48 74.31 68.81 73.39 72.48 68.81 70.64 67.89 72.48 70.64 71.19 2.06
FrozenEBERT-bs4-ep100-WRONG 66.97 66.97 69.72 54.13 67.89 55.05 68.81 68.81 72.48 66.97 65.78 5.82
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 68.81 67.89 64.22 70.64 70.64 63.30 68.81 66.06 65.14 70.64 67.61 2.63
BERT-bs16+Dropout0.1 70.64 67.89 57.80 69.72 69.72 64.22 70.64 66.97 62.39 68.81 66.88 3.99
EmbraceBERT-bs16-WRONG 66.97 64.22 66.06 64.22 66.97 66.06 72.48 61.47 70.64 70.64 66.97 3.23
EmbraceBERT-bs16+Dropout0.1-WRONG 67.89 66.06 63.30 67.89 66.97 66.06 66.97 66.06 66.06 70.64 66.79 1.78
EmbraceBERT-bs16+Dropout0.3-WRONG
CondensedEmbraceBERT-bs16 66.06 65.14 66.06 61.47 66.97 63.30 68.81 61.47 71.56 66.97 65.78 2.99
CondensedEmbraceBERT-bs16+Dropout0.1 67.89 66.06 66.06 67.89 70.64 64.22 68.81 62.39 71.56 68.81 67.43 2.67
CondensedEmbraceBERT-bs16+Dropout0.3 65.14 64.22 62.39 63.30 69.72 64.22 66.97 63.30 67.89 70.64 65.78 2.72
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 74.31 66.06 69.72 70.64 72.48 66.97 73.39 65.14 71.56 70.64 70.09 2.96
FrozenEBERT-bs16-ep100-WRONG 63.30 67.89 69.72 64.22 71.56 66.97 71.56 60.55 69.72 65.14 67.06 3.52
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 66.06 62.39 61.47 63.30 63.30 64.22 61.47 63.30 62.39 65.14 63.30 1.42
RoBERTa-bs4+Dropout0.1 60.55 65.14 61.47 60.55 62.39 66.06 63.30 59.63 66.06 65.14 63.03 2.32
EmbraceRoBERTa-bs4-WRONG 60.55 62.39 57.80 66.06 62.39 60.55 58.72 58.72 64.22 57.80 60.92 2.67
EmbraceRoBERTa-bs4+Dropout0.1-WRONG 59.63 66.97 60.55 63.30 63.30 63.30 63.30 62.39 60.55 62.39 62.57 1.96
EmbraceRoBERTa-bs4+Dropout0.3-WRONG 65.14 61.47 61.47 62.39 67.89 61.47 63.30 61.47 60.55 65.14 63.03 2.21
CondensedEmbraceRoBERTa-bs4 63.30 65.14 57.80 66.97 69.72 61.47 64.22 64.22 63.30 58.72 63.49 3.38
CondensedEmbraceRoBERTa-bs4+Dropout0.1 66.97 65.14 54.13 59.63 63.30 66.06 67.89 65.14 63.30 59.63 63.12 3.99
CondensedEmbraceRoBERTa-bs4+Dropout0.3 61.47 64.22 59.63 64.22 61.47 57.80 66.06 64.22 60.55 62.39 62.20 2.39
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 52.29 62.39 67.89 61.47 62.39 66.97 64.22 65.14 63.30 63.30 62.94 4.05
FrozenERoBERTa-bs4-ep100-WRONG 64.22 64.22 68.81 66.97 62.39 62.39 61.47 57.80 65.14 58.72 63.21 3.24
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 66.06 63.30 61.47 66.06 61.47 62.39 65.14 60.55 63.30 59.63 62.94 2.14
RoBERTa-bs16+Dropout0.1 51.38 67.89 58.72 60.55 62.39 64.22 66.06 59.63 68.81 61.47 62.11 4.84
EmbraceRoBERTa-bs16-WRONG 61.47 68.81 62.39 65.14 66.97 66.06 65.14 62.39 61.47 61.47 64.13 2.51
EmbraceRoBERTa-bs16+Dropout0.1-WRONG 64.22 68.81 58.72 64.22 67.89 66.06 61.47 63.30 63.30 63.30 64.13 2.80
EmbraceRoBERTa-bs16+Dropout0.3-WRONG 63.30 66.06 56.88 66.97 61.47 62.39 62.39 62.39 63.30 61.47 62.66 2.60
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1 64.22 66.97 59.63 61.47 62.39 61.47 64.22 63.30 63.30 63.30 63.03 1.88
CondensedEmbraceRoBERTa-bs16+Dropout0.3 66.97 65.14 61.47 66.97 66.97 59.63 63.30 61.47 62.39 60.55 63.49 2.68
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 64.22 66.06 59.63 63.30 59.63 60.55 64.22 61.47 62.39 61.47 62.29 2.03
FrozenERoBERTa-bs16-ep100 63.30 61.47 63.30 63.30 64.22 63.30 66.97 61.47 64.22 62.39 63.39 1.50
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 71.56 74.31 68.81 74.31 70.64 68.81 68.81 67.89 71.56 69.72 70.64 2.17
BERT-bs4+Dropout0.1 71.56 73.39 69.72 71.56 75.23 65.14 70.64 66.97 71.56 70.64 70.64 2.75
EmbraceBERT-bs4 68.81 67.89 70.64 69.72 72.48 64.22 71.56 71.56 74.31 66.97 69.82 2.80
EmbraceBERT-bs4+Dropout0.1 66.06 74.31 66.97 65.14 68.81 65.14 61.47 66.97 71.56 67.89 67.43 3.39
EmbraceBERT-bs4-p_selfatt 69.72 70.64 69.72 74.31 69.72 66.06 72.48 72.48 71.56 73.39 71.01 2.25
EmbraceBERT-bs4-p_att_clsquery_weights 71.56 71.56 69.72 68.81 64.22 70.64 65.14 66.97 66.97 74.31 68.99 3.01
CondensedEmbraceBERT-bs4 69.72 67.89 71.56 64.22 71.56 62.39 71.56 66.06 73.39 69.72 68.81 3.41
CondensedEmbraceBERT-bs4+Dropout0.1 66.97 73.39 68.81 62.39 68.81 65.14 64.22 66.06 71.56 66.97 67.43 3.16
CondensedEmbraceBERT-bs4+Dropout0.3 66.97 69.72 64.22 62.39 69.72 65.14 68.81 72.48 71.56 67.89 67.89 3.07
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 72.48 74.31 69.72 74.31 77.06 68.81 70.64 70.64 73.39 72.48 72.39 2.37
FrozenEBERT-bs4-ep100-WRONG 66.06 71.56 70.64 66.97 63.30 65.14 74.31 71.56 71.56 74.31 69.54 3.69
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3 69.72 75.23 75.23 72.48 63.30 68.81 74.31 67.89 69.72 73.39 71.01 3.63
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1 66.06 71.56 66.97 70.64 63.30 68.81 71.56 70.64 69.72 67.89 68.72 2.54
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10 75.23 71.56 73.39 74.31 71.56 69.72 71.56 74.31 74.31 70.64 72.66 1.78
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3 63.30 67.89 70.64 77.98 62.39 67.89 67.89 72.48 65.14 68.81 68.44 4.33
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10 73.39 70.64 70.64 74.31 73.39 68.81 74.31 66.97 71.56 71.56 71.56 2.28
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 58.72 61.47 70.64 71.56 70.64 63.30 64.22 68.81 76.15 69.72 67.52 5.09
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 74.31 66.06 67.89 68.81 74.31 66.06 71.56 66.97 67.89 72.48 69.63 3.08
BERT-bs16+Dropout0.1 72.48 70.64 60.55 72.48 71.56 65.14 70.64 66.97 69.72 75.23 69.54 4.04
EmbraceBERT-bs16 64.22 65.14 66.97 63.30 68.81 66.97 70.64 65.14 75.23 64.22 67.06 3.47
EmbraceBERT-bs16+Dropout0.1 66.97 68.81 63.30 66.97 71.56 65.14 66.97 64.22 76.15 68.81 67.89 3.58
CondensedEmbraceBERT-bs16 66.06 66.97 66.06 63.30 68.81 66.97 70.64 63.30 73.39 68.81 67.43 2.97
CondensedEmbraceBERT-bs16+Dropout0.1 66.97 68.81 63.30 68.81 71.56 67.89 66.97 62.39 73.39 68.81 67.89 3.15
CondensedEmbraceBERT-bs16+Dropout0.3 62.39 71.56 64.22 67.89 76.15 65.14 67.89 64.22 74.31 70.64 68.44 4.38
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 74.31 69.72 68.81 69.72 74.31 66.97 69.72 66.06 70.64 69.72 70.00 2.53
FrozenEBERT-bs16-ep100-WRONG 65.14 69.72 66.06 66.06 71.56 66.06 68.81 67.89 74.31 65.14 68.07 2.90
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3 69.72 66.06 68.81 73.39 60.55 69.72 69.72 64.22 69.72 67.89 67.98 3.39
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1 68.81 69.72 66.97 71.56 66.97 69.72 72.48 66.06 73.39 66.97 69.27 2.44
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10 71.56 67.89 76.15 68.81 74.31 68.81 71.56 69.72 75.23 76.15 72.02 3.05
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3 72.48 64.22 65.14 69.72 62.39 66.97 67.89 67.89 69.72 68.81 67.52 2.82
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10 70.64 71.56 75.23 68.81 71.56 64.22 72.48 68.81 77.06 74.31 71.47 3.49
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery 62.39 66.06 67.89 63.30 73.39 65.14 72.48 66.06 62.39 68.81 66.79 3.69
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 64.22 63.30 62.39 66.06 61.47 67.89 64.22 66.06 65.14 65.14 64.59 1.80
RoBERTa-bs4+Dropout0.1 64.22 66.97 61.47 62.39 61.47 64.22 63.30 55.96 68.81 62.39 63.12 3.28
EmbraceRoBERTa-bs4-WRONG 63.30 62.39 57.80 67.89 65.14 64.22 57.80 60.55 63.30 57.80 62.02 3.29
EmbraceRoBERTa-bs4+Dropout0.1-WRONG 65.14 66.06 63.30 66.97 59.63 60.55 62.39 63.30 63.30 62.39 63.30 2.17
EmbraceRoBERTa-bs4+Dropout0.3-WRONG 66.06 62.39 61.47 63.30 70.64 61.47 65.14 66.97 58.72 65.14 64.13 3.22
CondensedEmbraceRoBERTa-bs4
CondensedEmbraceRoBERTa-bs4+Dropout0.1 61.47 63.30 57.80 60.55 65.14 66.06 68.81 64.22 69.72 55.05 63.21 4.39
CondensedEmbraceRoBERTa-bs16+Dropout0.3 66.97 66.06 60.55 59.63 68.81 62.39 64.22 64.22 63.30 66.06 64.22 2.72
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 65.14 60.55 64.22 65.14 63.30 67.89 66.06 65.14 63.30 68.81 64.95 2.24
FrozenERoBERTa-bs4-ep100-WRONG 61.47 64.22 54.13 66.97 63.30 66.06 65.14 59.63 66.97 58.72 62.66 3.96
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 65.14 65.14 62.39 65.14 61.47 64.22 63.30 60.55 63.30 58.72 62.94 2.06
RoBERTa-bs16+Dropout0.1 56.88 66.97 58.72 64.22 62.39 63.30 65.14 62.39 63.30 62.39 62.57 2.78
EmbraceRoBERTa-bs16-WRONG 66.06 68.81 56.88 64.22 66.06 61.47 66.06 64.22 65.14 63.30 64.22 3.07
EmbraceRoBERTa-bs16+Dropout0.1-WRONG 65.14 71.56 56.88 60.55 66.06 61.47 66.06 68.81 66.97 59.63 64.31 4.31
EmbraceRoBERTa-bs16+Dropout0.3-WRONG 59.63 66.06 60.55 64.22 63.30 66.06 64.22 62.39 65.14 59.63 63.12 2.35
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1 66.97 66.06 56.88 62.39 62.39 62.39 64.22 67.89 67.89 66.06 64.31 3.24
CondensedEmbraceRoBERTa-bs16+Dropout0.3 66.97 65.14 59.63 65.14 61.47 61.47 65.14 67.89 62.39 67.89 64.31 2.77
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 67.89 66.06 60.55 66.97 66.06 65.14 65.14 59.63 64.22 62.39 64.40 2.59
FrozenERoBERTa-bs16-ep100 64.22 65.14 63.30 65.14 63.30 64.22 71.56 63.30 66.06 63.30 64.95 2.39
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
  • Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |70.64|71.56|63.30|72.48|71.56|68.81|70.64|67.89|70.64|68.81|69.63|2.51| | BERTwithAtt-bs8 |66.06|61.47|69.72|66.06|66.97|64.22|70.64|72.48|71.56|68.81|67.80|3.29| | BERTwithAttClsProjection-bs8 |70.64|69.72|68.81|71.56|69.72|73.39|68.81|77.98|75.23|65.14|71.10|3.46| | BERTwithProjection-bs8 |72.48|76.15|73.39|76.15|72.48|74.31|76.15|73.39|77.98|73.39|74.59|1.79| | BERTwithProjectionAtt-bs8 |69.72|66.97|69.72|65.14|67.89|66.97|68.81|67.89|71.56|67.89|68.26|1.70| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |66.97|67.89|66.97|65.14|67.89|68.81|68.81|63.30|71.56|70.64|67.80|2.30| | EmbraceBERT-bs8-p_att_clsquery_weights |71.56|70.64|68.81|66.06|66.06|68.81|70.64|67.89|61.47|70.64|68.26|2.91| | EmbraceBERTwithProj-bs8 |74.31|73.39|69.72|69.72|73.39|73.39|72.48|70.64|68.81|66.97|71.28|2.32| | EmbraceBERTwithProj-bs8-p_attclsqw |68.81|63.30|73.39|71.56|74.31|74.31|77.06|71.56|73.39|61.47|70.92|4.75| | EBERTconcatAtt-bs8 |63.30|67.89|70.64|68.81|67.89|67.89|68.81|69.72|71.56|70.64|68.72|2.19| | EBERTconcatAtt-bs8_attclsqw |72.48|69.72|72.48|64.22|74.31|66.06|71.56|66.97|66.06|68.81|69.27|3.21| | EBERTconcatAttwithProj-bs8 |65.14|69.72|75.23|68.81|68.81|71.56|69.72|67.89|73.39|74.31|70.46|2.98| | EBERTconcatAttwithProj-bs8_attclsqw |72.48|72.48|74.31|71.56|65.14|74.31|71.56|64.22|66.97|70.64|70.37|3.46| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |63.30|62.39|71.56|62.39|66.06|71.56|71.56|68.81|64.22|57.80|65.96|4.52| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |65.14|70.64|70.64|74.31|66.06|61.47|71.56|71.56|69.72|74.31|69.54|3.91| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |66.06|70.64|71.56|66.97|74.31|71.56|70.64|71.56|65.14|76.15|70.46|3.33| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |73.39|63.30|74.31|66.06|70.64|75.23|73.39|68.81|71.56|65.14|70.18|3.96| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |65.14|65.14|70.64|65.14|65.14|72.48|71.56|66.06|74.31|68.81|68.44|3.39| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |67.89|73.39|76.15|68.81|70.64|71.56|66.06|66.06|69.72|61.47|69.17|3.94| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |71.56|70.64|77.06|68.81|69.72|73.39|76.15|72.48|76.15|68.81|72.48|2.96| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |74.31|69.72|72.48|66.97|64.22|66.97|71.56|74.31|75.23|70.64|70.64|3.48|

macsay-witai

  • Epoch 30
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 88.99 89.91 88.99 88.07 88.07 84.40 91.74 88.99 89.91 90.83 88.99 1.88
BERT-bs4+Dropout0.1 88.07 90.83 92.66 88.99 88.07 88.07 88.07 89.91 92.66 92.66 90.00 1.94
EmbraceBERT-bs4-WRONG 85.32 86.24 91.74 86.24 88.99 84.40 88.07 90.83 83.49 79.82 86.51 3.38
EmbraceBERT-bs4+Dropout0.1-WRONG 88.07 89.91 88.99 88.99 91.74 86.24 89.91 88.07 87.16 83.49 88.26 2.16
EmbraceBERT-bs4+Dropout0.3-WRONG
CondensedEmbraceBERT-bs4 87.16 86.24 88.99 83.49 88.07 81.65 88.99 90.83 82.57 84.40 86.24 2.93
CondensedEmbraceBERT-bs4+Dropout0.1 85.32 89.91 89.91 87.16 88.07 82.57 87.16 84.40 88.99 85.32 86.88 2.32
CondensedEmbraceBERT-bs4+Dropout0.3 88.07 86.24 88.99 80.73 92.66 87.16 88.99 88.99 89.91 87.16 87.89 2.92
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 86.24 89.91 88.07 88.99 85.32 87.16 89.91 89.91 90.83 91.74 88.81 1.96
FrozenEBERT-bs4-ep100-WRONG 85.32 87.16 89.91 87.16 86.24 85.32 86.24 87.16 84.40 85.32 86.42 1.47
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 89.91 88.99 88.07 88.07 84.40 84.40 85.32 88.07 88.99 85.32 87.16 1.97
BERT-bs16+Dropout0.1 88.99 87.16 88.99 88.07 86.24 83.49 86.24 88.07 87.16 87.16 87.16 1.54
EmbraceBERT-bs16-WRONG 87.16 88.07 88.99 86.24 89.91 86.24 83.49 88.99 82.57 87.16 86.88 2.25
EmbraceBERT-bs16+Dropout0.1-WRONG 87.16 86.24 89.91 87.16 89.91 81.65 83.49 89.91 82.57 85.32 86.33 2.91
EmbraceBERT-bs16+Dropout0.3-WRONG
CondensedEmbraceBERT-bs16 85.32 87.16 86.24 85.32 89.91 82.57 84.40 88.99 85.32 85.32 86.06 2.04
CondensedEmbraceBERT-bs16+Dropout0.1 85.32 87.16 88.99 88.07 89.91 83.49 84.40 90.83 85.32 87.16 87.06 2.30
CondensedEmbraceBERT-bs16+Dropout0.3 85.32 85.32 85.32 86.24 87.16 82.57 87.16 87.16 84.40 85.32 85.60 1.36
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 91.74 88.07 90.83 88.99 87.16 88.07 90.83 88.99 90.83 89.91 89.54 1.43
FrozenEBERT-bs16-ep100-WRONG 86.24 85.32 88.99 87.16 92.66 88.07 86.24 87.16 84.40 87.16 87.34 2.16
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 91.74 89.91 88.99 91.74 88.07 88.99 87.16 88.99 89.91 89.91 89.54 1.37
RoBERTa-bs4+Dropout0.1 90.83 88.07 87.16 89.91 92.66 90.83 89.91 90.83 90.83 90.83 90.18 1.48
EmbraceRoBERTa-bs4-WRONG 89.91 89.91 88.07 89.91 88.99 90.83 89.91 88.07 87.16 89.91 89.27 1.09
EmbraceRoBERTa-bs4+Dropout0.1-WRONG 88.07 90.83 88.99 90.83 90.83 89.91 89.91 89.91 88.07 91.74 89.91 1.16
EmbraceRoBERTa-bs4+Dropout0.3-WRONG 88.07 90.83 88.99 88.99 88.99 89.91 88.07 90.83 91.74 89.91 89.63 1.16
CondensedEmbraceRoBERTa-bs4 90.83 90.83 88.99 88.07 92.66 88.99 90.83 88.07 90.83 88.07 89.82 1.50
CondensedEmbraceRoBERTa-bs4+Dropout0.1 89.91 88.07 88.07 88.07 91.74 90.83 90.83 91.74 90.83 91.74 90.18 1.48
CondensedEmbraceRoBERTa-bs4+Dropout0.3 89.91 89.91 88.07 88.99 88.99 88.07 91.74 92.66 90.83 90.83 90.00 1.45
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 90.83 88.99 88.99 89.91 88.07 90.83 88.99 88.07 91.74 90.83 89.72 1.22
FrozenERoBERTa-bs4-ep100-WRONG 90.83 90.83 89.91 91.74 89.91 89.91 90.83 86.24 88.99 88.07 89.72 1.52
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 88.99 87.16 89.91 90.83 87.16 88.07 89.91 88.99 88.07 90.83 88.99 1.30
RoBERTa-bs16+Dropout0.1 85.32 89.91 88.99 88.07 88.07 88.07 88.07 91.74 86.24 91.74 88.62 1.98
EmbraceRoBERTa-bs16-WRONG 89.91 89.91 88.07 88.07 89.91 89.91 88.07 85.32 88.07 85.32 88.26 1.68
EmbraceRoBERTa-bs16+Dropout0.1-WRONG 86.24 92.66 86.24 88.07 88.07 88.07 88.99 87.16 87.16 86.24 87.89 1.83
EmbraceRoBERTa-bs16+Dropout0.3-WRONG 86.24 87.16 85.32 88.07 88.07 87.16 88.07 88.07 86.24 85.32 86.97 1.07
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1 86.24 86.24 84.40 87.16 90.83 87.16 88.99 87.16 86.24 86.24 87.06 1.66
CondensedEmbraceRoBERTa-bs16+Dropout0.3 84.40 85.32 85.32 88.07 88.99 88.07 86.24 88.07 86.24 87.16 86.79 1.43
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 89.91 88.07 88.99 89.91 88.99 88.99 88.99 90.83 90.83 91.74 89.72 1.07
FrozenERoBERTa-bs16-ep100 88.99 89.91 88.07 88.99 91.74 89.91 88.99 88.07 88.07 88.99 89.17 1.07
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 88.99 88.99 88.99 88.99 87.16 88.07 91.74 89.91 88.99 91.74 89.36 1.37
BERT-bs4+Dropout0.1 88.99 90.83 92.66 88.99 88.07 87.16 89.91 89.91 92.66 93.58 90.28 2.02
EmbraceBERT-bs4 87.16 90.83 89.91 89.91 87.16 82.57 89.91 90.83 84.40 83.49 87.61 2.99
EmbraceBERT-bs4+Dropout0.1 88.07 88.99 91.74 90.83 87.16 84.40 87.16 86.24 88.07 83.49 87.61 2.44
EmbraceBERT-bs4-p_selfatt 89.91 89.91 90.83 87.16 88.07 87.16 89.91 87.16 93.58 85.32 88.90 2.26
EmbraceBERT-bs4-p_att_clsquery_weights 89.91 87.16 90.83 89.91 91.74 88.99 87.16 88.07 86.24 87.16 88.72 1.74
CondensedEmbraceBERT-bs4 86.24 88.99 90.83 84.40 92.66 85.32 90.83 90.83 85.32 87.16 88.26 2.78
CondensedEmbraceBERT-bs4+Dropout0.1 87.16 88.99 89.91 91.74 88.99 86.24 88.99 87.16 87.16 88.07 88.44 1.55
CondensedEmbraceBERT-bs4+Dropout0.3 87.16 88.99 91.74 85.32 94.50 87.16 89.91 89.91 90.83 86.24 89.17 2.65
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs4-ep100 88.99 88.07 88.99 88.99 88.99 88.99 86.24 89.91 88.99 91.74 88.99 1.30
FrozenEBERT-bs4-ep100-WRONG 86.24 89.91 89.91 85.32 89.91 86.24 88.07 88.07 85.32 86.24 87.52 1.80
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3 83.49 88.07 88.99 89.91 87.16 88.99 88.99 89.91 88.07 85.32 87.89 1.96
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1 86.24 87.16 87.16 88.99 88.99 85.32 88.07 88.99 88.07 84.40 87.34 1.52
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10 86.24 89.91 90.83 88.07 86.24 87.16 88.07 86.24 91.74 88.99 88.35 1.88
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3 87.16 86.24 88.99 86.24 86.24 86.24 88.99 89.91 92.66 82.57 87.52 2.60
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10 91.74 88.99 90.83 87.16 86.24 88.07 89.91 87.16 89.91 91.74 89.17 1.87
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 86.24 88.07 87.16 88.99 93.58 87.16 83.49 85.32 90.83 87.16 87.80 2.69
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 91.74 88.07 90.83 88.99 87.16 87.16 91.74 88.99 91.74 88.99 89.54 1.75
BERT-bs16+Dropout0.1 90.83 86.24 91.74 88.99 87.16 87.16 90.83 90.83 92.66 88.99 89.54 2.06
EmbraceBERT-bs16 86.24 88.07 88.99 89.91 93.58 86.24 88.07 90.83 84.40 90.83 88.72 2.56
EmbraceBERT-bs16+Dropout0.1 86.24 88.07 90.83 87.16 89.91 85.32 85.32 89.91 83.49 90.83 87.71 2.47
CondensedEmbraceBERT-bs16 87.16 88.07 89.91 87.16 92.66 87.16 88.07 87.16 85.32 86.24 87.89 1.96
CondensedEmbraceBERT-bs16+Dropout0.1 86.24 88.99 91.74 90.83 88.99 84.40 87.16 90.83 83.49 88.99 88.17 2.64
CondensedEmbraceBERT-bs16+Dropout0.3 88.07 86.24 88.07 88.99 90.83 85.32 86.24 89.91 86.24 87.16 87.71 1.70
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenBERT-bs16-ep100 90.83 88.07 90.83 88.07 88.07 91.74 90.83 89.91 92.66 88.99 90.00 1.56
FrozenEBERT-bs16-ep100-WRONG 86.24 87.16 88.07 90.83 94.50 88.07 87.16 91.74 85.32 85.32 88.44 2.85
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3 82.57 82.57 86.24 88.07 87.16 88.07 87.16 87.16 87.16 83.49 85.96 2.09
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1 83.49 84.40 88.99 82.57 86.24 88.99 86.24 86.24 88.07 86.24 86.15 2.07
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10 87.16 88.99 87.16 88.99 83.49 89.91 89.91 87.16 88.07 88.99 87.98 1.81
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3 83.49 84.40 87.16 90.83 86.24 86.24 85.32 86.24 87.16 85.32 86.24 1.88
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10 86.24 88.99 87.16 88.99 82.57 89.91 90.83 88.99 88.07 88.07 87.98 2.19
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery 87.16 90.83 85.32 87.16 90.83 86.24 87.16 85.32 88.99 88.99 87.80 1.93
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs4 90.83 88.99 89.91 90.83 88.07 88.99 88.07 88.07 88.99 89.91 89.27 1.01
RoBERTa-bs4+Dropout0.1 90.83 88.99 89.91 88.99 91.74 91.74 90.83 89.91 88.99 89.91 90.18 1.01
EmbraceRoBERTa-bs4-WRONG 89.91 89.91 88.99 88.07 89.91 90.83 90.83 88.99 88.99 89.91 89.63 0.83
EmbraceRoBERTa-bs4+Dropout0.1-WRONG 89.91 90.83 89.91 87.16 90.83 89.91 89.91 91.74 89.91 88.07 89.82 1.26
EmbraceRoBERTa-bs4+Dropout0.3-WRONG 87.16 92.66 91.74 89.91 88.99 89.91 89.91 89.91 89.91 89.91 90.00 1.39
CondensedEmbraceRoBERTa-bs4
CondensedEmbraceRoBERTa-bs4+Dropout0.1 86.24 87.16 88.07 87.16 90.83 91.74 89.91 91.74 93.58 92.66 89.91 2.46
CondensedEmbraceRoBERTa-bs4+Dropout0.3 87.16 91.74 88.99 87.16 90.83 88.99 88.99 91.74 88.99 91.74 89.63 1.69
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs4-ep100 90.83 88.99 90.83 91.74 88.07 88.99 88.07 88.07 87.16 90.83 89.36 1.49
FrozenERoBERTa-bs4-ep100-WRONG 88.99 90.83 88.99 88.99 87.16 91.74 90.83 88.07 88.99 89.91 89.45 1.31
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
RoBERTa-bs16 89.91 88.99 90.83 90.83 87.16 90.83 88.99 90.83 90.83 90.83 90.00 1.19
RoBERTa-bs16+Dropout0.1 89.91 90.83 90.83 89.91 90.83 88.99 89.91 89.91 90.83 89.91 90.18 0.59
EmbraceRoBERTa-bs16-WRONG 92.66 91.74 89.91 90.83 90.83 89.91 90.83 88.07 91.74 86.24 90.28 1.80
EmbraceRoBERTa-bs16+Dropout0.1-WRONG 88.07 93.58 88.07 88.07 89.91 90.83 88.07 89.91 88.99 88.99 89.45 1.65
EmbraceRoBERTa-bs16+Dropout0.3-WRONG 86.24 86.24 82.57 88.99 89.91 90.83 88.99 90.83 88.99 88.07 88.17 2.41
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1 88.07 91.74 85.32 89.91 90.83 89.91 89.91 90.83 90.83 89.91 89.72 1.73
CondensedEmbraceRoBERTa-bs16+Dropout0.3 85.32 88.07 83.49 86.24 89.91 89.91 89.91 91.74 88.99 88.99 88.26 2.39
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
FrozenRoBERTa-bs16-ep100 90.83 88.99 84.40 90.83 88.07 88.99 88.99 88.99 90.83 90.83 89.17 1.87
FrozenERoBERTa-bs16-ep100 90.83 88.99 88.99 90.83 88.99 89.91 91.74 87.16 91.74 88.07 89.72 1.47
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
  • Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |88.07|88.07|88.99|89.91|88.99|88.99|91.74|89.91|93.58|92.66|90.09|1.83| | BERTwithAtt-bs8 |87.16|90.83|91.74|88.07|88.07|85.32|87.16|89.91|89.91|88.99|88.72|1.84| | BERTwithAttClsProjection-bs8 |90.83|90.83|91.74|88.99|86.24|90.83|88.99|90.83|88.99|86.24|89.45|1.85| | BERTwithProjection-bs8 |84.40|84.40|87.16|90.83|88.07|88.07|82.57|89.91|84.40|90.83|87.06|2.83| | BERTwithProjectionAtt-bs8 |88.99|88.07|89.91|86.24|88.99|86.24|92.66|86.24|89.91|89.91|88.72|1.97| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |87.16|86.24|90.83|90.83|88.99|85.32|85.32|89.91|85.32|88.07|87.80|2.13| | EmbraceBERT-bs8-p_att_clsquery_weights |87.16|87.16|89.91|86.24|88.07|88.99|88.07|89.91|89.91|82.57|87.80|2.13| | EmbraceBERTwithProj-bs8 |89.91|89.91|87.16|90.83|88.99|88.99|92.66|89.91|90.83|89.91|89.91|1.36| | EmbraceBERTwithProj-bs8-p_attclsqw |91.74|88.07|88.99|88.99|90.83|90.83|91.74|89.91|90.83|89.91|90.18|1.16| | EBERTconcatAtt-bs8 |86.24|82.57|87.16|84.40|90.83|88.07|88.07|85.32|90.83|88.99|87.25|2.54| | EBERTconcatAtt-bs8_attclsqw |89.91|84.40|88.07|86.24|88.07|85.32|88.07|89.91|87.16|87.16|87.43|1.69| | EBERTconcatAttwithProj-bs8 |87.16|88.07|91.74|88.99|89.91|88.99|88.99|88.99|89.91|88.07|89.08|1.19| | EBERTconcatAttwithProj-bs8_attclsqw |84.40|86.24|89.91|90.83|88.07|92.66|88.07|86.24|90.83|87.16|88.44|2.43| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |88.99|90.83|88.07|87.16|91.74|89.91|87.16|85.32|88.07|90.83|88.81|1.92| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |88.07|88.99|90.83|85.32|88.99|85.32|88.99|91.74|85.32|87.16|88.07|2.17| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |88.99|91.74|86.24|88.07|89.91|92.66|87.16|88.99|88.07|90.83|89.27|1.93| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |88.99|88.99|91.74|92.66|88.99|88.07|90.83|88.99|88.99|88.07|89.63|1.48| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |88.99|88.07|86.24|89.91|84.40|88.99|88.07|91.74|89.91|88.99|88.53|1.94| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |88.99|87.16|88.99|87.16|87.16|88.99|89.91|83.49|85.32|87.16|87.43|1.84| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |88.07|88.07|88.07|89.91|90.83|86.24|86.24|90.83|91.74|86.24|88.62|1.98| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |88.99|90.83|91.74|91.74|89.91|88.07|88.07|88.07|89.91|88.07|89.54|1.43|

STT Error: trained and tested with INCOMPLETE and COMPLETE data

gtts-google

  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 74.31 77.98 77.98 76.15 78.90 77.98 78.90 72.48 77.98 76.15 76.88 2.00
BERT-bs4+Dropout0.1 76.15 80.73 78.90 77.06 77.06 75.23 79.82 77.06 77.98 77.98 77.80 1.58
EmbraceBERT-bs4 76.15 77.06 80.73 77.06 79.82 77.06 80.73 75.23 77.06 75.23 77.61 1.98
EmbraceBERT-bs4+Dropout0.1 75.23 75.23 77.98 76.15 75.23 77.98 77.06 77.98 76.15 76.15 76.51 1.10
EmbraceBERT-bs4-p_selfatt 76.15 78.90 79.82 77.98 74.31 74.31 78.90 79.82 77.98 77.06 77.52 1.94
EmbraceBERT-bs4-p_att_clsquery_weights 77.06 77.06 77.06 74.31 74.31 76.15 77.06 76.15 77.98 76.15 76.33 1.15
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ0 76.15 74.31 69.72 77.98 79.82 70.64 73.39 76.15 72.48 74.31 74.50 2.98
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3 78.90 76.15 73.39 74.31 71.56 76.15 75.23 73.39 73.39 75.23 74.77 1.94
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1 72.48 76.15 76.15 73.39 70.64 76.15 74.31 75.23 71.56 76.15 74.22 1.99
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10 77.06 76.15 74.31 73.39 71.56 77.98 77.06 74.31 74.31 77.06 75.32 1.94
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3 74.31 75.23 72.48 73.39 70.64 73.39 76.15 73.39 74.31 73.39 73.67 1.42
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10 74.31 76.15 77.98 76.15 74.31 77.06 77.06 73.39 76.15 79.82 76.24 1.81
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 77.06 76.15 78.90 77.06 79.82 78.90 76.15 80.73 75.23 77.98 77.80 1.68
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 77.98 79.82 77.98 76.15 77.98 77.98 79.82 75.23 77.98 79.82 78.07 1.45
BERT-bs16+Dropout0.1 73.39 76.15 78.90 77.98 76.15 77.06 77.06 76.15 77.98 79.82 77.06 1.69
EmbraceBERT-bs16 76.15 74.31 74.31 76.15 76.15 77.06 77.06 77.06 79.82 76.15 76.42 1.48
EmbraceBERT-bs16+Dropout0.1 76.15 78.90 77.06 77.06 77.06 77.06 77.98 77.06 77.06 76.15 77.16 0.76
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3 71.56 76.15 80.73 75.23 71.56 75.23 70.64 73.39 70.64 76.15 74.13 3.04
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1 73.39 76.15 77.98 74.31 75.23 73.39 69.72 70.64 74.31 74.31 73.94 2.29
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10 74.31 77.06 78.90 74.31 71.56 69.72 67.89 68.81 74.31 74.31 73.12 3.38
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3 72.48 75.23 77.98 75.23 73.39 73.39 71.56 72.48 71.56 73.39 73.67 1.88
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10 78.90 75.23 77.06 74.31 72.48 69.72 68.81 69.72 72.48 76.15 73.49 3.24
EmbraceBERT-bs16-p_multiheadatt_bertKVQ_epQ3
EmbraceBERT-bs16-p_multiheadatt_bertKVQ_epQ10
  • Epoch 100 - bs8
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs8 77.06 77.98 78.90 77.06 74.31 78.90 78.90 74.31 76.15 78.90 77.25 1.73
BERTwithAtt-bs8 74.31 76.15 78.90 77.06 76.15 75.23 71.56 77.06 77.98 79.82 76.42 2.25
BERTwithAttClsProjection-bs8 77.06 77.98 77.98 77.98 76.15 77.98 75.23 78.90 78.90 77.06 77.52 1.10
BERTwithProjection-bs8 79.82 76.15 73.39 77.06 78.90 76.15 74.31 77.06 79.82 77.06 76.97 2.03
BERTwithProjectionAtt-bs8 76.15 77.06 80.73 76.15 76.15 77.06 75.23 77.98 75.23 73.39 76.51 1.84
--------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs8 77.06 75.23 77.98 74.31 77.06 76.15 74.31 76.15 77.98 76.15 76.24 1.26
EmbraceBERT-bs8-p_att_clsquery_weights 73.39 77.98 79.82 75.23 78.90 76.15 78.90 74.31 73.39 76.15 76.42 2.25
EmbraceBERTwithProj-bs8 77.98 76.15 77.06 75.23 76.15 77.98 77.06 77.06 79.82 74.31 76.88 1.47
EmbraceBERTwithProj-bs8-p_attclsqw 77.98 77.06 76.15 80.73 76.15 74.31 78.90 77.06 78.90 77.98 77.52 1.70
EBERTconcatAtt-bs8 76.15 78.90 79.82 77.98 75.23 75.23 73.39 76.15 77.06 78.90 76.88 1.92
EBERTconcatAtt-bs8_attclsqw 75.23 75.23 73.39 79.82 76.15 77.98 77.06 74.31 76.15 75.23 76.06 1.76
EBERTconcatAttwithProj-bs8 77.98 80.73 77.06 77.98 74.31 74.31 76.15 81.65 77.06 71.56 76.88 2.87
EBERTconcatAttwithProj-bs8_attclsqw 77.06 77.98 78.90 79.82 77.98 77.06 77.98 77.98 76.15 80.73 78.17 1.28
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery 77.98 77.06 78.90 16.51 23.85 31.19 33.03 28.44 29.36 78.90 47.52 25.42
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 77.98 77.06 77.06 78.90 22.94 77.98 78.90 12.84 78.90 77.06 65.96 24.15
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery 77.06 77.98 76.15 74.31 10.09 18.35 79.82 77.98 77.98 76.15 64.59 25.29
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 77.06 77.98 76.15 12.84 16.51 14.68 13.76 76.15 77.06 76.15 51.83 30.54
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery 75.23 74.31 15.60 34.86 22.94 31.19 73.39 32.11 79.82 80.73 52.02 25.27
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 77.98 78.90 76.15 78.90 75.23 32.11 76.15 16.51 11.01 34.86 55.78 27.04
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery 76.15 11.01 20.18 77.98 15.60 74.31 75.23 30.28 39.45 27.52 44.77 26.49
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 77.06 76.15 74.31 74.31 10.09 36.70 25.69 22.02 77.06 75.23 54.86 26.22

gtts-sphinx

  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 84.40 84.40 85.32 85.32 84.40 82.57 84.40 85.32 85.32 82.57 84.40 1.00
BERT-bs4+Dropout0.1 85.32 83.49 86.24 83.49 85.32 87.16 84.40 84.40 84.40 82.57 84.68 1.30
EmbraceBERT-bs4 80.73 86.24 83.49 81.65 78.90 84.40 83.49 84.40 85.32 86.24 83.49 2.28
EmbraceBERT-bs4+Dropout0.1 84.40 81.65 85.32 83.49 83.49 80.73 82.57 88.99 84.40 84.40 83.94 2.14
EmbraceBERT-bs4-p_selfatt 88.07 81.65 82.57 82.57 87.16 84.40 84.40 84.40 83.49 80.73 83.94 2.18
EmbraceBERT-bs4-p_att_clsquery_weights 84.40 83.49 83.49 86.24 85.32 83.49 85.32 85.32 85.32 86.24 84.86 1.03
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ0 82.57 88.07 80.73 81.65 86.24 80.73 80.73 79.82 81.65 77.06 81.93 2.99
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3 79.82 82.57 81.65 79.82 79.82 81.65 72.48 80.73 80.73 84.40 80.37 2.96
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1 83.49 78.90 84.40 86.24 79.82 79.82 82.57 78.90 78.90 82.57 81.56 2.51
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10 86.24 83.49 82.57 82.57 87.16 84.40 82.57 84.40 80.73 88.07 84.22 2.20
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3 79.82 80.73 79.82 80.73 77.06 80.73 82.57 80.73 78.90 81.65 80.28 1.44
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10 85.32 82.57 86.24 84.40 83.49 83.49 87.16 83.49 84.40 88.07 84.86 1.70
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 83.49 81.65 87.16 79.82 83.49 84.40 83.49 83.49 81.65 86.24 83.49 2.05
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 83.49 83.49 85.32 80.73 88.99 82.57 86.24 83.49 88.07 86.24 84.86 2.44
BERT-bs16+Dropout0.1 86.24 85.32 83.49 85.32 83.49 83.49 86.24 82.57 88.07 85.32 84.95 1.60
EmbraceBERT-bs16 88.07 81.65 84.40 81.65 85.32 83.49 84.40 88.99 81.65 77.98 83.76 3.10
EmbraceBERT-bs16+Dropout0.1 85.32 83.49 84.40 81.65 82.57 84.40 83.49 86.24 83.49 81.65 83.67 1.41
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3 78.90 80.73 78.90 80.73 87.16 80.73 83.49 77.06 86.24 81.65 81.56 3.06
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1 79.82 79.82 81.65 77.98 84.40 81.65 75.23 74.31 88.07 81.65 80.46 3.87
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10 81.65 86.24 82.57 81.65 77.98 77.98 84.40 76.15 79.82 85.32 81.38 3.21
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3 78.90 84.40 81.65 81.65 87.16 81.65 82.57 74.31 86.24 81.65 82.02 3.46
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10 84.40 85.32 80.73 82.57 77.98 79.82 84.40 76.15 80.73 84.40 81.65 2.93
EmbraceBERT-bs16-p_multiheadatt_bertKVQ_epQ3
EmbraceBERT-bs16-p_multiheadatt_bertKVQ_epQ10
  • Epoch 100 - bs8
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs8 82.57 82.57 88.07 83.49 85.32 83.49 87.16 83.49 86.24 81.65 84.40 2.05
BERTwithAtt-bs8 83.49 87.16 86.24 83.49 84.40 83.49 80.73 89.91 82.57 82.57 84.40 2.53
BERTwithAttClsProjection-bs8 88.99 86.24 86.24 83.49 85.32 85.32 91.74 85.32 85.32 85.32 86.33 2.23
BERTwithProjection-bs8 87.16 83.49 88.99 82.57 85.32 86.24 86.24 85.32 85.32 87.16 85.78 1.75
BERTwithProjectionAtt-bs8 79.82 81.65 80.73 82.57 83.49 85.32 87.16 87.16 86.24 86.24 84.04 2.60
--------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs8 82.57 82.57 87.16 81.65 83.49 82.57 83.49 83.49 82.57 75.23 82.48 2.80
EmbraceBERT-bs8-p_att_clsquery_weights 84.40 80.73 83.49 85.32 82.57 82.57 85.32 84.40 84.40 83.49 83.67 1.35
EmbraceBERTwithProj-bs8 86.24 86.24 86.24 88.07 84.40 87.16 86.24 90.83 88.07 87.16 87.06 1.61
EmbraceBERTwithProj-bs8-p_attclsqw 88.07 84.40 87.16 86.24 86.24 84.40 87.16 84.40 84.40 82.57 85.50 1.63
EBERTconcatAtt-bs8 84.40 88.99 83.49 85.32 81.65 86.24 80.73 83.49 86.24 81.65 84.22 2.42
EBERTconcatAtt-bs8_attclsqw 86.24 83.49 88.99 83.49 81.65 82.57 82.57 82.57 81.65 82.57 83.58 2.19
EBERTconcatAttwithProj-bs8 82.57 87.16 83.49 82.57 85.32 84.40 87.16 83.49 88.99 81.65 84.68 2.29
EBERTconcatAttwithProj-bs8_attclsqw 86.24 84.40 84.40 84.40 85.32 84.40 89.91 86.24 86.24 83.49 85.50 1.73
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery 82.57 86.24 19.27 24.77 22.94 33.94 84.40 80.73 81.65 83.49 60.00 28.63
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 85.32 24.77 85.32 85.32 11.01 83.49 14.68 15.60 84.40 23.85 51.38 33.62
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery 39.45 11.01 14.68 34.86 12.84 83.49 83.49 85.32 34.86 18.35 41.83 29.23
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 86.24 86.24 85.32 19.27 85.32 15.60 11.93 11.93 33.94 12.84 44.86 33.95
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery 84.40 35.78 12.84 45.87 29.36 35.78 80.73 80.73 83.49 22.94 51.19 26.72
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 83.49 19.27 80.73 27.52 83.49 83.49 84.40 81.65 13.76 22.02 57.98 30.66
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery 36.70 85.32 86.24 89.91 22.94 87.16 86.24 87.16 45.87 87.16 71.47 24.34
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 88.99 85.32 13.76 33.03 6.42 88.99 11.93 21.10 14.68 16.51 38.07 33.20

gtts-witai

  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 90.83 93.58 92.66 90.83 92.66 90.83 92.66 91.74 90.83 92.66 91.93 0.99
BERT-bs4+Dropout0.1 90.83 92.66 90.83 91.74 92.66 90.83 92.66 93.58 91.74 91.74 91.93 0.90
EmbraceBERT-bs4 90.83 90.83 90.83 88.99 91.74 92.66 90.83 88.07 92.66 90.83 90.83 1.36
EmbraceBERT-bs4+Dropout0.1 91.74 90.83 91.74 89.91 90.83 90.83 90.83 89.91 90.83 89.91 90.73 0.64
EmbraceBERT-bs4-p_selfatt 92.66 90.83 92.66 88.07 92.66 88.99 90.83 90.83 89.91 88.99 90.64 1.58
EmbraceBERT-bs4-p_att_clsquery_weights 91.74 90.83 92.66 92.66 92.66 92.66 91.74 90.83 92.66 89.91 91.83 0.96
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ0 88.99 93.58 88.07 90.83 91.74 89.91 86.24 91.74 91.74 90.83 90.37 2.02
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3 82.57 93.58 92.66 89.91 88.99 85.32 90.83 91.74 90.83 88.07 89.45 3.21
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1 89.91 91.74 91.74 91.74 88.99 91.74 89.91 91.74 88.07 90.83 90.64 1.28
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10 90.83 93.58 93.58 89.91 91.74 89.91 91.74 91.74 90.83 91.74 91.56 1.22
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3 83.49 93.58 91.74 91.74 88.07 88.07 90.83 91.74 90.83 89.91 90.00 2.70
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10 91.74 92.66 92.66 89.91 91.74 86.24 89.91 91.74 90.83 92.66 91.01 1.87
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 90.83 90.83 90.83 90.83 91.74 92.66 90.83 91.74 91.74 93.58 91.56 0.90
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 92.66 92.66 92.66 91.74 91.74 90.83 92.66 92.66 88.99 92.66 91.93 1.15
BERT-bs16+Dropout0.1 89.91 92.66 92.66 89.91 88.99 90.83 91.74 92.66 89.91 92.66 91.19 1.37
EmbraceBERT-bs16 91.74 90.83 91.74 88.07 89.91 90.83 90.83 90.83 91.74 88.99 90.55 1.16
EmbraceBERT-bs16+Dropout0.1 89.91 91.74 90.83 90.83 90.83 91.74 87.16 89.91 91.74 88.99 90.37 1.38
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3 89.91 89.91 90.83 90.83 88.07 91.74 87.16 87.16 90.83 88.07 89.45 1.60
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1 89.91 88.99 89.91 87.16 89.91 92.66 88.99 88.07 88.07 88.07 89.17 1.47
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10 88.99 92.66 91.74 91.74 89.91 89.91 87.16 90.83 89.91 91.74 90.46 1.55
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3 88.99 92.66 89.91 90.83 88.99 91.74 86.24 88.99 89.91 88.99 89.72 1.68
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10 88.99 92.66 92.66 91.74 91.74 89.91 87.16 89.91 89.91 89.91 90.46 1.65
EmbraceBERT-bs16-p_multiheadatt_bertKVQ_epQ3
EmbraceBERT-bs16-p_multiheadatt_bertKVQ_epQ10
  • Epoch 100 - bs8
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs8 90.83 92.66 93.58 91.74 90.83 91.74 92.66 90.83 90.83 89.91 91.56 1.07
BERTwithAtt-bs8 89.91 89.91 90.83 91.74 88.99 90.83 89.91 88.07 89.91 89.91 90.00 0.96
BERTwithAttClsProjection-bs8 89.91 90.83 89.91 91.74 90.83 90.83 93.58 91.74 90.83 91.74 91.19 1.02
BERTwithProjection-bs8 91.74 90.83 89.91 86.24 90.83 90.83 91.74 92.66 89.91 92.66 90.73 1.76
BERTwithProjectionAtt-bs8 90.83 90.83 89.91 89.91 93.58 88.07 90.83 92.66 90.83 88.99 90.64 1.52
--------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs8 90.83 91.74 91.74 91.74 90.83 91.74 89.91 90.83 91.74 89.91 91.10 0.72
EmbraceBERT-bs8-p_att_clsquery_weights 90.83 92.66 91.74 90.83 89.91 92.66 90.83 92.66 92.66 89.91 91.47 1.09
EmbraceBERTwithProj-bs8 91.74 92.66 93.58 91.74 91.74 90.83 91.74 92.66 91.74 91.74 92.02 0.72
EmbraceBERTwithProj-bs8-p_attclsqw 92.66 89.91 89.91 90.83 92.66 90.83 92.66 91.74 90.83 90.83 91.28 1.03
EBERTconcatAtt-bs8 92.66 88.07 90.83 92.66 87.16 87.16 88.99 92.66 91.74 91.74 90.37 2.18
EBERTconcatAtt-bs8_attclsqw 90.83 93.58 88.99 89.91 88.07 88.07 94.50 87.16 89.91 88.99 90.00 2.26
EBERTconcatAttwithProj-bs8 90.83 91.74 91.74 91.74 91.74 90.83 88.99 92.66 91.74 91.74 91.38 0.94
EBERTconcatAttwithProj-bs8_attclsqw 88.99 89.91 91.74 92.66 92.66 91.74 91.74 92.66 92.66 91.74 91.65 1.19
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery 90.83 40.37 89.91 24.77 25.69 91.74 33.94 22.02 29.36 28.44 47.71 28.64
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 27.52 90.83 7.34 36.70 88.99 28.44 19.27 15.60 23.85 27.52 36.61 27.71
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery 91.74 92.66 92.66 91.74 13.76 16.51 89.91 33.94 33.94 92.66 64.95 33.54
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 91.74 92.66 92.66 89.91 17.43 11.93 14.68 92.66 37.61 15.60 55.69 36.83
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery 90.83 37.61 90.83 91.74 35.78 29.36 93.58 25.69 89.91 89.91 67.52 29.09
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 92.66 92.66 88.99 91.74 92.66 34.86 33.03 11.01 92.66 88.99 71.93 30.48
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery 35.78 91.74 90.83 91.74 16.51 33.94 91.74 30.28 39.45 92.66 61.47 30.79
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 89.91 91.74 11.01 92.66 92.66 89.91 90.83 92.66 92.66 17.43 76.15 31.01

macsay-google

  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 85.32 84.40 88.07 86.24 87.16 86.24 88.99 86.24 89.91 87.16 86.97 1.58
BERT-bs4+Dropout0.1 86.24 87.16 85.32 86.24 88.99 88.07 86.24 83.49 88.07 87.16 86.70 1.49
EmbraceBERT-bs4 85.32 86.24 84.40 84.40 86.24 85.32 87.16 88.07 88.07 87.16 86.24 1.30
EmbraceBERT-bs4+Dropout0.1 86.24 82.57 88.99 87.16 87.16 82.57 86.24 80.73 83.49 84.40 84.95 2.47
EmbraceBERT-bs4-p_selfatt 86.24 83.49 88.07 86.24 88.07 87.16 86.24 87.16 85.32 86.24 86.42 1.28
EmbraceBERT-bs4-p_att_clsquery_weights 86.24 87.16 85.32 86.24 84.40 86.24 89.91 84.40 85.32 84.40 85.96 1.59
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ0 87.16 85.32 82.57 82.57 83.49 78.90 84.40 78.90 82.57 80.73 82.66 2.51
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3 87.16 84.40 88.07 84.40 84.40 86.24 87.16 86.24 87.16 77.06 85.23 3.00
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1 84.40 88.07 87.16 82.57 88.07 69.72 86.24 87.16 86.24 86.24 84.59 5.20
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10 87.16 84.40 85.32 85.32 87.16 86.24 89.91 89.91 86.24 87.16 86.88 1.74
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3 86.24 87.16 85.32 86.24 84.40 86.24 84.40 84.40 88.99 84.40 85.78 1.44
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10 87.16 84.40 85.32 87.16 84.40 84.40 89.91 88.07 87.16 87.16 86.51 1.74
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 85.32 88.99 87.16 86.24 86.24 87.16 88.99 84.40 87.16 89.91 87.16 1.64
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 87.16 87.16 86.24 85.32 87.16 83.49 87.16 84.40 87.16 88.07 86.33 1.39
BERT-bs16+Dropout0.1 85.32 85.32 87.16 88.99 87.16 84.40 88.99 86.24 86.24 88.07 86.79 1.49
EmbraceBERT-bs16 82.57 86.24 87.16 85.32 86.24 84.40 84.40 87.16 86.24 86.24 85.60 1.36
EmbraceBERT-bs16+Dropout0.1 87.16 85.32 86.24 88.07 83.49 85.32 84.40 84.40 88.07 80.73 85.32 2.13
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3 84.40 85.32 83.49 87.16 84.40 86.24 83.49 82.57 86.24 81.65 84.50 1.66
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1 80.73 83.49 85.32 82.57 86.24 84.40 82.57 80.73 85.32 86.24 83.76 1.97
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10 84.40 84.40 86.24 84.40 85.32 88.99 85.32 84.40 87.16 87.16 85.78 1.49
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3 83.49 84.40 83.49 85.32 84.40 89.91 83.49 81.65 86.24 83.49 84.59 2.12
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10 83.49 85.32 88.07 88.99 83.49 87.16 82.57 84.40 88.07 87.16 85.87 2.18
EmbraceBERT-bs16-p_multiheadatt_bertKVQ_epQ3
EmbraceBERT-bs16-p_multiheadatt_bertKVQ_epQ10
  • Epoch 100 - bs8
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs8 86.24 85.32 86.24 85.32 87.16 86.24 88.07 83.49 88.99 87.16 86.42 1.47
BERTwithAtt-bs8 81.65 87.16 88.07 88.99 83.49 86.24 85.32 86.24 84.40 86.24 85.78 2.06
BERTwithAttClsProjection-bs8 86.24 86.24 87.16 90.83 87.16 87.16 86.24 85.32 84.40 84.40 86.51 1.74
BERTwithProjection-bs8 85.32 84.40 85.32 85.32 86.24 85.32 85.32 86.24 88.07 85.32 85.69 0.94
BERTwithProjectionAtt-bs8 84.40 87.16 86.24 84.40 84.40 84.40 85.32 85.32 87.16 85.32 85.41 1.04
--------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs8 84.40 85.32 88.99 84.40 86.24 82.57 82.57 83.49 88.07 84.40 85.05 2.05
EmbraceBERT-bs8-p_att_clsquery_weights 88.07 88.07 88.99 85.32 88.07 85.32 87.16 84.40 89.91 88.07 87.34 1.68
EmbraceBERTwithProj-bs8 87.16 87.16 87.16 87.16 85.32 87.16 85.32 86.24 88.99 88.07 86.97 1.07
EmbraceBERTwithProj-bs8-p_attclsqw 88.07 84.40 83.49 88.07 83.49 86.24 84.40 88.99 86.24 86.24 85.96 1.88
EBERTconcatAtt-bs8 85.32 87.16 87.16 86.24 84.40 86.24 83.49 84.40 86.24 85.32 85.60 1.16
EBERTconcatAtt-bs8_attclsqw 85.32 87.16 88.07 88.07 83.49 88.07 87.16 85.32 84.40 88.07 86.51 1.64
EBERTconcatAttwithProj-bs8 81.65 88.07 85.32 88.99 82.57 85.32 87.16 87.16 86.24 85.32 85.78 2.18
EBERTconcatAttwithProj-bs8_attclsqw 82.57 86.24 89.91 83.49 84.40 89.91 85.32 86.24 88.99 87.16 86.42 2.45
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery 20.18 84.40 88.07 85.32 83.49 85.32 87.16 29.36 84.40 87.16 73.49 24.48
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 87.16 20.18 8.26 84.40 23.85 88.07 83.49 18.35 18.35 87.16 51.93 34.35
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery 85.32 87.16 10.09 84.40 85.32 20.18 88.99 26.61 33.94 85.32 60.73 31.56
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 17.43 83.49 15.60 10.09 85.32 14.68 13.76 7.34 36.70 86.24 37.06 32.25
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery 84.40 88.07 15.60 88.07 86.24 85.32 84.40 29.36 28.44 25.69 61.56 30.26
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 86.24 18.35 16.51 88.99 84.40 88.07 83.49 14.68 10.09 88.07 57.89 35.18
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery 88.07 6.42 86.24 10.09 88.99 87.16 33.94 86.24 87.16 20.18 59.45 34.79
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 86.24 87.16 11.01 86.24 7.34 86.24 86.24 86.24 87.16 88.07 71.19 31.03

macsay-sphinx

  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 75.23 70.64 74.31 71.56 67.89 67.89 72.48 68.81 74.31 71.56 71.47 2.54
BERT-bs4+Dropout0.1 72.48 74.31 74.31 75.23 72.48 66.97 72.48 71.56 71.56 68.81 72.02 2.40
EmbraceBERT-bs4 69.72 68.81 75.23 65.14 68.81 67.89 63.30 73.39 65.14 68.81 68.62 3.48
EmbraceBERT-bs4+Dropout0.1 71.56 71.56 73.39 65.14 68.81 77.06 69.72 70.64 69.72 70.64 70.83 2.92
EmbraceBERT-bs4-p_selfatt 72.48 70.64 72.48 73.39 71.56 71.56 78.90 73.39 71.56 70.64 72.66 2.28
EmbraceBERT-bs4-p_att_clsquery_weights 72.48 67.89 66.06 67.89 73.39 67.89 70.64 69.72 66.06 73.39 69.54 2.68
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ0 66.97 71.56 71.56 71.56 67.89 67.89 66.97 72.48 70.64 69.72 69.72 2.01
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3 64.22 67.89 68.81 74.31 58.72 71.56 67.89 73.39 66.06 68.81 68.17 4.32
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1 66.06 71.56 66.97 70.64 63.30 68.81 71.56 70.64 69.72 67.89 68.72 2.54
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10 72.48 74.31 66.06 73.39 74.31 68.81 75.23 72.48 74.31 73.39 72.48 2.72
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3 63.30 67.89 70.64 77.98 62.39 67.89 67.89 72.48 65.14 68.81 68.44 4.33
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10 73.39 70.64 70.64 74.31 73.39 68.81 74.31 66.97 71.56 71.56 71.56 2.28
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 69.72 67.89 71.56 71.56 76.15 69.72 67.89 73.39 66.97 71.56 70.64 2.66
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 73.39 74.31 67.89 73.39 73.39 68.81 70.64 68.81 70.64 72.48 71.38 2.20
BERT-bs16+Dropout0.1 71.56 72.48 73.39 74.31 67.89 68.81 67.89 69.72 69.72 72.48 70.83 2.20
EmbraceBERT-bs16 68.81 72.48 67.89 63.30 73.39 67.89 65.14 68.81 70.64 68.81 68.72 2.89
EmbraceBERT-bs16+Dropout0.1 66.06 70.64 64.22 70.64 72.48 69.72 66.06 69.72 71.56 65.14 68.62 2.81
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3 69.72 66.06 68.81 73.39 60.55 69.72 69.72 64.22 69.72 67.89 67.98 3.39
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1 68.81 69.72 66.97 71.56 66.97 69.72 72.48 66.06 73.39 66.97 69.27 2.44
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10 71.56 67.89 76.15 68.81 74.31 68.81 71.56 69.72 75.23 76.15 72.02 3.05
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3 72.48 64.22 65.14 69.72 62.39 66.97 67.89 67.89 69.72 68.81 67.52 2.82
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10 70.64 71.56 75.23 68.81 71.56 64.22 72.48 68.81 77.06 74.31 71.47 3.49
EmbraceBERT-bs16-p_multiheadatt_bertKVQ_epQ3
EmbraceBERT-bs16-p_multiheadatt_bertKVQ_epQ10
  • Epoch 100 - bs8
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs8 72.48 76.15 68.81 71.56 74.31 63.30 68.81 69.72 73.39 70.64 70.92 3.41
BERTwithAtt-bs8 67.89 71.56 73.39 66.97 70.64 68.81 68.81 77.98 70.64 64.22 70.09 3.58
BERTwithAttClsProjection-bs8 72.48 71.56 71.56 68.81 73.39 69.72 70.64 71.56 70.64 68.81 70.92 1.42
BERTwithProjection-bs8 71.56 73.39 69.72 66.97 71.56 79.82 74.31 72.48 74.31 70.64 72.48 3.23
BERTwithProjectionAtt-bs8 68.81 72.48 73.39 71.56 72.48 67.89 69.72 67.89 71.56 70.64 70.64 1.88
--------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs8 66.06 69.72 72.48 65.14 75.23 66.97 67.89 66.06 72.48 67.89 68.99 3.20
EmbraceBERT-bs8-p_att_clsquery_weights 65.14 66.97 73.39 66.97 76.15 66.97 67.89 66.97 66.06 71.56 68.81 3.43
EmbraceBERTwithProj-bs8 73.39 73.39 72.48 68.81 73.39 71.56 75.23 70.64 64.22 74.31 71.74 3.06
EmbraceBERTwithProj-bs8-p_attclsqw 72.48 66.97 69.72 67.89 72.48 72.48 71.56 67.89 72.48 71.56 70.55 2.11
EBERTconcatAtt-bs8 63.30 73.39 70.64 66.06 77.06 70.64 74.31 70.64 68.81 68.81 70.37 3.78
EBERTconcatAtt-bs8_attclsqw 74.31 76.15 71.56 64.22 74.31 72.48 70.64 71.56 67.89 68.81 71.19 3.34
EBERTconcatAttwithProj-bs8 66.97 71.56 73.39 64.22 70.64 69.72 72.48 70.64 71.56 70.64 70.18 2.57
EBERTconcatAttwithProj-bs8_attclsqw 76.15 71.56 71.56 66.06 66.97 70.64 78.90 67.89 70.64 73.39 71.38 3.80
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery 66.97 68.81 23.85 22.02 17.43 69.72 35.78 72.48 66.06 72.48 51.56 22.37
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 30.28 68.81 71.56 74.31 16.51 63.30 67.89 62.39 67.89 15.60 53.85 22.19
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery 73.39 68.81 68.81 71.56 68.81 17.43 31.19 71.56 33.94 74.31 57.98 20.41
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 22.02 71.56 73.39 69.72 66.97 13.76 72.48 74.31 70.64 17.43 55.23 24.69
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery 33.03 70.64 71.56 26.61 16.51 73.39 30.28 64.22 66.97 70.64 52.39 21.55
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 73.39 76.15 68.81 71.56 72.48 34.86 66.06 68.81 65.14 67.89 66.51 11.03
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery 36.70 72.48 76.15 68.81 19.27 73.39 35.78 74.31 71.56 68.81 59.72 19.69
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 72.48 38.53 11.93 70.64 68.81 69.72 67.89 14.68 72.48 67.89 55.50 23.12

macsay-witai

  • Epoch 100
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs4 87.16 88.99 92.66 88.07 90.83 87.16 90.83 87.16 90.83 88.99 89.27 1.84
BERT-bs4+Dropout0.1 87.16 89.91 92.66 88.99 89.91 90.83 89.91 91.74 91.74 88.07 90.09 1.63
EmbraceBERT-bs4 88.07 89.91 88.99 88.07 88.99 90.83 90.83 85.32 89.91 88.07 88.90 1.56
EmbraceBERT-bs4+Dropout0.1 87.16 87.16 89.91 89.91 89.91 87.16 89.91 90.83 88.07 88.99 88.90 1.33
EmbraceBERT-bs4-p_selfatt 86.24 88.07 88.07 88.07 88.07 84.40 94.50 88.07 90.83 88.07 88.44 2.54
EmbraceBERT-bs4-p_att_clsquery_weights 88.07 86.24 89.91 88.07 91.74 89.91 87.16 88.07 89.91 90.83 88.99 1.64
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ0 89.91 84.40 88.07 88.07 85.32 84.40 86.24 89.91 84.40 86.24 86.70 2.06
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3 89.91 88.07 89.91 87.16 85.32 86.24 88.99 87.16 90.83 88.99 88.26 1.68
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1 86.24 87.16 87.16 88.99 88.99 85.32 88.07 88.99 88.07 84.40 87.34 1.52
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10 90.83 90.83 91.74 88.07 88.07 88.07 90.83 87.16 89.91 90.83 89.63 1.54
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3 87.16 86.24 88.99 86.24 86.24 86.24 88.99 89.91 92.66 82.57 87.52 2.60
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10 91.74 88.99 90.83 87.16 86.24 88.07 89.91 87.16 89.91 91.74 89.17 1.87
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery 87.16 87.16 87.16 88.99 88.07 88.99 91.74 87.16 88.99 91.74 88.72 1.69
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
BERT-bs16 87.16 89.91 90.83 88.07 88.07 88.99 89.91 88.07 88.99 89.91 88.99 1.09
BERT-bs16+Dropout0.1 88.99 88.99 90.83 88.07 89.91 88.99 89.91 88.99 89.91 88.99 89.36 0.73
EmbraceBERT-bs16 87.16 88.99 91.74 87.16 89.91 86.24 88.07 92.66 88.99 88.07 88.90 1.94
EmbraceBERT-bs16+Dropout0.1 86.24 90.83 90.83 89.91 90.83 88.99 85.32 89.91 88.99 86.24 88.81 2.00
----------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3 82.57 82.57 86.24 88.07 87.16 88.07 87.16 87.16 87.16 83.49 85.96 2.09
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1 83.49 84.40 88.99 82.57 86.24 88.99 86.24 86.24 88.07 86.24 86.15 2.07
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10 87.16 88.99 87.16 88.99 83.49 89.91 89.91 87.16 88.07 88.99 87.98 1.81
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3 83.49 84.40 87.16 90.83 86.24 86.24 85.32 86.24 87.16 85.32 86.24 1.88
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10 86.24 88.99 87.16 88.99 82.57 89.91 90.83 88.99 88.07 88.07 87.98 2.19
EmbraceBERT-bs16-p_multiheadatt_bertKVQ_epQ3
EmbraceBERT-bs16-p_multiheadatt_bertKVQ_epQ10
  • Epoch 100 - bs8
Model Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Run10 Mean Std
BERT-bs8 88.99 88.99 90.83 88.99 89.91 90.83 89.91 87.16 89.91 91.74 89.72 1.22
BERTwithAtt-bs8 90.83 88.07 88.99 89.91 87.16 88.99 88.07 91.74 88.99 87.16 88.99 1.42
BERTwithAttClsProjection-bs8 88.07 88.07 90.83 88.99 88.07 91.74 88.07 88.99 89.91 90.83 89.36 1.31
BERTwithProjection-bs8 89.91 88.07 85.32 88.99 88.99 91.74 77.06 87.16 88.07 88.99 87.43 3.81
BERTwithProjectionAtt-bs8 89.91 88.99 88.99 90.83 88.07 87.16 92.66 88.99 91.74 88.99 89.63 1.59
--------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs8 88.07 89.91 89.91 88.07 88.07 86.24 86.24 89.91 87.16 88.07 88.17 1.33
EmbraceBERT-bs8-p_att_clsquery_weights 87.16 89.91 89.91 85.32 91.74 88.99 88.99 89.91 88.07 88.07 88.81 1.68
EmbraceBERTwithProj-bs8 90.83 88.07 87.16 88.99 88.99 89.91 91.74 90.83 89.91 91.74 89.82 1.45
EmbraceBERTwithProj-bs8-p_attclsqw 90.83 89.91 90.83 88.07 88.07 92.66 88.07 88.07 88.07 88.99 89.36 1.55
EBERTconcatAtt-bs8 88.99 89.91 89.91 90.83 88.07 86.24 88.07 89.91 88.07 88.07 88.81 1.28
EBERTconcatAtt-bs8_attclsqw 90.83 88.07 88.07 92.66 85.32 88.07 88.99 89.91 87.16 88.07 88.72 1.93
EBERTconcatAttwithProj-bs8 88.07 89.91 89.91 88.99 90.83 89.91 87.16 88.07 88.07 88.99 88.99 1.09
EBERTconcatAttwithProj-bs8_attclsqw 88.07 88.07 91.74 87.16 88.99 90.83 88.99 89.91 89.91 88.07 89.17 1.35
------------------------------------- --- --- --- --- --- --- --- --- --- --- --- ---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery 88.99 85.32 88.07 22.02 90.83 31.19 33.94 89.91 88.07 28.44 64.68 29.38
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 90.83 88.99 12.84 90.83 87.16 88.99 90.83 88.07 85.32 87.16 81.10 22.82
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery 89.91 91.74 90.83 88.07 89.91 19.27 20.18 90.83 89.91 18.35 68.90 32.51
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 88.99 22.02 90.83 90.83 89.91 88.07 90.83 88.99 88.99 89.91 82.94 20.33
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery 25.69 88.99 88.07 38.53 34.86 87.16 32.11 88.99 33.03 89.91 60.73 28.06
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 88.07 88.07 88.07 89.91 88.99 88.07 88.07 86.24 88.07 85.32 87.89 1.22
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery 90.83 11.01 89.91 88.99 15.60 88.99 36.70 88.99 39.45 89.91 64.04 32.30
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw 88.99 88.99 87.16 88.07 89.91 88.99 88.07 88.99 90.83 87.16 88.72 1.09