Results on Nov. and Dec. 2010 for different combinations

[soroush-ziaeinejad]

@hosseinfani

This issue page is created to showcase the results of SEERa using different combinations of tml and gel methods. A line chart and some stats will be provided for each instance.

[soroush-ziaeinejad]

TML: LDA.gensim / GEL: DynAERNN

Gensim DynAERNN	Mean	Std
nDCG	0.0669	0.0349
map	0.0002	0.0005
success	0.7623	0.2355

[soroush-ziaeinejad]

TML: Gsdmm / GEL: DynAERNN

Gsdmm DynAERNN	Mean	Std
nDCG	0.0609	0.0320
map	0.0002	0.00004
success	0.7638	0.2388

[hosseinfani]

@soroush-ziaeinejad
please put the baselines for each metric in one figure, e.g., ndcgs for all combinations in one figure, ....

[soroush-ziaeinejad]

@soroush-ziaeinejad please put the baselines for each metric in one figure, e.g., ndcgs for all combinations in one figure, ....

I had it in mind but because we gradually achieve the results, I decided to put charts like these, and when all combinations are completed, I will draw and put that kind of chart to compare baselines.

[soroush-ziaeinejad]

TML: Gsdmm / GEL: DynAE

Gsdmm DynAE	Mean	Std
nDCG	0.0605	0.0321
map	0.0002	0.00004
success	0.7601	0.2334

[soroush-ziaeinejad]

TML: Gsdmm / GEL: DynRNN

Gsdmm DynAE	Mean	Std
nDCG	0.0618	0.0315
map	0.0002	0.00004
success	0.7761	0.2184

[soroush-ziaeinejad]

TML: LDA.gensim / GEL: DynAE

Gensim DynAERNN	Mean	Std
nDCG	0.0669	0.0360
map	0.0002	0.0005
success	0.7659	0.2271

[soroush-ziaeinejad]

TML: LDA.gensim / GEL: DynRNN

Gensim DynAERNN	Mean	Std
nDCG	0.0671	0.0354
map	0.0003	0.0005
success	0.7654	0.2319

[hosseinfani]

@soroush-ziaeinejad
Also, add the min-max to the plots to show the +/- std. Drop the max.

[soroush-ziaeinejad]

@hosseinfani,

Here is the comparison between these 6 combinations:
1. gsdmm / DynAE
2. gsdmm / DynAERNN
3. gsdmm / DynRNN
4. LDA/ DynAE
5. LDA/ DynAERNN
6. LDA/ DynRNN

for 3 metrics:
1. nDCG
2. map
3. success

nDCG

map

success

[hosseinfani]

@soroush-ziaeinejad
thanks. make them till 1,000, also, write your analysis of the figures here.

[soroush-ziaeinejad]

@hosseinfani,

Here is the comparison between these 6 combinations till k=1000:
1. gsdmm / DynAE
2. gsdmm / DynAERNN
3. gsdmm / DynRNN
4. LDA/ DynAE
5. LDA/ DynAERNN
6. LDA/ DynRNN

for 3 metrics:
1. nDCG
2. map
3. success

nDCG

map

success

[soroush-ziaeinejad]

@hosseinfani ,

I added +- std to the baselines chart and here is the result for success metric.

I checked the std values and they are greater than the mean (up to 10 times in some cases) and it causes wide shadows and high overlapping areas. Any suggestions?

[hosseinfani]

@soroush-ziaeinejad
drop the legends for stds. make the negative stds to zero. drill down for a sample cutoff like k=400 and double check the root cause of high variations.

[soroush-ziaeinejad]

[soroush-ziaeinejad]

Using variance instead of std leads to these results for success, ndcg, and map, respectively.

[hosseinfani]

@soroush-ziaeinejad so let's proceed with var then. btw, the metric values are very low for practical use though

[soroush-ziaeinejad]

@hosseinfani, btm_DynAERNN and btm_DynRNN are added (btm_DynAE is still running). A new legend to better show the label for each line is also added.

[soroush-ziaeinejad]

Meanwhile, I am putting the results for the toy dataset [1-4] Dec 2010. Right now, more combinations have results for this dataset compared to the main one.

[hosseinfani]

@soroush-ziaeinejad
Thanks, Soroush. I am going to allocate more time to your paper draft. We need to 80-20 time split, 80 to paper writeup :)

[soroush-ziaeinejad]

Thanks @hosseinfani