Loss and evaluation metrics for graded relevance labeled records (for public datasets)

[mohazahran]

Few public ranking datasets are labeled with graded relevance (score from 1 to 5) rather than clicks. Which means the target prediction is not necessarily binary vector. This means MRR is not suitable as an evaluation metric (NDCG is more natural in this case). Also, a special attention needs to be taken for the loss function as well.
@lastmansleeping
@jakemannix

[lastmansleeping]

Perfect. Let's prioritize adding these. @mohazahran

[git2gus]

This issue has been linked to a new work item: W-7983051