Refactoring ranking metrics helper and adding unit tests (#148)

* Adding reduce_lr_on_plateau scheduling learning rate * fix tests * Fixing sanity test * undo uncessery edits * Adding documentation and test for reduce_lr_on_plateau * moving to ranking_model * Fixing tests * Fix tests * Moving the scheduler to the callbacks build function * Refactoring * Refactoring * adding doc string and default patience * removing unnecessary arguments from relevance model * Fix test * Fix tests * Refactoring ranking metrics helper * Adding docs * Adding tests * Clean up * Adding docs and unit tests * Adding NDCG for secondary label (#149) * Adding NDCG for secondary label * Adding tests * Upgrading version * Fixing tests * Removing pdb Co-authored-by: Mohamed Abdelrahman Zahran Mohamed <[email protected]>
salesforce · Dec 2, 2021 · eeb15c6 · eeb15c6
1 parent 8f0b8a8
commit eeb15c6
Show file tree

Hide file tree

Showing 7 changed files with 641 additions and 160 deletions.
diff --git a/docs/source/misc/changelog.md b/docs/source/misc/changelog.md
@@ -5,6 +5,13 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.1.9] - 2021-11-29
+
+### Changed
+
+- Refactored secondary label metrics computation for ranking and added unit tests
+- Added NDCG metric for secondary labels
+
 ## [0.1.8] - 2021-10-21
 
 ### Added

diff --git a/docs/source/quickstart/feature_config.md b/docs/source/quickstart/feature_config.md
@@ -111,6 +111,10 @@ Value representing if the feature should be used for computing groupwise metrics
 
 Value representing if the feature is used as a secondary label to compute failure metrics. The usage of the feature to compute the failure metrics is left to the user to be customized. The Ranking models come prepackaged with failure metrics computation that can be used, for example, to compute rate of clicks on documents without a match on the subject field.
 
+In Ranking applications,
+
+A secondary label is any feature/value that serves as a proxy relevance assessment that the user might be interested to measure on the dataset in addition to the primary click labels. For example, this could be used with an exact query match feature. In that case, the metric sheds light on scenarios where the records with an exact match are ranked lower than those without. This would provide the user with complimentary information (to typical click metrics such as MRR and ACR) about the model to help make better trade-off decisions w.r.t. best model selection.
+
 -----
 
 The `FeatureConfig` can be extended to support additional attributes as necessary.