Merge pull request #12 from Kaleidophon/demo

Demo

CHANGELOG.md

@@ -0,0 +1,22 @@
+## 1.0.0
+
+- Massive changes to repository structure, including restructuing of APIs and bug fixes. See the points below:
+ - Demo notebook added, showcasing some common use cases. Take a look [here](https://github.com/Kaleidophon/nlp-uncertainty-zoo/blob/main/demo.ipynb) or [here](https://colab.research.google.com/drive/1-Pl5lvcnpbGL2ZXLGDDNqvJB7Ew8uIsS?usp=sharing).
+ - API changes:
+ - Instead of using a `model_params` dict when initializing a `Model` subclass, the necessary model arguments are spelled out 
+ explicitly, similarly to the API of scikit-learn` models. Furthermore, reasonable default values are set for most model parameters.
+ - The `Module` class implements additional functionalities:
+ - An `available_uncertainty_metrics` attribute gives some information about available uncertainty metrics, mapping from their names to their functions.
+ - `get_sequence_representation()` was renamed to `get_sequence_representation_from_hidden()`. This was done to distinguish it from the new `get_sequence_classification()` function, which retrieves sequence representation directly from the inputs.
+ - `get_hidden_representation()` defines how to obtain hidden representations from an input.
+ - An explicit function called `compute_loss_weights()` was added to the `Model` class to allow to customize loss weights for unbalanced problems.
+ - BERT-related models now have a `bert_class` argument, that allow the underlying model to be changed to a RoBERTa, DistilBert or similar model.'
+ - Split `utils.uncertainty_eval` into `utils.uncertainty_eval` and `utils.calibration_eval`.
+ - There are now explicit functions to evaluate calibration and uncertainty properties on model, specified in `utils.uncertainty_eval.evaluate_uncertainty()` and `utils.calibration_eval.evaluate_calibration()`.
+ - Bug fixes:
+ - Changing batch size during inference won't produce shape errors for LSTM models anymore.
+
+
+## 0.9.2
+
+- Fix import issues concerning the `transformers` and `protobuf` packages mentioned in issues [#8](https://github.com/Kaleidophon/nlp-uncertainty-zoo/issues/8) and [#9](https://github.com/Kaleidophon/nlp-uncertainty-zoo/issues/9).

README.md

@@ -7,18 +7,23 @@ implemented in PyTorch. You can install the repository using pip:
 
 If you are using the repository in your academic research, please cite the paper below:
 
- @article{ulmer2022exploring,
-  title={Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method \& Data Scarcity},
-  author={Ulmer, Dennis and Frellsen, Jes and Hardmeier, Christian},
-  journal={arXiv preprint arXiv:2210.15452},
-  year={2022}
- }
-
-
-Certain parts of this repository are still incomplete, but will come soon (I promise!):
-
-- [x] Build proper documentation
-- [ ] Add demo jupyter notebook
+ @inproceedings{ulmer-etal-2022-exploring,
+  title = "Exploring Predictive Uncertainty and Calibration in {NLP}: A Study on the Impact of Method {\&} Data Scarcity",
+  author = "Ulmer, Dennis and
+  Frellsen, Jes and
+  Hardmeier, Christian",
+  booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
+  month = dec,
+  year = "2022",
+  address = "Abu Dhabi, United Arab Emirates",
+  publisher = "Association for Computational Linguistics",
+  url = "https://aclanthology.org/2022.findings-emnlp.198",
+  pages = "2707--2735",
+ }
+
+To learn more about the package, consult the documentation [here](http://dennisulmer.eu/nlp-uncertainty-zoo/),
+check a Jupyter notebook demo [here](https://github.com/Kaleidophon/nlp-uncertainty-zoo/blob/main/demo.ipynb) or a Google 
+collab [here](https://colab.research.google.com/drive/1-Pl5lvcnpbGL2ZXLGDDNqvJB7Ew8uIsS?usp=sharing).
 
 ### Included models
 
@@ -50,7 +55,7 @@ model = LSTMEnsemble(**network_params, ensemble_size=10, is_sequence_classifer=F
 model.fit(train_split=train_dataloader)
 model.get_logits(X)
 model.get_predictions(X)
-model.get_sequence_representation(X)
+model.get_sequence_representation_from_hidden(X)
 model.get_uncertainty(X)
 model.get_uncertainty(X, metric_name="mutual_information")
 ```

docs/README_DOCS.html

@@ -5,7 +5,7 @@
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
 
- <title>|:robot:| |:speech_balloon:| |:question:| nlp-uncertainty-zoo &#8212; nlp-uncertainty-zoo 0.9.1 documentation</title>
+ <title>|:robot:| |:speech_balloon:| |:question:| nlp-uncertainty-zoo &#8212; nlp-uncertainty-zoo 1.0.0 documentation</title>
  <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
  <link rel="stylesheet" type="text/css" href="_static/bootstrap-sphinx.css" />
  <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
@@ -35,7 +35,7 @@
  </button>
  <a class="navbar-brand" href="index.html">
  nlp-uncertainty-zoo</a>
- <span class="navbar-text navbar-version pull-left"><b>0.9.1</b></span>
+ <span class="navbar-text navbar-version pull-left"><b>1.0.0</b></span>
  </div>
 
  <div class="collapse navbar-collapse nav-collapse">
@@ -89,7 +89,7 @@
 <li class="toctree-l1"><a class="reference internal" href="nlp_uncertainty_zoo.utils.custom_types.html">Custom Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="nlp_uncertainty_zoo.utils.custom_types.html#module-nlp_uncertainty_zoo.utils.custom_types">Custom Types Module Documentation</a></li>
 <li class="toctree-l1"><a class="reference internal" href="nlp_uncertainty_zoo.utils.data.html">Data</a></li>
-<li class="toctree-l1"><a class="reference internal" href="nlp_uncertainty_zoo.utils.data.html#data-module-documentation">Data Module Documentation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="nlp_uncertainty_zoo.utils.data.html#module-nlp_uncertainty_zoo.utils.data">Data Module Documentation</a></li>
 <li class="toctree-l1"><a class="reference internal" href="nlp_uncertainty_zoo.utils.metrics.html">Uncertainty metrics</a></li>
 <li class="toctree-l1"><a class="reference internal" href="nlp_uncertainty_zoo.utils.metrics.html#module-nlp_uncertainty_zoo.utils.metrics">Metric Module Documentation</a></li>
 <li class="toctree-l1"><a class="reference internal" href="nlp_uncertainty_zoo.utils.samplers.html">Samplers</a></li>
@@ -166,19 +166,24 @@ <h1>|:robot:| |:speech_balloon:| |:question:| nlp-uncertainty-zoo<a class="head
 </pre></div>
 </div>
 <p>If you are using the repository in your academic research, please cite the paper below:</p>
-<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="nd">@article</span><span class="p">{</span><span class="n">ulmer2022exploring</span><span class="p">,</span>
- <span class="n">title</span><span class="o">=</span><span class="p">{</span><span class="n">Exploring</span> <span class="n">Predictive</span> <span class="n">Uncertainty</span> <span class="ow">and</span> <span class="n">Calibration</span> <span class="ow">in</span> <span class="n">NLP</span><span class="p">:</span> <span class="n">A</span> <span class="n">Study</span> <span class="n">on</span> <span class="n">the</span> <span class="n">Impact</span> <span class="n">of</span> <span class="n">Method</span> \<span class="o">&amp;</span> <span class="n">Data</span> <span class="n">Scarcity</span><span class="p">},</span>
- <span class="n">author</span><span class="o">=</span><span class="p">{</span><span class="n">Ulmer</span><span class="p">,</span> <span class="n">Dennis</span> <span class="ow">and</span> <span class="n">Frellsen</span><span class="p">,</span> <span class="n">Jes</span> <span class="ow">and</span> <span class="n">Hardmeier</span><span class="p">,</span> <span class="n">Christian</span><span class="p">},</span>
- <span class="n">journal</span><span class="o">=</span><span class="p">{</span><span class="n">arXiv</span> <span class="n">preprint</span> <span class="n">arXiv</span><span class="p">:</span><span class="mf">2210.15452</span><span class="p">},</span>
- <span class="n">year</span><span class="o">=</span><span class="p">{</span><span class="mi">2022</span><span class="p">}</span>
-<span class="p">}</span>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="nd">@inproceedings</span><span class="p">{</span><span class="n">ulmer</span><span class="o">-</span><span class="n">etal</span><span class="o">-</span><span class="mi">2022</span><span class="o">-</span><span class="n">exploring</span><span class="p">,</span>
+ <span class="n">title</span> <span class="o">=</span> <span class="s2">&quot;Exploring Predictive Uncertainty and Calibration in </span><span class="si">{NLP}</span><span class="s2">: A Study on the Impact of Method {\&amp;} Data Scarcity&quot;</span><span class="p">,</span>
+ <span class="n">author</span> <span class="o">=</span> <span class="s2">&quot;Ulmer, Dennis and</span>
+ <span class="n">Frellsen</span><span class="p">,</span> <span class="n">Jes</span> <span class="ow">and</span>
+ <span class="n">Hardmeier</span><span class="p">,</span> <span class="n">Christian</span><span class="s2">&quot;,</span>
+ <span class="n">booktitle</span> <span class="o">=</span> <span class="s2">&quot;Findings of the Association for Computational Linguistics: EMNLP 2022&quot;</span><span class="p">,</span>
+ <span class="n">month</span> <span class="o">=</span> <span class="n">dec</span><span class="p">,</span>
+ <span class="n">year</span> <span class="o">=</span> <span class="s2">&quot;2022&quot;</span><span class="p">,</span>
+ <span class="n">address</span> <span class="o">=</span> <span class="s2">&quot;Abu Dhabi, United Arab Emirates&quot;</span><span class="p">,</span>
+ <span class="n">publisher</span> <span class="o">=</span> <span class="s2">&quot;Association for Computational Linguistics&quot;</span><span class="p">,</span>
+ <span class="n">url</span> <span class="o">=</span> <span class="s2">&quot;https://aclanthology.org/2022.findings-emnlp.198&quot;</span><span class="p">,</span>
+ <span class="n">pages</span> <span class="o">=</span> <span class="s2">&quot;2707--2735&quot;</span><span class="p">,</span>
 </pre></div>
 </div>
-<p>Certain parts of this repository are still incomplete, but will come soon (I promise!):</p>
-<ul class="simple">
-<li><p>[x] Build proper documentation</p></li>
-<li><p>[ ] Add demo jupyter notebook</p></li>
-</ul>
+<p>}</p>
+<p>To learn more about the package, consult the documentation <a class="reference external" href="http://dennisulmer.eu/nlp-uncertainty-zoo/">here</a>,
+check a Jupyter notebook demo <a class="reference external" href="https://github.com/Kaleidophon/nlp-uncertainty-zoo/blob/main/demo.ipynb">here</a> or a Google
+collab <a class="reference external" href="https://colab.research.google.com/drive/1-Pl5lvcnpbGL2ZXLGDDNqvJB7Ew8uIsS?usp=sharing">here</a>.</p>
 <section id="included-models">
 <h2>Included models<a class="headerlink" href="#included-models" title="Permalink to this heading">¶</a></h2>
 <p>The following models are implemented in the repository. They can all be imported by using <code class="docutils literal notranslate"><span class="pre">from</span> <span class="pre">nlp-uncertainty-zoo</span> <span class="pre">import</span> <span class="pre">&lt;MODEL&gt;</span></code>.
@@ -205,7 +210,7 @@ <h2>Usage<a class="headerlink" href="#usage" title="Permalink to this heading">
 <span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">train_split</span><span class="o">=</span><span class="n">train_dataloader</span><span class="p">)</span>
 <span class="n">model</span><span class="o">.</span><span class="n">get_logits</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
 <span class="n">model</span><span class="o">.</span><span class="n">get_predictions</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
-<span class="n">model</span><span class="o">.</span><span class="n">get_sequence_representation</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
+<span class="n">model</span><span class="o">.</span><span class="n">get_sequence_representation_from_hidden</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
 <span class="n">model</span><span class="o">.</span><span class="n">get_uncertainty</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
 <span class="n">model</span><span class="o">.</span><span class="n">get_uncertainty</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">metric_name</span><span class="o">=</span><span class="s2">&quot;mutual_information&quot;</span><span class="p">)</span>
 </pre></div>

docs/README_DOCS.md

@@ -7,18 +7,23 @@ implemented in PyTorch. You can install the repository using pip:
 
 If you are using the repository in your academic research, please cite the paper below:
 
- @article{ulmer2022exploring,
-  title={Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method \& Data Scarcity},
-  author={Ulmer, Dennis and Frellsen, Jes and Hardmeier, Christian},
-  journal={arXiv preprint arXiv:2210.15452},
-  year={2022}
- }
-
-
-Certain parts of this repository are still incomplete, but will come soon (I promise!):
-
-- [x] Build proper documentation
-- [ ] Add demo jupyter notebook
+ @inproceedings{ulmer-etal-2022-exploring,
+  title = "Exploring Predictive Uncertainty and Calibration in {NLP}: A Study on the Impact of Method {\&} Data Scarcity",
+  author = "Ulmer, Dennis and
+  Frellsen, Jes and
+  Hardmeier, Christian",
+  booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
+  month = dec,
+  year = "2022",
+  address = "Abu Dhabi, United Arab Emirates",
+  publisher = "Association for Computational Linguistics",
+  url = "https://aclanthology.org/2022.findings-emnlp.198",
+  pages = "2707--2735",
+ }
+
+To learn more about the package, consult the documentation [here](http://dennisulmer.eu/nlp-uncertainty-zoo/),
+check a Jupyter notebook demo [here](https://github.com/Kaleidophon/nlp-uncertainty-zoo/blob/main/demo.ipynb) or a Google 
+collab [here](https://colab.research.google.com/drive/1-Pl5lvcnpbGL2ZXLGDDNqvJB7Ew8uIsS?usp=sharing).
 
 ### Included models
 
@@ -50,7 +55,7 @@ model = LSTMEnsemble(**network_params, ensemble_size=10, is_sequence_classifer=F
 model.fit(train_split=train_dataloader)
 model.get_logits(X)
 model.get_predictions(X)
-model.get_sequence_representation(X)
+model.get_sequence_representation_from_hidden(X)
 model.get_uncertainty(X)
 model.get_uncertainty(X, metric_name="mutual_information")
 ```

docs/_build/html/.buildinfo

@@ -1,4 +1,4 @@
 # Sphinx build info version 1
 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: f61e5bc8c4291e3961dccbf4275f2c58
+config: 18c43478377692087fcbbbfc8f44aceb
 tags: 645f666f9bcd5a90fca523b33c5a78b7

docs/_sources/README_DOCS.md.txt

@@ -7,18 +7,23 @@ implemented in PyTorch. You can install the repository using pip:
 
 If you are using the repository in your academic research, please cite the paper below:
 
- @article{ulmer2022exploring,
- title={Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method \& Data Scarcity},
- author={Ulmer, Dennis and Frellsen, Jes and Hardmeier, Christian},
- journal={arXiv preprint arXiv:2210.15452},
- year={2022}
- }
-
-
-Certain parts of this repository are still incomplete, but will come soon (I promise!):
-
-- [x] Build proper documentation
-- [ ] Add demo jupyter notebook
+ @inproceedings{ulmer-etal-2022-exploring,
+ title = "Exploring Predictive Uncertainty and Calibration in {NLP}: A Study on the Impact of Method {\&} Data Scarcity",
+ author = "Ulmer, Dennis and
+ Frellsen, Jes and
+ Hardmeier, Christian",
+ booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
+ month = dec,
+ year = "2022",
+ address = "Abu Dhabi, United Arab Emirates",
+ publisher = "Association for Computational Linguistics",
+ url = "https://aclanthology.org/2022.findings-emnlp.198",
+ pages = "2707--2735",
+ }
+
+To learn more about the package, consult the documentation [here](http://dennisulmer.eu/nlp-uncertainty-zoo/),
+check a Jupyter notebook demo [here](https://github.com/Kaleidophon/nlp-uncertainty-zoo/blob/main/demo.ipynb) or a Google 
+collab [here](https://colab.research.google.com/drive/1-Pl5lvcnpbGL2ZXLGDDNqvJB7Ew8uIsS?usp=sharing).
 
 ### Included models
 
@@ -50,7 +55,7 @@ model = LSTMEnsemble(**network_params, ensemble_size=10, is_sequence_classifer=F
 model.fit(train_split=train_dataloader)
 model.get_logits(X)
 model.get_predictions(X)
-model.get_sequence_representation(X)
+model.get_sequence_representation_from_hidden(X)
 model.get_uncertainty(X)
 model.get_uncertainty(X, metric_name="mutual_information")
 ```

docs/_sources/nlp_uncertainty_zoo.utils.uncertainty_eval.rst.txt

@@ -1,19 +1,9 @@
 Uncertainty Eval
-=================
+================
 
-Calibration and the quality of uncertainty estimates can be tricky to evaluate, since there are no gold labels like for a classification tasks.
+The quality of uncertainty estimates can be tricky to evaluate, since there are no gold labels like for a classification tasks.
 For that reason, this module contains methods for exactly this purpose.
 
-For calibration, the module implements the expected calibration error `(Naeini et al., 2015) <https://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/download/9667/9958>`_, as well as the static calibration error and
-the adaptive calibration error by `Nixon et al. (2019) <https://openaccess.thecvf.com/content_CVPRW_2019/papers/Uncertainty%20and%20Robustness%20in%20Deep%20Visual%20Learning/Nixon_Measuring_Calibration_in_Deep_Learning_CVPRW_2019_paper.pdf>`_, where the former is an extension to multiple classes, and the latter uses ranges instead of bins,
-making sure that every range contains the same number of points.
-
-.. warning::
- In `Ulmer et al. (2022) <https://arxiv.org/pdf/2210.15452.pdf>`_, we found that the SCE is not very informative for a relatively large number of classes (> 5).
-
-Furthermore, we implement the evaluation of prediction sets by `Kompa et al. (2020) <http://arxiv.org/abs/2010.03039>`_: We determine the average width of prediction sets to reach 1 - alpha probability mass (:py:func:`nlp_uncertainty_zoo.utils.uncertainty_eval.coverage_width`),
-and what percentage of prediction sets contain the correct class (:py:func:`nlp_uncertainty_zoo.utils.uncertainty_eval.coverage_percentage`).
-
 To measure the quality of general uncertainty estimates, we use the common evaluation method of defining a proxy OOD detection tasks,
 where we quantify how well we can distinguish in- and out-of-distribution inputs based on uncertainty scores given by a model.
 This is realized using the area under the receiver-operator-characteristic (:py:func:`nlp_uncertainty_zoo.utils.uncertainty_eval.aupr`) and
@@ -22,6 +12,8 @@ the area under precision-recall curve (:py:func:`nlp_uncertainty_zoo.utils.uncer
 New in `Ulmer et al. (2022) <https://arxiv.org/pdf/2210.15452.pdf>`_, it is also evaluated how indicative the uncertainty score is with the potential loss of a model.
 For this reason, this module also implements the Kendall's tau correlation coefficient (:py:func:`nlp_uncertainty_zoo.utils.uncertainty_eval.kendalls_tau`).
 
+Instances of the :py:class:`nlp_uncertainty_zoo.models.model.Model` class can be evaluated in a single function using all the above metrics with :py:func:`nlp_uncertainty_zoo.uncertainty_eval.evaluate_uncertainty`.
+
 Uncertainty Eval Module Documentation
 =====================================
 

docs/conf.py

@@ -31,7 +31,7 @@
 
 sys.path.insert(0, os.path.abspath("../"))
 sys.path.insert(0, os.path.abspath("../nlp_uncertainty_zoo"))
-__version__ = "0.9.1"
+__version__ = "1.0.0"
 
 
 # -- General configuration ---------------------------------------------------