docs compiling (#132)

* cleanup environment and update setup.cfg metadata (#129) remove python 3.5 from testing (environment can't resolve) fixes #82 fixes #90 fixes #126 fixes #130 fixes #131 fixes #133 * update docs * minor api changes
erdc · Apr 12, 2019 · 9d5a856 · 9d5a856
1 parent 4553be9
commit 9d5a856
Show file tree

Hide file tree

Showing 44 changed files with 1,456 additions and 788 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -9,13 +9,8 @@ notifications:
 
 matrix:
   include:
-    - os: linux
-      env: PYTHON_VERSION="3.5"
     - os: linux
       env: PYTHON_VERSION="3.6"
-    - os: osx
-      osx_image: xcode9.4
-      env: PYTHON_VERSION="3.5"
     - os: osx
       osx_image: xcode9.4
       env: PYTHON_VERSION="3.6"

diff --git a/conda_environment.yml b/conda_environment.yml
@@ -17,14 +17,14 @@ dependencies:
 
     # core dependencies
     - geojson
-    - pandas<=0.22.0
+    - pandas
     - geopandas
     - numpy
-    - param>=1.8.1
+    - param>=1.9.0
     - pint
     - pony
     - pyyaml
-    - ulmo>=0.8.3.2
+    - ulmo>=0.8.5
 
     # task dependencies
     - tornado
@@ -44,11 +44,11 @@ dependencies:
 
     # notebook examples dependencies
     - jupyter
-    - pyviz/label/dev::holoviews
-    - pyviz/label/dev::geoviews
+    - holoviews
+    - geoviews
     - hvplot
     - paramnb
-    - parambokeh
+    - pyviz::panel
 
     # test dependencies
     - pytest

diff --git a/docs/_static/.gitkeep b/docs/_static/.gitkeep
diff --git a/docs/conf.py b/docs/conf.py
@@ -16,6 +16,8 @@
 import sphinx_rtd_theme
 from unittest.mock import MagicMock
 
+import quest
+
 
 class Mock(MagicMock):
     @classmethod
@@ -40,6 +42,7 @@ def __getattr__(cls, name):
     'param',
     'pint',
     'psutil',
+    'quest.util.param_util'
 ]
 sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)
 
@@ -62,6 +65,7 @@ def __getattr__(cls, name):
     'sphinx.ext.mathjax',
     'sphinx.ext.viewcode',
     'sphinx.ext.napoleon',
+    'sphinx.ext.todo',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -88,9 +92,10 @@ def __getattr__(cls, name):
 # built documents.
 #
 # The short X.Y version.
-version = '0.5'
+
+version = quest.__version__
 # The full version, including alpha/beta/rc tags.
-release = '0.5'
+release = version
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

diff --git a/docs/core_concepts.rst b/docs/core_concepts.rst
@@ -1,7 +1,33 @@
 Core Concepts
--------------
+=============
 
-Quest uses a hierarchical structure to organize and manage datasets, and data sources. The dataset hierarchy begins with `projects` which contains `collections` which contain `features` which have `datasets`. A more detailed description of each level is given below.
+Quest is a python library designed to automate the following data management tasks:
+
+    * Discovery
+    * Retrieval
+    * Organization
+    * Transformation
+    * Archival
+
+
+At the heart of all of these tasks are `datasets`_. Each of the tasks listed above involves finding, getting, storing, changing, or sharing a ``dataset``. The underlying concepts for how Quest accomplishes these five tasks will be described below and are grouped into the following three sections:
+
+    * `Local Data Organization`_
+    * `Data Transformations`_
+    * `Data Repositories`_
+
+        * Discovery
+        * Retrieval
+        * Archival
+
+.. _core-concepts-data-organization:
+
+Local Data Organization
+-----------------------
+
+Quest uses a hierarchical structure to organize and manage datasets, and data sources. The dataset hierarchy begins with `projects`_ which contains `collections`_ which have `datasets`_. A more detailed description of each level is given below.
+
+.. _core-concepts-projects:
 
 Projects
 ^^^^^^^^
@@ -12,40 +38,71 @@ currently the api does not allow copying data from one project to another.
 
 Physically, a project maps to a folder on the computer. All data and metadata
 associated with a project is stored under the project folder. The metadata is
-stored in a sqlite database.exi
+stored in a sqlite database.
+
+.. _core-concepts-collections:
 
 Collections
 ^^^^^^^^^^^
 
 Collections are a way of organizing data within a project. Collection names are
 unique and the collection name maps directly to a folder name in the project folder.
 
-Features
-^^^^^^^^
-
-Features are a unique identifiers that indicate a group of datasets. Typically,
-these are geospatial locations, i.e., monitoring stations, counties, lakes,
-roads at which data exists. Features can also just be a tag or name to group data
-that does not have a geospatial component (i.e. geotypical datasets). Features
-are always either part of a collection or part of a web service.
+.. _core-concepts-datasets:
 
 Datasets
 ^^^^^^^^
 
-These are the actual individual data files or in some cases a folder of data.
-Datasets are always located at a feature. Currently a dataset can only be
-attached to a single feature by design.
+These are the actual individual data files or in some cases a folder of data. Datasets have associated metadata that is stored in the project directory.
+
+.. _core-concepts-data-transformations:
+
+Data Transformations
+--------------------
 
-____
+Quest facilitates transforming data through the use of `tools`_. Some examples of the kinds of transformations that Quest can do include merging datasets, aggregating data within a dataset, or changing the format that the data is stored in.
 
-Data sources are organized into `providers` which contain one or more `services.
+.. _core-concepts-tools:
+
+Tools
+^^^^^
+Quest ``tools`` are a way to perform some kind of operation on data. It is important to note that a ``tool`` will never perform "in-place" changes the datasets that it operates on. This means that datasets that are passed to a ``tool`` will remain unchanged, and the ``tool`` will create new datasets that have the transformed data. New ``tools`` can be added to Quest through :ref:`tool-plugins`.
+
+Tools define a set of options that a user must specify when using the tool.
+
+.. _core-concepts-data-repositories:
+
+Data Repositories
+-----------------
+
+When Quest is used to search for data it searches among all of the data repositories or data `providers`_ that are registered with Quest. Similar to `Tools`_ `Providers`_ are added to Quest as plugins (see :ref:`provider-plugins`). `Providers`_ contain one or more `services`_. `Services`_ provide an interface for a single data product. Each service has a :ref:`core-concepts-catalogs`, which stores metadata about the datasets that are available from that service and is what enables Quest to search for data.
+
+.. _core-concepts-providers:
 
 Providers
-~~~~~~~~~
+^^^^^^^^^
 
-Data providers are the top level source of data. Providers are composed of one or more `Services`_, and typically represent an organization or specific part of an organization that provides data. In Quest Providers are a way of grouping related services.
+Data `providers` are the top level source of data. Providers are composed of one or more `Services`_, and typically represent an organization or specific part of an organization that provides data. In Quest, `providers` are a way of grouping related services.
+
+.. _core-concepts-services:
 
 Services
 ^^^^^^^^
 
-A data service is a specific type or channel of data that is offered from a `Provider`_, and are the primary means of ingesting data into Quest.
+A data service is a specific type or channel of data that is offered from a `Providers`_, and are the primary means of ingesting data into Quest.
+
+.. _core-concepts-catalogs:
+
+Catalogs
+^^^^^^^^
+
+.. _core-concepts-catalog-entries:
+
+Catalog Entries
+^^^^^^^^^^^^^^^
+
+Catalog Entries are a unique identifiers that indicate a group of datasets. Typically,
+these are geospatial locations, i.e., monitoring stations, counties, lakes,
+roads at which data exists. Features can also just be a tag or name to group data
+that does not have a geospatial component (i.e. geotypical datasets). Features
+are always either part of a collection or part of a web service.
diff --git a/docs/developers/design.rst b/docs/developers/design.rst
@@ -36,16 +36,16 @@ Any settings that are not set explicitly are given default values
 
 Description of Settings:
 
-======================= ======================================================================= ====================================
-Variable Name           Description                                                             Default
------------------------ ----------------------------------------------------------------------- ------------------------------------
-QUEST_BASE_DIR            Base directory to save quest data/metadata                                determined by appdirs python package
+========================= ======================================================================= ====================================
+Variable Name             Description                                                             Default
+------------------------- ----------------------------------------------------------------------- ------------------------------------
+QUEST_BASE_DIR            Base directory to save quest data/metadata                              determined by appdirs python package
 QUEST_CACHE_DIR           Location to save cached data/metadata                                   QUEST_BASE_DIR/cache/
 QUEST_PROJECT_FILE        Name of project metadata file                                           quest_project.yml
 QUEST_PROJECTS_INDEX_FILE Name of projects index file listing available projects and their paths  quest_projects_index.yml
-QUEST_CONFIG_FILE         Name of quest_config file that these settings are saved in                quest_config.yml
-QUEST_USER_SERVICES       list of web/file uris to user defined Quest services                      None
-======================= ======================================================================= ====================================
+QUEST_CONFIG_FILE         Name of quest_config file that these settings are saved in              quest_config.yml
+QUEST_USER_SERVICES       list of web/file uris to user defined Quest services                    None
+========================= ======================================================================= ====================================
 
 You can add any extra settings needed by a plugin here as well using the keyword:arg structure.
 

diff --git a/docs/developers/index.rst b/docs/developers/index.rst
@@ -1,5 +1,5 @@
-Quest Developer Documentation
-=============================
+Developer Documentation
+=======================
 
 This documentation is geared toward those wanting to contribute to the Quest source code.
 

diff --git a/docs/developers/plugin.rst b/docs/developers/plugin.rst
@@ -60,21 +60,21 @@ A data service plugin must subclass the ``ServiceBase`` class (or one of it's su
 
 The content of ``ExampleServiceBase`` has not yet been fully implemented. The above example simply illustrates the structure. All of the attributes and methods shown in the ``ExampleServiceBase`` will need to be implemented either in this class directly or in the services that subclass this base class. The specifics of how this are done will be different for each plugin, but the next step, `3. Service Classes`_ will demonstrate one way to do it.
 
-.. tip::
 
-    Specialized Service Base Subclasses
+Specialized Service Base Subclasses
+...................................
 
-    There are a couple of special cases that apply to services from various providers. To allow all of these services to use the same codebase a couple of other base classes are available that can be used in place of the ``ServiceBase``.
+There are a couple of special cases that apply to services from various providers. To allow all of these services to use the same codebase a couple of other base classes are available that can be used in place of the ``ServiceBase``.
 
-    TimePeriodServiceBase
-    ~~~~~~~~~~~~~~~~~~~~~
+TimePeriodServiceBase
+~~~~~~~~~~~~~~~~~~~~~
 
-    This base class simply adds two parameters, a `start` and `end` date to represent the time period for the data being requested (see `d. Specify the Download Options`_).
+This base class simply adds two parameters, a `start` and `end` date to represent the time period for the data being requested (see `d. Specify the Download Options`_).
 
-    SingleFileServiceBase
-    ~~~~~~~~~~~~~~~~~~~~~
+SingleFileServiceBase
+~~~~~~~~~~~~~~~~~~~~~
 
-    This base class implements the `download` method for services where there is simply a download url that links to a single zip file that contains the data.
+This base class implements the `download` method for services where there is simply a download url that links to a single zip file that contains the data.
 
 3. Service Classes
 ------------------
@@ -189,7 +189,7 @@ d. Specify the Download Options
 ...............................
 Data sources's APIs often allow various options to be specified to determine what data to download, what format it should be in, etc.
 
-The download options that are needed for each service are defined using the Python library `Param (https://ioam.github.io/param/)`_. This library enables parameters to have features like type and range checking, documentation strings, default values, etc. Refer to the `Param documentation (https://ioam.github.io/param/)`_ for more information.
+The download options that are needed for each service are defined using the Python library `Param <https://ioam.github.io/param/>`_. This library enables parameters to have features like type and range checking, documentation strings, default values, etc. Refer to the `Param documentation <https://ioam.github.io/param/>`_ for more information.
 
 
 .. todo::