Skip to content
This repository has been archived by the owner on Apr 16, 2024. It is now read-only.

Commit

Permalink
NEW: adds new distro-specific install instructions (#65)
Browse files Browse the repository at this point in the history
  • Loading branch information
lizgehret authored Oct 12, 2023
1 parent 37e985a commit 919fd65
Show file tree
Hide file tree
Showing 8 changed files with 85 additions and 33 deletions.
1 change: 1 addition & 0 deletions source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
'sphinx.ext.autodoc',
'sphinx.ext.napoleon',
'sphinx.ext.doctest',
'sphinx_tabs.tabs'
]

# Add any paths that contain templates here, relative to this directory.
Expand Down
11 changes: 11 additions & 0 deletions source/glossary.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,13 @@ it without the accompanying context provided by the rest of this documentation.
A string that represents a particular layout of files and or directories
as well as how their contents will be structured.

Distribution
A collection of QIIME 2 plugins that are designed to be installed together.
These are generally grouped by a theme. For example, the `Amplicon Distribution`_
provides a collection of plugins for analysis of microbiome amplicon data,
while the `Shotgun Distribution`_ provides a collection of plugins
for analysis of microbiome shotgun metagenomics data.

Format
A string that represents a particular file format.

Expand Down Expand Up @@ -133,3 +140,7 @@ it without the accompanying context provided by the rest of this documentation.
A visualizer accepts some combination of QIIME 2 :term:`artifacts
<Artifact>` and :term:`parameters <Parameter>` as :term:`input`, and
produces exactly one :term:`visualization` as :term:`output`.


.. _`Amplicon Distribution`: https://docs.qiime2.org/2023.9/install/#qiime-2-2023-9-amplicon-distribution
.. _`Shotgun Distribution`: https://docs.qiime2.org/2023.9/install/#qiime-2-2023-9-shotgun-distribution
1 change: 0 additions & 1 deletion source/img/packaging-infra.svg

This file was deleted.

1 change: 1 addition & 0 deletions source/img/packaging_infra.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 2 additions & 1 deletion source/metadata.rst
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ How can the Metadata API Help Me?
---------------------------------
The :doc:`Metadata API <api-reference/metadata>` has many interesting features - here are some of the more
commonly utlitized elements amongst the core plugins.
commonly utlitized elements amongst the plugins within the `Amplicon Distribution`_.
Merging Metadata
................
Expand Down Expand Up @@ -174,6 +174,7 @@ called |metadata_tabulate|_, which renders an interactive table of the metadata
in question. Cool!
.. _`Amplicon Distribution`: https://docs.qiime2.org/2023.9/install/#qiime-2-2023-9-amplicon-distribution
.. _`Metadata in QIIME 2`: https://docs.qiime2.org/2018.4/tutorials/metadata/
.. _`familiarizing yourself`: https://docs.qiime2.org/2018.4/tutorials/metadata/
.. |pd.Dataframe| replace:: ``pd.Dataframe``
Expand Down
27 changes: 13 additions & 14 deletions source/publishing/infrastructure.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ There are many connected components in the QIIME 2 packaging infrastructure
topology. The following diagram attempts to summarize the aspects of these
elements that have been discussed elsewhere in this section of the dev-docs.

.. figure:: ../img/packaging-infra.svg
.. figure:: ../img/packaging_infra.svg
:width: 350

Detailed description of diagram elements
Expand Down Expand Up @@ -46,23 +46,20 @@ listens for new package builds from instances of ``action-library-packaging``,
and when a new build is found, it triggers all of the downstream steps
necessary for publishing the new package build.

Every plugin is associated with one or more distributions - a collective of
Every plugin is associated with one or more `Distributions`_ - a collective of
related QIIME 2 packages - when a new plugin package build is found, Library
will prepare new versions of the ``conda_build_config.yaml`` file. More on this
below.

node: packages.qiime2.org/qiime2/20XY.Z/tested
----------------------------------------------
node: packages.qiime2.org/qiime2/20XY.Z/<distribution>/staged
-------------------------------------------------------------

This node corresponds to a self-hosted conda server run by the QIIME 2 team,
for example: https://packages.qiime2.org/qiime2/2022.2/tested/.
for example: https://packages.qiime2.org/qiime2/2023.9/amplicon/staged.

This is where the packages go after ``action-library-packaging`` informs
Library of the package build's existence.

Please note, the "tested" channel is for packages builds that haven't yet been
integrated into a QIIME 2 distribution (see following node steps).

node: package integration repo
------------------------------

Expand All @@ -83,15 +80,15 @@ dictionary that keeps track of individual package build versions, as well as
version specifications for common external dependencies, such as ``pandas`` and
``scikit-bio``. An example ``conda_build_config.yaml`` can be `seen here`_.

node: packages.qiime2.org/qiime2/20XY.Z/staged
node: packages.qiime2.org/qiime2/20XY.Z/<distribution>/passed
----------------------------------------------

This node corresponds to a self-hosted conda server run by the QIIME 2 team,
for example: https://packages.qiime2.org/qiime2/2022.2/staged/core/.
for example: https://packages.qiime2.org/qiime2/2023.9/amplicon/passed/.

This is where the packages go after ``package-integration`` informs Library
that the package build was able to successfully "integrate" with all the other
packages in a given distribution. This is no small feat - QIIME 2 plugins have
packages in a given `Distribution`_. This is no small feat - QIIME 2 plugins have
all sorts of different dependencies that must be coordinated with each other,
and it is easy for conflicts to occur.

Expand All @@ -101,7 +98,7 @@ Final publication
There is one final step, not listed in the diagram above, because it occurs on
a fixed schedule, as opposed to being triggered by new package builds - the
"cron" workflows that run in ``package-integration`` will find the latest
metapackage for all known distributions, and "destructure" the metapackage,
metapackage for all known `Distributions`_, and "destructure" the metapackage,
testing each individual packages that was bundled up inside the metapackage.

This step is important because even though a package version might've
Expand All @@ -112,9 +109,11 @@ dependencies.

Once this step is successful, ``package-integration`` alerts Library, which
then publishes the final package build to
``https://packages.qiime2.org/qiime2/20XY.Z/passed/core/`` - for example:
``https://packages.qiime2.org/qiime2/20XY.Z/<distribution>/released/`` - for example:

https://packages.qiime2.org/qiime2/2022.2/passed/core/
https://packages.qiime2.org/qiime2/2022.2/amplicon/released/

.. _`Distribution`: https://docs.qiime2.org/2023.9/install/#qiime-2-2023-9-distributions
.. _`Distributions`: https://docs.qiime2.org/2023.9/install/#qiime-2-2023-9-distributions
.. _`PEP 440`: https://peps.python.org/pep-0440/
.. _`seen here`: https://github.com/qiime2/package-integration/blob/c521d68d9c66e9c309214d5b2aac7474192b324f/2022.2/tested/conda_build_config.yaml
7 changes: 4 additions & 3 deletions source/publishing/tutorial.rst
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ This particular recipe file is stored in a directory called ``ci/meta``.
script: make install
Next we actually tell conda what command to run in order to build this plugin.
For the plugins in the core distribution, we like to use Makefiles for
For the plugins in the `Amplicon Distribution`_, we like to use Makefiles for
standardizing a lot of our commands, this particular make directive is short
for ``python setup.py install``.

Expand Down Expand Up @@ -173,7 +173,7 @@ The full copy of the GitHub workflow that uses this actions
This first section instructs GitHub when to run this workflow. At the very least
it needs to run on pushes to whatever the default branch is in your repository.
For core distribution plugins, we like to also run the workflow any time a
For `Amplicon Distribution`_ plugins, we like to also run the workflow any time a
pull request is opened - this has the benefit of ensuring the conda package can
successfully build, but won't actually deploy the changes to Library.

Expand Down Expand Up @@ -205,7 +205,7 @@ ensure that GitHub checks out the latest copy of the source code.
- name: set up git repo for versioneer
run: git fetch --depth=1 origin +refs/tags/*:refs/tags/*
For core distribution plugins, we use a tool called `versioneer`_ which allows
For `Amplicon Distribution`_ plugins, we use a tool called `versioneer`_ which allows
us to manage our version identifiers using git tags, but that is not a
requirement for your plugin. Because of this, we have to tell this step how
much of the repo to check out.
Expand Down Expand Up @@ -238,6 +238,7 @@ repo: https://github.com/qiime2/package-integration.

Congratulations!

.. _`Amplicon Distribution`: https://docs.qiime2.org/2023.9/install/#qiime-2-2023-9-amplicon-distribution
.. _`GitHub Actions`: https://github.com/features/actions
.. _`recipe file`: https://github.com/qiime2/q2-cutadapt/blob/a6c7b6a26c373791edbfc73d13688996550e8233/ci/recipe/meta.yaml
.. _`GitHub Secret`: https://docs.github.com/en/actions/security-guides/encrypted-secrets
Expand Down
67 changes: 53 additions & 14 deletions source/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,28 +21,66 @@ After installing Miniconda and opening a new terminal, make sure you're running
Install QIIME 2 within a ``conda`` environment
----------------------------------------------

Once you have Miniconda installed, create a ``conda`` environment and install the latest QIIME 2 staging distribution within the environment. We **highly** recommend creating a *new* environment specifically for development purposes. You can choose whatever name you'd like for the environment. In this example, we'll name the environment ``qiime2-dev``.
Once you have Miniconda installed, create a ``conda`` environment and install the latest passing QIIME 2 `Distribution`_ of your choice within the environment. We **highly** recommend creating a *new* environment specifically for the QIIME 2 Distribution you will be using for development purposes. You can choose whatever name you'd like for the environment. In this example, we'll name the environment ``qiime2-dev``.

Please choose the installation instructions that are appropriate for your platform (macOS vs Linux).
Please choose the installation instructions that are appropriate for your platform (macOS vs Linux) and desired QIIME 2 Distribution.


macOS
.....
.. tabs::

.. code-block:: bash
.. tab:: Amplicon Distribution

wget https://raw.githubusercontent.com/qiime2/environment-files/master/latest/staging/qiime2-latest-py38-osx-conda.yml
conda env create -n qiime2-dev --file qiime2-latest-py38-osx-conda.yml
rm qiime2-latest-py38-osx-conda.yml
macOS

Linux
.....
.. code-block:: bash
.. code-block:: bash
wget https://raw.githubusercontent.com/qiime2/distributions/dev/latest/passed/qiime2-amplicon-macos-latest-conda.yml
conda env create -n qiime2-dev --file qiime2-amplicon-macos-latest-conda.yml
rm qiime2-amplicon-macos-latest-conda.yml
Linux

.. code-block:: bash
wget https://raw.githubusercontent.com/qiime2/distributions/dev/latest/passed/qiime2-amplicon-ubuntu-latest-conda.yml
conda env create -n qiime2-dev --file qiime2-amplicon-ubuntu-latest-conda.yml
rm qiime2-amplicon-ubuntu-latest-conda.yml
.. tab:: Shotgun Distribution

macOS

.. code-block:: bash
wget https://raw.githubusercontent.com/qiime2/distributions/dev/latest/passed/qiime2-shotgun-macos-latest-conda.yml
conda env create -n qiime2-dev --file qiime2-shotgun-macos-latest-conda.yml
rm qiime2-shotgun-macos-latest-conda.yml
Linux

.. code-block:: bash
wget https://raw.githubusercontent.com/qiime2/distributions/dev/latest/passed/qiime2-shotgun-ubuntu-latest-conda.yml
conda env create -n qiime2-dev --file qiime2-shotgun-ubuntu-latest-conda.yml
rm qiime2-shotgun-ubuntu-latest-conda.yml
.. tab:: Tiny Distribution

macOS

.. code-block:: bash
wget https://raw.githubusercontent.com/qiime2/distributions/dev/latest/passed/qiime2-tiny-macos-latest-conda.yml
conda env create -n qiime2-dev --file qiime2-tiny-macos-latest-conda.yml
rm qiime2-tiny-macos-latest-conda.yml
Linux

.. code-block:: bash
wget https://raw.githubusercontent.com/qiime2/environment-files/master/latest/staging/qiime2-latest-py38-linux-conda.yml
conda env create -n qiime2-dev --file qiime2-latest-py38-linux-conda.yml
rm qiime2-latest-py38-linux-conda.yml
wget https://raw.githubusercontent.com/qiime2/distributions/dev/latest/passed/qiime2-tiny-ubuntu-latest-conda.yml
conda env create -n qiime2-dev --file qiime2-tiny-ubuntu-latest-conda.yml
rm qiime2-tiny-ubuntu-latest-conda.yml
Activate the ``conda`` environment
----------------------------------
Expand Down Expand Up @@ -95,6 +133,7 @@ If you want to make changes to the `framework`_, `q2cli`_, or any of the `offici
Congratulations! You should now have a working development environment - time to start hacking!

.. _`Miniconda`: https://conda.io/miniconda.html
.. _`Distribution`: https://docs.qiime2.org/2023.9/install/#qiime-2-2023-9-distributions
.. _`framework`: https://github.com/qiime2/qiime2
.. _`q2cli`: https://github.com/qiime2/q2cli
.. _`official plugins`: https://github.com/qiime2?q=plugin+in%3Areadme

0 comments on commit 919fd65

Please sign in to comment.