Sentinel-3 (#182)

* Added sentinel-3

* Added summaries workflows

* feat: first pass at synergy-aod Item updates

* Removed summaries

* update sentinel-3 synergy aod collection

* update synergy-aod and add synergy-syn

* Started olci-lfr

* [olci-lfr]: fixups

* [olci-lfr]: fixups

* wip - start olci-wfr

* [olci-wfr]: Added to dataset.yaml

* [synergy] adds the vg1, v10, and vgp collections

- also adds a base Collection class that handles most changes

* [olci]: revert valid / invalid change

* [olci]: Fixed geometries

* [olci]: Fixed platform international designator

* [olci-wfr]: Clean up summaries, added new ones

* [sral] Add SRAL LAN and WAT collections

* [olci]: Use base class

* [olci]: Moved by to description

* [olci]: Moved tests

* Removed percentage summaries

* wip: adds slstr collections, many updates

* fix: eo_bands -> eo:bands

* updates and fixes

* normalize asset keys, update product names

* fix: update sral and some synergy spatial extents

* updated tests

* update: update antimeridian fix, some synergy extents

* [aod] fix asset key error and template item_asset

* fix: s3:resolution -> s3:spatial_resolution

* refactor: move winding logic to separate module

* refattor: recursive_round returns new list instead of modifying in place

* fix: correct olci-lfr description per Tom's comment

* update: update antimeridian package version

* remove collection summary data

* add Dockerfile

* make slstr-lst strip geometries cover the whole globe

- patch for bad geometries

* [slstr-frp] add description

* update frp, add lst

* [slstr-wst] add description

* update: use "bad" slstr-lst geometries

* fix: fix winding detection

- accomodate point on center latitude

* add sral descriptions, standardize existing descriptions

* Updated image

* removed limits

* add synergy vegetation descriptions, update templates

* fix: minor title and short_description edits

* Added update workflow

* Add msft:group_id to all Sentinel-3 collections

---------

Co-authored-by: pjhartzell <[email protected]>
Co-authored-by: Rob Emanuele <[email protected]>

datasets/sentinel-3/Dockerfile

@@ -0,0 +1,74 @@
+FROM ubuntu:20.04
+
+# Setup timezone info
+ENV TZ=UTC
+
+ENV LC_ALL=C.UTF-8
+ENV LANG=C.UTF-8
+
+RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
+
+RUN apt-get update && apt-get install -y software-properties-common
+
+RUN add-apt-repository ppa:ubuntugis/ppa && \
+    apt-get update && \
+    apt-get install -y build-essential python3-dev python3-pip \
+    jq unzip ca-certificates wget curl git && \
+    apt-get autoremove && apt-get autoclean && apt-get clean
+
+RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
+
+# See https://github.com/mapbox/rasterio/issues/1289
+ENV CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
+
+# Install Python 3.8
+RUN curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh" \
+    && bash "Mambaforge-$(uname)-$(uname -m).sh" -b -p /opt/conda \
+    && rm -rf "Mambaforge-$(uname)-$(uname -m).sh"
+
+ENV PATH /opt/conda/bin:$PATH
+ENV LD_LIBRARY_PATH /opt/conda/lib/:$LD_LIBRARY_PATH
+
+RUN mamba install -y -c conda-forge python=3.8 gdal=3.3.3 pip setuptools cython numpy==1.21.5
+
+RUN python -m pip install --upgrade pip
+
+# Install common packages
+COPY requirements-task-base.txt /tmp/requirements.txt
+RUN python -m pip install --no-build-isolation -r /tmp/requirements.txt
+
+#
+# Copy and install packages
+#
+
+COPY pctasks/core /opt/src/pctasks/core
+RUN cd /opt/src/pctasks/core && \
+    pip install .
+
+COPY pctasks/cli /opt/src/pctasks/cli
+RUN cd /opt/src/pctasks/cli && \
+    pip install .
+
+COPY pctasks/task /opt/src/pctasks/task
+RUN cd /opt/src/pctasks/task && \
+    pip install .
+
+COPY pctasks/client /opt/src/pctasks/client
+RUN cd /opt/src/pctasks/client && \
+    pip install .
+
+COPY pctasks/ingest /opt/src/pctasks/ingest
+RUN cd /opt/src/pctasks/ingest && \
+    pip install .
+
+COPY pctasks/dataset /opt/src/pctasks/dataset
+RUN cd /opt/src/pctasks/dataset && \
+    pip install .
+
+COPY ./datasets/sentinel-3/requirements.txt /opt/src/datasets/sentinel-3/requirements.txt
+RUN python3 -m pip install -r /opt/src/datasets/sentinel-3/requirements.txt
+
+# Setup Python Path to allow import of test modules
+ENV PYTHONPATH=/opt/src:$PYTHONPATH
+
+WORKDIR /opt/src

datasets/sentinel-3/README.md

@@ -1 +1,47 @@
-# Work in progress - in a broken state
+# Sentinel-3
+
+## Dataset structure
+
+This dataset has many different products. We create one STAC collection per product. The collections are under the `collection` folder.
+
+```
+collection/sentinel-3-olci-lfr-l2-netcdf
+collection/sentinel-3-olci-wfr-l2-netcdf
+collection/sentinel-3-slstr-frp-l2-netcdf
+collection/sentinel-3-slstr-lst-l2-netcdf
+collection/sentinel-3-slstr-wst-l2-netcdf
+collection/sentinel-3-sral-lan-l2-netcdf
+collection/sentinel-3-sral-wat-l2-netcdf
+collection/sentinel-3-synergy-aod-l2-netcdf
+collection/sentinel-3-synergy-syn-l2-netcdf
+collection/sentinel-3-synergy-v10-l2-netcdf
+collection/sentinel-3-synergy-vg1-l2-netcdf
+collection/sentinel-3-synergy-vgp-l2-netcdf
+```
+
+The package `sentinel_3` contains the python code for making the STAC items.
+Each collection has its own module
+(`sentinel_3/sentinel_3_olci_lfr_l2_netcdf.py`). The name of the module should
+match the name of the collection. The module should have a class named
+`Collection` that is the pctasks `dataset.Collection` subclass.
+
+## Tests
+
+The `datasets/sentinel-3/tests` directory contains some tests. Run those with
+
+
+```
+$ PYTHONPATH=datasets/sentinel-3 python -m pytest datasets/sentinel-3/tests/
+```
+
+### Dynamic updates
+
+```console
+$ ls datasets/sentinel-3/collection/ | xargs -I {}  \
+    pctasks dataset process-items '${{ args.since }}' \
+        -d datasets/sentinel-3/update.yaml \
+        -c {} \
+        --workflow-id={}-update \
+        --is-update-workflow \
+        --upsert
+```

datasets/sentinel-3/collection/sentinel-3-olci-lfr-l2-netcdf/description.md

@@ -0,0 +1,27 @@
+This collection provides Sentinel-3 Full Resolution [OLCI Level-2 Land][olci-l2] products containing data on global vegetation, chlorophyll, and water vapor.
+
+## Data files
+
+This dataset includes data on three primary variables:
+
+* OLCI global vegetation index file
+* terrestrial Chlorophyll index file
+* integrated water vapor over water file.
+
+Each variable is contained within a separate NetCDF file, and is cataloged as an asset in each Item.
+
+Several associated variables are also provided in the annotations data files:
+
+* rectified reflectance for red and NIR channels (RC681 and RC865)
+* classification, quality and science flags (LQSF)
+* common data such as the ortho-geolocation of land pixels, solar and satellite angles, atmospheric and meteorological data, time stamp or instrument information. These variables are inherited from Level-1B products.
+
+This full resolution product offers a spatial sampling of approximately 300 m.
+
+## Processing overview
+
+The values in the data files have been converted from Top of Atmosphere radiance to reflectance, and include various corrections for gaseous absorption and pixel classification. More information about the product and data processing can be found in the [User Guide](https://sentinel.esa.int/web/sentinel/user-guides/sentinel-3-olci/product-types/level-2-land) and [Technical Guide](https://sentinel.esa.int/web/sentinel/technical-guides/sentinel-3-olci/level-2/processing).
+
+This Collection contains Level-2 data in NetCDF files from April 2016 to present.
+
+[olci-l2]: https://sentinel.esa.int/web/sentinel/technical-guides/sentinel-3-olci/level-2/land-products