From 66fca3cd0e8ccea02b037922a2b58dc9fbfd7ca4 Mon Sep 17 00:00:00 2001 From: Roberto Del Prete <71963566+sirbastiano@users.noreply.github.com> Date: Wed, 2 Aug 2023 15:00:02 +0200 Subject: [PATCH 1/3] commit --- README.md | 344 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 177 insertions(+), 167 deletions(-) diff --git a/README.md b/README.md index 0fd17db..c52e2b1 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,13 @@ [![Tests](https://github.com/ESA-PhiLab/PyRawS/actions/workflows/run_tests.yml/badge.svg)](https://github.com/ESA-PhiLab/PyRawS/actions/workflows/run_tests.yml) # PyRawS -![Alt Text](resources/images/PyRawS_logo.png) + +

+ Alt Text +

+ +
+ ## About the project `Python for RAW Sentinel2 data (PyRawS)` is a powerful open-source Python package that provides a comprehensive set of tools for working with [Sentinel-2 Raw data](#sentinel-2-raw-data)🔬. 1 @@ -16,9 +22,11 @@ The package is written in Python and is open source💻, making it easy to use a The systme is based on [pytorch]("https://pytorch.org/"), which be installed with `CUDA` support, to enable GPU acceleation. -NB: What we call raw data in this project are Sentinel-2 data generated by the decompression and metadata addition of Sentinel-2 L0 data. Because of that, with the exception of the effects due to onboard equalization and lossy compression, they are the most similar version of the rawest form of data acquired by the satellite's sensors. Both the compression and equalization are applied onboard the satellite to reduce the amount of data transmitted to the ground station. For easy naming convention, this repo refer to the term "Raw" as the products decompressed with ancillary information appended. For further information browse our paper at https://arxiv.org/abs/2305.11891 ↩ +> NB: What we call raw data in this project are Sentinel-2 data generated by the decompression and metadata addition of Sentinel-2 L0 data. Because of that, with the exception of the effects due to onboard equalization and lossy compression, they are the most similar version of the rawest form of data acquired by the satellite's sensors. Both the compression and equalization are applied onboard the satellite to reduce the amount of data transmitted to the ground station. For easy naming convention, this repo refer to the term "Raw" as the products decompressed with ancillary information appended. For further information browse our paper at https://arxiv.org/abs/2305.11891 ↩ + +> *(Disclaimer: 🚧 This project is currently under development 🚧)* + -*(Disclaimer: This project is currently under development.)* @@ -29,24 +37,24 @@ NB: What we call raw data in this project are Sentinel-2 data generated by the d
  • Content of the repository
  • Installation -
  • PyRawS databases
    • +
  • PyRawS databases
  • Raw events and Raw granules
  • Quickstart
  • Glossary
    • @@ -57,120 +65,122 @@ NB: What we call raw data in this project are Sentinel-2 data generated by the d -## Sentinel-2 Raw granules and events +
    +
    ## Content of the repository -The PyRawS repository includes the following directories: +The PyRawS repository is structured in the following directories: +| Directory Name | Description | Subdirectories and Contents | +|-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| +| quickstart | Contains some subdirectories including examples. | - `API demonstration`: A [jupyter notebook](https://jupyter-notebook.readthedocs.io/en/stable/index.html) to demonstrate PyRawS API. | +| pyraws | Contains the PyRawS package. | - `database`: Various [PyRawS database](#PyRawS-databases) and other databases. | +| | | - `raw`: Includes the `Raw_event` and `Raw_granule` classes used to model [Sentinel-2 Raw events](#sentinel-2-raw-event) and [Sentinel-2 Raw granules](#sentinel-2-raw-granule). | +| | | - `l1`: Contains the `L1_event` and `L1_tiles` classes used to model [Sentinel-2 L1C events](#sentinel-2-l1c-event) and [Sentinel-2 L1C tiles](#sentinel-2-l1c-tile). | +| | | - `utils`: Contains some utilities for the PYRAW package. | +| resources | Contains various resources (i.e., images for the README). | - | +| scripts_and_studies | Contains various scripts and code for different studies realized to create the `THRAWS` dataset. | - `coregistration_study`: Contains utils used to perform the coregistration study and design the [coarse coregistration](#coarse-coregistration) technique. | +| | | - `dataset_preparation`: Contains scripts and other files used to design the `THRAWS` files (i.e., download data, select events, and others). | +| | | - `hta_detection_algorithms`: Contains custom and simplified implementation of various high-thermal-anomalies-detection algorithms, including [1](#ref1) used to design the `THRAWS` dataset. | +| | | - `runscripts`: Contains some runscripts and utils to crop [Sentinel-2 L1C tiles](#sentinel-2-l1c-tile) and generate useful images and export tif. | +| | | - `granules_filtering`: Contains a script to run and [[1]](#ref1) on [cropped Sentinel-2 L1-C tiles](#sentinel-2-l1c-tile), extract bounding boxes and map them to the correspondent [Raw granules](#sentinel-2-raw-granule). | +| | | - `download_thraws`: Contains the utility for the download of the dataset [THRAWS](https://zenodo.org/record/7908728). | -* **quickstart**: it contains some subdirectories including examples: - 1. `API demonstration`: it contains a [jupyter notebook](https://jupyter-notebook.readthedocs.io/en/stable/index.html) to demonstrate PyRawS API. -* **PyRawS**: it contains the PyRawS package. The latter is made of the following subdirectories: - 1. `database`: it contains various [PyRawS database](#PyRawS-databases) and other databases. - 2. `raw`: it includes the `Raw_event` and `Raw_granule` classes used to model, respectively, [Sentinel-2 Raw events](#sentinel-2-raw-event) and [Sentinel-2 Raw granules](#sentinel-2-raw-granule). - 3. `l1`: it contains the `L1_event` and `L1_tiles` classes used to model, respectively, [Sentinel-2 L1C events](#sentinel-2-l1c-event) and [Sentinel-2 L1C tiles](#sentinel-2-l1c-tile). - 4. `utils`: it contains some utilities for the PYRAW package. - -* **resources**: it contains various resources (i.e., images for the README) -* **scripts_and_studies**: it contains various scripts and code for different studies realized to create the `THRAWS` dataset. In particular: - 1. `coregistration_study`: it contains utils used to perform the coregistration study and desing the [coarse coregistration](#coarse-coregistration) technique. - 2. `dataset_preparation`: it contains scripts and other files used to design the `THRAWS` files (i.e., download data, select events, and others). - 3. `hta_detection_algorithms`: it contains custom and simplified implementation of various high-thermal-anomalies-dection algorithms, including [1](#ref1) used to design the `THRAWS` dataset. - 4. `runscripts`: it contains some runscripts and utils to crop [Sentinel-2 L1C tiles](#sentinel-2-l1c-tile) and generate useful images and export tif. - 5. `granules_filtering`: it contains a script to run and [[1]](#ref1) on [cropped Sentinel-2 L1-C tiles](#sentinel-2-l1c-tile), extract bounding boxes and map them to the correspondent [Raw granules](#sentinel-2-raw-granule). - 6. `download_thraws`: it contains the utility for the download of the dataset [THRAWS](https://zenodo.org/record/7908728). +
    +
    ## Installation -### Pre-requirements -Before all, clone this repository. We suggest using git from CLI, execute: - -``` git clone https://github.com/ESA-PhiLab/PyRawS ``` - -### Create the PyRawS environment -#### - On Linux -``` +### **A) On Linux** +```bash # For Linux, the installation is straightforward. # You just need to run the following command from the main directory: \bin\bash\ source pyraws_install.sh -# NB: Your data should be placed in the data directory in the main. ``` +> NB: Your data should be placed in the data directory in the main. -#### - On Other Os -``` +### **B) Common** +```bash # To install the environment, we suggest to use anaconda. # You can create a dedicated conda environment by using the `environment.yml` file by running the following command from the main directory: +# Before all, clone this repository. We suggest using git from CLI, execute: +git clone https://github.com/ESA-PhiLab/PyRawS ``` +# Create the environment conda env create -f environment.yml - -# To activate your environment, please execute: - +# Activate your environment: conda activate PyRawS - ``` +> Create a file `sys_cfg.py` in the directory `pyraws\pyraws`, and add two variables as follows: +> ```cmd +> PYRAWS_HOME_PATH="Absolute path to the main pyraws directory." +> DATA_PATH="Absolute path to the data directory." +> ``` -Create a file `sys_cfg.py` in the directory `pyraws\pyraws`, and add two variables as follows: - -``` PYRAWS_HOME_PATH="Absolute path to the main pyraws directory.``` - -``` DATA_PATH="Absolute path to the data directory. " ``` - -By default the data directory is located in PyRawS main directory. +> By default the data directory is located in PyRawS main directory. -### Docker - -To use PyRawS with docker, use one of the following methods. - - -#### Method 1: Pull the docker image from Docker Hub +### **C) Docker** [![Docker](https://fontawesome.com/icons/docker?style=brands)](https://www.docker.com/) -``` docker pull sirbastiano94/pyraws:latest ``` +PyRawS comes also delivered in a docker image. To use PyRawS with docker, use one of the following methods. -#### Method 2: Build docker image -Follow these steps: +>#### **Method 1**:       Pull the docker image from Docker Hub +>>```bash +>> docker pull sirbastiano94/pyraws:latest +>>``` -1. Clone the repository and build the docker image by running the following command from the main directory: - -``` docker build -t pyraws:latest --build-arg CACHEBUST=$(date +%s) -f dockerfile . ``` - -2. Run the docker image by executing: - -``` docker run -it --rm -p 8888:8888 pyraws:latest ``` +>#### Method 2:       Build docker image from dockerfile +>Follow these steps: +>1. Clone the repository and build the docker image by running the following command from the main directory: +>>```bash +>> docker build -t pyraws:latest --build-arg CACHEBUST=$(date +%s) -f dockerfile . +>> ``` +> 2. Run the docker image by executing: +>>```bash +>>docker run -it --rm -p 8888:8888 pyraws:latest +>>``` +
    +
    +
    -### Set-up for coregistration study +## *Set-up for coregistration study* If you want to perform the coregistration study, you need to: -* clone the repository [SuperGlue Inference and Evaluation Demo Script](https://github.com/magicleap/SuperGluePretrainedNetwork); -* rename the subdirectory `models` to `superglue_models`; -* move `superglue_models` into `coregistration_study`. +1. clone the repository [SuperGlue Inference and Evaluation Demo Script](https://github.com/magicleap/SuperGluePretrainedNetwork); +2. rename the subdirectory `models` to `superglue_models`; +3. move `superglue_models` into `coregistration_study`. + +
    Coregistration study results can be generated by using the `pyraws_coregistration_study.ipynb` notebook. The database `coregistration_study_db.csv` containing info on the discarded eruption events is used to generate results in the notebook. -Coregistration notebook results can be generated by using the `pyraws_coregistration_profiling.ipynb` notebook. +> #### _Aside_: Coregistration notebook results can be generated by using the `pyraws_coregistration_profiling.ipynb` notebook. -### Data directory +>## Data directory +>> +>>All the data used by PyRawS need to be located in a directory called `data`, containing all >>the different datasets. You can place the `data` directory where you want and update the >>`DATA_PATH` variable in the `sys_cfg.py` file with its path (please, refer to: [Set-up for >>coregistration study](#set-up-for-coregistration-study)). +>>By default the data directory is located in PyRawS main directory. +>>
    +>>To create a specific dataset, you need to create subfolder with the dataset name (e.g., >>`THRAWS`) in the `data` directory specific for your dataset. Such directory shall contain the >>following subdirectories: +>> +>>* `raw`: it will contain [decompressed raw](#sentinel-2-raw-data). +>>* `l1c`: it will contain [L1C](#sentinel-2-l1c-data). +>> +>>Every [raw data](#sentinel-2-decompressed-raw-data) and [L1C](#sentinel-2-l1c-data) data shall >>be contained in a specific subdirectory --placed in `raw`or `l1c`-- whose name shall match one >>of the **ID_event** entries of the **.csv** file, located in `database` directory. -All the data used by PyRawS need to be located in a directory called `data`, containing all the different datasets. You can place the `data` directory where you want and update the `DATA_PATH` variable in the `sys_cfg.py` file with its path (please, refer to: [Set-up for coregistration study](#set-up-for-coregistration-study)). -By default the data directory is located in PyRawS main directory.
    -To create a specific dataset, you need to create subfolder with the dataset name (e.g., `THRAWS`) in the `data` directory specific for your dataset. Such directory shall contain the following subdirectories: - -* `raw`: it will contain [decompressed raw](#sentinel-2-raw-data). -* `l1c`: it will contain [L1C](#sentinel-2-l1c-data). - -Every [raw data](#sentinel-2-decompressed-raw-data) and [L1C](#sentinel-2-l1c-data) data shall be contained in a specific subdirectory --placed in `raw`or `l1c`-- whose name shall match one of the **ID_event** entries of the **.csv** file, located in `database` directory. +
    ## PyRawS databases @@ -183,13 +193,13 @@ Please note that it is important to carefully follow the instructions in the not This respository contains an example of database (`THRAWS`) that is used by PyRawS to read and process [Sentinel-2 Raw data](#sentinel-2-raw-data) and [Sentinel-2 L1 data](#sentinel-2-l1-data) correctly. The minimal fields of the database include: -* **ID_event**: ID of the event (e.g., volcanic-eruption, wildfire, not-event). All the other fields of the row are referred to that `Sentinel-2` acquisition. -* **class**: class of the event (e.g., eruption, fire, not-event). Leave it **empty** -* **Raw_useful_granules**: list of [Raw useful granules](#raw-useful-granule). Set to `None` or leave it empty if you do not know what are the [Raw useful granules](#raw-useful-granule). -* **Raw_complementary_granules**: list of [Raw complementry granules](#raw-complementary-granule). Set to `None` or leave it empty if you do not know what are the [Raw complementry granules](#raw-complementary-granule). -* **Raw_files**: list of [Raw granules](#sentinel-2-raw-granule) (**mandatory**). -* **l1c_files**: list of [L1 tiles](#sentinel-2-l1c-tile) (mandatory if you need L1C data). -* **bbox_list**: dictionary {[Raw useful granules](#raw-useful-granule) : [bounding box list for that granule]}. Set to `None` or leave it **empty** if you do not know the bounding box location. +>* **ID_event**: ID of the event (e.g., volcanic-eruption, wildfire, not-event). All the other >fields of the row are referred to that `Sentinel-2` acquisition. +>* **class**: class of the event (e.g., eruption, fire, not-event). Leave it **empty** +>* **Raw_useful_granules**: list of [Raw useful granules](#raw-useful-granule). Set to >`None` or leave it empty if you do not know what are the [Raw useful granules]>(#raw-useful-granule). +>* **Raw_complementary_granules**: list of [Raw complementry granules]>(#raw-complementary-granule). Set to `None` or leave it empty if you do not know what are the >[Raw complementry granules](#raw-complementary-granule). +>* **Raw_files**: list of [Raw granules](#sentinel-2-raw-granule) (**mandatory**). +>* **l1c_files**: list of [L1 tiles](#sentinel-2-l1c-tile) (mandatory if you need L1C data). +>* **bbox_list**: dictionary {[Raw useful granules](#raw-useful-granule) : [bounding box list for that granule]}. Set to `None` or leave it **empty** if you do not know the bounding box location. To create a new database (e.g., `my_database_name`), please, proceed as follows: @@ -197,10 +207,16 @@ To create a new database (e.g., `my_database_name`), please, proceed as follows: 2. Create subdirectory `my_database_name` in the `data` subdirectory and populate it with the correspondent [Sentinel-2 Raw data](#sentinel-2-raw-data) and [Sentinel-2 L1 data](#sentinel-2-l1-data) as described in [Data directory](#data-directory). 3. Update the `DATABASE_FILE_DICTIONARY` in `PyRawS/utils/constants.py` as follows: -```DATABASE_FILE_DICTIONARY={"THRAWS" : "thraws_db.csv", "my_database_name" : "my_db.csv"}``` +```cmd +DATABASE_FILE_DICTIONARY={"THRAWS" : "thraws_db.csv", "my_database_name" : "my_db.csv"} +``` +**Now, you should be able to use your new database! ⭐️** +
    + +>The creation of a database is not mandatory. However, it is strongly advisable. >Indeed, without creating a database you can still open `Raw data` as described in [Open a Raw >event from path](#open-a-raw-event-from-path). However, some pieces of information such as the [Raw useful granules](#raw-useful-granule) associated to a specific event, the event bounding boxes or the image class can be retrieved only when the database is set-up. -Now, you should be able to use your new database.
    -**N.B** The creation of a database is not mandatory. However, it is strongly advisable. Indeed, without creating a database you can still open `Raw data` as described in [Open a Raw event from path](#open-a-raw-event-from-path). However, some pieces of information such as the [Raw useful granules](#raw-useful-granule) associated to a specific event, the event bounding boxes or the image class can be retrieved only when the database is set-up. +
    +
    ## Raw events and Raw granules ![Alt Text](resources/images/etna_00_granules.png) @@ -209,57 +225,64 @@ Downloading [Sentinel-2 Raw data](#sentinel-2-raw-data) requires to specify a po since we use the bands [`B8A`, `B11`, `B12`] to detect wildfires and volcanic eruptions, we consider [Raw useful granules](#raw-useful-granule) those granules whose band `B8A` interesects the event. This is true for the yellow rectangular but not for the pink one (you need to trust us here, since the bands are not displaced in the image above). We take the band `B8A` only because after the coregistration, the other bands will be moved to match the area of `B8A`.
    Finally, for some [Raw useful granules](#raw-useful-granule) part of the eruptions or the wildfire could extend until the top or the bottom edge of the polygon. In this case, some of the bands could be missing for a portion of the area of interest. To be sure that this is not happening, in addition to the [Raw useful granules](#raw-useful-granule), it is important to consider [Raw complementary granules](#raw-complementary-granule), which fills the missing part of the interest bands of the [Raw useful granules](#raw-useful-granule).
    For each [Sentinel-2 Raw event](#Sentinel-2-RAW-event), the `THRAWS` dataset clearly states those [Raw granules](#sentinel-2-raw-granule) that are [Raw useful granules](#raw-useful-granule) or [Raw complementary granules](#raw-complementary-granule). However, the entire [Raw granules](#sentinel-2-raw-granule) collection is provided for each [Raw event](#sentinel-2-raw-event) to permit users that wants to use other bands to detect warm temeprature anomalies to do it. +
    +
    -## Quickstart -The next examples (with the exception of [Open a Raw event from path](#open-a-raw-event-from-path)) will exploit the `THRAWS` database to showcase the use of PyRawS, but they are applicable to any [databases compatible with PyRawS](#PyRawS-databases). - -### Open a Raw event from path -The next code snipped will showcase how to use PyRawS to open a [Raw_event](#sentinel-2-raw-event) `My_Raw_data`, included in the `THRAWS` database by using its `PATH`. We assume to have the `My_RAW_data` directory in the same directory where you execute the code snippet below.
    -To manipulate [Raw events](#sentinel-2-raw-event), `PyRawS` offer a class called `Raw_event`. To open an avent, we will use the [Raw_event](#sentinel-2-raw-event) class method `from_path(...)`, which parses the database specified, retrieves the event specified by `id_event` and opens it with the requested bands (`bands_list`). -When you open an event, you can specify which bands to use. If `bands_list` is not specified, the method `from_path(...)` will return all the bands. - -```py -from pyraws.raw.raw_event import Raw_event -#Instantiate an empty Raw_event -raw_event=Raw_event() -#Bands to open. -bands_list=["B8A", "B11", "B12"] +## Quickstart ⚡️ +The next examples (with the exception of [Open a Raw event from path](#open-a-raw-event-from-path)) will exploit the `THRAWS` database to showcase the use of PyRawS, but they are applicable to any [databases compatible with PyRawS](#PyRawS-databases). -#Read "Etna_00" from THRAWS -raw_event.from_path(#Path to the Etna_00 event - raw_dir_path="Path_to_my_RAW_data", - #Bands to open. Leave to None to use all the bands. - bands_list=bands_list, - #If True, verbose mode is on. - verbose=True) -``` -The example above can be used directly, **even if you did not set up a** [PyRawS database](#PyRawS-databases). However, `PyRawS` offer some API to parse directly [Raw_event](#sentinel-2-raw-event) parsing a database. with no need to specify a path. Please, check [Open a Raw event from database](#open-a-raw-event-from-database). -### Open a Raw event from database -The next code snipped will showcase how to use PyRawS to open the [Raw_event](#sentinel-2-raw-event) `Etna_00` included in the `THRAWS` database.
    To do that, -To manipulate [Raw events](#sentinel-2-raw-event) objects, `PyRawS` will exploits the `Raw_event` class method `from_database(...)`, which parses the associated `.csv` file located in `PyRawS/database` with no need to specify the `PATH` from the user. To execute the next code snipped, we assume to you have already downloaded and set-up the `THRAWS` database as specificied in [databases compatible with PyRawS](#PyRawS-databases). -As for the method `from_path(...)` described in [Open a Raw event from path](#open-an-raw-event-from-path), you can specify which bands to use. If `bands_list` is not specified, the method `from_database(...)` will return all the bands. +>### A) Open a Raw event from path +>The next code snipped will showcase how to use PyRawS to open a [Raw_event]>(#sentinel-2-raw-event) `My_Raw_data`, included in the `THRAWS` database by using its `PATH`. >We assume to have the `My_RAW_data` directory in the same directory where you execute the code >snippet below.
    +>To manipulate [Raw events](#sentinel-2-raw-event), `PyRawS` offer a class called `Raw_event`. >To open an avent, we will use the [Raw_event](#sentinel-2-raw-event) class method `from_path>(...)`, which parses the database specified, retrieves the event specified by `id_event` and >opens it with the requested bands (`bands_list`). +>When you open an event, you can specify which bands to use. If `bands_list` is not specified, >the method `from_path(...)` will return all the bands. +> +>```py +>from pyraws.raw.raw_event import Raw_event +>#Instantiate an empty Raw_event +>raw_event=Raw_event() +> +>#Bands to open. +>bands_list=["B8A", "B11", "B12"] +> +>#Read "Etna_00" from THRAWS +>raw_event.from_path(#Path to the Etna_00 event +> raw_dir_path="Path_to_my_RAW_data", +> #Bands to open. Leave to None to use all the bands. +> bands_list=bands_list, +> #If True, verbose mode is on. +> verbose=True) +>``` +>The example above can be used directly, **even if you did not set up a** [PyRawS database]>(#PyRawS-databases). However, `PyRawS` offer some API to parse directly [Raw_event]>(#sentinel-2-raw-event) parsing a database. with no need to specify a path. Please, check >[Open a Raw event from database](#open-a-raw-event-from-database). + +>### B) Open a Raw event from database +>The next code snipped will showcase how to use PyRawS to open the [Raw_event]>(#sentinel-2-raw-event) `Etna_00` included in the `THRAWS` database.
    To do that, +>To manipulate [Raw events](#sentinel-2-raw-event) objects, `PyRawS` will exploits the >`Raw_event` class method `from_database(...)`, which parses the associated `.csv` file located >in `PyRawS/database` with no need to specify the `PATH` from the user. To execute the next >code snipped, we assume to you have already downloaded and set-up the `THRAWS` database as >specificied in [databases compatible with PyRawS](#PyRawS-databases). +>As for the method `from_path(...)` described in [Open a Raw event from path]>(#open-an-raw-event-from-path), you can specify which bands to use. If `bands_list` is not >specified, the method `from_database(...)` will return all the bands. +> +>```py +>from pyraws.raw.raw_event import Raw_event +>#Instantiate an empty Raw_event +>raw_event=Raw_event() +> +>#Bands to open. +>bands_list=["B8A", "B11", "B12"] +> +>#Read "Etna_00" from THRAWS +>raw_event.from_database(#Database ID_EVENT +> id_event="Etna_00", +> #Bands to open. Leave to None to use all the bands. +> bands_list=bands_list, +> #If True, verbose mode is on. +> verbose=True, +> #Database name +> database="THRAWS") +>``` -```py -from pyraws.raw.raw_event import Raw_event -#Instantiate an empty Raw_event -raw_event=Raw_event() - -#Bands to open. -bands_list=["B8A", "B11", "B12"] +
    -#Read "Etna_00" from THRAWS -raw_event.from_database(#Database ID_EVENT - id_event="Etna_00", - #Bands to open. Leave to None to use all the bands. - bands_list=bands_list, - #If True, verbose mode is on. - verbose=True, - #Database name - database="THRAWS") -``` -All the next examples will assume you already have downloaded and set-up the `THRAWS` database as specificied in [databases compatible with PyRawS](#PyRawS-databases). However, they can work by using `from_path(...)` instead of `from_database(...)` and specifying the `PATH` to the `Etna_00` event manually. +_NB_: All the next examples will assume you already have downloaded and set-up the `THRAWS` database as specificied in [databases compatible with PyRawS](#PyRawS-databases). However, they can work by using `from_path(...)` instead of `from_database(...)` and specifying the `PATH` to the `Etna_00` event manually. ### Show Raw granules information of a Raw event As specified in [Raw events and Raw granules](#raw-events-and-raw-granules), an [Raw event](#sentinel-2-raw-event) is a collection of [Raw granules](#sentinel-2-raw-granule). As for [Raw_event](#sentinel-2-raw-event), [Raw granules](#sentinel-2-raw-granule) are modelled in PyRawS through a dedicated class `Raw_granule`.
    @@ -323,7 +346,7 @@ raw_granule_0.show_granule_info() granule_0_info_dict=raw_granule_0.get_granule_info() ``` -### Access a Raw_granule pixels +### Access a Raw\_granule pixels To visualize the values of an `Raw_data` object, it is possible to return it as a [PyTorch](https://pytorch.org/) tensor. However, since the different bands have different resolutions, depending on the bands that we want to shape as a tensor, it is necessary to upsample/downsample some of them to adapt them to the band with higher/smaller resolution. The next code snippet will open the `Etna_00` with bands `B02` (10 m), `B8A` (20 m), `B11` (20 m), get the granule with index 1, and will return the first two bands in the collection as tensor by performing upsample. ```py @@ -596,39 +619,27 @@ output_tif_folder="raw_target_folder" raw_granule_0.export_to_tif(save_path=output_tif_folder) ``` -## Glossary -### Coarse coregistration - Lightweight spatial coregistration method optimized for onboard-satellite applications. It simply shifts the various bands of a fixed factor that depends only on the bands, the satellite and detector number. - -### Sentinel-2 L0 data - Sentinel-2 data at `level-0` (`L0`) are data that are transmitted to Ground from Sentinel-2 satellites. The `L0` format is compressed to diminish downlink bandwidth requirements. For more information, refer to the [Sentinel-2 Products Specification Document](https://sentinel.esa.int/documents/247904/685211/sentinel-2-products-specification-document) - -### Sentinel-2 Raw data - In the frame of this project, the [Sentinel-2 Raw](#sentinel-2-raw-data) represents a particular product in the Sentinel-2 processing chain that matches a decompressed version of [Sentinel-2 L0 data](#sentinel-2-l0-data) with additional metadata that are produced on ground. Once decompressed, `Sentinel-2 Raw data` are the data available on Ground that better emulate the one produced by Sentinel-2 detectors with the exception of the effects due to compression and onboard equalization, which are not compensated at this stage. Therefore, `Sentinel-2 raw data` are those exploited in this project. For more information, refer to the [Sentinel-2 Products Specification Document](https://sentinel.esa.int/documents/247904/685211/sentinel-2-products-specification-document).
    -**N.B**: the nomenclature ```raw data``` and its location in the Sentinel-2 processing chain is specific for this project only. - -### Sentinel-2 Raw granule - A `granule` is the image acquired by a Sentinel-2 detector during a single acquisition lasting 3.6 s. Granules are defined at [L0](#sentinel-2-l0-data) level. However, since the processing perfomed on the ground between L0 and raw data does not alter the image content (with the exception of the decompression process) but just provide additional metadata, granules are defined also at [Sentinel-2 Raw](#sentinel-2-raw-data) level. - Given the pushbroom nature of the Sentinel-2 sensor, bands do not look at the same area at [Raw](#sentinel-2-raw-data) level. For more information, refer to the [Sentinel-2 Products Specification Document](https://sentinel.esa.int/documents/247904/685211/sentinel-2-products-specification-document) +
    +
    +
    -### Sentinel-2 Raw event - [Sentinel-2 Raw data](#sentinel-2-raw-data) are produced by decompressing [Sentinel-2 L0 data](#sentinel-2-l0-data). To download L0 data, it is necessary to specify one polygon that surrounds a particular area-of-interest. This leads to download all those [Sentinel-2 Raw granules](#sentinel-2-raw-granule) whose reference band intersects the specified polygon. Such collection is a `Raw-event`. Each `Raw-event` matches one of the `ID_event` entry of the database.
    - For each `Raw-event`, we do not provide all the collection of [Sentinel-2 Raw granules](#sentinel-2-Raw-granule), but only the set of [Raw data useful granules](#raw-data-useful-granule) and [Raw data complementary granules](#Raw-data-complementary-granule). For an intuitive example, please, check [Raw events and granules](#raw-events-and-raw-granules). +## Glossary -### Sentinel-2 L1C data - The `Level 1-C` (`L1C`) is one format for `Sentinel-2` data. To convert [Sentinel-2 Raw data](#sentinel-2-raw-data) to `L1C` data, numerous processing steps are applied to correct defects, including bands coregistration, ortho-rectification, decompression, noise-suppression and other. For more information, refer to the [Sentinel-2 Products Specification Document](https://sentinel.esa.int/documents/247904/685211/sentinel-2-products-specification-document). +Some clarifications about the terms used in the repo 📚 -### Sentinel-2 L1C event - Same concept for [Sentinel-2 Raw events](#sentinel-2-raw-event) but applied on [Sentinel-2 L1C data](#sentinel-2-l1c-data). +
    -### Sentinel-2 L1C tile - The `Sentinel-2 L1C tile` is the minimum `L1C` product that can be downloaded. +| Term | Description | Reference | +|--------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------| +| Sentinel-2 Raw granule | A granule is the image acquired by a Sentinel-2 detector during a single acquisition lasting 3.6 s. Granules are defined at L0 level. However, since the processing performed on the ground between L0 and raw data does not alter the image content (with the exception of the decompression process) but just provide additional metadata, granules are defined also at Sentinel-2 Raw level. Given the pushbroom nature of the Sentinel-2 sensor, bands do not look at the same area at Raw level. | Sentinel-2 Products Specification Document | +| Sentinel-2 Raw event | Sentinel-2 Raw data are produced by decompressing Sentinel-2 L0 data. To download L0 data, it is necessary to specify one polygon that surrounds a particular area-of-interest. This leads to download all those Sentinel-2 Raw granules whose reference band intersects the specified polygon. Such collection is a Raw-event. Each Raw-event matches one of the ID_event entry of the database.
    For each Raw-event, we do not provide all the collection of Sentinel-2 Raw granules, but only the set of Raw data useful granules and Raw data complementary granules. For an intuitive example, please, check Raw events and granules. | - | +| Sentinel-2 L1C data | The Level 1-C (L1C) is one format for Sentinel-2 data. To convert Sentinel-2 Raw data to L1C data, numerous processing steps are applied to correct defects, including bands coregistration, ortho-rectification, decompression, noise-suppression, and others. For more information, refer to the Sentinel-2 Products Specification Document. | - | +| Sentinel-2 L1C event | Same concept for Sentinel-2 Raw events but applied on Sentinel-2 L1C data. | - | +| Sentinel-2 L1C tile | The Sentinel-2 L1C tile is the minimum L1C product that can be downloaded. | - | +| Raw complementary granule| Given a certain set of bands of interest [Bx,By,...,Bz], Raw complementary granules are the granules adjacents at Raw-useful-granules that that can be used to fill missing pixels of [By,...,Bz] bands due to their coregistration with respect to the band Bx. For an intuitive example, please, check Raw events and granules. | - | +| Raw useful granule | Given a certain set of bands of interest [Bx,By,...,Bz], where Bx is the first band in the set, a Raw useful granule is one of the collection of Sentinel-2 Raw granules that compose a Sentinel-2 Raw event whose band Bx includes (or intersects) a certain area of interest (e.g., an eruption or an area covered by a fire). For an intuitive example, please, check Raw data events and granules. | - | -### Raw complementary granule - Given a certain set of bands of interest `[Bx,By,...,Bz]`, `Raw complementarey granules` are the granules adjacents at [Raw-useful-granules](#raw-useful-granule) that that can be used to fill missing pixels of `[By,...,Bz]` bands due to their coregistration with respecto the band `Bx`. For an intuitive example, please, check [Raw events and granules](#raw-events-and-raw-granules). -### Raw useful granule - Given a certain set of bands of interest `[Bx,By,...,Bz]`, where `Bx` is the first band in the set, an `Raw useful granule` is one of the collection of [Sentinel-2 Raw granules](#sentinel-2-raw-granule) that compose a [Sentinel-2 Raw event](#sentinel-2-raw-event) whose band `Bx` include (or intersects) a certain area of interest (e.g., an eruption or an area covered by a fire). For an intuitive example, please, check [Raw data events and granules](#raw-events-and-raw-granules). ## Contributing The ```PyRawS``` project is open to contributions. To discuss new ideas and applications, please, reach us via email (please, refer to [Contacts](#contacts)). To report a bug or request a new feature, please, open an [issue](https://github.com/ESA-PhiLab/PyRawS/issues) to report a bug or to request a new feature. @@ -653,5 +664,4 @@ Created by the European Space Agency $\Phi$-[lab](https://phi.esa.int/). * Federico Serva - federico.serva at ext.esa.int ## References - ### Ref1 - [Massimetti, Francesco, et al. ""Volcanic hot-spot detection using SENTINEL-2: a comparison with MODIS–MIROVA thermal data series."" Remote Sensing 12.5 (2020): 820."](https://www.mdpi.com/2072-4292/12/5/820) + >### [1]: [Massimetti, Francesco, et al. ""Volcanic hot-spot detection using SENTINEL-2: a comparison with MODIS–MIROVA thermal data series."" Remote Sensing 12.5 (2020): 820."](https://www.mdpi.com/2072-4292/12/5/820) From f0cb2a7c5d2dc29264b5482b62bcf36fc90e7ac4 Mon Sep 17 00:00:00 2001 From: Roberto Del Prete <71963566+sirbastiano@users.noreply.github.com> Date: Wed, 2 Aug 2023 15:11:11 +0200 Subject: [PATCH 2/3] update hyperlinks --- README.md | 29 ++++++++++++----------------- 1 file changed, 12 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index c52e2b1..f534bb0 100644 --- a/README.md +++ b/README.md @@ -36,25 +36,20 @@ The systme is based on [pytorch]("https://pytorch.org/"), which be installed wit
  • About the Project
  • Content of the repository
  • Installation - -
  • PyRawS databases
    • +
  • PyRawS databases
  • Raw events and Raw granules
  • Quickstart
  • Glossary
    • @@ -128,7 +123,7 @@ conda activate PyRawS -### **C) Docker** [![Docker](https://fontawesome.com/icons/docker?style=brands)](https://www.docker.com/) +### **C) Docker** PyRawS comes also delivered in a docker image. To use PyRawS with docker, use one of the following methods. @@ -229,11 +224,11 @@ Finally, for some [Raw useful granules](#raw-useful-granule) part of the eruptio
    -## Quickstart ⚡️ +## Quickstart The next examples (with the exception of [Open a Raw event from path](#open-a-raw-event-from-path)) will exploit the `THRAWS` database to showcase the use of PyRawS, but they are applicable to any [databases compatible with PyRawS](#PyRawS-databases). ->### A) Open a Raw event from path +>### Open a Raw event from path >The next code snipped will showcase how to use PyRawS to open a [Raw_event]>(#sentinel-2-raw-event) `My_Raw_data`, included in the `THRAWS` database by using its `PATH`. >We assume to have the `My_RAW_data` directory in the same directory where you execute the code >snippet below.
    >To manipulate [Raw events](#sentinel-2-raw-event), `PyRawS` offer a class called `Raw_event`. >To open an avent, we will use the [Raw_event](#sentinel-2-raw-event) class method `from_path>(...)`, which parses the database specified, retrieves the event specified by `id_event` and >opens it with the requested bands (`bands_list`). >When you open an event, you can specify which bands to use. If `bands_list` is not specified, >the method `from_path(...)` will return all the bands. @@ -256,7 +251,7 @@ The next examples (with the exception of [Open a Raw event from path](#open-a-ra >``` >The example above can be used directly, **even if you did not set up a** [PyRawS database]>(#PyRawS-databases). However, `PyRawS` offer some API to parse directly [Raw_event]>(#sentinel-2-raw-event) parsing a database. with no need to specify a path. Please, check >[Open a Raw event from database](#open-a-raw-event-from-database). ->### B) Open a Raw event from database +>### Open a Raw event from database >The next code snipped will showcase how to use PyRawS to open the [Raw_event]>(#sentinel-2-raw-event) `Etna_00` included in the `THRAWS` database.
    To do that, >To manipulate [Raw events](#sentinel-2-raw-event) objects, `PyRawS` will exploits the >`Raw_event` class method `from_database(...)`, which parses the associated `.csv` file located >in `PyRawS/database` with no need to specify the `PATH` from the user. To execute the next >code snipped, we assume to you have already downloaded and set-up the `THRAWS` database as >specificied in [databases compatible with PyRawS](#PyRawS-databases). >As for the method `from_path(...)` described in [Open a Raw event from path]>(#open-an-raw-event-from-path), you can specify which bands to use. If `bands_list` is not >specified, the method `from_database(...)` will return all the bands. From 94977e1633943f05997948fb46ce68c4fc9e4ff4 Mon Sep 17 00:00:00 2001 From: Gabriele Meoni <70584239+GabrieleMeoni@users.noreply.github.com> Date: Fri, 6 Sep 2024 10:25:06 +0200 Subject: [PATCH 3/3] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f534bb0..572e541 100644 --- a/README.md +++ b/README.md @@ -68,7 +68,7 @@ The PyRawS repository is structured in the following directories: | Directory Name | Description | Subdirectories and Contents | |-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| -| quickstart | Contains some subdirectories including examples. | - `API demonstration`: A [jupyter notebook](https://jupyter-notebook.readthedocs.io/en/stable/index.html) to demonstrate PyRawS API. | +| quickstart | Contains some subdirectories including examples. | `API demonstration`: A [jupyter notebook](https://jupyter-notebook.readthedocs.io/en/stable/index.html) to demonstrate PyRawS API. | | pyraws | Contains the PyRawS package. | - `database`: Various [PyRawS database](#PyRawS-databases) and other databases. | | | | - `raw`: Includes the `Raw_event` and `Raw_granule` classes used to model [Sentinel-2 Raw events](#sentinel-2-raw-event) and [Sentinel-2 Raw granules](#sentinel-2-raw-granule). | | | | - `l1`: Contains the `L1_event` and `L1_tiles` classes used to model [Sentinel-2 L1C events](#sentinel-2-l1c-event) and [Sentinel-2 L1C tiles](#sentinel-2-l1c-tile). |