This repository is now depracated and no longer maintained. Please find the latest version at github.com/dcalc.
Reduction software for SO/PHI-HRT instrument on the ESA Solar Orbiter
- read in science data (+scaling) (one or multiple files)
- read in flat field (+scaling) - accepts only one flat field fits file
- read in dark field (+scaling)
- prefilter correction
- apply dark field (to only science - assumes flat is dark fielded)
- clean flat field with unsharp masking (Stokes QUV, UV or V)
- normalise flat field
- apply flat field
- apply field stop
- apply hot pixels mask
- polarimetric registration
- demodulate with const demod matrix
a) option to output demod to fits file - normalise to quiet sun
- calibration
a) ItoQUV cross talk correction
b) VtoQU cross talk correction - wavelengths registration
- rte inversion with cmilos
a) output rte data products to fits files
Operating System: Linux
##############################################################
If running on bob MPS server (and you have access)
##############################################################
Otherwise:
- Compile milos in both cmilos-fits folder and cmilos folder (cmilos-fits):
make clear
make
- P-MILOS - multiple steps required - assumes running on BOB server
- module load openmpi_gcc
- module load fftw
- load cfitsio (or export through .bashrc)
make clean
make
- Setup virtual environment from requirements.txt with PIP or CONDA
using pip - REQUIRES PYTHON >= 3.6
pip install -r requirements.txt
OR using conda (Anaconda3) - creates virtual environment called 'hrt_pipeline_env'
conda env create -f environment.yml
- Activate 'hrt_pipeline_env'
source activate hrt_pipeline_env
- Install sophi_hrt_pipe package
pip install . #or "conda develop ."
##############################################################
(once environment loaded and activated - need the environment to start and use all the steps in the notebook)
First let jupyter notebook know the environment exists:
python -m ipykernel install --user --name hrt_pipeline_env
Then start the notebook
##############################################################
**(DEPRECATED)** 5. Download files - see Download Input Files Section
-
Generate json files with the science, dark and flat you desire to reduce, ensure that all keywords from in the example are used (for limb images you must know the limb in the FOV - can only reduce multiple limb files at the same runtime if all the same limb in FOV)
-
Make sure the correct input.json file is given to
hrt_pipe
inrun.py
-
Execute
run.py
python run.py
Any and all steps can be turned on or off as you wish using the keywords in the input json file passed to the phihrt_pipe
function
See /input_jsons/create_input_json.py
for example to create json file
** In input.json file, cant insert directly .fits.gz files from the PHI database **
This needs the 'dataproc' environment - see Setup
EITHER: download from the PHI Image Database (recommended): https://www2.mps.mpg.de/services/proton/phi/imgdb/
Suggested filters for HRT science data:
- Keyword Detector = 'HRT'
- Filename* like *L1_phi-hrt-ilam_date*
To download via the command line (eg: if you want to save the files on a server and not locally)
wget --user yourusername --password yourpassword file_web_address
gunzip file.gz
Gunzip used to unpack the .gz to the file you want
Can also use /download/download_from_db.py
to perform multi download from database:
Instructions:
- From the database find the files you wish to download
- Copy the 'Download File List' that the database will generate
- Paste into the
file_names.txt
file - Create a
.env
file with your MPS Windows login:
USER_NAME = PHIDATAPASSWORD =
- Set the target download folder in the
download_from_db.py
file - Run the file (will require dotenv python module to be installed - included in
hrt_pipeline_env
)
OR : use download_files.py
to download images from the attic repository: https://www2.mps.mpg.de/services/proton/phi/fm/attic/
Filename: solo_L2_phi-hrt-stokes.....fits.gz
Shape: [Y,X,POL,WAVE]
-
File:
solo_L2_phi-hrt-bmag.....fits.gz
Shape: [Y,X]
|B| (Gauss) -
File:
solo_L2_phi-hrt-binc.....fits.gz
Shape: [Y,X]
Inclination (Degrees) -
File:
solo_L2_phi-hrt-bazi.....fits.gz
Shape: [Y,X]
Aimuth (Degrees) -
File:
solo_L2_phi-hrt-blos.....fits.gz
Shape: [Y,X]
Blos (Gauss) -
File:
solo_L2_phi-hrt-vlos.....fits.gz
Shape: [Y,X]
Vlos (km/s) -
File:
solo_L2_phi-hrt-icnt.....fits.gz
Shape: [Y,X]
Continuum Intensity -
File:
_rte_data_products.fits
Shape: [7,Y,X]
First Index:
- 0: Continuum Intensity
- 1: Magnetic Field Strength |B| (Gauss)
- 2: Inclination (degrees)
- 3: Azimuth (degrees)
- 4: Vlos (km/s)
- 5: Blos (Gauss)
- 6: ChiSq
-
File:
_dark_corrected.fits.gz
Shape: [Y,X]
Dark corrected science data -
File:
_flat_corrected.fits.gz
Shape: [Y,X]
Flat corrected science data -
File:
copy_{flat_field_name}.fits.gz
Shape: [Y,X]
Flat field before US -
File:
_prefilter_corrected.fits.gz
Shape: [Y,X]
Prefilter corrected science data -
File:
_demodulated.fits.gz
Shape: [Y,X]
Demodulated science data
Jonas Sinjan - Max Planck Institute for Solar System Research, Goettingen, Germany
Daniele Calchetti - Max Planck Institute for Solar System Research, Goettingen, Germany
Fatima Kahil - Max Planck Institute for Solar System Research, Goettingen, Germany
- SPGPylibs for the foundation, from which it was expanded upon
- CMILOS: RTE INVERSION C code for SO-PHI (based on the IDL code MILOS by D. Orozco) Author: juanp (IAA-CSIC)
- CMILOS-FITS: CMILOS with fits interace, fits interfacing developed by Philipp Loeschl (MPS)
- P-MILOS: Parallellised RTE INVERSION C code Authors: Manuel Cabrera, Juan P. Cobos, Luis Bellot Rubio (IAA-CSIC)
- PyPD for the PSF deconvolution algorithm, by F. Kahil (MPS)