v0.5.0

Summary

This is a major cleanup release, mostly thanks to @HealthyPear who put it in a reasonable state to be maintained by others in the future! It has been tested against CTA Prod3b and CTA Prod5b for CTA-North point-like on-axis sources and should work as a baseline for analysis optimization.

This release will brings huge improvements to the performance of the pipeline and the software package.

Some of the changes are related to the improvements in the interface to the DIRAC grid. Please, consult that for more details about its latest release.

The performance of the pipeline has been validated successfully on a Prod5b simulation for the Alpha configuration at the North site under specific observational settings. Tests on a similar configuration for the South site are ongoing. From now on, performance results will be showcased at this website.

The benchmarking has become a package module (protopipe.benchmarking) and a new protopipe-BENCHMARK script has been added to the suite of available commands.

The pipeline makes now use of ctapipe v0.11.0 and pyirf v0.5.0. For more details, please consult the merged Pull Requests listed below.

Please, remember that protopipe and its interface are open-source projects and anyone can contribute to it. It is possible to perform full-scale analyses and contribute to the production of CTA performances for different science cases or perform studies that require big amounts of data.

Contributors

• Michele Peresano (@HealthyPear)
• Karl Kosack (@kosack)

What is changed since v0.4.0.post1

Pull-requests containing changes of multiple nature are repeated.

🚀 General features

• Add benchmarking notebook template (#181) @HealthyPear
• Add book template material (#174) @HealthyPear
• Add new script for benchmarking (#157) @HealthyPear
• Update the entire benchmark suite (#166) @HealthyPear
• Improve data scripts output and update analysis default settings (#165) @HealthyPear
• Improve protopipe.mva and protopipe-MODELS (#164) @HealthyPear
• Add image extraction status for TwoPassWindowSum (#163) @HealthyPear
• Add support for prod5N files (#167) @HealthyPear
• Add I/O functions to simplify benchmarking (#147) @HealthyPear
• Apply CALIB_SCALE directly from ctapipe.io.SimtelEventSource (#145) @HealthyPear
• Add progress bar to both TRAINING and DL2 production scripts (#150) @HealthyPear
• Add option to enable/disable charge integration integration correction (#146) @HealthyPear
• Calculate impact parameter always in the tilted frame (#143) @HealthyPear
• Add support for calibscale (#140) @HealthyPear
• Add optional LST stereo trigger requirement (#131) @HealthyPear
• Update to ctapipe 0.11.0 (#136) @HealthyPear
• Add configuration option to choose which cleaning to use to get training data (#135) @HealthyPear
• Add choice of estimation weigths and standard deviation for RandomForestRegressor models (#134) @HealthyPear
• Add concentration (#133) @HealthyPear
• Improve models generation (#96) @HealthyPear
• Support for ctapipe 0.10.5 (#124) @HealthyPear
• Small improvements to modeling script (#114) @HealthyPear

🔭 Pipeline applications

• Add support for prod5N files (#167) @HealthyPear

🌐 GRID support

• Update README and documentation (grid interface) (#152) @HealthyPear
• Make progress bar optional and add it also to DL2 script (#154) @HealthyPear

🐛 Bug Fixes

• fixed missing import of QTable (#188) @kosack
• try to fix build (mamba error) (#189) @kosack
• correctly count SSTs, MSTs, and LSTs. (#184) @kosack
• Add book template material (#174) @HealthyPear
• Set effective area HDU name as gammapy wants (#144) @HealthyPear
• Update CameraFrame-to-TelescopeFrame transformation and HillasReconstructor (#151) @HealthyPear
• Fix shower's core transformation to TiltedFrame (#160) @HealthyPear
• Fix CTAMARS-like energy estimation (#156) @HealthyPear
• Make progress bar optional and add it also to DL2 script (#154) @HealthyPear
• Output NaN concentrations in case of HillasParameterizationError or similar (#155) @HealthyPear
• Fix CALIB_SCALE key in analysis.yaml (#153) @HealthyPear
• Ensure that estimated energy is always recorded in linear scale (#141) @HealthyPear
• Add pandas to pip installation (#138) @HealthyPear
• Fix classifier integration test (#120) @HealthyPear
• Bugfixes and improvements to protopipe-MODELS (#122) @HealthyPear
• Fix pipeline integration test workflow (#118) @HealthyPear
• Fix documentation development version in docs (#112) @HealthyPear

🧰 Maintenance

• Update documentation for release 0.5.0 (#175) @HealthyPear
• Add Pull Request template (#187) @HealthyPear
• Add new script for benchmarking (#157) @HealthyPear
• Update README and documentation (grid interface) (#152) @HealthyPear
• Fix documentation build (#158) @HealthyPear
• Add integration test for DL2-to-DL3 step (#137) @HealthyPear
• Add pandas to pip installation (#138) @HealthyPear
• Implement DL2 integration tests (#126) @HealthyPear
• Add issue templates (#121) @HealthyPear
• Fix classifier integration test (#120) @HealthyPear
• Improve debugging of integration testing (#119) @HealthyPear
• Fix pipeline integration test workflow (#118) @HealthyPear
• Setup of pipeline integration testing up to modeling (#116) @HealthyPear
• Update training integration tests (#113) @HealthyPear
• Fix documentation development version in docs (#112) @HealthyPear
• Versioning, packaging and continuous deployment on PyPI (#105) @HealthyPear
• Update TRAINING integration tests (#111) @HealthyPear
• Update DOIs after latest release (#109) @HealthyPear

v0.4.0.post1

Summary

This is a post-release that takes care of project maintenance, so it doesn’t change the performance of the code.

Contributors

• Michele Peresano (@HealthyPear)

What is changed since v0.4.0

Pull-requests that contain changes belonging to multiple classes are repeated.

🐛 Bug Fixes

• Fix zenodo configuration file and add LICENSE file (#106) @HealthyPear

🧰 Maintenance

• Update CHANGELOG (#108) @HealthyPear
• Prepare first upload to PyPI (#107) @HealthyPear
• Fix zenodo configuration file and add LICENSE file (#106) @HealthyPear

v0.4.0

Summary

This release brings many improvements of which the most relevant are summarised
here depending on their scope within the pipeline workflow.

Performance-wise, protopipe caught up with the EventDisplay and CTAMARS historical pipelines starting from about 500 GeV onwards.
Below this threshold, even if compatible with requirements, the sensitivity diverges.

• All pipeline

  • upgrade to the API of ctapipe 0.9.1
  • documentation also on readthedocs and link to Zenodo
  • Continuous Integration is now performed on GitHub
  • New benchmarks have been added
  • Reference analysis and benchmarks results have been updated

• Data training

  • calibration benchmarks need only ctapipe-stage1-process
  • write_dl1 has become data_training
  • DL1 parameters and (optionally) images are merged in a single file
  • DL1 parameters names as in ctapipe and they are in degrees (TelescopeFrame)
  • scale correction with the effective focal length
  • fixed bugs and wrong behaviors

• Modeling and DL2 production

  • fixed bugs and wrong behaviors
  • Added missing features to get closer to CTAMARS

• DL3

  • the performance step is now based on the pyirf library
  • performance results are stored here

Contributors

• Michele Peresano (@HealthyPear)
• Gaia Verna (@gaia-verna)
• Alice Donini (@adonini)

What is changed since v0.3.0

Pull-requests that contain changes belonging to multiple classes are repeated.

🚀 General features

• Performance using Pyirf (#83) @gaia-verna
• Towards using Pyirf (#79) @gaia-verna
• Upgrade of DL2 production (#77) @HealthyPear
• Upgrade calibration benchmarks (#59) @HealthyPear
• Upgrade of data training (#58) @HealthyPear

🐛 Bug Fixes

• Fix zenodo configuration file and add LICENSE file (#106) @HealthyPear
• Fix calibration benchmarking settings (#100) @HealthyPear
• Fix plot of simulated signal and noise of 2nd pass image extraction (#99) @HealthyPear
• Upgrade of DL2 production (#77) @HealthyPear
• Upgrade of data training (#58) @HealthyPear

🧰 Maintenance

• Fix zenodo configuration file and add LICENSE file (#106) @HealthyPear
• Update documentation + general maintenance (#62) @HealthyPear
• Use mamba to create virtual enviroment for the CI (#101) @HealthyPear
• Upgrade all other notebooks and their docs version (#76) @HealthyPear
• Upgrade calibration benchmarks (#59) @HealthyPear
• Upgrade of data training (#58) @HealthyPear
• Enable CI from GitHub actions (#84) @HealthyPear

v0.3.0

Summary

• compatible with ctapipe 0.7.0
• early improvements related to the DL1 comparison against the CTAMARS pipeline
• improvements to basic maintenance
• a more consistent approach for full-scale analyses
• bug fixes

Contributors

• Michele Peresano (@HealthyPear)
• Thierry Stolarczyk (@tstolarczyk)
• Gaia Verna (@gaia-verna)
• Karl Kosack (@kosack)
• Thomas Vuillaume (@vuillaut)

What is changed since v0.2.1

🚀 General features

• Add missing variables in write_dl2 (#66) @HealthyPear
• Add missing dl1 parameters (#41) @HealthyPear
• Updates on notebooks (#47) @HealthyPear
• New plots for calibration benchmarking (#43) @HealthyPear
• Double-pass image extractor (#48) @HealthyPear
• Notebooks for low-level benchmarking (#42) @HealthyPear
• Improved handling of sites, arrays and cameras for all Prod3b simtel productions (#33) @HealthyPear
• Change gain selection (#35) @HealthyPear
• Changes for adding Cameras beyond LSTCam and NectarCam (#29) @tstolarczyk

🌐 GRID support

• Update configuration files (#74) @HealthyPear
• Update documentation for GRID support (#54) @HealthyPear
• Rollback for GRID support (#52) @HealthyPear

🐛 Bug Fixes

• Bugfix in Release Drafter workflow file (#71) @HealthyPear
• Convert pointing values to float64 at reading time (#68) @HealthyPear
• Rollback for GRID support (#52) @HealthyPear
• Fix recording of DL1 image and record reconstruction cleaning mask (#46) @gaia-verna
• consistent definition of angular separation to the source with config (#39) @vuillaut
• Update write_dl1.py (#30) @tstolarczyk

🧰 Maintenance

• Update documentation release 0.3.0 (#80) @HealthyPear
• Add mailmap file (#72) @HealthyPear
• Update benchmarks and documentation (#75) @HealthyPear
• Bugfix in Release Drafter workflow file (#71) @HealthyPear
• Add release drafter (#67) @HealthyPear
• Add benchmark notebooks for medium and late stages (#55) @HealthyPear
• Update documentation for GRID support (#54) @HealthyPear
• Updated documentation (#50) @HealthyPear
• Implementation of a first unit test (DL1) (#34) @HealthyPear
• Updated documentation (Closes #23) (#32) @HealthyPear
• Added Travis CI configuration file (#18) @HealthyPear
• Update README.md (#28) @tstolarczyk
• Added versioning to init.py and setup.py using the manual approach. (#20) @HealthyPear
• Update README.md (#21) @tstolarczyk

v0.2.1

Bugs fixed:

• the ctapipe-based cleaning algorithm for the biggest cluster was crashing in case of cleaned images with no surviving pixel clusters.

v0.2

This release includes:

• full support for the stable release of ctapipe 0.7.0
• improved instructions for installation, use, and development
• additional auxiliary scripts

Detailed information is stored in the CHANGELOG.

0.1 with optional image storage

This version includes:

• optional storage of DL1 photoelectron images both from calibrated and raw data available for at both DL1 and DL2 data-levels
• PNG image format added to diagnostic plots output
• additional PEP8-compliant formatting.

First official development version

This version of protopipe is based on ctapipe 0.6.2 (conda package stable version).
Its performance has been shown at the CTAC meeting in Lugano 2019.