v1.14.0

Added

• Added total job duration to extraction report header
• IsIdentifiableReviewer rule review screen
• Added CSV input support for IsIdentifiable, use verb file from command line
• Updater microservice now audits performance of queries (cumulative affected rows, queries executed etc)
• Added -f option to DicomTagReader to read a single zip/dicom file
• Added some Python library code

Changed

• Clarified the CLI help text for --format in CohortPackager
• CTP calls an external program to anonymise Structured Reports
• Requires an addition to default.yaml: CTPAnonymiserOptions.SRAnonTool (adding this does not break existing programs).
• Consolidate System.IO.Abstractions.TestingHelpers package references into the Smi.Common.Tests package
• Tidy common csproj options into Directory.Build.props files for all, src, and test projects

Fixed

• Fixed a bug where newlines would never be correctly parsed from the config option in CohortPackager
• CohortPackager: Don't try and create the jobId file when recreating an existing report
• CohortPackager.Tests: Fix a flaky test caused by NUnit setup/teardown code when running tests in parallel
• CohortPackager.Tests: Fix a flaky test caused by using the same MongoDB database name when running tests in parallel
• Fixed the existing CTPAnonymiser tests which had not been updated for the SRAnonTool changes
• Fixed executable name on UpdateValues microservice

Dependencies

• Bump CsvHelper from 17.0.1 to 22.1.1
• Bump HIC.RDMP.Dicom from 2.1.11 to 2.2.2
• Bump HIC.RDMP.Plugin from 4.1.9 to 4.2.3
• Bump HIC.RDMP.Plugin.Test from 4.1.9 to 4.2.3
• Bump Magick.NET-Q16-AnyCPU 7.22.2.2 to 7.23.1
• Bump Microsoft.NET.Test.Sdk 16.8.0 to 16.8.3
• Bump Moq from 4.15.2 to 4.16.0
• Bump NUnit from 3.12.0 to 3.13.1
• Bump NunitXml.TestLogger from 2.1.80 to 3.0.91
• Bump System.IO.Abstractions from 13.2.2 to 13.2.9
• Bump System.IO.Abstractions.TestingHelpers from 13.2.2 to 13.2.9
• Bump YamlDotNet from 9.1.0 to 9.1.4

v1.13.0

Changelog

Added

• Added new command line application TriggerUpdates for detecting and issuing UpdateValuesMessages (e.g. ECHI mapping changes)
• Added new service UpdateValues which propagates changes (e.g. ECHI mapping changes) throughout the deployed database tables.
• ConsensusRule for combining 2+ other rules e.g. SocketRules (See IsIdentifiable Readme.md for more details)
• Added runtime and total failures count to IsIdentifiable logs
• Added NoSuffixProjectPathResolver which generates anonymous image path names that do not contain "-an" (which is the default behaviour).
  • To use, set CohortExtractorOptions.ProjectPathResolverType to Microservices.CohortExtractor.Execution.ProjectPathResolvers.NoSuffixProjectPathResolver
  • For identifiable extractions, the NoSuffixProjectPathResolver is now used
• Validation reports can now be created as either "Combined" (single report as before" or "Split" (a pack of reports including CSVs suitable for post-processing). This is configurable in the YAML config and can also be specified on the CLI when recreating reports for an extraction
• Added JobCompletedAt to the validation reports
• IsIdentifiable: Add support for ignoring OCR output less than n characters in length
• IsIdentifiable: Add a test case for burned-in image text

Changed

• Update docs and make more keywords links to the relevant docs (#440)
• Reduce memory usage on long-running microservices even when .Net assumes RAM is plentiful
• Validation reports are now written to the project reports directory, instead of to a central reports directory

Fixed

• Fix mismatch in Java/C# messages for ExtractionModality
• ExtractionFileCopier: Copy files relative to the extraction root not the global filesystem root
• Fix implementation of minimum OCR length (before being reported) #471

Dependencies

• Bump CsvHelper from 17.0.0 to 17.0.1
• Bump System.IO.Abstractions from 13.2.1 to 13.2.2
• Bump Moq from 4.15.1 to 4.15.2
• Bump System.IO.Abstractions.TestingHelpers from 13.2.1 to 13.2.2
• Bump CsvHelper from 16.2.0 to 17.0.0
• Bump JetBrains.Annotations from 2020.1.0 to 2020.3.0
• Bump jackson-dataformat-yaml from 2.11.3 to 2.12.0
• Bump jackson-databind from 2.11.3 to 2.12.0

v1.12.2

Changelog

Fixed

• Fix missing JSON fields from CTP output

v1.12.1

Changelog

• Remove reference to MongoDB.Driver in Smi.Common.MongoDb.csproj since it caused a version conflict in the output packages

v1.12.0

NOTE - package smi-services-linux-x64-v1.12.0.tar.gz below contains a broken version of the MongoDBPopulator service. Please use a version of this in a later release

Changelog

Added

• [breaking] Add identifiable extraction support
  • New service "FileCopier" which sits in place of CTP for identifiable extractions and copies source files to their output dirs
  • Changes to MongoDB extraction schema, but backwards compatibility has been tested
  • RabbitMQ extraction config has been refactored. Queues and service config files need to be updated
• Add SecurityCodeScan tool to build chain for .NET code
• Add "no filters" extraction support. If specified when running ExtractorCLI, no file rejection filters will be applied by CohortExtractor. True by default for identifiable extractions
• Added caching of values looked up in NLP/rulesbase for IsIdentifiable tool
• Added new rejector that throws out values (e.g. patient IDs) whose IDs are stored in a database table. Set RejectColumnInfos option in yaml to enable this
• Added a check to QueryToExecuteResult for RejectReason being null when Reject is true.

Changed

• [breaking] Environment variables are no longer required. Previous settings now appear in configuration file
  • Environment variable SMI_LOGS_ROOT is now GlobalOptions.LogsRoot
  • Environment variable MONGO_SERVICE_PASSWORD is now MongoDbOptions.Password
  • Removed ISIDENTIFIABLE_NUMTHREADS as it didn't work correctly anyway
• Extraction report: Group PixelData separately and sort by length
• IsIdentifiable Reviewer 'Symbols' rule factory now supports digits only or characters only mode (e.g. use \d for digits but leave characters verbatim)
• IsIdentifiable Reviewer 'symbols' option when building Regex now builds capture groups and matches only the failing parts of the input string not the full ProblemValue. For example MR Head 12-11-20 would return (\d\d-\d\d-\d\d)$

Fixed

• Fix the extraction output directory to be <projId>/extractions/<extractname>

Dependencies

• Bump fo-dicom.Drawing from 4.0.5 to 4.0.6
• Bump fo-dicom.NetCore from 4.0.5 to 4.0.6
• Bump HIC.BadMedicine.Dicom from 0.0.6 to 0.0.7
• Bump HIC.DicomTypeTranslation from 2.3.0 to 2.3.1
• Bump HIC.FAnsiSql from 1.0.2 to 1.0.5
• Bump HIC.RDMP.Dicom from 2.1.6 to 2.1.10
• Bump HIC.RDMP.Plugin from 4.1.6 to 4.1.8
• Bump HIC.RDMP.Plugin.Test from 4.1.6 to 4.1.8
• Bump Microsoft.CodeAnalysis.CSharp.Scripting from 3.6.0 to 3.7.0
• Bump Microsoft.Extensions.Caching.Memory from 3.1.6 to 3.1.8
• Bump Microsoft.NET.Test.Sdk from 16.6.1 to 16.7.1
• Bump MongoDB.Driver from 2.11.0 to 2.11.2
• Bump System.IO.Abstractions from 12.1.1 to 12.1.9
• Bump System.IO.Abstractions.TestingHelpers from 12.1.1 to 12.1.9
• Bump Terminal.Gui from 0.81.0 to 0.89.4

v1.11.1: Hotfix - publish trimmed (#348)

* Disable publishtrimmed since it breaks things

* bump for v1.11.0

Co-authored-by: James A Sutherland <[email protected]>

v1.11.0

Changelog

Added

• DicomDirectoryProcessor and TagReader support for zip archives
  • Expressed in notation /mydrive/myfolder/myzip.zip!somesubdir/my.dcm
  • Requires command line -f zips

Changed

• Improved the extraction report by summarising verification failures
• Start MongoDB in replication mode in the Travis builds
• Switch to self-contained .Net binaries to avoid dependency on host runtime package
• NationalPACSAccessionNumber is now allowed to be null in all messages

Dependencies

• Bump HIC.RDMP.Plugin from 4.1.5 to 4.1.6
• Bump MongoDB.Driver from 2.10.4 to 2.11.0
• Bump System.IO.Abstractions from 12.0.10 to 12.1.1
• Bump System.IO.Abstractions.TestingHelpers from 12.0.10 to 12.1.1
• Bump jackson-dataformat-yaml from 2.11.1 to 2.11.2

v1.10.0

Changed

• Updated the extraction report to be more human-readable #320, #328
• Add CLI option to CohortPackager to allow an existing report to be recreated #321
• Added a runsettings file for NUnit to allow configuration of test output. Fixes an issue with TravisCI and NUnit3TestAdapter v3.17.0, which caused the test output to spill to over 20k lines.

Dependencies

• Bump HIC.FAnsiSql from 0.11.1 to 1.0.2
• Bump HIC.RDMP.Dicom from 2.1.5 to 2.1.6
• Bump HIC.RDMP.Plugin from 4.1.3 to 4.1.5
• Bump Magick.NET-Q16-AnyCPU from 7.20.0 to 7.21.1
• Bump Microsoft.Extensions.Caching.Memory from 3.1.5 to 3.1.6
• Bump System.IO.Abstractions from 12.0.1 to 12.0.10
• Bump System.IO.Abstractions from 12.0.1 to 12.0.2
• Bump System.IO.Abstractions.TestingHelpers from 12.0.1 to 12.0.2
• Bump com.fasterxml.jackson.dataformat.jackson-dataformat-yaml from 2.11.0 to 2.11.1
• Bump org.mockito.mockito-core from 3.3.3 to 3.4.6

v1.9.0

Changelog

Added

• Added image extraction blacklist rejector.
  • Configure with Blacklists option (specify a list of Catalogue IDs)
  • Catalogues listed must include one or more column(s) StudyInstanceUID, SeriesInstanceUID, SOPInstanceUID.
  • Records in the referenced table will blacklist where any UID is found (StudyInstanceUID, SeriesInstanceUID or SOPInstanceUID). This allows blacklisting an entire study or only specific images.
  • [breaking] Config on live system may need updated
• Change the extraction directory generation to be <projname>/image-requests/<extractname>. Fixes MVP Service #159

Fixed

• Fixed IsIdentifiable rule order being the order the files are detected in rules directory (Now goes IgnoreRules=>ReportRules=>SocketRules)
• Adjust log handling in CTP anonymiser to use SMIlogging setup
• IsIdentifiable case-sensitive rules now implemented with property
• Bufix for fo-dicom image handling race condition in Release mode builds (issue #238)

Changed

• Refactored WhiteListRule to inherit from IsIdentifiableRule (affects serialization).
  • Parent property As replaces IfClassification
  • CaseSensitive replaces IfPatternCaseSensitive and IfPartPatternCaseSensitive (Also fixes serialization bug)
• Bump CommandLineParser from 2.7.82 to 2.8.0
• Bump CsvHelper from 15.0.4 to 15.0.5
• Bump HIC.BadMedicine.Dicom from 0.0.5 to 0.0.6
• Bump HIC.DicomTypeTranslation from 2.2.2 to 2.3.0
• Bump HIC.RDMP.Dicom from 2.0.9 to 2.1.5
• Bump HIC.RDMP.Plugin from 4.0.2 to 4.1.3
• Bump Magick.NET-Q16-AnyCPU from 7.16.0 to 7.20.0
• Bump Microsoft.CodeAnalysis.CSharp.Scripting from 3.5.0 to 3.6.0
• Bump Microsoft.Extensions.Caching.Memory from 3.1.3 to 3.1.5
• Bump MongoDB.Driver from 2.10.3 to 2.10.4
• Bump StackExchange.Redis from 2.1.30 to 2.1.58
• Bump System.IO.Abstractions from 10.0.8 to 12.0.1
• Bump YamlDotNet from 8.1.0 to 8.1.2
• Bump fo-dicom.Drawing from 4.0.4 to 4.0.5
• Pinned fo-dicom.NetCore to 4.0.5

v1.8.1

Changelog

Fixed

• Fix null check bug in CohortPackager when no files match the extraction filter