2.7 release (#16)

* Add developer onboarding guide
* Increment emap versions
* Remove unnecessary check for PR in comment action
* Add health endpoint to hl7 reader

.github/workflows/comment.yaml

@@ -21,7 +21,6 @@ jobs:
           template: SOP/pr_checklist.md
       - name: Comment
         uses: peter-evans/create-or-update-comment@v1
-        if: ${{ github.event_name == 'pull_request' }}
         with:
           issue-number: ${{ github.event.pull_request.number }}
           body: ${{ steps.get-pr-checklist.outputs.result }}

README.md

@@ -1,7 +1,12 @@
 # Emap
+
 A monorepo for all core Emap functions
 
-# Basic layout
+# Setup
+
+The EMAP project follows this structure, for deploying a live instance of EMAP follow the instructions
+in [docs/core.md](docs/core.md).
+
 ```
 EMAP [your root emap directory]
 ├── config [config files passed to docker containers, not in any repo]
@@ -14,8 +19,12 @@ EMAP [your root emap directory]
 │   ├── [etc.]
 ```
 
-# Using IntelliJ with emap
-How to [configure IntelliJ](docs/intellij.md) to build emap and run tests.
+## Developer onboarding
+
+- How to [configure IntelliJ](docs/intellij.md) to build emap and run tests.
+- [Onboarding](docs/dev/onboarding.md) gives details on how data items are processed and the test strategies used.
+
 
 # Monorepo migration
+
 How were [old repos migrated into this repo?](docs/migration.md)

core/pom.xml

@@ -5,23 +5,23 @@
   <groupId>uk.ac.ucl.rits.inform</groupId>
   <artifactId>core</artifactId>
   <packaging>jar</packaging>
-  <version>2.6</version>
+  <version>2.7</version>
   <name>Emap Core Processor</name>
   <url>http://maven.apache.org</url>
 
   <parent>
     <groupId>org.springframework.boot</groupId>
     <artifactId>spring-boot-starter-parent</artifactId>
-    <version>2.7.13</version>
+    <version>2.7.18</version>
   </parent>
 
   <properties>
     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
     <java.version>17</java.version>
     <checkstyle.version>10.3.1</checkstyle.version>
     <checkstyle.plugin.version>3.3.0</checkstyle.plugin.version>
-    <emap-star.version>2.6</emap-star.version>
-    <emap-interchange.version>2.6</emap-interchange.version>
+    <emap-star.version>2.7</emap-star.version>
+    <emap-interchange.version>2.7</emap-interchange.version>
     <go-offline-maven-plugin.version>1.2.8</go-offline-maven-plugin.version>
   </properties>
 

docs/README.md

@@ -12,6 +12,7 @@ This is a work in progress.
 * [2021-09-27](changelogs/2021-09-change_log.md)
 * [2021-11-30](changelogs/2021-11-change_log.md)
 * [2022-02-14](changelogs/2022-02-change_log.md) 💝
+* [2024-02-14](changelogs/2024-02-change_log.md) 💝
 
 
 ### Data and Validation

docs/SOP/release_procedure.md

@@ -41,4 +41,6 @@ At this point we create a list in the planner with vital things to be completed
 
    1. Prepare and do demo.
 
-   1. Shut down old instance.  
+   1. Shut down old instance, changing the HL7_READER_PORT in the global-configuration.yaml to something other than 8080.  
+
+   1. Update the HL7_READER_PORT in the global-configuration.yaml in the new instance to 8080 and re-up the hl7-reader.

docs/changelogs/2024-02-change_log.md

@@ -0,0 +1,86 @@
+# EMAP Release
+
+**Date: 2024-02-14 Changes in this release**
+
+---
+
+### Changes/fixes
+
+- We have added a healthcheck endpoint for the hl7-reader, so that users can check the status of the hl7-reader without
+  needing to access the server directly.
+    - The current live version can be found at: `http://<GAE hosting EMAP>:8080/actuator/health`
+    - Please contact the developer team to find out the GAE hostname.
+- All services now run using Java 17 (LTS)
+- Open source release using mono-repo
+- Improved testing to ensure database consistency
+
+### Tables changed
+
+Department specialities can change over time, we've updated the database to reflect this.
+The following tables have been updated:
+
+| Table           | Attributes added | Attributes removed |
+|:----------------|:-----------------|:-------------------|
+| DepartmentState | speciality       | -                  |
+| Department      | -                | speciality         
+
+
+```mermaid
+erDiagram
+bed {
+    varchar(255) hl7string
+    bigint room_id
+    bigint bed_id
+}
+department {
+    varchar(255) hl7string
+    bigint internal_id
+    varchar(255) name
+    bigint department_id
+}
+department_state {
+  timestamp-with-time-zone stored_from
+  timestamp-with-time-zone valid_from
+  timestamp-with-time-zone stored_until
+  timestamp-with-time-zone valid_until
+  varchar(255) speciality
+  varchar(255) status
+  bigint department_id
+  bigint department_state_id
+}
+location {
+  varchar(255) location_string
+  bigint bed_id
+  bigint department_id
+  bigint room_id
+  bigint location_id
+}
+room {
+  varchar(255) hl7string
+  varchar(255) name
+  bigint department_id
+  bigint room_id
+}
+
+department  ||--|{  department_state: department_id
+location |{--o| department: department_id
+location |{--o| room: room_id
+location |{--o| bed: bed_id
+department  ||--o{  room: room_id 
+room  ||--o{  bed: bed_id
+```
+
+---
+<!--
+## Data sources
+
+### Repository Versions
+
+| Repository            | Version |
+| :-                    | :-:     |
+|Hl7-processor          | 2.7     |
+|Emap_interchange       | 2.7     |
+|Emap-Core              | 2.7     |
+|Inform-DB              | 2.7     |
+|Hoover                 | 2.7     |
+>

docs/dev/img/core-message-processing.mmd

@@ -0,0 +1,66 @@
+classDiagram
+  direction BT
+  class InformDbOperations {
+    + processMessage(AdtMessage) void
+    + processMessage(LabOrderMsg) void
+  }
+  class LabCache {
+    + saveEntityAndUpdateCache(RowState~LabBattery, LabBatteryAudit~) LabBattery
+    + updateLabTestDefinitionCache(RowState~LabTestDefinition, LabTestDefinitionAudit~) LabTestDefinition
+    + findExistingLabTestDefinition(String, String) LabTestDefinition
+    + createLabBatteryElementIfNotExists(LabTestDefinition, LabBattery, Instant, Instant) LabBatteryElement
+  }
+  class LabController {
+    + processLabOrder(Mrn, HospitalVisit?, LabOrderMsg, Instant) void
+    + deleteLabOrdersForVisit(HospitalVisit, Instant, Instant) void
+    + processLabMetadata(LabMetadataMsg, Instant) void
+  }
+  class LabOrderAuditRepository {
+    <<Interface>>
+    + findAllIds(Long, Long, Instant, String) List~Long~
+    + previouslyDeleted(Long, Long, InterchangeValue~Instant~, InterchangeValue~String~) boolean
+    + findAllByHospitalVisitIdIn(Iterable~Long~) List~LabOrderAudit~
+  }
+  class LabOrderController {
+    + processSampleAndOrderInformation(Mrn, HospitalVisit, LabBattery, LabOrderMsg, Instant, Instant) LabOrder
+    + deleteLabOrder(LabOrder, Instant, Instant) void
+    + findLabBatteryOrThrow(String, String) LabBattery
+    + getLabOrdersForVisit(HospitalVisit) List~LabOrder~
+    + processLabSampleAndDeleteLabOrder(Mrn, LabBattery, HospitalVisit, LabOrderMsg, Instant, Instant) void
+    + getOrCreateLabBattery(String, String, Instant, Instant) LabBattery
+  }
+  class LabOrderRepository {
+    <<Interface>>
+    + findByLabBatteryIdAndLabSampleId(LabBattery, LabSample) Optional~LabOrder~
+    + findByLabBatteryIdAndLabSampleIdAndValidFromBefore(LabBattery, LabSample, Instant) Optional~LabOrder~
+    + findByLabBatteryIdBatteryCodeAndLabSampleId(String, LabSample) Optional~LabOrder~
+    + findByLabSampleIdExternalLabNumber(String) Optional~LabOrder~
+    + findAllByHospitalVisitId(HospitalVisit) List~LabOrder~
+  }
+  class LabProcessor {
+    + processMessage(LabMetadataMsg, Instant) void
+    + processMessage(LabOrderMsg, Instant) void
+  }
+  class LabTestDefinitionAuditRepository {
+    <<Interface>>
+  }
+  class LabTestDefinitionRepository {
+    <<Interface>>
+    + findByLabProviderAndTestLabCode(String, String) Optional~LabTestDefinition~
+    + findByTestLabCode(String) Optional~LabTestDefinition~
+  }
+  class EmapOperationMessageProcessor {
+    <<Interface>>
+    + processMessage(LabMetadataMsg) void
+    + processMessage(AdtMessage) void
+  }
+
+  InformDbOperations *--> LabProcessor
+  LabCache *--> LabTestDefinitionAuditRepository
+  LabCache *--> LabTestDefinitionRepository
+  LabController *--> LabCache
+  LabController *--> LabOrderController
+  LabOrderController *--> LabOrderAuditRepository
+  LabOrderController *--> LabOrderRepository
+  LabProcessor *--> LabController
+  InformDbOperations ..> EmapOperationMessageProcessor

docs/dev/img/hl7-message-processing.mmd

@@ -0,0 +1,82 @@
+classDiagram
+    direction BT
+    class AblLabBuilder {
+        + build(String, ORU_R30) Collection~LabOrderMsg~
+    }
+    class AppHl7 {
+        + main(String[]) void
+        + mainLoop(Publisher, IdsOperations) CommandLineRunner
+    }
+    class ConsultFactory {
+        ~ makeConsult(String, ORM_O01) ConsultRequest
+    }
+    class IdsOperations {
+        + getHl7DataSource() EmapDataSource
+        + getNextHL7IdsRecordBlocking(int) IdsMaster
+        + close() void
+        + parseAndSendNextHl7(Publisher, PipeParser) void
+        + getNextHL7IdsRecord(int) IdsMaster
+        + messageFromHl7Message(Message, int) List~EmapOperationMessage~
+        + populateIDS() CommandLineRunner
+        + getIdsEmptyOnInit() boolean
+        ~ getLatestProcessedId() IdsProgress
+    }
+    class LabFunnel {
+        + buildMessages(String, ORU_R30, OrderCodingSystem) Collection~LabOrderMsg~
+        + buildMessages(String, ORR_O02, OrderCodingSystem) Collection~LabOrderMsg~
+        + buildMessages(String, ORU_R01, OrderCodingSystem) Collection~LabOrderMsg~
+        + buildMessages(String, ORM_O01, OrderCodingSystem) List~LabOrderMsg~
+    }
+    class LabOrderBuilder {
+        ~ getCodingSystem() OrderCodingSystem
+        ~ getEpicCareOrderNumberOrc() String
+        ~ setQuestions(Collection~NTE~, String, Pattern) void
+        + getMsg() LabOrderMsg
+        ~ getEpicCareOrderNumberObr() String
+        ~ setBatteryCodingSystem() void
+        ~ populateOrderInformation(ORC, OBR) void
+        ~ populateObrFields(OBR, boolean) void
+        ~ setSourceAndPatientIdentifiers(String, PatientInfoHl7) void
+        ~ populateObrFields(OBR) void
+        ~ setOrderTemporalInformation(Instant) void
+    }
+    class OrderAndResultService {
+        ~ buildMessages(String, ORR_O02) Collection~EmapOperationMessage~
+        ~ buildMessages(String, ORM_O01) Collection~EmapOperationMessage~
+        ~ buildMessages(String, ORU_R30) Collection~EmapOperationMessage~
+        ~ buildMessages(String, ORU_R01) Collection~EmapOperationMessage~
+    }
+    class WinPathLabBuilder {
+        + build(String, ORU_R01) Collection~LabOrderMsg~
+        + build(String, ORR_O02) Collection~LabOrderMsg~
+        + build(String, ORM_O01) List~LabOrderMsg~
+    }
+    class ConsultRequest {
+        + getEpicConsultId() Long
+        + getMrn() String
+        + setQuestions(Map~String, String~) void
+        + getQuestions() Map~String, String~
+        + setMrn(String) void
+        + processMessage(EmapOperationMessageProcessor) void
+    }
+    class LabOrderMsg {
+        + getSpecimenType() InterchangeValue~String~
+        + getOrderControlId() String
+        + getLabResultMsgs() List~LabResultMsg~
+        + setOrderDateTime(InterchangeValue~Instant~) void
+        + setOrderControlId(String) void
+        + setSpecimenType(InterchangeValue~String~) void
+        + processMessage(EmapOperationMessageProcessor) void
+        + addLabResult(LabResultMsg) void
+    }
+
+    AblLabBuilder --|> LabOrderBuilder
+    AppHl7 ..> IdsOperations
+    IdsOperations *--> OrderAndResultService
+    LabFunnel ..> AblLabBuilder
+    LabFunnel ..> WinPathLabBuilder
+    OrderAndResultService *--> ConsultFactory
+    OrderAndResultService ..> LabFunnel
+    WinPathLabBuilder --|> LabOrderBuilder
+    LabOrderBuilder ..> LabOrderMsg: «create»
+    ConsultFactory ..> ConsultRequest: «create»

docs/dev/img/hoover-development.mmd

@@ -0,0 +1,63 @@
+classDiagram
+    direction BT
+    class Application {
+        + main(String[]) void
+        + locationMetadataProcessor(LocationMetadataQueryStrategy) Processor
+        + flowsheetProcessor(FlowsheetQueryStrategy) Processor
+        + runBatchProcessor(Processor, Processor, ...) BatchProcessor
+    }
+    class LocationMetadata {
+        + getHl7String() String
+        + getDepartmentId() Long
+        + getRoomMetadata() RoomMetadata
+        + getBedMetadata() BedMetadata
+        + setHl7String(String) void
+        + setDepartmentId(Long) void
+        + setRoomMetadata(RoomMetadata) void
+        + setBedMetadata(BedMetadata) void
+        + processMessage(EmapOperationMessageProcessor) void
+    }
+    class LocationMetadataDTO {
+        + getInterchangeMessage() LocationMetadata
+    }
+    class LocationMetadataQueryStrategy {
+        + getSqlQueryFilename() String
+        + getInstantCalculator() InstantCalculator
+        + fixInitialProgress(Instant) Instant
+        ~ getClarityData(Instant) List~LocationMetadataDTO~
+        + getBatchOfInterchangeMessages(Instant, Instant, String) List~ImmutablePair~ EmapOperationMessage, String~~
+        + getUpdatedProgress() EtlHooverProgress
+        + getName() String
+    }
+    class QueryStrategy {
+        <<Interface>>
+        + getBatchOfInterchangeMessages(Instant, Instant, String) List~ImmutablePair~ EmapOperationMessage, String~~
+        + fixInitialProgress(Instant) Instant
+        + getInstantCalculator() InstantCalculator
+        + getName() String
+        + getUpdatedProgress() EtlHooverProgress
+        + getBatchOfInterchangeMessages(DataTypeProgress) List~ImmutablePair~ EmapOperationMessage, String~~
+        + getSqlQueryFilename() String
+        + interpretPreviousProgress(EtlHooverProgress) DataTypeProgress
+        + getSqlQuery() String
+    }
+    class Processor {
+        + getDataType() String
+        + getPreviousProgress() List~EtlHooverProgress~
+        + execute() boolean
+        + executeOne(EtlHooverProgress) boolean
+    }
+
+    class BatchProcessor {
+        + registerDataTypeProcessor(Processor) void
+    }
+
+    Application ..> LocationMetadataQueryStrategy
+    LocationMetadataDTO ..> LocationMetadata: «create»
+    LocationMetadataQueryStrategy ..> LocationMetadata
+    LocationMetadataQueryStrategy ..> LocationMetadataDTO
+    LocationMetadataQueryStrategy ..|> QueryStrategy
+    Processor *--> QueryStrategy
+    BatchProcessor "1" *--> "dataTypeProcessors *" Processor
+    Application ..> Processor
+    Application ..> BatchProcessor: «create»