Kitodo Community Meeting - Change the workflow for existing processes

[BartChris]

Several Kitodo community members articulated the desire to be able to change the workflow of already existing processes. This is necessary for several reasons:

• a additional task needs to be added for generating a URN/DOI for an object or for executing an OCR recognition on an already existing processes
• a task needs to be changed in a specific way (specifying permissions, changing script parameters).

Right now (in the current Master branch) it is possible to update an existing workflow and to generate new processes with the changed workflow. The changes however are not transfered to the old processes.

The topic has already been discussed in several occasions and been added to the potential candidates for the development fund (#4840)

During the community meeting in Braunschweig in October 2022 (https://www.kitodo.org/community/praxistreffen) it became clear that before the implementation can take place, it has to be specified, what the actual requirements are. I will try to summarize as much of the discussed aspects as possible.

The discussions in Braunschweig concluded that Kitodo Production should at least allow the following workflow changes to existing processes:

• add a single new workflow step
• delete a single workflow step
• change a single workflow step

The idea behind this is to reduce complexity by allowing only smaller changes to workflow of existing processes.

When those changes are applied to old processes the implementation needs to be written in a way that it could handle changes done to a lot (thousands) of processes.

The meeting concluded that the goal should be to come up with a precise enough specification as basis of the technical implementation. What would maybe also greatly help might be mockups visualising how the update of workflow steps could actually work in practice. This Github discussion could be use to collect ideas related to that goal.

[BartChris]

One important point in the discussion is the question, wether workflows should be modifiable or wether used workflows are immutable and changing an already used workflow requires to copy the old workflow and create a new one.
The discussion at the community meeting made clear that there are arguments for both:

• Changing an existing workflows (as it is possible in current master) allows for quick changes to the workflow without having to generate a new workflow and coming up with the new name for that. In order for using a newly created workflow it is also necessary to create a new process template and also to come up with a name for that since processes are always generated from templates (which refer to a workflow) not directly from workflows.
• Disallowing changes to existing workflows allows to clearly associate a process to a specific workflow. When a workflow is changed it is not clear on which version of that workflow this processesd was based on. It might also lead to confusion across projects. @matthias-ronge elaborates this here Workflow creation and changes - Additional Dev-Mode to allow for more flexibility #4334 (comment) . Requiring to always create a new workflow (and a new process template!) prevents that confusion.

The community meeting outlined more aspects which should be considered here

• If already used workflows can be changed it is probably necessary that the systems informs the user about which processes do conform to a changed workflow and which processes are based on an outdated workflow, so that the user can make informed decisions about the processes.
• if workflows can be changed all the time, specific scenarios are probably not possible. This includes changing the same workflow in different ways, e.g. constructing a changed workflow with a new OCR step or creating a workflow with a changed script step, by starting from the same old workflow which is used by all not yed converted old processes.

Another important aspect is the question what kind of changes should be allowed at all and how much information needs to be preserved.

1. Is it possible to restrict changes to specific changes? As outlined in Handling of subsequent changes to workflows #5351: detecting the nature of a change to a workflow and identifying which tasks changed in which way is hard.

2. Is it necessary to preserve information attached to the old workflow steps and transfer them to the new workflow? What about the names of the editor and the state of the tasks.

3. What would be lost with a pragmatic approach? Assigning a changed workflow to a task could for example be implemented by deleting all the old tasks of a process, adding all the new tasks and let the user decide which task of the new workflow should be the open task for the updated processes.

[matthias-ronge]

From a technical point of view, here is what we have: In Production 2, the workflow steps were set up in the production template as a table, and for each process, the steps were copied. In Production 3, you first create the workflow in the fancy graphical editor, and from that you create a production template. Technically, this is when the table of workflow steps in the production template is created based on the graphic. (You are not allowed to create things in the graphical editor, that cannot be represented as a table.) Then the same thing happens: when creating a process, the workflow steps are copied into it. Technically, this means that you can change the workflow or the steps of the production template without changing the tasks in existing processes. This is the technical reality on which we must base all our considerations.

@stroetgen correctly remarked at the practice meeting, that if we delete the tasks in existing processes, we can lose important information, for example about workers, or correction runs.

Against this background, the following procedure seems to me to be the most plausible: Active workflows and production templates cannot be changed. (This is also to protect other clients using the same workflow when working in a multi-client system.) There should be one exception to this rule: the roles [which are client-specific] at each workflow step should be changeable later, and the corresponding change should be reflected in the production template, and should affect all running processes. (Technically, the tasks then must not determine the role by themselves, but by their linked task from the production template.) Further changes should not be possible to a workflow in use. (Remember that the workflow is also used by other clients. They probably don't want the changed behaviour.)

If other properties of a workflow in use are to be changed, this generally requires creating a copy of the workflow and creating a new production template. (Perhaps we can automate the creation of production templates in a copy, if a generic title of the production template is accepted.) Existing processes can then be moved to a specific task of the new production template by an authorized administrator, for example with a KitodoScript. This would also allow processes to be moved into an entirely new workflow. In the process, the existing tasks should be put into an aging state:

• finished → petrified
• inwork → canceled
• open / close → skipped

The tasks of the new workflow are appended. In this way, the loss of information mentioned at the beginning can be avoided.

Changing the workflow would have to be implemented as a long-running task in the task manager. If one of the newly open tasks is an automatic task, its processing must be processed in another separate thread in the task manager. (The internal logic of the task manager ensures that not too many tasks are started at the same time.)

[BartChris]

This sounds reasonable and probably implies some general ideas on how to use Kitodo in an ideal way (reflecting the restrictions of the workflow engine), which i would describe as following:

The development of a workflow mostly happens on the test system and is transferred to the production system after a longer conception phase. So changes on the production system should mostly happen because of a real change in a workflow, not because of wrong assumptions or errors (those should be elimated while testing or before creating a lot of processes). So it is very justifiable to create a new worfklow and template for the new procedure. In order to keep the consistency in the system, workflows should never be changed. Making Copy pasting of workflows and templates easier on the the other hand probably eases development in the test system, where workflows could change rapidly.

This probably is the most coherent way of using Kitodo. One question here: Are workflows really exchanged between/accessible by muliple clients (Mandanten)? On my system both clients have seperate workflows and do not see workflows of the other client, so changes to workflows should not affect other clients or?

Preserving the old tasks (in a changed state) and attach them before the new workflow is probably a good idea as well.

For me the question would be, if there should be a little bit more flexibility

• maybe the Kitodo admin can decide if he wants to keep steps based on old workflows or not. I am thinking here about multiple changes to the workflow of existing processes: Do we want to chain all old (now no longer relevant) tasks if the user does multiple workflow changes at different point in time? The admin should decide if he or she wants to keep the old information or not
• i am wondering wether one could extend your idea of allowing changes to the permissions of each task. One option would maybe also be to allow changes only to the right side of the workflow edit screen. (colored red) The sequence of tasks then could not be changed, only the specific attributes. This would allow to make changes to the attributes of a task, not the sequence of tasks itself. But this is just an idea, wich i haven' really thought through.