Errors due to wrong archive version

[bernt-matthias]

I have users reporting the following error (with strage line breaks)

Unexpected error loading arguments in
q2galaxy: /gpfs1/data/galaxy_server/g
alaxy/database/files/000/575/dataset_
575964.dat was created by 'QIIME
2023.5.1'. The currently installed
framework cannot interpret archive
version '6'.

I guess there is some versioning in the qza files and the users used a wrong one.

I think the metadata stored in the galaxy datatype https://github.com/galaxyproject/galaxy/blob/29290dea0cc78566947fc871fcc5634f11b4ee48/lib/galaxy/datatypes/qiime2.py#L67 can be used by adding a metadata validator https://docs.galaxyproject.org/en/master/dev/schema.html#tool-inputs-param-validator to data parameters that take qza input

[Oddant1]

This is expected behavior on our end. In QIIME 2 2023.5 we changed the provenance archive version from 5 to 6 because we made breaking changes to provenance to add support for output collections. Artifacts produced in QIIME 2 2023.5 and on are not backwards compatible with QIIME 2 releases older than 2023.5, but older artifacts are of course forwards compatible.

We should be able to do that validation, but whether the input is valid or not will depend on which version of QIIME 2 a given galaxy deployment is running. Is it possible for the validator to get clever and determine at run time what it can and cannot accept, or do we need to do it at tool creation time?

Given the versions of QIIME 2 realistically in use it's currently either going to be <=5 or <=6

[bernt-matthias]

The validator has access to the metadata defined here, i.e. the format version. So I guess the supported format version (bound) needs to be set at tool creation time.

[Oddant1]

@bernt-matthias We don't actually have the archive version stored on the metadata there. We have the version of the framework that was used to produce the artifact, so in order for this validator to work we're going to need to add another field to that metadata.

[bernt-matthias]

Can one framework version write different archive versions?

[Oddant1]

No, a given framework version will always write the same archive version, but it can read all archive versions up to and including the one it writes. Old artifacts are compatible with newer versions of QIIME 2, but new artifacts may not be compatible with older versions of QIIME 2.

We can't just use the framework version to determine compatibility though because compatibility isn't broken every time we update QIIME 2, it's only broken every time we create a new archive version. Last time we did that was 2023.5 the last time before that was October of 2018. So artifacts produced by versions of QIIME 2 from 2018.11 to 2023.2 all use archive version 5 meaning you could in principle use an artifact produced by QIIME 2 2023.2 as an input to QIIME 2 2018.11.

We MIGHT be able to fudge things a bit by knowing what version of QIIME 2 we are using and what archive version that supports, but at this point the next release of q2galaxy and the galaxy-tools will support up to version 6. We would need to re-render the old pre-archive version 6 tools. There's no telling when archive version 7 will be a thing, but it likely won't take 5.5 years like the last bump did.

[ebolyen]

This will generally be fixed by updating the the latest toolshed version. We'll have 2024.5 going up next week (which also will support archive 6).

Otherwise, there's not much we can do about it at the moment.