issue with metadata files when using cleanup plugin

[shilpakallaganad1]

The cleanup plugin deletes important meta-data files resulting in corrupting the Conan repositories. The problem is that meta-data files (eg. index.json) are not downloaded by a client and therefore the download count stays always on zero, resulting in these artifacts being returned by the search artifactsNotDownloadedSince() and therefore removed afterwards. This causes Conan clients to not detect which recipe revisions (and packages) are available for a specific package.
This issue is occurring for multiple repositories which are aggregated below:
npm: under the repo root there is a .npm folder that contains a package.json for each package. This file includes all the revisions of an npm package published in the repository. This file has as well a download count of zero and would be deleted by the plugin.
cocoapods: inside the .specs folder in the repository root, each package has a .podspec file that contains meta-data about the package. The download count of this file is as well zero even if the corresponding .tar.gz file was already downloaded by the pod cli tool.
conda: seems to be as well safe since current_repodata.json, repodata.json were downloaded. But based on the download count it does not match the download count of the artifacts themselves. So, seems that it might happen that those files have as well a download count of zero and could potentially be removed by the cleanup plugin.
docker: not supported by the plugin (at least this is documented)
debian: I have seen situations where the Packages file was not downloaded but the other files like Packages.bz2 or Packages.gz not. Not sure whether this could cause any issues on a Debian repository.

[DavidRadoorHummel]

Yes, the cleanUp plugin only works with packages types where a package consists of a single artifact (like nuget and chocolatey). We have modified the plugin greatly - so that it ignores meta data for relevant package types.

We had to create our own plugin for cleaning up conan packages. You are not entirely correct though. the top most "index.json" is updated with download statistics every time someone downloads the package in question. It is this file that we're using to evaluate, whether the given conan package should be deleted or not.

I would love to share our plugins and modifications, but JFrog does not review pull requests so the changes will never be merged. I have complained to JFrog but this git repository seems to be very lowly prioritized.

If you really want the plugins we have modified and developed, let me know and I will ask my manager if we're allowed to share them (NDA and all)

I hope my pointers can at least be of some help for you

Best regards
David

[martinm82]

Indeed, I have summarised this issue for package types we use in this issue as well: #319 (comment)

If you could share your solution that would be great. We started to use already a different approach (running cleanup through REST API) since we are on SaaS and user plugins are not allowed there.