move requests creating failed with a list of 200K files

[zhangxiaomei]

There is a use case in JUNO to move data from disk to tape in large scale. I tried with dirac-dms-create-moving-request, but it looks like it doesn't work with too large data more than 200K files.

"dirac-dms-create-moving-request xxx.txt --List prd109.txt -S IHEP-JUNOEOS -T IHEP-TAPE -X
Will create request(s) with 204754 Ifns
Unable to read metadata for lfns: FileCatalog error ( 1604 : Failed to perform getFileMetadata from any catalog)ERROR: Failed to create Moving Request: Could not read metadata: Filecatalog error ( 1604 : Failed to perform getFileletadata from any catalog)"

Any better solutions? or methods to avoid these bottleneck?

[andresailer]

Split the txt file.

https://stackoverflow.com/a/2016923

[andresailer]

You can also implement the splitting of the large lfnlist in the dirac_dms_create_moving_request command directly.

[zhangxiaomei]

OK, thanks. This is the way we are using now. I am wondering there is a way just in one step.