-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory problem #87
Comments
Been working on improvements to get around RAM-limitation based memory issues when using A couple quick questions:
There is the possibility that you can downsample the raw movies (https://bahanonu.github.io/ciatah/pipeline_detailed_downsample_raw/) to reduce file/matrix size, as often the higher pixels/micron resolution is not needed for cell detection and activity trace extraction. |
Hi!
Maximum possible array: 139597 MB (1.464e+11 bytes) * Limited by System Memory (physical + swap file) available. We tried downsampling in the inscopix software and that decreased the size of the videos and made it work. Thanks again for your help!! |
Great to hear the downsampling works! For Inscopix miniscope movies often 2x or 4x downsampling in each dimension usually works fine. Did a movie play in ImageJ or MATLAB? Can you send me the |
20210910_113823_preprocess.log Here the file! A snippet played in ImageJ, yes. |
Looks like a turboreg error occurred during 2nd round of motion correction. Do you see Also, can you run with the below Else, if you can run with a single turboreg iteration and send me a copy of the movie (only a couple frames, e.g. see below), that can help me diagnose what happened with projective. e.g. the thisPath='full_path_to_output_movie';
[thisDir,filename1,ext1]=fileparts(thisPath);
saveMatrixToFile(loadMovieList(thisPath,'frameList',1:500),[thisDir filesep filename1 '_snippet' ext1]); |
We managed in the end to make it work: 5 iterations with a 4x downsampled video (size 5Gb approx.) managed to be preprocessed on a 512GB RAM computer and the results of motion correction are really great. After preprocessing we need to use extract as an extraction method. For our application, people in our lab have worked in conjuction with the extract team to finetune the config and get results. Nonetheless, this finetuning was done on non-downsampled videos. So we went back and tried to run ciatah preprocessing on non downsampled files (approx. 80Gb) on a 512Gb RAM computer but this led to the out memory error again. Is there any way we can improve RAM efficiency? Very sorry for the trouble and thanks a lot for your help. |
Thanks for the update on this and glad to hear the downsampled movies worked well for you-all. re: non-downsampled movies. It is possible to improve the RAM efficiency by running ciatah pre-processing via the command-line (e.g. https://bahanonu.github.io/ciatah/api_example_pipeline/) so that certain steps (e.g. motion correction) are only done on chunks of the movie at a time (this is what we do for some of our cell extraction algorithms or other steps). Could you let me know what steps you are running for ciatah pre-processing? I can get back with a script that can reduce RAM usage as still working to update CIAtah GUI for other users to handle this issue. |
Hey, The preprocess steps we currently wish to use are:
Thanks again! |
When I try to preprocess videos bigger than my JAVA heap memory (34Gb, max allowed) I get the error below and I was wondering how can I analyse videos with size bigger than that.
Thank you very much in advance for your help
pre-allocating single ones matrix: 800 1280 38801
@@@@@@@
Error using ones
Requested 800x1280x38801 (148.0GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.
Error in loadMovieList (line 472)
outputMovie = ones(preallocSize,imgClass);
Error in ciatah/modelPreprocessMovieFunction (line 729)
thisMovie = loadMovieList(movieList,'convertToDouble',0,'frameList',thisFrameList,'inputDatasetName',options.datasetName,'treatMoviesAsContinuous',options.turboreg.treatMoviesAsContinuousSwitch,'loadSpecificImgClass','single');
Error in ciatah/modelPreprocessMovie (line 37)
obj.modelPreprocessMovieFunction('folderListPath',folderListInfo,'fileFilterRegexp',options.fileFilterRegexp,'datasetName',options.datasetName,'frameList',[]);
Error in ciatah/runPipeline (line 202)
obj.(thisFxn{1});
Error in ciatah/display (line 6)
obj.runPipeline;
@@@@@@@
pre-allocating movies to display...
+++++++
no movie!
MIJ 1.3.9: Matlab to ImageJ Interface
More Info: http://bigwww.epfl.ch/sage/soft/mij/
Help: MIJ.help
JVM> 1.3.9
JVM> Version: 1.8.0_181
JVM> Total amount of memory: 387584 Kb
JVM> Amount of free memory: 244514 Kb
ImageJ> Version:1.53k
ImageJ> Memory:141MB of 29964MB (<1%)
ImageJ> Directory plugins: Not specified
ImageJ> Directory macros: Not specified
ImageJ> Directory luts: C:\Users\realtime\Documents\MATLAB\luts
ImageJ> Directory image: Not specified
ImageJ> Directory imagej: C:\Users\realtime\Documents\MATLAB
ImageJ> Directory startup: C:\Users\realtime\Documents\MATLAB
ImageJ> Directory home: C:\Users\realtime\
Status> ImageJ is running.
Warning: The DrawMode property will be removed in a future release. Use the
SortMethod property instead.
The text was updated successfully, but these errors were encountered: