Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved 2D preprocessing script #287

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

hamzaMahdi
Copy link

Thank you for this repo. I ran into issues with the 2D script so i modified it:
Optimize image and mask processing for improved performance and storage efficiency

  1. Multiprocessing: Implemented parallel processing using Python's multiprocessing module, significantly reducing execution time by utilizing all available CPU cores.

  2. Image data type optimization: Changed from implicit float32 (4 bytes/pixel) to explicit uint8 (1 byte/pixel), reducing image data storage by approximately 75%.

  3. Compressed NPZ format: Implemented np.savez_compressed() instead of np.save(), further reducing file sizes through compression.

  4. Single file storage: Combined image and mask data into a single compressed NPZ file, reducing overhead and improving file management.

These changes significantly reduce storage requirements, especially for large datasets. For example, a 1024x1024 RGB image that previously required about 12 MB (4 bytes * 1024 * 1024 * 3) now requires only about 3 MB (1 byte * 1024 * 1024 * 3), before additional compression from the NPZ format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant