Major memory control refactoring #199
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
dmitriyb
commented
Aug 27, 2024
dmitriyb
commented
- Added option for manual array allocation control
- Allocator storage refactoring for less memory consumption and faster access
- Controlling mechanisms enhanced, including timed cache clearing
- Abandoning array wrapper structures in favor of working with raw primitive arrays
…sm from special function referencing to simple output copying.
…y management: use of primitive arrays directly.
…ling: streamlined array type handling and improved memory limit checks within create and reset methods; KIModel predict improved for NoAllocator case.
…ling: added new type for limiter which works with manually managed ndarrays, added manual ndarray handling in Attention and TensorExtensions, moved to use standard DataType enum instead of ArrayTypes.
…ling Added manual NDArray handling, refactored existing operations to use standard DataType enum instead of ArrayTypes, and optimized memory allocations across multiple modules.
Replaced static method calls with instance method calls for NDArray storage wrappers. Introduced internal storage fields in `ManualAllocatorContext` to streamline access and management of different data types.
Deleted obsolete memory storage and allocator context classes. Added new classes with improved manual and auto handling storage for arrays, enhancing memory management and performance.
… for better primitive types handling: this solution gives less double primitive array allocations when Array<Any> changes to actual type.
…ve handling in primitive array storage classes.
cupertank
requested changes
Aug 30, 2024
...mMain/kotlin/io/kinference/ndarray/arrays/memory/contexts/BaseAllocatorContextWithStorage.kt
Outdated
Show resolved
Hide resolved
.../inference-core/src/jvmMain/kotlin/io/kinference.core/operators/layer/attention/Attention.kt
Outdated
Show resolved
Hide resolved
.../inference-core/src/jvmMain/kotlin/io/kinference.core/operators/layer/attention/Attention.kt
Outdated
Show resolved
Hide resolved
...rc/jvmMain/kotlin/io/kinference.core/operators/layer/normalization/SkipLayerNormalization.kt
Show resolved
Hide resolved
...e/src/jvmMain/kotlin/io/kinference/ndarray/arrays/memory/storage/AutoArrayHandlingStorage.kt
Outdated
Show resolved
Hide resolved
...src/jvmMain/kotlin/io/kinference/ndarray/arrays/memory/storage/ManualArrayHandlingStorage.kt
Outdated
Show resolved
Hide resolved
ndarray/ndarray-core/src/jvmMain/kotlin/io/kinference/ndarray/arrays/memory/MemoryManager.kt
Outdated
Show resolved
Hide resolved
ndarray/ndarray-core/src/jvmMain/kotlin/io/kinference/ndarray/extensions/PrimitiveExtensions.kt
Outdated
Show resolved
Hide resolved
ndarray/ndarray-core/src/jvmMain/kotlin/io/kinference/ndarray/extensions/PrimitiveExtensions.kt
Outdated
Show resolved
Hide resolved
...y/ndarray-core/src/jvmMain/kotlin/io/kinference/ndarray/extensions/gelu/BiasGeluPrimitive.kt
Outdated
Show resolved
Hide resolved
…constants, removed commented-out and redundant code.
…phic, merge ParallelismLimiterContext and its thread limiter behavior into PredictionContext.
cupertank
previously approved these changes
Sep 2, 2024
...array-core/src/jvmMain/kotlin/io/kinference/ndarray/extensions/constants/BooleanConstants.kt
Outdated
Show resolved
Hide resolved
…n block handling (avoid Integer boxing operations).
cupertank
approved these changes
Sep 2, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.