You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Depending on the application - model and problem sizes - it's possible to make the inference very much faster by doing it in batches (packing vector-sized inputs into a 2D array) and replacing the matrix-vector multiplications by matrix-matrix which are delegated to a BLAS library. I have a Fortran application based on FKB, or actually its earlier incarnation neural-Fortran, where I did such that (I referenced neural-Fortran in my paper). It works well, and the nice thing is it's trivial to run the code on GPU's too:
#ifdef USE_CUDA
#define sgemm cublassgemm
#endif
plus some OpenACC directives above the bias addition and activation loops. You can find my code here. I think a similar batched output procedure for 2D arrays would be a valuable contribution to the main repo. I am happy to work on a pull request if you agree. If so let me know if you'd like to keep the GPU stuff: I'd have to add a few things to make it more general, like copying the input array to device, and creating the intermediate arrays for hidden layers (in my code I can get away with just two intermediate arrays where I do pointer swapping, because my models had the same number of neurons in all hidden layers).
There are a few other points too:
should DGEMM be called if input data is in double precision?
Hi,
Depending on the application - model and problem sizes - it's possible to make the inference very much faster by doing it in batches (packing vector-sized inputs into a 2D array) and replacing the matrix-vector multiplications by matrix-matrix which are delegated to a BLAS library. I have a Fortran application based on FKB, or actually its earlier incarnation neural-Fortran, where I did such that (I referenced neural-Fortran in my paper). It works well, and the nice thing is it's trivial to run the code on GPU's too:
plus some OpenACC directives above the bias addition and activation loops. You can find my code here. I think a similar batched output procedure for 2D arrays would be a valuable contribution to the main repo. I am happy to work on a pull request if you agree. If so let me know if you'd like to keep the GPU stuff: I'd have to add a few things to make it more general, like copying the input array to device, and creating the intermediate arrays for hidden layers (in my code I can get away with just two intermediate arrays where I do pointer swapping, because my models had the same number of neurons in all hidden layers).
There are a few other points too:
The text was updated successfully, but these errors were encountered: