You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PR #663 is going to bring support for batch suggest operations.
The STWFSA backend could benefit from implementing _suggest_batch instead of _suggest. It could process a batch of texts with parallel and/or vector operations.
The text was updated successfully, but these errors were encountered:
Unfortunately, the results were not very encouraging. Batched suggest done this way seems to be slower than the original. Maybe switching to a new representation for suggestion results (see #678) could help.
I also tried using the predict_proba method of stwfsapy, which returns the results as a sparse matrix. But here the problem is that stwfsapy internally uses different numeric IDs for concepts than Annif, so there would have to be an ID mapping mechanism to convert the results into something that Annif can use.
I'm too lazy to make a table, but here are the main test results. I'm evaluating a YSO STWFSA English model on jyu-theses/eng-test on my 4 core laptop.
Before (master)
1 job
User time (seconds): 201.96
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:23.56
4 jobs
User time (seconds): 288.02
Elapsed (wall clock) time (h:mm:ss or m:ss): 2:19.72
PR #663 is going to bring support for batch suggest operations.
The STWFSA backend could benefit from implementing _suggest_batch instead of _suggest. It could process a batch of texts with parallel and/or vector operations.
The text was updated successfully, but these errors were encountered: