OpenVINO vs Torch Inference

[alexriedel1]

Hello,
considering using a GPU for inferencing, will the inference using a openvino IR outperform the torch cuda inferencing in terms of speed?

Thanks, Alex

[samet-akcay]

Hi @alexriedel1, depends on the model. The use of GPU is advantageous in many cases.
OpenVINO accelerates the CPU inference, though.

Here is an example that I ran some time ago using NVIDIA 3090 GPU and i9-10980XE CPU. Please note that these stats are not official and experimental only.

[ke-dev]

Hi @alexriedel1, depends on the model. The use of GPU is advantageous in many cases. OpenVINO accelerates the CPU inference, though.

Here is an example that I ran some time ago using NVIDIA 3090 GPU and i9-10980XE CPU. Please note that these stats are not official and experimental only.

Hi, @samet-akcay how to get How to get the FP32 and INT8 models? I want to do more similar tests, can you provide some methods?

best regards

[alexriedel1]

Thanks @samet-akcay !
Though these stats are only experimental, I'd be highly interested to see more of these 😄

[samet-akcay]

We plan to add some jupyter notebooks to show this sort of experimentation

[samet-akcay]

Meanwhile, you could use the following benchmarking script to reproduce the throughput results for gpu and cpu
https://github.com/openvinotoolkit/anomalib/blob/development/tools/benchmarking/benchmark.py