How to track chips usage over time?

[lapinM]

I'm trying to use the n300 to train my models. However, I can't watch how much memory my model takes up on cards.
Are there similar nvidia-smi methods for tracking memory usage and utilization for each process?

[sbansalTT]

Hello, thanks for the issue. Yup that is a feature that is currently on our radar to add into SMI and is a work in progress :)

[lapinM]

What are the current ways to solve this problem?
Is there any way to track the memory usage of a chip without tt-smi?

[warthog9]

The best way, right this minute, to track the memory usage of what's going on in the card is in-band with what's running on the card directly. There's some hardware limitations that make it hard to read it out of band where tt-smi is at, and the solutions there are not going to be quick to surface right now.