#TODO tags?
Synaptic pruning improves efficiency by removing redundant neurons and strengthening synaptic connections that are most useful for the environment. 'use it or lose it'.
At face value, pruning does appear to promise you can can (almost) have it all. State of art pruning methods remove the majority of the weights with minimal degradation to top-1 accuracy. These newly slimmed down networks require less memory, energy consumption and are faster at producing predictions. All these attributes make pruned models ideal for deploying deep neural networks to resource constrained environments.
The cost to top-1 accuracy appears minimal if it is spread uniformally across all classes, but what if the cost is concentrated in only a few classes? Are certain types of examples or classes disproportionately impacted by pruning?
In this work, we propose a formal framework to identify the classes and images where there is a high level of disagreement or difference in generalization performance between pruned and non-pruned models. We find that certain examples, which we term pruning identified exemplars (PIEs), and classes are systematically more impacted by the introduction of sparsity.
To better understand why PIEs are more sensitive to capacity, we conducted a limited human study (85 participants) and find that PIEs from the ImageNet test set are more likely to be mislabelled, depict multiple objects or require fine-grained classification.