How to exclude "resnet152_v2" in my choice

[Anselmoo]

Dear community,

is there a way to explicitly exclude models like the memory-hungry "resnet152_v2"??

Thx for the help a lot

[haifeng-jin]

I don't think we have a way to do that.
Does it break if you have not enough memory, or it just skip the model and search the next one?

[Anselmoo]

Dear @haifeng-jin

thx for a super fast answer.

It breaks because of not having enough memory (>32GB) for 5GB data. Sometimes it crashed, sometimes I even have to restart the node.

In addition to that using image_net in combination with any resnet_50/100/152_v2 seems to be also a memory eater as well. First-generation resnet makes no trouble.

[Anselmoo]

I don't think we have a way to do that.
Does it break if you have not enough memory, or it just skip the model and search the next one?

Would it be possible to skip in case of out-of-memory?

I should mention, that I am only using CPU cores.

[haifeng-jin]

Unfortunately, I don't have the time to fix this recently.
Would you like to make the contribution?
Current behavior:
When out-of-memory, we reduce batch size:
https://github.com/keras-team/autokeras/blob/master/autokeras/utils/utils.py#L87
The fix would be if batch size is redueced to 1 and still OOM, we print a warning and skip the model.

[Anselmoo]

I will take a look and care.
The last question, is there is already skip a model function implemented?
Thx

[Anselmoo]

I am just wondering, if AutoKeras run out of memory during the cycle, this can be actually fixed?
Or is that more related to TensorFlow itself?

Epoch 2/200
162/162 [==============================] - 62s 383ms/step - loss: 4679.4224 - mean_squared_error: 4679.4224 - val_loss: 3188.2500 - val_mean_squared_error: 3188.2500 
Epoch 3/200
 68/162 [===========>..................] - ETA: 16s - loss: 3397.3586 

There is no message coming, you just see via htop that it is close to starting swapping and then disappearing.

Well, I am working on a fix.

[Anselmoo]

@haifeng-jin I was taking a closer look and it seems to be an effect of fish-shell in combination with nohup. bash in combination with tmux seems to run fine.

#1722 might be closed?

thx for help in advance

[Anselmoo]

see also fish-shell/fish-shell#2810

[Anselmoo]

@haifeng-jin recently Keras and TensorFlow are updated to 2.9.

If I am using AutoKeras with the new Keras and TensorFlow Versions. The error printout for the resnet_152_v2 is a little bit extended:

WARNING:tensorflow:Using a while_loop for converting AdjustContrastv2
WARNING:tensorflow:Using a while_loop for converting AdjustContrastv2
WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 82). These functions will not be directly callable after loading.
WARNING:tensorflow:Using a while_loop for converting RngReadAndSkip
WARNING:tensorflow:Using a while_loop for converting RngReadAndSkip
WARNING:tensorflow:Using a while_loop for converting Bitcast
WARNING:tensorflow:Using a while_loop for converting Bitcast
WARNING:tensorflow:Using a while_loop for converting Bitcast
WARNING:tensorflow:Using a while_loop for converting Bitcast
WARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformFullIntV2
WARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformFullIntV2
WARNING:tensorflow:Using a while_loop for converting StatelessRandomGetKeyCounter
WARNING:tensorflow:Using a while_loop for converting StatelessRandomGetKeyCounter
WARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformV2
WARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformV2
WARNING:tensorflow:Using a while_loop for converting AdjustContrastv2
WARNING:tensorflow:Using a while_loop for converting AdjustContrastv2
WARNING:tensorflow:Using a while_loop for converting RngReadAndSkip
WARNING:tensorflow:Using a while_loop for converting RngReadAndSkip
WARNING:tensorflow:Using a while_loop for converting BitcastWARNING:tensorflow:Using a while_loop for converting Bitcast
WARNING:tensorflow:Using a while_loop for converting Bitcast
WARNING:tensorflow:Using a while_loop for converting Bitcast
WARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformFullIntV2
WARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformFullIntV2
WARNING:tensorflow:Using a while_loop for converting StatelessRandomGetKeyCounter
WARNING:tensorflow:Using a while_loop for converting StatelessRandomGetKeyCounter
WARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformV2
WARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformV2
WARNING:tensorflow:Using a while_loop for converting AdjustContrastv2
WARNING:tensorflow:Using a while_loop for converting AdjustContrastv2
WARNING:tensorflow:Using a while_loop for converting RngReadAndSkip
WARNING:tensorflow:Using a while_loop for converting RngReadAndSkipWARNING:tensorflow:Using a while_loop for converting Bitcast
WARNING:tensorflow:Using a while_loop for converting Bitcast
WARNING:tensorflow:Using a while_loop for converting Bitcast
WARNING:tensorflow:Using a while_loop for converting BitcastWARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformFullIntV2WARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformFullIntV2
WARNING:tensorflow:Using a while_loop for converting StatelessRandomGetKeyCounter
WARNING:tensorflow:Using a while_loop for converting StatelessRandomGetKeyCounter
WARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformV2
WARNING:tensorflow:Using a while_loop for converting StatelessRandomUniformV2
WARNING:tensorflow:Using a while_loop for converting AdjustContrastv2
WARNING:tensorflow:Using a while_loop for converting AdjustContrastv2
WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 82). These functions will not be directly callable after loading.

And it seems to be this happens before starting a new EPOCH

Have you any idea what's going on?

Thx for feedback

[jwliou]

A quick answer is to define your own "ResNetBlock" for removing unwanted options such as "resnet152"-what-so-ever.
How to do that? Referred to autokeras/blocks/basic.py for possible way to implement custom block class.

[Anselmoo]

There might be the explicitly exclude RESNET as proposed https://autokeras.com/tutorial/customized/.

input_node = ak.ImageInput()
output_node = ak.Normalization()(input_node)
output_node1 = ak.ConvBlock()(output_node)
output_node2 = ak.ResNetBlock(version="v2")(output_node)
output_node = ak.Merge()([output_node1, output_node2])
output_node = ak.ClassificationHead()(output_node)

auto_model = ak.AutoModel(
    inputs=input_node, outputs=output_node, overwrite=True, max_trials=1
)

However, it seems to strongly interact with the principles of AutoML?

[jwliou]

well, one possible example in my mind for custom block may be something like following:
since memory hungry is related to model size instead of model version,
152 from both version will not be recommended in such scenario as far as I had seen.

But if such way will violate so-called AutoML principles, I have no idea.

# only modify two dicts from ResNetBlock, but that means custom definition for yet another block class.

RESNET_V1S = {
    "resnet50": applications.ResNet50,
    "resnet101": applications.ResNet101,
    # "resnet152": applications.ResNet152,
}

RESNET_V2S = {
    "resnet50_v2": applications.ResNet50V2,
    "resnet101_v2": applications.ResNet101V2,
    # "resnet152_v2": applications.ResNet152V2,
}

class ResLightBlock(KerasApplicationBlock):
    """Block for ResNet.
    # Arguments
        version: String. 'v1', 'v2'. The type of SMALLER ResNet to use.
            If left unspecified, it will be tuned automatically.
        pretrained: Boolean. Whether to use ImageNet pretrained weights.
            If left unspecified, it will be tuned automatically.
    """

    def __init__(
        self,
        version: Optional[str] = None,
        pretrained: Optional[bool] = None,
        **kwargs,
    ):
        if version is None:
            models = {**RESNET_V1S, **RESNET_V2S}
        elif version == "v1":
            models = RESNET_V1S
        elif version == "v2":
            models = RESNET_V2S
        else:
            raise ValueError(
                'Expect version to be "v1", or "v2", but got '
                "{version}.".format(version=version)
            )
        super().__init__(pretrained=pretrained, models=models, min_size=32, **kwargs)
        self.version = version

    def get_config(self):
        config = super().get_config()
        config.update({"version": self.version})
        return config