Update serving code to enable `saved_model=True` #18153

amyeroberts · 2022-07-15T18:47:38Z

What does this PR do?

Fixes and adds any missingserving and serving_output code to our TF models to enable
model.save_pretrained(path, saved_model=True)

I've added comments throughout the code to explain any areas where the TF refactor might no be obvious.

I'm aware the diff of this PR is quite big, but most of it repetitive changes to enable passing of tests, so I hope acceptable.

Specifically:

1. Adds missing serving logic to: ResNet, Swin, TAPAS

2. Adds missing serving_output logic to models
Some vision models didn't have serving_output implemented - serving returned the model outputs directly. This was to enable testing (see 4.) and to keep consistent with the rest of the library,

3. Update or add input_signature decorator for models
4. Adds a test to check serving_output is implemented and return types are as expected

We can't test model.serving directly i.e. this is not possible:

model = model_class(config)
inputs = self._prepare_for_class(inputs_dict, model_class)
serving_outputs = model.serving(inputs)

Running this on vision models raises the following:

E                           ValueError: The channel dimension of the inputs should be defined. The input_shape received is (None, None, None, None), where axis -1 (0-based) is the channel dimension, which found to be `None`.

This is because the input signature defined in the tf.function decorator for the serving method has all of the input dimensions defined as None:

 @tf.function(input_signature=[{
     "pixel_values": tf.TensorSpec((None, None, None, None), tf.float32, name="pixel_values"),
            }]
    )
    def serving(self, inputs):
        ....

We can't hard code the channel dimensions as e.g. 3 as we want to support both RGB and greyscale images (although testing this locally does work).

5. Moves test_saved_model_creation back into test_modeling_tf_common and add explicit skips

There were quite a few models that couldn't be saved with model.save_pretrained(path, saved_model=True) and quite a few whose input signature or return types from serving_output were broken or inconsistent with the model.

I think this is in part because the relevant test was moved to only be applied to certain core models, and those models didn't explicitly skip. See #14415, #9478

I've:

added it back to common so that it's added to models by default. CI currently running and passing,
added unittest.skip decorator so it's counted as a skipped rather than passed test on all models that were previously skipping.

6. Update logic in models such that their graph can be created and saved.

Adds serving logic to enable saving of models and ensures their outputs are transformed in line with the rest of the library

Fixes

#18179
#18164
#17233
#17285
https://discuss.huggingface.co/t/tfresnetforimageclassification-fails-with-save-pretrained-when-saved-model-is-true/20404

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

HuggingFaceDocBuilderDev · 2022-07-15T18:59:57Z

The documentation is not available anymore as the PR was closed or merged.

sayakpaul · 2022-07-20T08:06:44Z

@amyeroberts thank you for digging into this one. An important one albeit.

We can't hard code the channel dimensions as e.g. 3 as we want to support both RGB and greyscale images (although testing this locally does work).

Could you shed more details on the locally testing you're referring to here?

amyeroberts · 2022-07-20T14:26:03Z

Could you shed more details on the locally testing you're referring to here?

@sayakpaul Sure :) In the test test_prepare_serving_output, if serving outputs are got by calling serving directly i.e.

            inputs = self._prepare_for_class(inputs_dict, model_class)
            serving_outputs = model.serving(inputs)

This will fail for vision models with the channel dim error I posted above. If I changed the input signature such that channel dimension is hard coded then it runs e.g.

    @tf.function(
        input_signature=[
            {
                "pixel_values": tf.TensorSpec((None, 3, None, None), tf.float32, name="pixel_values"),
            }
        ]
    )

I run the test with
pytest tests/models/{model_name}/test_modeling_tf_{model_name}.py::TF{ModelName}ModelTest::test_prepare_serving_output

sayakpaul · 2022-07-20T14:28:23Z

Thanks. A follow-up question. So if you run with (pytest tests/models/{model_name}/test_modeling_tf_{model_name}.py::TF{ModelName}ModelTest::test_prepare_serving_output) how does it get the hardcoded value for channels? Or do you first hard-code it and then run it?

amyeroberts · 2022-07-20T14:51:11Z

src/transformers/models/hubert/modeling_tf_hubert.py

@@ -227,7 +227,7 @@ def _compute_mask_indices(
            f" `sequence_length`: {sequence_length}`"
        )
    # compute number of masked spans in batch
-    num_masked_spans = int(mask_prob * sequence_length / mask_length + tf.random.uniform((1,)))
+    num_masked_spans = int(mask_prob * sequence_length / mask_length + np.random.uniform(size=(1,)))


Using tf.random.uniform here creates issues with int casting here, max on L231 and other logic that assumes an int downstream when the graph is created when saving.

@amyeroberts won't using np create problems in graph mode? For e.g. if the model was compiled?

Yep. I've removed it and added all the necessary tf logic.

amyeroberts · 2022-07-20T14:55:04Z

src/transformers/models/convnext/modeling_tf_convnext.py

@@ -383,7 +383,8 @@ def serving(self, inputs):
            inputs (`Dict[str, tf.Tensor]`):
                The input of the saved model as a dictionary of tensors.
        """
-        return self.call(inputs)


Adding in self.serving(output) return to:

keep in line with the rest of the models call pattern and return types e.g. not including loss (I don't know why this is done)

allow for testing (see PR desc.)

amyeroberts · 2022-07-20T14:57:14Z

src/transformers/models/funnel/modeling_tf_funnel.py

@@ -1127,12 +1127,12 @@ def call(
            training=training,
        )

-    # Copied from transformers.models.distilbert.modeling_tf_distilbert.TFDistilBertModel.serving_output


Removing copied from across models as the tensors in the tuples for hidden_states and activations are different sizes so can't be called with tf.convert_to_tensor

A quick look in Funnel docstring, I could not find why they have different sizes.

transformers/src/transformers/models/funnel/modeling_funnel.py

Lines 843 to 845 in 2c5747e

hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`):

Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of

shape `(batch_size, sequence_length, hidden_size)`.

Maybe the docstring is wrong, and we should update it (in another PR for sure) ..?

It seems because it uses pooling, the sequence length is different after each block:

transformers/docs/source/en/model_doc/funnel.mdx

Line 38 in 2c5747e

- Since Funnel Transformer uses pooling, the sequence length of the hidden states changes after each block of layers.

amyeroberts · 2022-07-20T15:32:34Z

src/transformers/models/speech_to_text/modeling_tf_speech_to_text.py

-        hidden_states = tf.identity(input_features)  # TF Conv1D assumes Batch x Time x Channels, same as the input
+        hidden_states = tf.cast(
+            input_features, tf.float32
+        )  # TF Conv1D assumes Batch x Time x Channels, same as the input


I'm not sure what the tf.identity was doing here - will removing this break anything @gante?

Casting to float to enable the tf.concat in graph mode in L156

If I remember correctly, this line only existed to rename the variable (and probably hidden_states = input_features would be enough for that 🤔 )

Could it be a mutation thing? tf.identity returns a copy

To be frank I can't remember if that was the cause :D But since it was my first HF PR, I would err towards a bad choice on my end

amyeroberts · 2022-07-20T15:41:07Z

src/transformers/models/swin/modeling_tf_swin.py

@@ -227,7 +227,7 @@ def window_reverse(windows: tf.Tensor, window_size: int, height: int, width: int
    Merges windows to produce higher resolution features.
    """
    x = shape_list(windows)[0]
-    y = tf.cast(height * width / window_size / window_size, tf.int32)
+    y = tf.cast(height * width / (window_size * window_size), tf.int32)


The previous calculation fails because:

height * width / widow_size is calculated first, before the final window_size division.

height, width are ints, window_size is a tensor of type int

Therefore, the resulting height * width / widow_size is a tensor of type float

This fails when dividing by window_size again as the tensors are of different types

amyeroberts · 2022-07-20T15:42:31Z

src/transformers/models/swin/modeling_tf_swin.py

@@ -1396,7 +1414,7 @@ def __init__(self, config: SwinConfig):
        self.classifier = (
            tf.keras.layers.Dense(config.num_labels, name="classifier")
            if config.num_labels > 0
-            else tf.identity(name="classifier")
+            else tf.keras.layers.Activation("linear", name="classifier")


This was a bug

amyeroberts · 2022-07-20T17:56:33Z

src/transformers/models/speech_to_text/modeling_tf_speech_to_text.py

    def build(self, input_shape: tf.TensorShape):
        """
        Build shared token embedding layer Shared weights logic adapted from
        https://github.com/tensorflow/models/blob/a009f4fb9d2fc4949e32192a944688925ef78659/official/transformer/v2/embedding_layer.py#L24
        """
-        self._resize_embeddings()


_resize_embeddings causes issues when in graph mode because it re-adds the parameter "weights", meaning the model doesn't know what can happen to the node. Instead here, we add once and then reassign values.

Yup, embeddings in general have to be rewritten -- they still use TF 1 code :(

amyeroberts · 2022-07-20T17:57:13Z

src/transformers/models/speech_to_text/modeling_tf_speech_to_text.py

+        self.embeddings = self.add_weight(
+            name="weights",  # name also used in PT
+            shape=shape_list(self.embedding_weights),
+            trainable=False,


As far as I can tell from PT model, this should be fixed. Is this right @gante ?

Right, it is a non-trainable positional embedding that is obtained from sines and cosines 👍

amyeroberts · 2022-07-20T18:00:32Z

src/transformers/models/speech_to_text/modeling_tf_speech_to_text.py

@@ -942,22 +942,6 @@ def get_embed_tokens(self):
    def set_embed_tokens(self, embed_tokens):
        self.embed_tokens = embed_tokens

-    def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length):


This function causes issues in graph mode, as combined_attention_mask can be either a tensor or None. Instead, the equivalent logic in bart call method is used in this model's to safely create combined_attention_mask

amyeroberts · 2022-07-20T18:01:13Z

src/transformers/models/swin/modeling_tf_swin.py

@@ -659,64 +659,46 @@ def __init__(
        self.drop_path = (
            TFSwinDropPath(config.drop_path_rate, name="drop_path")
            if config.drop_path_rate > 0.0
-            else tf.identity(name="drop_path")
+            else tf.keras.layers.Activation("linear", name="drop_path")


This was a bug

amyeroberts · 2022-07-21T11:23:39Z

Thanks. A follow-up question. So if you run with (pytest tests/models/{model_name}/test_modeling_tf_{model_name}.py::TF{ModelName}ModelTest::test_prepare_serving_output) how does it get the hardcoded value for channels? Or do you first hard-code it and then run it?

@sayakpaul Yes, I hardcode then run the tests. In this case, they pass.

amyeroberts · 2022-07-21T11:25:05Z

src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py


    # make sure num masked indices <= sequence_length
-    if num_masked_spans * mask_length > sequence_length:
-        num_masked_spans = sequence_length // mask_length
+    num_masked_spans = tf.cond(


Same as for hubert. In graph mode, we can't used a tensor in an if/else statement. You get the following error:
OperatorNotAllowedInGraphError: using a tf.Tensoras a Pythonbool is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.

(same comment as for hubert)

gante

In general, looks good to me 👍 Thanks for fixing all these failing cases!

P.S.: the changes to swin were pretty big and probably should have been a PR on their own, but I'm also often guilty of this sin :D Have you double-checked the slow tests for the models with the most changes?

gante · 2022-07-21T11:58:34Z

src/transformers/models/hubert/modeling_tf_hubert.py


    # make sure num masked indices <= sequence_length
-    if num_masked_spans * mask_length > sequence_length:
-        num_masked_spans = sequence_length // mask_length
+    num_masked_spans = tf.cond(


Looks good 👍 Perhaps for readability, if it is allowed, it can be rewritten as tf.math.minimum(sequence_length // mask_length, num_masked_spans)?

src/transformers/models/hubert/modeling_tf_hubert.py

src/transformers/models/speech_to_text/modeling_tf_speech_to_text.py

gante · 2022-07-21T12:02:39Z

src/transformers/models/speech_to_text/modeling_tf_speech_to_text.py

    def build(self, input_shape: tf.TensorShape):
        """
        Build shared token embedding layer Shared weights logic adapted from
        https://github.com/tensorflow/models/blob/a009f4fb9d2fc4949e32192a944688925ef78659/official/transformer/v2/embedding_layer.py#L24
        """
-        self._resize_embeddings()


Yup, embeddings in general have to be rewritten -- they still use TF 1 code :(

gante · 2022-07-21T12:08:47Z

src/transformers/models/speech_to_text/modeling_tf_speech_to_text.py

+        self.embeddings = self.add_weight(
+            name="weights",  # name also used in PT
+            shape=shape_list(self.embedding_weights),
+            trainable=False,


Right, it is a non-trainable positional embedding that is obtained from sines and cosines 👍

src/transformers/models/speech_to_text/modeling_tf_speech_to_text.py

src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py

gante · 2022-07-21T12:28:12Z

src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py


    # make sure num masked indices <= sequence_length
-    if num_masked_spans * mask_length > sequence_length:
-        num_masked_spans = sequence_length // mask_length
+    num_masked_spans = tf.cond(


(same comment as for hubert)

tests/models/bart/test_modeling_tf_bart.py

ydshieh · 2022-07-21T13:12:52Z

Before merge, could you measure the timing for the tests test_saved_model_creation on CPU? You can run like

python -m pytest -v tests -k "test_saved_model_creation" --durations=0 --make-reports=tests_timing

and copy-paste the results from reports/tests_timing/durations.txt

sgugger · 2022-07-21T13:24:06Z

@ydshieh We don't run the slow tests on CPU, only on GPU/multi-GPU.

ydshieh · 2022-07-21T13:52:41Z

@ydshieh We don't run the slow tests on CPU, only on GPU/multi-GPU.

I should double check the latest version. My memory was in a previoius commit Add in tests (505cb774b1b7eb5c9a6c8e2bc63f12061824b8bd) while I asked the question 😢

ydshieh

Thank you @amyeroberts for this great work! I left a few questions.

ydshieh · 2022-07-21T13:54:31Z

src/transformers/models/hubert/modeling_tf_hubert.py

+    @tf.function(
+        input_signature=[
+            {
+                "input_values": tf.TensorSpec((None, None), tf.int32, name="input_ids"),


I have some doubt here: input_values should be float, I think? Same for Wav2Vec2.

Also, the name should be changed too.

Updated the name 👍 Following your comment below, input_values are set to float and attention_maks and token_types_ids as int.

ydshieh · 2022-07-21T13:55:28Z

src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py

+    @tf.function(
+        input_signature=[
+            {
+                "input_values": tf.TensorSpec((None, None), tf.int32, name="input_ids"),


input_values should be float? Also to change name.

Updated the name 👍 Following your comment below, input_values are set to float and attention_maks and token_types_ids as int.

ydshieh · 2022-07-21T13:57:30Z

src/transformers/models/speech_to_text/modeling_tf_speech_to_text.py

@@ -608,7 +608,7 @@ def _get_feat_extract_output_lengths(self, input_lengths: tf.Tensor):
    @tf.function(
        input_signature=[
            {
-                "input_ids": tf.TensorSpec((None, None), tf.int32, name="input_ids"),
+                "input_features": tf.TensorSpec((None, None, None), tf.int32, name="input_ids"),


input_features should be float? Also to change name?

I've updated the name.
For the types, from the pytorch model, input_features, decoder_inputs and decoder_attention_mask should be int32. attention_mask isn't specified, but test_saved_model_creation fails if it's set to float32.

From the 2 places

transformers/tests/test_modeling_common.py

Lines 1720 to 1721 in 2c5747e

elif key == "input_features":

tf_inputs_dict[key] = tf.convert_to_tensor(tensor.cpu().numpy(), dtype=tf.float32)

and

transformers/tests/test_modeling_tf_common.py

Lines 531 to 532 in 2c5747e

elif name == "input_features":

pt_inputs_dict[name] = torch.from_numpy(key.numpy()).to(torch.float32)

as well as the docstring (the 2nd line below)

transformers/src/transformers/models/speech_to_text/modeling_speech_to_text.py

Lines 602 to 603 in 2c5747e

input_features (`torch.LongTensor` of shape `(batch_size, sequence_length, feature_size)`):

Float values of fbank features extracted from the raw speech waveform. Raw speech waveform can be obtained

I still think input_features should be float. The part torch.LongTensor seems to be a mistake.

(cc @patrickvonplaten @anton for a confirmation.)

OK, that's a good point. I've updated it to float32 (passes tests locally)

ydshieh · 2022-07-21T14:01:46Z

src/transformers/models/funnel/modeling_tf_funnel.py

@@ -1127,12 +1127,12 @@ def call(
            training=training,
        )

-    # Copied from transformers.models.distilbert.modeling_tf_distilbert.TFDistilBertModel.serving_output


A quick look in Funnel docstring, I could not find why they have different sizes.

transformers/src/transformers/models/funnel/modeling_funnel.py

Lines 843 to 845 in 2c5747e

hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`):

Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of

shape `(batch_size, sequence_length, hidden_size)`.

Maybe the docstring is wrong, and we should update it (in another PR for sure) ..?

@tooslow

Add suggestions made by Joao Gante: * Use tf.shape instead of shape_list * Use @tooslow decorator on tests * Simplify some of the logic

Address Yih-Dar Sheih comments - making tensor names consistent and make types float

ydshieh · 2022-07-21T15:46:11Z

@amyeroberts The attention_mask and token_type_ids in TFHubert / TFWav2Vec2 should be int32 I believe. I think we don't put this clearly in their docstrings in TF models, but we have this information in their PyTorch model files.

amyeroberts · 2022-07-21T16:36:33Z

@ydshieh I know from @sgugger's comment, we don't run on CPU, but I ran the tests for reference (Macbook Pro 2021 M1 Max)
Based on this I disabled the test for Swin. The slowest tests - test_saved_model_creation_extended - are independent of this PR.

slowest durations
242.92s call     tests/models/convbert/test_modeling_tf_convbert.py::TFConvBertModelTest::test_saved_model_creation_extended
202.38s call     tests/models/bert/test_modeling_tf_bert.py::TFBertModelTest::test_saved_model_creation_extended
177.82s call     tests/models/gptj/test_modeling_tf_gptj.py::TFGPTJModelTest::test_saved_model_creation_extended
150.56s call     tests/models/bart/test_modeling_tf_bart.py::TFBartModelTest::test_saved_model_creation_extended
82.65s call     tests/models/gpt2/test_modeling_tf_gpt2.py::TFGPT2ModelTest::test_saved_model_creation_extended
61.12s call     tests/models/lxmert/test_modeling_tf_lxmert.py::TFLxmertModelTest::test_saved_model_creation_extended
26.88s call     tests/models/swin/test_modeling_tf_swin.py::TFSwinModelTest::test_saved_model_creation
20.14s call     tests/models/clip/test_modeling_tf_clip.py::TFCLIPTextModelTest::test_saved_model_creation_extended
19.51s call     tests/models/convbert/test_modeling_tf_convbert.py::TFConvBertModelTest::test_saved_model_creation
19.40s call     tests/models/gptj/test_modeling_tf_gptj.py::TFGPTJModelTest::test_saved_model_creation
18.54s call     tests/models/deberta_v2/test_modeling_tf_deberta_v2.py::TFDebertaModelTest::test_saved_model_creation
16.77s call     tests/models/speech_to_text/test_modeling_tf_speech_to_text.py::TFSpeech2TextModelTest::test_saved_model_creation
16.72s call     tests/models/clip/test_modeling_tf_clip.py::TFCLIPVisionModelTest::test_saved_model_creation_extended
14.51s call     tests/models/roformer/test_modeling_tf_roformer.py::TFRoFormerModelTest::test_saved_model_creation
14.37s call     tests/models/deberta/test_modeling_tf_deberta.py::TFDebertaModelTest::test_saved_model_creation
13.15s call     tests/models/wav2vec2/test_modeling_tf_wav2vec2.py::TFWav2Vec2ModelTest::test_saved_model_creation
13.12s call     tests/models/hubert/test_modeling_tf_hubert.py::TFHubertModelTest::test_saved_model_creation
12.79s call     tests/models/mpnet/test_modeling_tf_mpnet.py::TFMPNetModelTest::test_saved_model_creation
12.47s call     tests/models/tapas/test_modeling_tf_tapas.py::TFTapasModelTest::test_saved_model_creation
12.16s call     tests/models/layoutlm/test_modeling_tf_layoutlm.py::TFLayoutLMModelTest::test_saved_model_creation
12.16s call     tests/models/electra/test_modeling_tf_electra.py::TFElectraModelTest::test_saved_model_creation
12.12s call     tests/models/xlm/test_modeling_tf_xlm.py::TFXLMModelTest::test_saved_model_creation
11.92s call     tests/models/flaubert/test_modeling_tf_flaubert.py::TFFlaubertModelTest::test_saved_model_creation
11.54s call     tests/models/transfo_xl/test_modeling_tf_transfo_xl.py::TFTransfoXLModelTest::test_saved_model_creation
11.42s call     tests/models/bert/test_modeling_tf_bert.py::TFBertModelTest::test_saved_model_creation
11.36s call     tests/models/rembert/test_modeling_tf_rembert.py::TFRemBertModelTest::test_saved_model_creation
11.12s call     tests/models/distilbert/test_modeling_tf_distilbert.py::TFDistilBertModelTest::test_saved_model_creation
10.85s call     tests/models/ctrl/test_modeling_tf_ctrl.py::TFCTRLModelTest::test_saved_model_creation
10.77s call     tests/models/roberta/test_modeling_tf_roberta.py::TFRobertaModelTest::test_saved_model_creation
10.51s call     tests/models/wav2vec2/test_modeling_tf_wav2vec2.py::TFWav2Vec2RobustModelTest::test_saved_model_creation
10.44s call     tests/models/gpt2/test_modeling_tf_gpt2.py::TFGPT2ModelTest::test_saved_model_creation
10.30s call     tests/models/hubert/test_modeling_tf_hubert.py::TFHubertRobustModelTest::test_saved_model_creation
10.23s call     tests/models/deit/test_modeling_tf_deit.py::TFDeiTModelTest::test_saved_model_creation
10.20s call     tests/models/t5/test_modeling_tf_t5.py::TFT5EncoderOnlyModelTest::test_saved_model_creation
10.14s call     tests/models/albert/test_modeling_tf_albert.py::TFAlbertModelTest::test_saved_model_creation
10.14s call     tests/models/clip/test_modeling_tf_clip.py::TFCLIPTextModelTest::test_saved_model_creation
9.53s call     tests/models/openai/test_modeling_tf_openai.py::TFOpenAIGPTModelTest::test_saved_model_creation
9.11s call     tests/models/vit_mae/test_modeling_tf_vit_mae.py::TFViTMAEModelTest::test_saved_model_creation
8.71s call     tests/models/vit/test_modeling_tf_vit.py::TFViTModelTest::test_saved_model_creation
8.68s call     tests/models/clip/test_modeling_tf_clip.py::TFCLIPVisionModelTest::test_saved_model_creation
8.44s call     tests/models/dpr/test_modeling_tf_dpr.py::TFDPRModelTest::test_saved_model_creation
8.14s call     tests/models/xlnet/test_modeling_tf_xlnet.py::TFXLNetModelTest::test_saved_model_creation
7.54s call     tests/models/data2vec/test_modeling_tf_data2vec_vision.py::TFData2VecVisionModelTest::test_saved_model_creation
6.70s call     tests/models/convnext/test_modeling_tf_convnext.py::TFConvNextModelTest::test_saved_model_creation
6.21s call     tests/models/regnet/test_modeling_tf_regnet.py::TFRegNetModelTest::test_saved_model_creation
4.62s call     tests/models/resnet/test_modeling_tf_resnet.py::ResNetModelTest::test_saved_model_creation

gante · 2022-07-21T18:02:29Z

I noticed a cool thing -- assuming the models with @tooslow also pass the tests (which I'm assuming they do, from this comment), this PR fixes:

bug in modeling_tf_wav2vec2 #17233, as the problematic line was rewritten with TF code in this PR
TF: all models can run in Graph mode #17285, as we know we can create a SavedModel (which contains a graph) for all models.

@amyeroberts can you add these issues to your Fixes list above? :D

amyeroberts · 2022-07-22T10:16:09Z

@gante Nice spot :D Yep - I just double checked, and all the models with the @tooslow decorator can be saved with saved_model=True - and so their graph can be built. Added the fixes.

The only model we can't save at the moment is CLIP, due to the nested dict of outputs.

amyeroberts · 2022-07-22T12:08:09Z

@ydshieh @Rocketknight1 Am I OK to merge?

ydshieh

Thanks again!

Rocketknight1

Overall, this is an extremely strong PR. The model changes were comprehensive and I'm impressed that you got the SavedModel test back into common!

@tooslow

* Add serving_output and serving methods to some vision models * Add serving outputs for DeiT * Don't convert hidden states - differing shapes * Make saveable * Fix up * Make swin saveable * Add in tests * Fix funnel tests (can't convert to tensor) * Fix numpy call * Tidy up a bit * Add in hidden states - resnet * Remove numpy * Fix failing tests - tensor shape and skipping tests * Remove duplicated function * PR comments - formatting and var names * PR comments Add suggestions made by Joao Gante: * Use tf.shape instead of shape_list * Use @tooslow decorator on tests * Simplify some of the logic * PR comments Address Yih-Dar Sheih comments - making tensor names consistent and make types float * Types consistent with docs; disable test on swin (slow) * CI trigger * Change input_features to float32 * Add serving_output for segformer * Fixup Co-authored-by: Amy Roberts <amyeroberts@users.noreply.github.com>

amyeroberts added 2 commits July 15, 2022 19:23

Add serving_output and serving methods to some vision models

cc93471

Add serving outputs for DeiT

0a8ed80

amyeroberts mentioned this pull request Jul 18, 2022

Cannot save TFSwinForImageClassification as SavedModel #18164

Closed

4 tasks

amyeroberts changed the title ~~Add serving_output and serving methods to some vision models~~ [WIP] Add serving_output and serving methods to some vision models Jul 18, 2022

amyeroberts added the WIP Label your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress label Jul 18, 2022

amyeroberts added 3 commits July 19, 2022 20:55

Don't convert hidden states - differing shapes

2e948ca

Make saveable

bf684e7

Fix up

c230382

amyeroberts added 3 commits July 20, 2022 12:11

Make swin saveable

ac7019b

Add in tests

505cb77

Fix funnel tests (can't convert to tensor)

a86b369

Fix numpy call

cf747bf

amyeroberts commented Jul 20, 2022

View reviewed changes

amyeroberts added 2 commits July 20, 2022 16:09

Tidy up a bit

0ff7d8c

Add in hidden states - resnet

88e7888

amyeroberts commented Jul 20, 2022

View reviewed changes

amyeroberts added 2 commits July 20, 2022 17:27

Remove numpy

899c76c

Fix failing tests - tensor shape and skipping tests

57cf29a

amyeroberts commented Jul 20, 2022

View reviewed changes

PR comments - formatting and var names

5178ab8

amyeroberts commented Jul 21, 2022

View reviewed changes

gante approved these changes Jul 21, 2022

View reviewed changes

ydshieh reviewed Jul 21, 2022

View reviewed changes

amyeroberts added 2 commits July 21, 2022 15:39

PR comments

6f0aa42

Add suggestions made by Joao Gante: * Use tf.shape instead of shape_list * Use @tooslow decorator on tests * Simplify some of the logic

PR comments

e9486db

Address Yih-Dar Sheih comments - making tensor names consistent and make types float

Types consistent with docs; disable test on swin (slow)

db264ea

amyeroberts added 2 commits July 21, 2022 18:23

CI trigger

006d9e5

Change input_features to float32

b6e7d06

amyeroberts added 3 commits July 22, 2022 11:19

Merge branch 'main' into fix-tf-resnet-saving-bug

a0b70e9

Add serving_output for segformer

02caeae

Fixup

2ed2025

ydshieh approved these changes Jul 22, 2022

View reviewed changes

Rocketknight1 approved these changes Jul 22, 2022

View reviewed changes

amyeroberts merged commit 8e83846 into huggingface:main Jul 22, 2022

amyeroberts deleted the fix-tf-resnet-saving-bug branch July 22, 2022 17:05

This was referenced Jul 22, 2022

Cannot save TFTapasModel as SavedModel #18179

Closed

bug in modeling_tf_wav2vec2 #17233

Closed

gante mentioned this pull request Jul 22, 2022

TF: all models can run in Graph mode #17285

Closed

	hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`):
	Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of
	shape `(batch_size, sequence_length, hidden_size)`.

	elif key == "input_features":
	tf_inputs_dict[key] = tf.convert_to_tensor(tensor.cpu().numpy(), dtype=tf.float32)

	elif name == "input_features":
	pt_inputs_dict[name] = torch.from_numpy(key.numpy()).to(torch.float32)

	input_features (`torch.LongTensor` of shape `(batch_size, sequence_length, feature_size)`):
	Float values of fbank features extracted from the raw speech waveform. Raw speech waveform can be obtained

Update serving code to enable saved_model=True #18153

Update serving code to enable saved_model=True #18153

Conversation

amyeroberts commented Jul 15, 2022 • edited Loading

What does this PR do?

Specifically:

Fixes

Before submitting

HuggingFaceDocBuilderDev commented Jul 15, 2022 • edited Loading

sayakpaul commented Jul 20, 2022

amyeroberts commented Jul 20, 2022

sayakpaul commented Jul 20, 2022

amyeroberts Jul 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amyeroberts Jul 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amyeroberts commented Jul 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gante left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ydshieh commented Jul 21, 2022

sgugger commented Jul 21, 2022

ydshieh commented Jul 21, 2022

ydshieh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ydshieh Jul 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ydshieh Jul 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ydshieh Jul 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ydshieh commented Jul 21, 2022

amyeroberts commented Jul 21, 2022

gante commented Jul 21, 2022 • edited Loading

amyeroberts commented Jul 22, 2022

amyeroberts commented Jul 22, 2022

ydshieh left a comment

Choose a reason for hiding this comment

Rocketknight1 left a comment

Choose a reason for hiding this comment

Update serving code to enable `saved_model=True` #18153

Update serving code to enable `saved_model=True` #18153

amyeroberts commented Jul 15, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented Jul 15, 2022 •

edited

Loading

amyeroberts Jul 20, 2022 •

edited

Loading

amyeroberts Jul 20, 2022 •

edited

Loading

ydshieh Jul 21, 2022 •

edited

Loading

ydshieh Jul 21, 2022 •

edited

Loading

ydshieh Jul 21, 2022 •

edited

Loading

gante commented Jul 21, 2022 •

edited

Loading