Why does the JsonInputAdapter wrap the json request into a list when served via flask ?

[jiyer2016]

I know this has got something to do with the Micro-batching feature.
At runtime - when served via flask - the JsonInput Adapter adds the json request to a list.
I am assuming this is because we want the API to be aware that it may be handling multiple requests (from a micro-batch) related to completely different users.

I have two questions on the above:

1. Why do I not see this behavior in integration test cases (without flask) ? Might be related to Why do Input Adapters not take effect in test cases ? #1066 - where input adapters dont take effect i test cases
2. Why do I not observe similar behavior for DataframeInput Adapter ? I was thinking that the API should receive a list of Dataframes (to be in sync with the JsonInput behavior)

[bojiang]

For Q2:
In an ordinary model training procedure, we often see a DataFrame as a batch of data, and the batch dimension is the rows of the DataFrame.
For data in JSON format, generally it is an object like this: {"name": "jeff"}, which represents a single record.
So for the former, what micro-batching does is concat(df1, df2); for the latter, it is [json1, json2].

[parano]

Great questions @jiyer2016

2 DataframeInput in BentoML treats each row in the pandas.DataFrame object as one inference input. And BentoML merges HTTP requests of Dataframes in JSON format into one Dataframe before feeding it to the user-defined inference API function.

1 Adapters do not take effect when calling directly from the BentoService's class method in Python. In tests, you will need to convert the array to a pandas.DataFrame and then pass it to the inference API method.