DataType::Dictionary is out of spec

[jorgecarleitao]

Describe the bug

The schema.fbs and corresponding generated code has no concept of a Dictionary datatype.

However, we declare a DataType::Dictionary.

Additional context

As a user, I would like to be able to not have to change my DataType whenever I want to change an arrays' encoding. In the context of DataFusion, that uses DataType to declare the schema of the logical plan, this forbids optimizations at the physical level that e.g. would convert an array to a dictionary-encoded array, which is useful in any group-by or hashing operation.

[tustvold]

table DictionaryEncoding is part of the definition of table Field, theoretically we could move the dictionary-ness onto Field. However, this would complicate downcasting, and given Field is part of Schema regardless, this wouldn't change the fact that dictionary-ness is an inherent part of an array's schema.

Feel free to re-open if I have missed something