Dictionary Uniqueness and Sortedness

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

A schema Field currently has a dict_is_ordered field. Much like dict_id this is non-trivial to use in practice (#1206) as it is part of the schema which may need to remain constant across multiple RecordBatch that might otherwise wish to use different dictionaries. It is also not visible to compute kernels.

There doesn't appear to be a notion of if a dictionary contains unique values.

Describe the solution you'd like

We would ideally like to associated dictionary properties with the dictionary arrays themselves. This would allow compute kernels to exploit their properties, in addition to avoiding schema incompatibility issues.

There is an assumption in many places that the arrays themselves are sugar on top of ArrayData, with it possible to construct a DictionaryArray from a ArrayData. As such I would like to propose:

• Remove dict_is_ordered from Field
• Add dict_is_ordered bool to ArrayData (default false)
• Add dict_is_unique bool to ArrayData (default false)

The various compute kernels can then progressively be updated to exploit these properties.