-
Notifications
You must be signed in to change notification settings - Fork 0
UnsafeCrailSerializer
Animesh Trivedi edited this page Apr 25, 2017
·
1 revision
Here we document our concerns with the UnsafeCrailSerializer
.
The default SQL serializer is UnsafeRowSerializer
. This can be changed now with :
spark.sql.serializer UnsafeCrailSerializer
It has two variants buffered and buffer-less. Thse can be activated for serialization and deserialization separately by :
spark.sql.serializer.buffer true
spark.sql.deserializer.buffer true
Next important parameter is the buffer size - which will be the same for both serialization and de-serialization. Set using
spark.sql.serializer.buffer.size 1048576
To use crail serializer or not. For now it does not do much apart from the right file type casting.
spark.sql.serializer.buffer.size