[SPARK-23723][SPARK-23724][SQL] Flexible format for the lineSep option of CSV datasource

[MaxGekk]

What changes were proposed in this pull request?

I propose flexible format for the lineSep option used in text datasources like Json. New format of the option has the following syntax:

lineSep ::= (selector separator-spec) | text-separator
selector := 'x' | '\' | reserved-selector
reserved-selector ::= '\' | 'r'
separator-spec ::= < valid string literal in Python, R, Java and Scala>
text-separator ::= first-char separator-spec
first-char ::= ! selector

Examples of lineSep in the new format:

x0a.00.00.00 0d.00.00.00
x5445
|^|
\r\n
-
sep

The '\' and 'r' are reserved for future usage. For instance, 'r' could be used for regular expressions line r[0-9]+ or r(x1E|x0Ax1E|x0A) for parsing Json Streaming

New format addresses the use cases:

1. Hexadecimal format allows to specify lineSep independently from encoding. It gives opportunity for reading json files with BOM in per-line mode. See [SPARK-23723] New charset option for json datasource apache/spark#20849 (comment)

2. Jsons coming usually from embedded systems have not-standard separators (invisible in some cases). It is very convenient to open a file in hex editor and copy bytes between }{ to the lineSep option. This is the use case for the format with 'x' selector like: x0d 54 45

3. In Json Streaming, records could be separated in pretty different ways. We should leave room for improvement I believe. See 'r' (for regexp) and '/' reserved selectors

4. Some UTF-8 chars could cause errors from style (format) checkers. It is easier to represent such chars in hexadecimal format instead of disabling the checkers.

5. In near future, json datasource will support input json in different charsets. If the source code in UTF-8 but input json in different charset, it is slightly hard to put such chars as values for the lineSep option. The x<hexs> format is more convenient here again.

How was this patch tested?

• The .option("mode", "DROPMALFORMED") was replaced by the lineSep option like .option("lineSep", "x0d 00 0a 00")

• Additional cases for hexadecimal format were added to existing tests