Accept unicode letters in keyword search terms

[philrz]

At the time of the filing of this issue, Zed is at commit b8e1b18.

A user said in a community Slack thread:

I've started to use simplified search expressions. It annoys me that when search for a person with non-ascii names I have to quote the name. For instance that zq 'bjorndal | {id,name}' is allowed, but if I do zq 'bjørndal | {id, name} I get this error:

zq: error parsing Zed at column 3:
bjørndal | {id, name}
= ^ ===

and I have to go back and edit this expression to zq '"bjørndal" | {id, name}'. It all seems to depend on the definition of https://zed.brimdata.io/docs/language/search-expressions#keyword-search-term which says that only a very limited set of ASCII letters are part of keywords. I wish you would extend this. Most modern languages just follow the Unicode classifications of letters to define stuff like this.

We discussed this one as a team and consensus was that this should not be too difficult to extend in the parser.

Here's a repro using data input.csv.

$ cat input.csv 
id,name
1,bjorndal
2,bjørndal

$ zq -version
Version: v1.9.0-7-gb8e1b188

$ zq 'bjorndal' input.csv
{id:1.,name:"bjorndal"}

$ zq 'bjørndal' input.csv
zq: error parsing Zed at column 3:
bjørndal
= ^ ===

[artemklevtsov]

The same with yield:

$ echo '{"тест": "значение"}' | zq -Z 'yield тест' -
# zq: error parsing Zed at column 7:
# yield тест
#   === ^ ===

[philrz]

Verified in Zed commit 24b9cc9.

Repeating the original repro steps, now I can search for and successfully find the unicode text.

$ zq -version
Version: v1.10.0-10-g24b9cc95

$ zq 'bjørndal' input.csv
{id:2.,name:"bjørndal"}

Thanks @mattnibs!