Groovy removed from Elasticsearch 6

[r3dham]

https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_60_scripting_changes.html

As TimeSketch is still using Groovy, annotation and stars don't work on Elasticsearch 6.
Could you update the script to Painless?

[mari0d]

@r3dham I'm working on a PR for this. I have TimeSketch working cleanly with ES 6.x at https://github.com/mari0d/timesketch/tree/mari0d/painless and we are using that fork internally, but it currently isn't fully compatible with ES 5.x. Once I have the ES 5.x support back in I will open the PR (may need some handholding).

[mari0d]

@berggren is it worth me taking the time to do a PR for the work above? Not sure how far along you are with it; just noticed that you mentioned in issue 557 that you had the work queued up.

[berggren]

@mari0d Sorry for the delay. We already have painless scripts ready in the repository. We just need to update the documentation and make some minor changes to the loading.

Example:
https://github.com/google/timesketch/blob/master/contrib/add_label.painless

I will make this a priority to have Timesketch be ES 6.x ready by default.

[r3dham]

Same error for annotation as bellow with painless scripts in @mari0d's and master branch

[2018-05-10 06:16:31,746] ERROR in app: Exception on /api/v1/sketches/2/event/annotate/ [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/flask_restful/__init__.py", line 480, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask/views.py", line 84, in view
    return self.dispatch_request(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask_restful/__init__.py", line 595, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask_login/utils.py", line 228, in decorated_view
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/timesketch/api/v1/resources.py", line 860, in post
    toggle=toggle)
  File "/usr/local/lib/python2.7/dist-packages/timesketch/lib/datastores/elastic.py", line 417, in set_label
    body=script)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 73, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 525, in update
    doc_type, id, '_update'), params=params, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 312, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 128, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, u'illegal_argument_exception', u'[script] unknown field [file], parser not found')

[adamjnichols]

Looks like the ‘file’ key is now ‘source’, but that change has not been made in the fork you mention.

[r3dham]

@berggren ES 6.x removed file scripts itself.(elastic/elasticsearch#24627)
It means that coping painless scripts into the scripts directory doesn't work.

We need to use "Stored Script" with "id" parameter. https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-using.html#modules-scripting-stored-scripts

Or embedded painless scripts with "source" parameter.

[berggren]

Yeah, thanks for the heads-up. We will need to figure out a clean migration to 6.x without brake all 5.x users. Will take a look asap.

[ptitdoc]

Please note that the commit attempt above only support Elasticsearch 6 using embedded painless scripts as suggested by @r3dham.
I currently use it as a workaround because I cannot install Elasticsearch 5.X.

[ptitdoc]

Side note: Elasticsearch version can probably retrieved from self.client.info(). How about using scripts embedded in code with a selection based on Elasticsearch version ?

I also noted that the histogram is only showing a single bar using this commit. I don't know yet if the commit is the cause.

[ptitdoc]

Also elasticsearch 6 now apparently requises specifying path:timesketch_label in nested querie. This is required for star or hidden queries.
However I managed to make star queries work several times before fixing this. I really dont understand why.