Scroll memory improvement

[rattuscz]

Remove reference to current result set before running another call to ES - without this it was effectively doubling memory requirement for data as json parsing and processing to ResultSet was done before previous ResultSet was released.

As i'm working with quite large dataset i hit memory limit and found i can fix it by just removing the reference before next search() call.

[ruflin]

Could you see any side effects of this change?

[rattuscz]

I've not seen any side effects, the _currentResultSet field is nulled just before next search is called, and so it is again set to next result inside the _setScrollId() method.
It should therefore be contained within the call of next() or rewind() and will be valid after the method returns.
I guess there is change in that last resultset is not available if search throws exception.

Also it may be good to include nulling the result in the clear() method, if for some reason not all results are iterated and it is therefore not removed during last next() call.
It's not my usecase though so i did not actually test it.

[thePanz]

@ruflin LGTM, should I squash-and-merge this one?

[ruflin]

@thePanz Go for it.

@rattuscz Perhaps we can follow up with the clear() part?

[thePanz]

Handling the "nulling" of the property in the clear() call would be great @rattuscz .
Could you open another PR for that? thank you!

[rattuscz]

@ruflin thought about the clear() and I believe it's mostly useless to manually set to null.

• Either you use the scroll object again for iteration in which case it will be nulled before pulling new data
• or you just discard the whole Scroll object therefore also removing the last ResultSet

So I'm not seeing any usecase for that after all.