Add the ability to execute query without fetching the result (SNOW-502598)

[TimShilov]

What is the current behavior?

At the moment there seems to be no way to execute a query and only get queryId. Every time you execute the query the whole response is still loaded which leads to memory issues (like this one).

What is the desired behavior?

I would like to execute the request, wait for its completion and only get queryId in response for further usage with RESULT_SCAN().

How would this improve snowflake-connector-nodejs?

That will greatly optimize the network usage of the client and will allow complex transformations to be done.

References, Other Background

I was talking about it with Support (Case Number 00510759) and they have pointed me to #43 which seems to be in the works and should help resolve memory issues as I will be able to stream data (to nowhere I guess) more effectively without using too much memory but I feel like it is wrong to fetch something I don't need in the first place. I want to further process the data inside Snowflake (as mentioned in documentation). Responses in my case can be huge (tens of millions of rows) and even memory aside, that's going to waste time and network traffic.

What is your Snowflake account identifier, if any?

xe29001.us-central1.gcp

[sfc-gh-dszmolka]

hello and thank you for submitting this issue ! if i understand correctly, you would like async query execution capabilities for the nodejs driver, something similar to what for example the JDBC driver and the SQL REST API already has .

this request is already with the driver team under the internal jira ID with which i just updated the Issue title. sadly, at this moment the improvement is not prioritised and therefore i am unable to give even an estimation for the expected implementation and availability timeline. Since you're a Snowflake customer, i recommend as of next step to contact your Sales rep. and tell them how important this capability would be for you. This helps the product team with prioritisation and resource allocation between the many requests.

Closing this issue for now but if you feel your request would be something entirely different from the aforementioned async query execution capability (submit request -> immediately return queryID and control to the caller -> execute further calls in arbitrary later time to retrieve the results/do something with the already-executed query) , please comment and I can reopen if necessary.

[TimShilov]

So you are saying that you're not going to support this in SDK and point me to the API? Isn't that the whole purpose of an SDK to allow people to not use the API directly and have all the best practices built-in?
Also, isn't the fact that it is implemented in the JDBC driver mean that it is possible to achieve and also that there's a need for that?

I can't see any logical sense in all that. You have the documentation talking about a workflow, and you have support for this for Java but when it comes to JavaScript that's just not possible and it's not a priority?
If the narrative here is to "go use API" then I'm fine with that, I'm just really struggling to understand that this position.

Will you accept a pull request if our team will add support for this functionality ourselves?

[sfc-gh-dszmolka]

No, that was not what i was meant to say, apologies if my response was confusing. Let me try to word it another way.

1. there is a definite need for async query execution capability in nodejs driver too
2. especially that other Snowflake drivers already support it (mentioned for example: JDBC, REST API. not as a replacement, not as a narrative - but an example).
3. we already plan to support async query execution capability in nodejs driver too
4. currently i'm not able to estimate when 3. will be implemented. It is indeed at this very moment of me writing this comment, not a priority. This can be changed very quickly however, if the product team sees how much a certain feature is needed. that's why i recommended to word your needs to your Snowflake Sales representative who has additional means of making requests more important.

Pull requests are of course more than welcome and much appreciated. It can speed up implementation tremendously.

[sfc-gh-dszmolka]

#208