[FEATURE REQ] spring-data-cosmos paging without incurring cost of count query for custom query using paginationQuery method.

[anath-onetrust]

Is your feature request related to a problem? Please describe.
I want to page through a large result set using continuation tokens and the azure-spring-data-cosmos library method which will accept my query with return argument. However, CosmosTemplate repository methods(paginationQuery) always issue a count query to populate the total elements in the page response, and when the query is complex and the result set is large, the count query can be very expensive.

Describe the solution you'd like
I would like to be able to use CosmosPageRequest to fetch a page of data with a continuation token without issuing a count query to populate the total elements. There is a private slice method in CosmosTemplate which is called from paginationQuery method after executing count query , would like a wrapper public method of that so that it can call directly to avoid count query .Existing private method private Slice sliceQuery(SqlQuerySpec querySpec,Pageable pageable, Sort sort,Class returnType, String containerName)
Describe alternatives you've considered
Considered extending CosmosTemplate, but too many fields and methods are defined with private access modifiers.

Additional context
Add any other context or screenshots about the feature request here.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Description Added
• Expected solution specified

[joshfree]

@backwind1233 @yiliuTo could you please take a look?

[backwind1233]

Hi, @kushagraThapar could you please take a look?

[kushagraThapar]

@anath-onetrust - I see that sliceQuery in CosmosTemplate is actually a public API. So that can be used if needed.
Also, this code snippet shows how to use slice query.

SliceQueriesUserRepository - https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-spring-data-cosmos/src/samples/java/com/azure/spring/data/cosmos/SliceQueriesUserRepository.java

[anath-onetrust]

@kushagraThapar Yes there is a public sliceQuery method in CosmosTemplate but that does not meet our requirement .We need to pass a return argument in sliceQuery which is not supported by public method but private method supports it .Sharing both method signature for reference .

private Slice sliceQuery(SqlQuerySpec querySpec,
Pageable pageable, Sort sort,
Class returnType, String containerName)

public Slice sliceQuery(CosmosQuery query, Class domainType, String containerName)

[kushagraThapar]

@anath-onetrust - in the public API, domainType is the actual return type. So you should be able to use it.
See this for example -
final Slice<Person> slice = cosmosTemplate.sliceQuery(query, Person.class, containerName);
https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-spring-data-cosmos-test/src/test/java/com/azure/spring/data/cosmos/core/CosmosTemplateIT.java#L597

[anath-onetrust]

@kushagraThapar I saw the API but here we have an issue to use existing public API since the API only supports CosmosQuery in query parameter but our requirement is to pass SqlQuerySpec in API call which is not supported .Is there any way to achieve this.

public Slice sliceQuery(CosmosQuery query, Class domainType, String containerName)

[anath-onetrust]

@kushagraThapar Could you please update us we are waiting for it .

[kushagraThapar]

@anath-onetrust while we try to implement this in spring / or investigate the possibility of it, can you use cosmos java SDK client and query directly using java SDK instead of spring ?
For example, you can get the cosmosClient bean through applicationContext in spring data SDK, and then using that cosmosClient bean, you can directly query through cosmos java SDK using SqlQuerySpec

[anath-onetrust]

@kushagraThapar what if we add a new public method in CosmosTemplate that internally call private sliceQuery method something like .

public Slice runSliceQuery(SqlQuerySpec querySpec, Pageable pageable, Class<?> domainType, Class returnType) {
final String containerName = getContainerName(domainType);
return sliceQuery(querySpec, pageable, pageable.getSort(), returnType, containerName);
}

I believe this is a straight forward change if you accept this I can raise a PR for same .
Actually here we are depending on requestContinuation for pagination so not sure if we use cosmosClient we will get it or not .

[kushagraThapar]

@anath-onetrust - That seems like a possible solution for the time being. We can try it out and see how it works for you guys. Please raise a PR and I will look into it. Thank you!

[anath-onetrust]

@kushagraThapar I have raised a PR please let me know if I have missed anything .
#24540

[kushagraThapar]

@anath-onetrust - Thanks for raising the PR, it looks good to me.
However my concern is that it might not support complete pagination APIs, like getTotalCount / getTotalPages / next / previous, etc.

Please read this github issue, where we are having the discussion on how to support pagination on spring data cosmos SDK - #24262

[johnmannix]

For issue #24262 it is worth pointing out that you can add a custom method to the repository returning Slice<T> and it will automatically return results that support pagination via continuation token without issuing a count query, and without having to call custom methods on the cosmosTemplate.

Even if you are planning to adjust the way the main pageable implementation works, it makes sense to include the runSliceQuery api for parity with runPageQuery, and that would be enough to unblock us.

[kushagraThapar]

@johnmannix @anath-onetrust - this has been done and will be released today. Thank you again for your contribution as always :)
Appreciate it!