Implement filtering and sorting for the listTransactions endpoint.

[KtorZ]

Overview

This task will make it possible for callers of the listTransactions API endpoint:

1. to specify a time range, so that the endpoint will return only transactions that fall within the given range.

2. to specify a sort order, so that returned transactions can be returned in either ascending or descending order of time.

Equivalent filtering and sorting functionality will also be provided through the transaction list CLI command.

Acceptance Criteria

1. The listTransactions endpoint (and transaction list CLI command) must provide a way to specify a range of times, that when specified will restrict the list of transactions returned to just those transactions whose times are within the range. The range can be open or closed at either end.
2. The listTransactions endpoint (and transaction list CLI command) must provide a way to specify a sort order (either ascending or descending).

Design

Input

The listTransactions endpoint will accept the following query parameters, all optional:

Parameter	Definition
start	a start time, in ISO 8601 format. Basic and extended formats are both accepted. Times can be local (with a timezone offset) or UTC.
end	an end time, in ISO 8601 format. Basic and extended formats are both accepted. Times can be local (with a timezone offset) or UTC.
order	a sort order, either ascending or descending.

The transaction list CLI command will accept the same parameter names prefixed with --, with the same definitions.

Notes

1. The start and end parameters, if both are provided, are expected to be ordered so that start ≤ end. If start > end, the endpoint must return a 400 error.
2. The start and end parameters should be inclusive bounds.
3. If start is not provided, the endpoint should use the time of the earliest available historical transaction as a start time.
4. If end is not provided, the endpoint should use the time of the latest available historical transaction as an end time.
5. If order is not provided, the endpoint should use descending order by default.

Output

Queries in ascending order

The endpoint should return a list of all available transactions in the range start (inclusively) to end (inclusively) in ascending order of time.

Queries in descending order

The endpoint should return a list of all available transactions in the range end (inclusively) to start (inclusively) in descending order of time.

QA

Range Boundary Conditions

We should provide tests to confirm that:

• The endpoint doesn't return transactions that lie outside the range:

  for all transactions t in the returned list: start ≤ time (t) ≤ end.

• The endpoint doesn't omit transactions that lie within the range:

  for all historical transactions t, if start ≤ time (t) ≤ end, then t is included in the returned list.

• If start is not specified by the caller, then the time of the earliest historical transaction is used as a starting point.

• If end is not specified by the caller, then the time of the latest historical transaction is used as an ending point.

Development Plan

• @jonathanknowles Add start, end, and order query parameters to the API.
• @jonathanknowles Make the API return a 400 error if start > end.
• @jonathanknowles Add appropriate API tests to cover the case when start > end.
• @Anviking Add support for filtering in readTxHistory of the DB layer (perhaps dealing with SlotIds instead of UTCTimes)
• @Anviking / @KtorZ Convert given time range to slot and filter tx by slot ids.
• @jonathanknowles Update the API documentation to indicate the default order: descending.
• @jonathanknowles Update the API documentation to indicate that start must not be later than end.
• @jonathanknowles Update the API documentation to indicate the intended semantics if start or end are not provided.
• @jonathanknowles Provide tests to check that marginal range boundary conditions are respected.
• @piotr-iohk Provide integration tests for all functionality.
• @KtorZ Align transaction creation time with the start of the slot

PRs

Number	Base
#541	master
#543	master
#544	master
#588	master
#592	#588
#595	master
#605	master
#606	master
#608	master
#613	master
#617	master
#618	master
#626	master
#629	master
#637	master
#653	master

[KtorZ]

*review acceptance criteria 3 -> use previous range

[rvl]

I'm not sure I like using the Range header to define sort direction. It's a bit too implicit, and if the client has a bug that requests incorrect ranges, this will amplify their problems. Also, there is no way to set the sort order for an (half-) open range. I think we should use a query parameter to provide sort field and direction.

[jonathanknowles]

A decision was made to implement the proposal outlined here:

#467 (comment):

As a result of this decision, we created PR #580, which uses the start, end, and order query parameters to specify a range in listTransactions (with CLI equivalents).

However, we still need to define what behaviour we expect when start and end are specified, but when order is not specified.

Obvious Cases

In the following cases, it seems reasonable to choose ascending order as a default:

• When neither start nor end are specified.
• When only start is specified.
• When only end is specified.
• When both start and end are specified, and start < end.

Less Obvious Case

However, if both start and end are specified, but start > end, then the choice of default is less obvious.

We have three clear options:

Option 1

Treat start and end as directionless boundary markers when both are specified, and always sort according to the order parameter (with a default of ascending, if no order is specified).

Examples:

• --start 2019 --end 2000
  specifies all times between 2000 and 2019 in ascending order.
• --start 2019 --end 2000 --order ascending
  specifies all times between 2000 and 2019 in ascending order.
• --start 2019 --end 2000 --order descending
  specifies all times between 2000 and 2019 in descending order.

Option 2

Use the implied order, if none is specified.

Examples:

• --start 2019 --end 2000
  specifies all times between 2000 and 2019 in descending order.
• --start 2019 --end 2000 --order ascending
  specifies all times between 2000 and 2019 in ascending order.
• --start 2019 --end 2000 --order descending
  specifies all times between 2000 and 2019 in descending order.

Option 3

Disallow start > end in the both CLI and the API, returning a suitable error to the caller.

Question

@KtorZ @Anviking @rvl Which of these options would you prefer?

Whichever way we choose, our documentation and testing should be explicit and consistent.

[jonathanknowles]

Decision from team discussion: use Option #3 from above.

In addition:

If start > end, regardless of the ordering, the API returns either an explicit error 400.

[KtorZ]

@jonathanknowles As mentioned on slack, this story has taken way more time than originally. A reason for this has been the change from header to query parameters, but also, the introduction of a limit parameter, which gives birth to many more questions about pagination.

Yet, the initial requirements said nothing about the ability to limit the number of items in the response, and also this would be practical for a particular client and, something we are looking forward to implementing in the future, it is rather out-of-scope here.

Hence, trying to manage a bit this story, let's not yet do anything about the limit and pagination and allocate more time in a different story to do so. I'll add a U/S on the roadmap regarding this.

This also removes the need for any X-Next-Query header, or Content-Range header and leave room for discussing a solution for pagination properly.

[jonathanknowles]

I've now removed the part of the design relating to paging.

I've created a new issue here to capture the discussion on how to provide a paging solution: https://github.com/input-output-hk/cardano-wallet-adr/issues/1