Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[#17872] YSQL: Planner support for distinct pushdown
Summary: #### Objective While the LSM index itself supports prefix based distinct skip scans, the planning layer lacks awareness of such support. We make an Index Scan variant called “Distinct Index Scan'' to express such scans. The primary objective of this change is to integrate Distinct Index Scans into YSQL. #### Design Implementing distinct pushdown in YSQL requires three broad changes. First, the planner must generate distinct index scan path nodes. Then, these path nodes must be integrated properly into the overall query plan. Last, a few minor changes are required in the execution layer. ##### Generating Distinct Index Paths Distinct Index Paths are generated along with other index paths. This is generally useful only when generating paths for a DISTINCT query, bar a few exceptions such as semi joins and UNION queries (unsupported at the moment). To avoid creating another scan path node, the index scan path data structure is reused for distinct index scans as well. However, there are a few fundamental differences. First, distinct index scans are parameterized by the prefix length on the index. Hence, the planner can distinguish a distinct index scan from a normal index scan by simply checking if the prefix length is non-zero. Additionally, recall that a distinct index scan fetches fewer rows from the storage and as a consequence, has a different path cost. Finally, the distinct operation cannot always be pushed down to the base relations because of which distinct index scans cannot always be generated alongside the regular index paths. These differences are discussed in more detail in the following sections. ###### Computing the prefix length To compute prefix length, first, we determine the set of keys that need to be unique. This change operates only on index-only scan scenarios. Moreover, this change does not apply in the presence of unsupported expressions such as aggregates and volatile functions within the requested column references. With our distinct key set we have just determined, use the shortest prefix that encompasses all these key columns. We look for the shortest possible prefix since using a longer one leads to a poorly performant scan due to fetching more rows than necessary. We can use distinct index scans for queries that request any subset of index columns, not just prefixes, albeit requiring further processing on top. ###### Range partitioned tables There is a small caveat with the current implementation of distinct pushdown. Range partitioned tables can return duplicate tuples since the DISTINCTification only happens within a tablet. To combat this, we generate a unique node on top of the distinct index scan, as a workaround. Hash partitioned tables do not exhibit this quirk. ###### Path Cost Accurate costing of distinct index path is not the primary focus of the change. Regardless, we adjust the cost so that it makes a bit more sense. We simply scale the cost down by a rough estimation of the number of duplicate values of the prefix. ##### Distinct Index Scan Integration In this section, we discuss how distinct queries were previously supported in YSQL, next we understand how pathkeys help in generating these query plans, then we reason why pathkeys do not mesh well with distinct pushdown, and finally we see how distinct queries are supported now with skip scan integration. ###### Distinct query plan generation before distinct pushdown There are two primary mechanisms to create paths for distinct queries. One is sort based and the other hash based. The hash based method uses a hash aggregation method to remove duplicate values, not too relevant to distinct pushdown. The sort based method, on the other hand, is similar to a skip scan. Here, the input is first sorted and the duplicate values are now easily removed since they are adjacent to each other. This is useful when we expect the input to be already sorted like in the output of index scans or merge sort joins. The planner identifies the correct ordering of such scans/joins by looking at their pathkeys. Pathkeys provide useful functionality such as optimizing away constants and equivalent columns. However, they have their limitations when it comes to pushing down distinct. ###### Using skip scans with distinct queries With skip scans, there is now a third way to generate a plan for distinct queries. A distinct query can use skip scans to generate a candidate distinct path once they are proven to be sufficiently distinct. While pathkeys provide a good mechanism for ordering proofs, they are inadequate for distinct pushdown. For example, not all distinct queries request leading columns of an index. In such cases, using a skip scan is still useful since they probably fetch fewer rows. We can then use the above techniques to deduplicate the output of the skip scan. Moreover, a distinct query need not request leading columns of an index in the same order as the index. To elaborate, unlike ordering, distinct does not care about the order in which the target keys are specified. This distinction is also visible when it comes to joins. Here, the distinct operation distributes over the join operation but the sort operation does not. For all these reasons, we use a different mechanism to propagate distinctness information to parent pathnodes. ##### Changes during execution time The changes here are minimal. The primary objective here is to propagate the prefix length computed at the planning time to the underlying storage layer. Moreover, DocDB should use a HybridScan whenever the query layer requests for a prefix based skip scan. #### Future Work We will extend support for more complex queries that may involve join trees or those using DISTINCT ON queries. Jira: DB-6955 Test Plan: ./yb_build.sh --java-test TestPgRegressDistinctPushdown Reviewers: smishra, mihnea, tnayak Reviewed By: tnayak Subscribers: jason, yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D26566
- Loading branch information
