SQL conversion of relative forms #3419

jnsrnhld · 2024-05-03T16:41:35Z

No description provided.

This reverts commit 55c33eb.

…s/README.md Co-authored-by: Torben Meyer <torben.meyer@bakdata.com>

…ect/HanaSqlFunctionProvider.java Co-authored-by: Torben Meyer <torben.meyer@bakdata.com>

…sion' into sql/feature/absolute-form-conversion

…s/README.md Co-authored-by: awildturtok <1553491+awildturtok@users.noreply.github.com>

…conversion # Conflicts: # backend/src/main/java/com/bakdata/conquery/sql/conversion/dialect/HanaSqlFunctionProvider.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/dialect/PostgreSqlFunctionProvider.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/dialect/SqlDialect.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/forms/FormCteStep.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/forms/HanaStratificationFunctions.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/forms/README.md # backend/src/main/java/com/bakdata/conquery/sql/conversion/forms/StratificationFunctions.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/forms/StratificationTableFactory.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/query/AbsoluteFormQueryConverter.java

jnsrnhld · 2024-05-03T16:42:57Z

...end/src/main/java/com/bakdata/conquery/sql/conversion/forms/HanaStratificationFunctions.java

+			// end date of validity dates is exclusive, but we need the inclusive one
+			case LATEST -> functionProvider.addDays(DSL.max(validityDate.getEnd()), DSL.val(-1));
+			case RANDOM -> {
+				// TODO seed required?


Ich glaube wir hatten darüber mal gesprochen, dass es einen Seed dafür bei euch gibt?

Genau, der ist als Konstante in ConqueryConstants.RANDOM_SEED festgelegt.

Es war nur wichtig, dass der Seed konstant war zur Reproduzierbarkeit von eventuellen Fehlern.

Also das Problem ist, dass nach meinem Kenntnisstand für HANA keine Möglichkeit existiert einen Seed zu setzen. Für Postgres schon.

thoniTUB · 2024-05-06T06:59:59Z

...src/main/java/com/bakdata/conquery/sql/conversion/forms/PostgresStratificationFunctions.java

+	protected Field<Date> jumpToNextQuarterStart(Field<Date> date) {
+		Field<Timestamp> yearStart = dateTruncate(DSL.val("year"), date);
+		Field<Date> quarterEndInclusive = date.minus(1);
 		Field<Integer> quarter = functionProvider.extract(DatePart.QUARTER, quarterEndInclusive);
 		return addQuarters(yearStart, quarter, Offset.NONE);
 	}


Kann es sein, dass die Funktion mit einem Datum wie z.B. 01.01.2024 das falsche Ergebnis rausgibt.

Statt 01.10.2023, kommt dann 01.10.2024 heraus.

thoniTUB · 2024-05-06T07:00:41Z

...src/main/java/com/bakdata/conquery/sql/conversion/forms/PostgresStratificationFunctions.java

+			case EARLIEST -> DSL.min(lower(validityDate));
+			// upper returns the exclusive end date, we want to inclusive one, so we add -1 day
+			case LATEST -> functionProvider.addDays(DSL.max(upper(validityDate)), DSL.val(-1));
+			case RANDOM -> {


Hier bitte auch einen konstanten Seed nutzen

Man kann in SQL den seed nicht so setzen, wie wir das machen, das müsste dann als CTE in jeder Query passieren

...src/main/java/com/bakdata/conquery/sql/conversion/forms/PostgresStratificationFunctions.java

awildturtok · 2024-05-07T10:26:04Z

...end/src/main/java/com/bakdata/conquery/sql/conversion/forms/HanaStratificationFunctions.java

+	}
+
+	@Override
+	protected Field<Date> jumpToNextQuarterStart(Field<Date> date) {


Quartal 4 landet im nächsten Jahr

Ist in dem Fall beabsichtigt.

awildturtok · 2024-05-07T10:26:24Z

...end/src/main/java/com/bakdata/conquery/sql/conversion/forms/HanaStratificationFunctions.java

-		Field<Date> yearStart = jumpToYearStart(dateRange.getEnd());
-		Field<Date> inclusiveEnd = functionProvider.addDays(dateRange.getEnd(), DSL.val(-1));
-		Field<Integer> quartersInMonths = getMonthsInQuarters(inclusiveEnd, Offset.NONE);
+		return jumpToNextQuarterStart(dateRange.getEnd());


Good catch 👍 Danke dir.

awildturtok · 2024-05-07T10:29:37Z

...src/main/java/com/bakdata/conquery/sql/conversion/forms/PostgresStratificationFunctions.java

 	}

 	@Override
 	public Field<Date> quarterEnd(ColumnDateRange dateRange) {
-		Field<Timestamp> yearStart = dateTruncate(DSL.val("year"), upper(dateRange));
-		Field<Date> quarterEndInclusive = upper(dateRange).minus(1);
+		return jumpToNextQuarterStart(upper(dateRange));


links hattest du auch noch minus 1, kannst du bitte einen kommentar dran machen, wrum du das nicht brauchst?

Ich brauche sie, hatte es allerdings an der falschen Stelle in der jumpToNextQuarterStart() Methode.

awildturtok · 2024-05-07T10:30:27Z

...src/main/java/com/bakdata/conquery/sql/conversion/forms/PostgresStratificationFunctions.java

+			case EARLIEST -> DSL.min(lower(validityDate));
+			// upper returns the exclusive end date, we want to inclusive one, so we add -1 day
+			case LATEST -> functionProvider.addDays(DSL.max(upper(validityDate)), DSL.val(-1));
+			case RANDOM -> {


Man kann in SQL den seed nicht so setzen, wie wir das machen, das müsste dann als CTE in jeder Query passieren

backend/src/main/java/com/bakdata/conquery/sql/conversion/forms/RelativeStratification.java

…s/RelativeStratification.java Co-authored-by: awildturtok <1553491+awildturtok@users.noreply.github.com>

…ure/relative-form-conversion

…s/PostgresStratificationFunctions.java Co-authored-by: MT <12283268+thoniTUB@users.noreply.github.com>

This reverts commit 37d441c6efce4e423b09e694af19b2eca45029ac.

…ersion # Conflicts: # backend/src/main/java/com/bakdata/conquery/sql/conversion/dialect/SqlDialect.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/forms/FormCteStep.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/forms/HanaStratificationFunctions.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/forms/PostgresStratificationFunctions.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/forms/README.md # backend/src/main/java/com/bakdata/conquery/sql/conversion/forms/StratificationFunctions.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/forms/StratificationTableFactory.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/query/AbsoluteFormQueryConverter.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/query/EntityDateQueryConverter.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/query/FormConversionHelper.java

jnsrnhld · 2024-05-07T16:55:31Z

@thoniTUB @awildturtok Vielen dank nochmal für die Hinweise zu den Edge-Cases, da gab es noch einige Fehler. Ich habe dahingehend auch nochmal die Testcases angepasst. Da ist jetzt auch der Edge-Case mit 31.12./01.01. mit dabei.

jnsrnhld and others added 30 commits April 8, 2024 14:44

Implement absolute form conversion

941c5f4

Cleanup

389a4fe

Fix alignment

5c5cc49

No-Op commit

55c33eb

Revert "No-Op commit"

c3f8f46

This reverts commit 55c33eb.

refactoring

12b80e2

New approach

135d7a1

...

74d093e

Cleanup

cbfc427

Adjust docs

17e0c70

Update backend/src/main/java/com/bakdata/conquery/sql/conversion/form…

f62ec0c

…s/README.md Co-authored-by: Torben Meyer <torben.meyer@bakdata.com>

Update backend/src/main/java/com/bakdata/conquery/sql/conversion/form…

4dcaa4e

…s/README.md Co-authored-by: Torben Meyer <torben.meyer@bakdata.com>

Update backend/src/main/java/com/bakdata/conquery/sql/conversion/dial…

e90e729

…ect/HanaSqlFunctionProvider.java Co-authored-by: Torben Meyer <torben.meyer@bakdata.com>

Refactoring

d3f8eb2

Fix

8f509cf

Merge branch 'develop' into sql/feature/absolute-form-conversion

e16e731

Fix

c162133

Merge branch 'develop' into sql/feature/absolute-form-conversion

623af5f

Fix error message

9ce680c

Cleanup

e830a15

Merge branch 'develop' into sql/feature/absolute-form-conversion

5bc4b01

Merge branch 'develop' into sql/feature/absolute-form-conversion

7dcf72a

Fix calculating date diff counts

5f3dffb

Merge remote-tracking branch 'origin/sql/feature/absolute-form-conver…

27e6d88

…sion' into sql/feature/absolute-form-conversion

Merge branch 'develop' into sql/feature/absolute-form-conversion

e6b39e9

PR review changes

4e90719

Update backend/src/main/java/com/bakdata/conquery/sql/conversion/form…

6508ffd

…s/README.md Co-authored-by: awildturtok <1553491+awildturtok@users.noreply.github.com>

Cleanup

fd01238

Use Offset for HANA functions

f1f9311

Merge branch 'develop' into sql/feature/absolute-form-conversion

2ca950e

jnsrnhld and others added 4 commits April 25, 2024 07:47

Merge branch 'develop' into sql/feature/entity-date-query-conversion

142b1d4

More cleanup

3ca0ee9

Implement relative form conversion

8da6e63

jnsrnhld added backend Project SQL labels May 3, 2024

jnsrnhld commented May 3, 2024

View reviewed changes

jnsrnhld requested a review from awildturtok May 3, 2024 16:43

thoniTUB requested changes May 7, 2024

View reviewed changes

awildturtok approved these changes May 7, 2024

View reviewed changes

jnsrnhld and others added 6 commits May 7, 2024 14:08

Merge branch 'develop' into sql/feature/entity-date-query-conversion

1c3aed4

Fix DSL.val usage

e8388c2

Simplify feature joining

0cf33dc

Update backend/src/main/java/com/bakdata/conquery/sql/conversion/form…

81784f2

…s/RelativeStratification.java Co-authored-by: awildturtok <1553491+awildturtok@users.noreply.github.com>

Merge branch 'sql/feature/entity-date-query-conversion' into sql/feat…

6f379f7

…ure/relative-form-conversion

Update backend/src/main/java/com/bakdata/conquery/sql/conversion/form…

b0c5774

…s/PostgresStratificationFunctions.java Co-authored-by: MT <12283268+thoniTUB@users.noreply.github.com>

Base automatically changed from sql/feature/entity-date-query-conversion to develop May 7, 2024 14:33

jnsrnhld added 8 commits May 7, 2024 18:34

Rename function

db0be9f

Fix -1 day calc

4c15e43

Fix year end quarter aligned calc

a0cc258

Revert "Fix year end quarter aligned calc"

e024003

This reverts commit 37d441c6efce4e423b09e694af19b2eca45029ac.

Fix year end quarter aligned calc

5f9a25d

Fix upper bound calc

1df4baa

Fix day date distance calcs

b8c79d1

jnsrnhld requested a review from thoniTUB May 7, 2024 16:55

thoniTUB approved these changes May 8, 2024

View reviewed changes

jnsrnhld merged commit f26749d into develop May 8, 2024
6 checks passed

delete-merged-branch bot deleted the sql/feature/relative-form-conversion branch May 8, 2024 08:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SQL conversion of relative forms #3419

SQL conversion of relative forms #3419

jnsrnhld commented May 3, 2024

jnsrnhld May 3, 2024

thoniTUB May 6, 2024

jnsrnhld May 6, 2024

thoniTUB May 6, 2024

thoniTUB May 6, 2024

awildturtok May 7, 2024

awildturtok May 7, 2024

jnsrnhld May 7, 2024

awildturtok May 7, 2024

jnsrnhld May 7, 2024

awildturtok May 7, 2024

jnsrnhld May 7, 2024

awildturtok May 7, 2024

jnsrnhld commented May 7, 2024

SQL conversion of relative forms #3419

SQL conversion of relative forms #3419

Conversation

jnsrnhld commented May 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnsrnhld commented May 7, 2024