Split imports on preprocess #3389

awildturtok · 2024-04-10T12:03:14Z

No description provided.

backend/src/main/java/com/bakdata/conquery/models/jobs/ImportJob.java

backend/src/main/java/com/bakdata/conquery/models/preproc/Preprocessed.java

backend/src/main/java/com/bakdata/conquery/models/jobs/ImportJob.java

…ob.java

extract methods for PreprocessorCommand#run

…, due to a bug)

awildturtok · 2024-04-23T16:42:05Z

backend/src/main/java/com/bakdata/conquery/models/preproc/PreprocessedData.java


 	private final Map<String, Integer> starts;
-	private final Map<String, Integer> lengths;
+	// TODO make sure that everyone respects this is an end not a length


noch offen?

lol, war an mich selber gerichtet, habe es aber noch nicht angeschaut.

Hier ist noch kein outdated dranne

backend/src/test/java/com/bakdata/conquery/util/support/StandaloneSupport.java

thoniTUB

Ich finde das mit dem ConsistentHash cool, aber muss es nochmal durch denken.

backend/src/main/java/com/bakdata/conquery/commands/PreprocessorCommand.java

thoniTUB · 2024-06-05T05:34:29Z

backend/src/main/java/com/bakdata/conquery/commands/PreprocessorCommand.java

+		preprocessJobs(jobs, buckets, config);
+
+
+		log.info("Successfully Preprocess {} Jobs:", success.size());


Suggested change

log.info("Successfully Preprocess {} Jobs:", success.size());

log.info("Successfully preprocessed {} jobs:", success.size());

thoniTUB · 2024-06-05T05:38:00Z

backend/src/main/java/com/bakdata/conquery/io/storage/NamespaceStorage.java

-		entity2Bucket.add(entity, bucket);
-
-		return bucket;
+	public void assignEntityBucket(String entity, int bucket) {


Suggested change

public void assignEntityBucket(String entity, int bucket) {

public void assignEntityToBucket(String entity, int bucket) {

backend/src/main/java/com/bakdata/conquery/io/storage/xodus/stores/CachedStore.java

backend/src/main/java/com/bakdata/conquery/models/jobs/ImportJob.java

backend/src/main/java/com/bakdata/conquery/models/preproc/PreprocessedHeader.java

backend/src/test/java/com/bakdata/conquery/integration/tests/RestartTest.java

backend/src/main/java/com/bakdata/conquery/models/preproc/Preprocessed.java

backend/src/main/java/com/bakdata/conquery/models/preproc/PreprocessedReader.java

backend/src/main/java/com/bakdata/conquery/commands/PreprocessorCommand.java

backend/src/main/java/com/bakdata/conquery/io/storage/xodus/stores/CachedStore.java

awildturtok · 2024-06-05T07:40:06Z

backend/src/main/java/com/bakdata/conquery/mode/cluster/ClusterImportHandler.java

@@ -31,17 +31,15 @@ class ClusterImportHandler implements ImportHandler {
 	@SneakyThrows
 	@Override
 	public void updateImport(Namespace namespace, InputStream inputStream) {
-		ImportJob job = ImportJob.createOrUpdate(
+		final Table table = ImportJob.createOrUpdate(


ja vorher war es besser, hat aber alles auf einmal im RAM gehalten, das will ich ja mit dem PR vermeiden. Aber der Name ist tatsächlich nicht mehr passend.

backend/src/main/java/com/bakdata/conquery/models/events/Bucket.java

backend/src/main/java/com/bakdata/conquery/models/preproc/Preprocessed.java

thoniTUB · 2024-06-06T11:53:50Z

backend/src/main/java/com/bakdata/conquery/mode/cluster/ClusterImportHandler.java

+	/**
+	 * Handle validity and update logic.
+	 */
+	public static Table createOrUpdate(DistributedNamespace namespace, boolean update, PreprocessedHeader header) {


Vielleicht nochmal splitten, der Kommentar passt besser als der Funktionsname

backend/src/main/java/com/bakdata/conquery/models/preproc/Preprocessed.java

backend/src/main/java/com/bakdata/conquery/mode/cluster/ClusterImportHandler.java

# Conflicts: # backend/src/main/java/com/bakdata/conquery/mode/cluster/ClusterImportHandler.java # backend/src/main/java/com/bakdata/conquery/models/jobs/ImportJob.java # backend/src/main/java/com/bakdata/conquery/models/preproc/PreprocessedHeader.java

thoniTUB

Bitte ausgiebig testen mit großen daten vor dem Mergen

thoniTUB · 2024-06-13T07:48:39Z

backend/src/main/java/com/bakdata/conquery/models/preproc/PreprocessedData.java


 	private final Map<String, Integer> starts;
-	private final Map<String, Integer> lengths;
+	// TODO make sure that everyone respects this is an end not a length


…e/split-cqpps

…terfere with submission

backend/src/main/java/com/bakdata/conquery/io/storage/StoreMappings.java

thoniTUB · 2024-06-27T13:02:55Z

backend/src/main/java/com/bakdata/conquery/mode/cluster/ClusterImportHandler.java

+	/**
+	 * select, then send buckets.
+	 */
+	public static WorkerId sendBucket(Bucket bucket, WorkerInformation responsibleWorker) {


Suggested change

public static WorkerId sendBucket(Bucket bucket, WorkerInformation responsibleWorker) {

private static WorkerId sendBucket(Bucket bucket, WorkerInformation responsibleWorker) {

backend/src/main/java/com/bakdata/conquery/mode/cluster/ClusterImportHandler.java

thoniTUB · 2024-06-27T13:20:00Z

Bitte noch mal über die offenen Konversationen schauen

# Conflicts: # backend/src/main/java/com/bakdata/conquery/models/jobs/CalculateCBlocksJob.java

…itions with high parallelism.

…tHashMap

…ort process that loads concepts last, batching CalculateCBlocksJobs

# Conflicts: # backend/src/main/java/com/bakdata/conquery/models/datasets/concepts/tree/ConceptTreeCache.java

awildturtok commented Apr 10, 2024

View reviewed changes

backend/src/main/java/com/bakdata/conquery/models/jobs/ImportJob.java Outdated Show resolved Hide resolved

thoniTUB reviewed Apr 10, 2024

View reviewed changes

backend/src/main/java/com/bakdata/conquery/models/preproc/Preprocessed.java Outdated Show resolved Hide resolved

backend/src/main/java/com/bakdata/conquery/models/jobs/ImportJob.java Outdated Show resolved Hide resolved

awildturtok added 3 commits April 23, 2024 16:19

initial impl to split cqpps on write

faaa7f8

Update backend/src/main/java/com/bakdata/conquery/models/jobs/ImportJ…

ff1499c

…ob.java

adds bucket as preprocess param

f38d321

extract methods for PreprocessorCommand#run

awildturtok force-pushed the feature/split-cqpps branch from f1964db to f38d321 Compare April 23, 2024 15:30

awildturtok added 2 commits April 23, 2024 17:30

minor fix to CachedStore

0f2a151

use new store in RestartTest (old environments were closed but worked…

4917e0a

…, due to a bug)

awildturtok marked this pull request as ready for review April 23, 2024 16:01

awildturtok requested a review from thoniTUB April 23, 2024 16:01

awildturtok commented Apr 23, 2024

View reviewed changes

awildturtok added 2 commits May 2, 2024 16:00

fix bollean parameters

ec06b0e

count buckets instead of exception-oriented flow

09618e5

thoniTUB requested changes Jun 5, 2024

View reviewed changes

awildturtok commented Jun 5, 2024

View reviewed changes

awildturtok added 2 commits June 5, 2024 12:49

review feedback

729f41e

rearrange ImportJob methods into ClusterImportHandler.

8b77f38

awildturtok requested a review from thoniTUB June 6, 2024 08:30

thoniTUB reviewed Jun 6, 2024

View reviewed changes

awildturtok added 4 commits June 12, 2024 15:18

cleanup

a1c22d4

map validation exceptions to http error

bd3a5c7

fix dropped param

9601c87

awildturtok requested a review from thoniTUB June 12, 2024 13:55

thoniTUB approved these changes Jun 13, 2024

View reviewed changes

awildturtok added 5 commits June 13, 2024 15:29

Merge branch 'develop' into feature/split-cqpps

b5c49d4

delay CalulateCBlocksJob to speedup import AND batching of CBlocksJob

83bd498

cleanup of toString and logging in ImportJob.java

b897477

Merge remote-tracking branch 'origin/feature/split-cqpps' into featur…

428646c

…e/split-cqpps

adds missing CPSType anno

bff7c07

awildturtok force-pushed the feature/split-cqpps branch from 220f5ab to 92c3561 Compare June 24, 2024 13:14

awildturtok added 13 commits June 24, 2024 15:15

fixes ENTITY_TO_BUCKET type signature

bdb2048

fix NonPersistentStoreFactory types

88090ba

experiment: drop porgressReporter. We suspect directExecutor might in…

0fddfa3

…terfere with submission

log pool information while waiting

5f47569

reduce logging

e3046e3

drop way too coarse mutex

32bd184

fixes logging

6937f16

restructure entities registration

b33e282

simplify CalculateCBlocksJob

d7a4dd7

fix ProgrammaticTests

858bb9b

adds new adminEndpointInfo

753aa1e

adds missing fix CPSBase of StartCalculateCblocks phase

735aec8

fixes a bug with entities not getting registered

5de8f20

awildturtok requested a review from thoniTUB June 26, 2024 14:10

logging cleanup

1e3a077

thoniTUB approved these changes Jun 27, 2024

View reviewed changes

awildturtok and others added 11 commits July 3, 2024 15:46

reverts some schema changes to make migration to old storage possible

020e08e

Merge remote-tracking branch 'origin/develop' into feature/split-cqpps

be059f1

# Conflicts: # backend/src/main/java/com/bakdata/conquery/models/jobs/CalculateCBlocksJob.java

change log-level of WorkerStorage messages for Bucket/CBlock management

ee1a236

makes ProgressReporterImpl.java multithreaded

23f745b

use ConcurrentHashMap in ConceptTreeCache as it might cause race-cond…

f38c0cc

…itions with high parallelism.

store Cache using Optional to circumvent null limitation of Concurren…

488130d

…tHashMap

minor logging fixes for CalculateCBlocksJob.java

7ebff2a

revert deferral of calculating CBlocks. Instead we use an adapted imp…

5b959b3

…ort process that loads concepts last, batching CalculateCBlocksJobs

Merge branch 'develop' into feature/split-cqpps

67033ed

# Conflicts: # backend/src/main/java/com/bakdata/conquery/models/datasets/concepts/tree/ConceptTreeCache.java

fix missed calculateCBlocks reference

f439350

Merge remote-tracking branch 'origin/develop' into feature/split-cqpps

7cce1c0

awildturtok merged commit bcad24d into develop Aug 5, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split imports on preprocess #3389

Split imports on preprocess #3389

awildturtok commented Apr 10, 2024

awildturtok Apr 23, 2024

thoniTUB Jun 13, 2024

awildturtok Jun 13, 2024

thoniTUB Jun 27, 2024

thoniTUB left a comment

thoniTUB Jun 5, 2024

thoniTUB Jun 5, 2024

awildturtok Jun 5, 2024

thoniTUB Jun 6, 2024

thoniTUB left a comment

thoniTUB Jun 13, 2024

thoniTUB Jun 27, 2024

thoniTUB commented Jun 27, 2024

		preprocessJobs(jobs, buckets, config);


		log.info("Successfully Preprocess {} Jobs:", success.size());

	log.info("Successfully Preprocess {} Jobs:", success.size());
	log.info("Successfully preprocessed {} jobs:", success.size());

	public void assignEntityBucket(String entity, int bucket) {
	public void assignEntityToBucket(String entity, int bucket) {

	public static WorkerId sendBucket(Bucket bucket, WorkerInformation responsibleWorker) {
	private static WorkerId sendBucket(Bucket bucket, WorkerInformation responsibleWorker) {

Split imports on preprocess #3389

Split imports on preprocess #3389

Conversation

awildturtok commented Apr 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thoniTUB left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thoniTUB left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thoniTUB commented Jun 27, 2024