Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Infocard for Entity Preview #2616

Merged
merged 25 commits into from
Aug 22, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
0320294
Rough sketch of how to create infoCard query
awildturtok Jul 21, 2022
0626d96
Directly use the involved Workers
awildturtok Jul 21, 2022
9aaf318
pull Selects from PreviewConfig, asserting their resolvability and ma…
awildturtok Jul 26, 2022
f94bcdb
also embed Infocard Execution into Status
awildturtok Jul 26, 2022
4b14122
rough sketch, to evaluate and embed infocard result
awildturtok Aug 2, 2022
86d05b3
Fix expecting SingleLineResutl, where ArrayFormQueryPlan always retur…
awildturtok Aug 16, 2022
e30f3e4
Adapt test for EntityPreviewCard
awildturtok Aug 16, 2022
a7fe3f3
Merge branch 'develop' into feature/entity-extra-info
awildturtok Aug 16, 2022
b264cc8
Restructure EntityPreview as Form creating two subqueries
awildturtok Aug 17, 2022
47c348f
Update AutoDoc
github-actions[bot] Aug 17, 2022
0ac87ea
fix tests
awildturtok Aug 17, 2022
84c6eb3
adds types and semantics to EntityPreviewStatus.Info
awildturtok Aug 17, 2022
e53ec44
Update AutoDoc
github-actions[bot] Aug 17, 2022
aebff7c
cleanup and move queries to form construction
awildturtok Aug 18, 2022
ae79d2e
Update AutoDoc
github-actions[bot] Aug 18, 2022
4e9ae37
adds documentation
awildturtok Aug 18, 2022
57128ca
adds documentation
awildturtok Aug 18, 2022
5cbb457
Update AutoDoc
github-actions[bot] Aug 18, 2022
166e9fd
Merge branch 'develop' into feature/entity-extra-info
awildturtok Aug 22, 2022
acc0ab4
copy paste TestForm splitting into Rel and Abs
awildturtok Aug 22, 2022
6175260
fixup prior commit
awildturtok Aug 22, 2022
254e201
also allow labelling of InfoCardSelects by virtue of PrintSettings co…
awildturtok Aug 22, 2022
66c1dff
Merge branch 'develop' into feature/entity-extra-info
awildturtok Aug 22, 2022
c597b95
Update AutoDoc
github-actions[bot] Aug 22, 2022
94158d8
adds documentation
awildturtok Aug 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.TimeUnit;
Expand All @@ -29,11 +28,7 @@
import com.bakdata.conquery.apiv1.query.Query;
import com.bakdata.conquery.apiv1.query.QueryDescription;
import com.bakdata.conquery.apiv1.query.SecondaryIdQuery;
import com.bakdata.conquery.apiv1.query.TableExportQuery;
import com.bakdata.conquery.apiv1.query.concept.filter.CQTable;
import com.bakdata.conquery.apiv1.query.concept.specific.CQAnd;
import com.bakdata.conquery.apiv1.query.concept.specific.CQConcept;
import com.bakdata.conquery.apiv1.query.concept.specific.CQDateRestriction;
import com.bakdata.conquery.apiv1.query.concept.specific.external.CQExternal;
import com.bakdata.conquery.io.result.ResultRender.ResultRendererProvider;
import com.bakdata.conquery.io.storage.MetaStorage;
Expand All @@ -57,6 +52,8 @@
import com.bakdata.conquery.models.query.ExecutionManager;
import com.bakdata.conquery.models.query.ManagedQuery;
import com.bakdata.conquery.models.query.Visitable;
import com.bakdata.conquery.models.query.preview.EntityPreviewExecution;
import com.bakdata.conquery.models.query.preview.EntityPreviewForm;
import com.bakdata.conquery.models.query.visitor.QueryVisitor;
import com.bakdata.conquery.models.worker.DatasetRegistry;
import com.bakdata.conquery.models.worker.Namespace;
Expand All @@ -65,9 +62,7 @@
import com.google.common.collect.ClassToInstanceMap;
import com.google.common.collect.MutableClassToInstanceMap;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;

@Slf4j
Expand Down Expand Up @@ -215,10 +210,7 @@ public Stream<ExecutionStatus> getQueriesFiltered(Dataset datasetId, UriBuilder
.filter(q -> q.getState().equals(ExecutionState.DONE) || q.getState().equals(ExecutionState.NEW))
.filter(q -> subject.isPermitted(q, Ability.READ))
.map(mq -> {
OverviewExecutionStatus status = mq.buildStatusOverview(
uriBuilder.clone(),
subject
);
OverviewExecutionStatus status = mq.buildStatusOverview(uriBuilder.clone(), subject);
if (mq.isReadyToDownload(datasetAbilities)) {
status.setResultUrls(getDownloadUrls(config.getResultProviders(), mq, uriBuilder, allProviders));
}
Expand All @@ -241,7 +233,8 @@ public static List<URL> getDownloadUrls(List<ResultRendererProvider> renderer, M

return renderer.stream()
.map(r -> r.generateResultURLs(exec, uriBuilder.clone(), allProviders))
.flatMap(Collection::stream).collect(Collectors.toList());
.flatMap(Collection::stream)
.collect(Collectors.toList());

}

Expand Down Expand Up @@ -324,9 +317,7 @@ public void reexecute(Subject subject, ManagedExecution<?> query) {
log.info("User[{}] reexecuted Query[{}]", subject.getId(), query);

if (!query.getState().equals(ExecutionState.RUNNING)) {
datasetRegistry.get(query.getDataset().getId())
.getExecutionManager()
.execute(datasetRegistry, query, config);
datasetRegistry.get(query.getDataset().getId()).getExecutionManager().execute(datasetRegistry, query, config);
}
}

Expand Down Expand Up @@ -359,12 +350,13 @@ public FullExecutionStatus getQueryFullStatus(ManagedExecution<?> query, Subject
*/
public ExternalUploadResult uploadEntities(Subject subject, Dataset dataset, ExternalUpload upload) {

final CQExternal.ResolveStatistic statistic =
CQExternal.resolveEntities(upload.getValues(), upload.getFormat(),
datasetRegistry.get(dataset.getId()).getStorage().getIdMapping(),
config.getFrontend().getQueryUpload(),
config.getLocale().getDateReader(),
upload.isOneRowPerEntity()
final CQExternal.ResolveStatistic
statistic =
CQExternal.resolveEntities(upload.getValues(), upload.getFormat(), datasetRegistry.get(dataset.getId())
.getStorage()
.getIdMapping(), config.getFrontend()
.getQueryUpload(), config.getLocale()
.getDateReader(), upload.isOneRowPerEntity()

);

Expand All @@ -378,8 +370,10 @@ public ExternalUploadResult uploadEntities(Subject subject, Dataset dataset, Ext
final ConceptQuery query = new ConceptQuery(new CQExternal(upload.getFormat(), upload.getValues(), upload.isOneRowPerEntity()));

// We only create the Query, really no need to execute it as it's only useful for composition.
final ManagedQuery execution =
((ManagedQuery) datasetRegistry.get(dataset.getId()).getExecutionManager()
final ManagedQuery
execution =
((ManagedQuery) datasetRegistry.get(dataset.getId())
.getExecutionManager()
.createExecution(datasetRegistry, query, subject.getUser(), dataset, false));

execution.setLastResultCount((long) statistic.getResolved().size());
Expand All @@ -390,60 +384,36 @@ public ExternalUploadResult uploadEntities(Subject subject, Dataset dataset, Ext

execution.initExecutable(datasetRegistry, config);

return new ExternalUploadResult(
execution.getId(),
statistic.getResolved().size(),
statistic.getUnresolvedId(),
statistic.getUnreadableDate()
);
return new ExternalUploadResult(execution.getId(), statistic.getResolved().size(), statistic.getUnresolvedId(), statistic.getUnreadableDate());
}

/**
* Execute a {@link TableExportQuery} for a single Entity on some Connectors.
*
* @return
* @implNote we don't do anything special here, this request could also be made manually. We however want to encapsulate this behaviour to shield the frontend from knowing too much about the query engine.
* Create and submit {@link EntityPreviewForm} for subject on to extract sources for entity, and extract some additional infos to be used as infocard.
*/
public FullExecutionStatus getSingleEntityExport(Subject subject, UriBuilder uriBuilder, String idKind, String entity, List<Connector> sources, Dataset dataset, Range<LocalDate> dateRange) {
EntityPreviewForm form =
EntityPreviewForm.create(entity, idKind, dateRange, sources, config.getPreview().resolveInfoCardSelects(dataset, datasetRegistry));

final ConceptQuery entitySelectQuery =
new ConceptQuery(new CQDateRestriction(Objects.requireNonNullElse(dateRange, Range.all()), new CQExternal(List.of(idKind), new String[][]{{"HEAD"}, {entity}}, false)));

final TableExportQuery exportQuery = new TableExportQuery(entitySelectQuery);
exportQuery.setRawConceptValues(false);

exportQuery.setTables(
sources.stream()
.map(source -> {
final CQConcept cqConcept = new CQConcept();
cqConcept.setElements(List.of(source.getConcept()));
final CQTable cqTable = new CQTable();
cqTable.setConcept(cqConcept);
cqTable.setConnector(source);
cqConcept.setTables(List.of(cqTable));

return cqConcept;
})
.collect(Collectors.toList())
);

final ManagedExecution<?> execution = postQuery(dataset, exportQuery, subject, true);

// TODO make sure that subqueries are also system
// TODO do not persist system queries
final EntityPreviewExecution execution = (EntityPreviewExecution) postQuery(dataset, form, subject, true);

// collect id immediately so it does not get sucked into closure
final ManagedExecutionId id = execution.getId();

while (execution.awaitDone(10, TimeUnit.SECONDS) == ExecutionState.RUNNING) {
log.trace("Still waiting for {}", id);
if (execution.awaitDone(10, TimeUnit.SECONDS) == ExecutionState.RUNNING) {
log.warn("Still waiting for {} after 10 Seconds.", execution.getId());
throw new ConqueryError.ExecutionProcessingTimeoutError();
}


if (execution.getState() == ExecutionState.FAILED) {
throw ConqueryError.ContextError.fromErrorInfo(execution.getError());
}


// Use the provided format name to find the respective provider.
return getQueryFullStatus(execution, subject, uriBuilder, true);

FullExecutionStatus status = execution.buildStatusFull(storage, subject, datasetRegistry, config);
status.setResultUrls(getDownloadUrls(config.getResultProviders(), execution, uriBuilder, true));
return status;
}


}
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
package com.bakdata.conquery.apiv1.query;

import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
Expand All @@ -10,7 +9,6 @@
import com.bakdata.conquery.models.auth.entities.User;
import com.bakdata.conquery.models.datasets.Dataset;
import com.bakdata.conquery.models.execution.ExecutionState;
import com.bakdata.conquery.models.execution.ManagedExecution;
import com.bakdata.conquery.models.identifiable.ids.specific.ManagedExecutionId;
import com.bakdata.conquery.models.query.ManagedQuery;
import com.bakdata.conquery.models.query.QueryPlanContext;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
import com.bakdata.conquery.apiv1.query.concept.specific.external.CQExternal;
import com.bakdata.conquery.io.cps.CPSBase;
import com.bakdata.conquery.io.storage.MetaStorage;
import com.bakdata.conquery.models.auth.entities.User;
import com.bakdata.conquery.models.auth.entities.Subject;
import com.bakdata.conquery.models.auth.entities.User;
import com.bakdata.conquery.models.auth.permissions.Ability;
import com.bakdata.conquery.models.config.ConqueryConfig;
import com.bakdata.conquery.models.datasets.Dataset;
Expand All @@ -28,6 +28,7 @@
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import com.google.common.collect.ClassToInstanceMap;
import lombok.NonNull;
import org.jetbrains.annotations.NotNull;

@JsonTypeInfo(use = JsonTypeInfo.Id.CUSTOM, property = "type")
@CPSBase
Expand Down Expand Up @@ -71,8 +72,12 @@ default void addVisitors(@NonNull ClassToInstanceMap<QueryVisitor> visitors) {
* Check implementation specific permissions. Is called after all visitors have been registered and executed.
*/
default void authorize(Subject subject, Dataset submittedDataset, @NonNull ClassToInstanceMap<QueryVisitor> visitors, MetaStorage storage) {
authorizeQuery(this, subject, submittedDataset, visitors, storage);
}

public static void authorizeQuery(QueryDescription queryDescription, Subject subject, Dataset submittedDataset, @NotNull ClassToInstanceMap<QueryVisitor> visitors, MetaStorage storage) {
NamespacedIdentifiableCollector nsIdCollector = QueryUtils.getVisitor(visitors, NamespacedIdentifiableCollector.class);
ExternalIdChecker externalIdChecker = QueryUtils.getVisitor(visitors, QueryUtils.ExternalIdChecker.class);
ExternalIdChecker externalIdChecker = QueryUtils.getVisitor(visitors, ExternalIdChecker.class);
if (nsIdCollector == null) {
throw new IllegalStateException();
}
Expand All @@ -93,11 +98,11 @@ default void authorize(Subject subject, Dataset submittedDataset, @NonNull Class
subject.authorize(concepts, Ability.READ);

// Check reused query permissions
final Set<ManagedExecution<?>> collectedExecutions = collectRequiredQueries().stream()
.map(storage::getExecution)
.filter(Objects::nonNull)
.collect(Collectors.toSet());
final Set<ManagedExecution<?>> collectedExecutions = queryDescription.collectRequiredQueries().stream()
.map(storage::getExecution)
.filter(Objects::nonNull)
.collect(Collectors.toSet());

subject.authorize(collectedExecutions, Ability.READ);

// Check if the query contains parts that require to resolve external IDs. If so the subject must have the preserve_id permission on the dataset.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import com.bakdata.conquery.models.datasets.Column;
import com.bakdata.conquery.models.datasets.concepts.Concept;
import com.bakdata.conquery.models.datasets.concepts.ConceptElement;
import com.bakdata.conquery.models.datasets.concepts.Connector;
import com.bakdata.conquery.models.datasets.concepts.select.Select;
import com.bakdata.conquery.models.identifiable.ids.NamespacedIdentifiable;
import com.bakdata.conquery.models.query.DateAggregationMode;
Expand Down Expand Up @@ -86,6 +87,8 @@ public class CQConcept extends CQElement implements NamespacedIdentifiableHoldin
@JsonView(View.InternalCommunication.class)
private boolean aggregateEventDates;



@Override
public String defaultLabel(Locale locale) {
if (elements.isEmpty()) {
Expand Down Expand Up @@ -295,4 +298,30 @@ public void setDefaultExists() {
t.setSelects(conSelects);
}
}

public static CQConcept forSelect(Select select) {
CQConcept cqConcept = new CQConcept();
cqConcept.setElements(List.of(select.getHolder().findConcept()));
CQTable table = new CQTable();
cqConcept.setTables(List.of(table));

table.setConnector(((Connector) select.getHolder()));

table.setSelects(List.of(select));

return cqConcept;
}

public static CQConcept forConnector(Connector source) {
final CQConcept cqConcept = new CQConcept();
cqConcept.setElements(List.of(source.getConcept()));
final CQTable cqTable = new CQTable();
cqTable.setConcept(cqConcept);
cqTable.setConnector(source);
cqConcept.setTables(List.of(cqTable));

return cqConcept;
}
Comment on lines +302 to +324
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Würde ich ein eine Util Klasse packen und doku bitte



}
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ public class ConqueryConfig extends Configuration {
@NotNull
private FrontendConfig frontend = new FrontendConfig();

private PreviewConfig preview = new PreviewConfig();

@NotNull
@Valid
private SearchConfig search = new SearchConfig();
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
package com.bakdata.conquery.models.config;

import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;

import javax.ws.rs.core.UriBuilder;

import com.bakdata.conquery.models.auth.entities.Subject;
import com.bakdata.conquery.models.common.Range;
import com.bakdata.conquery.models.datasets.Dataset;
import com.bakdata.conquery.models.datasets.concepts.Connector;
import com.bakdata.conquery.models.datasets.concepts.select.Select;
import com.bakdata.conquery.models.error.ConqueryError;
import com.bakdata.conquery.models.identifiable.ids.specific.SelectId;
import com.bakdata.conquery.models.query.PrintSettings;
import com.bakdata.conquery.models.query.resultinfo.SelectResultInfo;
import com.bakdata.conquery.models.worker.DatasetRegistry;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.extern.slf4j.Slf4j;

@Data
@Slf4j
@AllArgsConstructor
@NoArgsConstructor
public class PreviewConfig {
/**
* Selects to be used in {@link com.bakdata.conquery.apiv1.QueryProcessor#getSingleEntityExport(Subject, UriBuilder, String, String, List, Dataset, Range)}.
* To be displayed as additional infos.
*
* @implSpec the order of selects is the order of the output fields.
*/
private List<InfoCardSelect> infoCardSelects = List.of();

@Data
public static class InfoCardSelect {
/**
* User facing label of the select.
*/
private final String label;
/**
* Id (without dataset) of the select.
*
* @implNote this id will be resolved at runtime and cannot be validated at startup.
*/
private final String id;
}

/**
* Used to map {@link SelectResultInfo} to {@link InfoCardSelect#getLabel()} via {@link PrintSettings#getColumnNamer()}.
*/
public String resolveSelectLabel(SelectResultInfo info) {
Comment on lines +52 to +55
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wo spielt der PrintSettings#getColumnNamer() hier rein?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Schau mal in com.bakdata.conquery.models.query.preview.EntityPreviewExecution#transformQueryResultToInfos

final String id = info.getSelect().getId().toStringWithoutDataset();

for (InfoCardSelect infoCardSelect : getInfoCardSelects()) {
if (infoCardSelect.getId().equals(id)) {
return infoCardSelect.getLabel();
}
}

throw new IllegalArgumentException(String.format("%s is not an InfoCard Select", info));
}

/**
* Find infoCard-selects by id within Dataset.
*/
public List<Select> resolveInfoCardSelects(Dataset dataset, DatasetRegistry registry) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

laut gh wird die methode in dieser Datei Zeile 395 benutzt 🤔

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Werden die Funktionen noch benutzt?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

com.bakdata.conquery.apiv1.QueryProcessor#getSingleEntityExport Zeile 395

final List<Select> infoCardSelects = new ArrayList<>();

for (InfoCardSelect select : getInfoCardSelects()) {
final SelectId selectId = SelectId.Parser.INSTANCE.parsePrefixed(dataset.getName(), select.getId());
final Select resolved = registry.resolve(selectId);

infoCardSelects.add(resolved);
}

final Set<Select> nonConnectorSelects = infoCardSelects.stream()
.filter(select -> !(select.getHolder() instanceof Connector))
.collect(Collectors.toSet());

if (!nonConnectorSelects.isEmpty()) {
log.error("The selects {} are not connector-Selects", nonConnectorSelects);
throw new ConqueryError.ExecutionCreationErrorUnspecified();
}

return infoCardSelects;

}
}
Loading