Add a `format` option to `docvalue_fields`. #29639

jpountz · 2018-04-20T16:09:15Z

This commit adds the ability to configure how a docvalue field should be
formatted, so that it would be possible eg. to return a date field
formatted as the number of milliseconds since Epoch.

Closes #27740

This commit adds the ability to configure how a docvalue field should be formatted, so that it would be possible eg. to return a date field formatted as the number of milliseconds since Epoch. Closes elastic#27740

elasticmachine · 2018-04-20T16:09:16Z

Pinging @elastic/es-search-aggs

colings86

@jpountz I left some comments but I like the change in general

colings86 · 2018-04-23T09:43:16Z

docs/reference/search/request/docvalue-fields.asciidoc

 }
 --------------------------------------------------
 // CONSOLE
+<1> the name of the field
+<2> the special `use_field_mapping` format tells Elasticsearch to use the format from the mapping


Could we not make the format field optional and indicate to use the field mapping by the option not being present? This would be in line with other places we support format such as in the date range query and date range aggregation

This is the goal. The issue is that if we do this in 6.4 this will be a breaking change. For instance today date fields are returned as the output of Joda's ReadableInstant#toString(). I plan to make using the format from mappings the default in a follow-up PR that will only target 7.0.

ok great, thanks for clarifying

colings86 · 2018-04-23T09:46:18Z

...in/java/org/elasticsearch/search/aggregations/metrics/tophits/TopHitsAggregationBuilder.java

-                builder.value(fieldDataField);
+            for (FieldAndFormat dvField : docValueFields) {
+                if (dvField.format == null) {
+                    builder.value(dvField.field);


I wonder if we should no always render this with the expanded object form to be explicit?

Fine with me. Should we skip the format when it is null or emit an explicit null?

I would skip format when its null but always use the object variant

colings86 · 2018-04-23T09:46:59Z

server/src/main/java/org/elasticsearch/search/builder/SearchSourceBuilder.java

-                builder.value(docValueField);
+            for (FieldAndFormat docValueField : docValueFields) {
+                if (docValueField.format == null) {
+                    builder.value(docValueField);


I wonder if we should no always render this with the expanded object form to be explicit?

colings86 · 2018-04-23T09:48:13Z

server/src/main/java/org/elasticsearch/search/fetch/subphase/DocValueFieldsContext.java

+         * Parse a {@link FieldAndFormat} from some {@link XContent}.
+         */
+        public static FieldAndFormat fromXContent(XContentParser parser) throws IOException {
+            Token token = parser.currentToken();


Could we maybe use the ObjectParser or ConstructingObjectParser here since that is now the preferred way to parse JSON?

my understanding was that it is tricky when you need to be able to handle either a string or an object

Maybe we could do something similar to what Script does?

colings86

LGTM

Relates elastic#29658

jpountz · 2018-05-09T12:38:32Z

@elasticmachine run gradle build tests

This commit adds the ability to configure how a docvalue field should be formatted, so that it would be possible eg. to return a date field formatted as the number of milliseconds since Epoch. Closes #27740

Relates #29639

* master: (25 commits) [DOCS] Splits auditing.asciidoc into smaller files Reintroduce mandatory http pipelining support (elastic#30820) Painless: Types Section Clean Up (elastic#30283) Add support for indexed shape routing in geo_shape query (elastic#30760) [test] java tests for archive packaging (elastic#30734) Revert "Make http pipelining support mandatory (elastic#30695)" (elastic#30813) [DOCS] Fix more edit URLs in Stack Overview (elastic#30704) Use correct cluster state version for node fault detection (elastic#30810) Change serialization version of doc-value fields. [DOCS] Fixes broken link for native realm [DOCS] Clarified audit.index.client.hosts (elastic#30797) [TEST] Don't expect acks when isolating nodes Add a `format` option to `docvalue_fields`. (elastic#29639) Fixes UpdateSettingsRequestStreamableTests mutate bug Mustes {p0=snapshot.get_repository/10_basic/*} YAML test Revert "Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled" Only allow x-pack metadata if all nodes are ready (elastic#30743) Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled Use original settings on full-cluster restart (elastic#30780) Only ack cluster state updates successfully applied on all nodes (elastic#30672) ...

* 6.x: [DOCS] Fixes typos in security settings Add support for indexed shape routing in geo_shape query (#30760) [DOCS] Splits auditing.asciidoc into smaller files Painless: Types Section Clean Up (#30283) [test] java tests for archive packaging (#30734) Deprecate http.pipelining setting (#30786) [DOCS] Fix more edit URLs in Stack Overview (#30704) Use correct cluster state version for node fault detection (#30810) [DOCS] Fixes broken link for native realm [DOCS] Clarified audit.index.client.hosts (#30797) Change serialization version of doc-value fields. Add a `format` option to `docvalue_fields`. (#29639) [TEST] Don't expect acks when isolating nodes Fixes UpdateSettingsRequestStreamableTests mutate bug Revert "Add more yaml tests for get alias API (#29513)" Revert "Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled" Only allow x-pack metadata if all nodes are ready (#30743) Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled Use original settings on full-cluster restart (#30780) Only ack cluster state updates successfully applied on all nodes (#30672) Replace Request#setHeaders with addHeader (#30588) [TEST] remove endless wait in RestClientTests (#30776) QA: Add xpack tests to rolling upgrade (#30795) Add support for search templates to the high-level REST client. (#30473) Reduce CLI scripts to one-liners on Windows (#30772) Fold RestGetAllSettingsAction in RestGetSettingsAction (#30561) Add more yaml tests for get alias API (#29513) [Docs] Fix script-fields snippet execution (#30693) Convert FieldCapabilitiesResponse to a ToXContentObject. (#30182) Remove assert statements from field caps documentation. (#30601) Fix a bug in FieldCapabilitiesRequest#equals and hashCode. (#30181) Add support for field capabilities to the high-level REST client. (#29664) [DOCS] Add SAML configuration information (#30548) [DOCS] Remove X-Pack references from SQL CLI (#30694) [Docs] Fix typo in circuit breaker docs (#29659) [Feature] Adding a char_group tokenizer (#24186) Increase the maximum number of filters that may be in the cache. (#30655) [Docs] Fix broken cross link in documentation Test: wait for netty threads in a JUnit ClassRule (#30763) [Security] Include an empty json object in an json array when FLS filters out all fields (#30709) [DOCS] fixed incorrect default [TEST] Wait for CS to be fully applied in testDeleteCreateInOneBulk Enable installing plugins from snapshots.elastic.co (#30765) Ignore empty completion input (#30713) Fix docs failure on language analyzers (#30722) [Docs] Fix inconsistencies in snapshot/restore doc (#30480) Add Delete Repository High Level REST API (#30666) Reduce CLI scripts to one-liners (#30759)

* master: [DOCS] Fixes typos in security settings Fix GeoShapeQueryBuilder serialization after backport [DOCS] Splits auditing.asciidoc into smaller files Reintroduce mandatory http pipelining support (#30820) Painless: Types Section Clean Up (#30283) Add support for indexed shape routing in geo_shape query (#30760) [test] java tests for archive packaging (#30734) Revert "Make http pipelining support mandatory (#30695)" (#30813) [DOCS] Fix more edit URLs in Stack Overview (#30704) Use correct cluster state version for node fault detection (#30810) Change serialization version of doc-value fields. [DOCS] Fixes broken link for native realm [DOCS] Clarified audit.index.client.hosts (#30797) [TEST] Don't expect acks when isolating nodes Add a `format` option to `docvalue_fields`. (#29639) Fixes UpdateSettingsRequestStreamableTests mutate bug Mustes {p0=snapshot.get_repository/10_basic/*} YAML test Revert "Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled" Only allow x-pack metadata if all nodes are ready (#30743) Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled Use original settings on full-cluster restart (#30780) Only ack cluster state updates successfully applied on all nodes (#30672) Expose Lucene's FeatureField. (#30618) Fix a grammatical error in the 'search types' documentation. Remove http pipelining from integration test case (#30788)

Doc-value fields now return a value that is based on the mappings rather than the script implementation by default. This deprecates the special `use_field_mapping` docvalue format which was added in elastic#29639 only to ease the transition to 7.x and it is not necessary anymore in 7.0.

* es/ccr: (55 commits) [DOCS] Fixes typos in security settings Fix GeoShapeQueryBuilder serialization after backport [DOCS] Splits auditing.asciidoc into smaller files Reintroduce mandatory http pipelining support (elastic#30820) Painless: Types Section Clean Up (elastic#30283) Add support for indexed shape routing in geo_shape query (elastic#30760) [test] java tests for archive packaging (elastic#30734) Revert "Make http pipelining support mandatory (elastic#30695)" (elastic#30813) [DOCS] Fix more edit URLs in Stack Overview (elastic#30704) Use correct cluster state version for node fault detection (elastic#30810) Change serialization version of doc-value fields. [DOCS] Fixes broken link for native realm [DOCS] Clarified audit.index.client.hosts (elastic#30797) [TEST] Don't expect acks when isolating nodes Mute CorruptedFileIT in CCR Add a `format` option to `docvalue_fields`. (elastic#29639) Fixes UpdateSettingsRequestStreamableTests mutate bug Mustes {p0=snapshot.get_repository/10_basic/*} YAML test Revert "Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled" Only allow x-pack metadata if all nodes are ready (elastic#30743) ...

In elastic#29639 we added a `format` option to doc-value fields and deprecated usage of doc-value fields without a format so that we could migrate doc-value fields to use the format that comes with the mappings by default. However I missed to fix the machine-learning datafeed extractor.

…1463) In #29639 we added a `format` option to doc-value fields and deprecated usage of doc-value fields without a format so that we could migrate doc-value fields to use the format that comes with the mappings by default. However I missed to fix the machine-learning datafeed extractor.

Doc-value fields now return a value that is based on the mappings rather than the script implementation by default. This deprecates the special `use_field_mapping` docvalue format which was added in #29639 only to ease the transition to 7.x and it is not necessary anymore in 7.0.

Add a format option to docvalue_fields.

b25ccfc

This commit adds the ability to configure how a docvalue field should be formatted, so that it would be possible eg. to return a date field formatted as the number of milliseconds since Epoch. Closes elastic#27740

jpountz added >feature :Search/Search Search-related issues that do not fall into other categories labels Apr 20, 2018

jpountz added 3 commits April 20, 2018 18:12

iter

c62e5bc

iter

3e373e1

iter

9991b80

colings86 reviewed Apr 23, 2018

View reviewed changes

iter

46ba35e

colings86 approved these changes Apr 23, 2018

View reviewed changes

jpountz added 13 commits April 25, 2018 15:48

Merge branch 'master' into feature/doc_value_format

ad7130c

iter

b2d3de8

Make the format an optional constructor arg.

ae28857

Merge branch 'master' into feature/doc_value_format

a177292

Fix docs of the _ignored meta field.

b38cde6

Relates elastic#29658

Post backport of elastic#29658.

c422c36

Merge branch 'master' into feature/doc_value_format

9080853

iter

886bae6

Fix changelog.

88374e1

Merge branch 'master' into feature/doc_value_format

b228a29

iter

677c75d

iter

99e2718

Merge branch 'master' into feature/doc_value_format

a5932de

jpountz added 6 commits May 9, 2018 14:39

Merge branch 'master' into feature/doc_value_format

20c863c

iter

1b74854

Merge branch 'master' into feature/doc_value_format

0098d6c

iter

710a420

iter

d106a26

Merge branch 'master' into feature/doc_value_format

defe800

jpountz added 4 commits May 17, 2018 15:29

Merge branch 'master' into feature/doc_value_format

4294d1d

Merge branch 'master' into feature/doc_value_format

ff2df6a

iter

2e33fb5

iter

99a0249

jpountz added v7.0.0 v6.4.0 labels May 23, 2018

jpountz merged commit a19df4a into elastic:master May 23, 2018

jpountz deleted the feature/doc_value_format branch May 23, 2018 12:39

jpountz added a commit that referenced this pull request May 23, 2018

Change serialization version of doc-value fields.

fe9251e

Relates #29639

jpountz added a commit that referenced this pull request May 23, 2018

Change serialization version of doc-value fields.

405eb7a

Relates #29639

jpountz mentioned this pull request May 24, 2018

Use mappings to format doc-value fields by default. #30831

Merged

jpountz mentioned this pull request Jun 20, 2018

Avoid deprecation warning when running the ML datafeed extractor. #31463

Merged

Mpdreamz mentioned this pull request Sep 21, 2018

[meta] 6.4.0 release elastic/elasticsearch-net#3397

Closed

89 tasks

Mpdreamz mentioned this pull request Oct 22, 2018

[meta] 6.5.0 Release elastic/elasticsearch-net#3457

Closed

russcam mentioned this pull request Nov 19, 2018

Add a format option to docvalue_fields elastic/elasticsearch-net#3490

Closed

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a `format` option to `docvalue_fields`. #29639

Add a `format` option to `docvalue_fields`. #29639

jpountz commented Apr 20, 2018

elasticmachine commented Apr 20, 2018

colings86 left a comment

colings86 Apr 23, 2018

jpountz Apr 23, 2018

colings86 Apr 23, 2018

colings86 Apr 23, 2018

jpountz Apr 23, 2018

colings86 Apr 23, 2018

colings86 Apr 23, 2018

colings86 Apr 23, 2018

jpountz Apr 23, 2018

colings86 Apr 23, 2018

colings86 left a comment

jpountz commented May 9, 2018

Add a format option to docvalue_fields. #29639

Add a format option to docvalue_fields. #29639

Conversation

jpountz commented Apr 20, 2018

elasticmachine commented Apr 20, 2018

colings86 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

colings86 left a comment

Choose a reason for hiding this comment

jpountz commented May 9, 2018

Add a `format` option to `docvalue_fields`. #29639

Add a `format` option to `docvalue_fields`. #29639