Add a normed options on fields #1001

lpouget · 2021-03-29T16:08:07Z

Make fieldnorm indexation optional:

for all types except text: added a NORMED options
for text field:
- if STRING, field has not fieldnorm retained
- if TEXT, field has fieldnorm computed

fulmicoton · 2021-05-19T09:19:38Z

@lpouget Will review next week. Sorry for having taken so much time!

src/schema/flags.rs

src/fieldnorm/mod.rs

src/schema/flags.rs

fulmicoton

See comments inline.

Make fieldnorm indexation optional: * for all types except text => added a NORMED options * for text field ** if STRING, field has not fieldnorm retained ** if TEXT, field has fieldnorm computed

fmassot · 2021-11-30T23:11:15Z

I decided to finish it as I think @lpouget has a baby to take care :)
I thought the issue was easy but... I needed to touch many parts of the code and potentially I missed some stuff.
I think the PR is almost ready, I need to add some specific tests when with fields without field norms.

@fulmicoton @PSeitz When you have the time (it's not urgent at all), you can have a look.

src/indexer/merger.rs

src/indexer/segment_writer.rs

src/schema/bytes_options.rs

src/schema/text_options.rs

fulmicoton · 2021-12-08T02:21:52Z

@lpouget I will take over that PR and merge it. I hope this is ok with you.

* Remove unused sort for segment meta list * Fix segment meta order dependent test

In addition this PR: - removes unnecessary flushes and fsyncs on files. - replace all fsync by fdatasync. The latter triggers a meta sync if a metadata required to read the file has changed. It is therefore sufficient for us. Closes quickwit-oss#1224

Removes the indexed option for facets. Facets are now always indexed. Closes quickwit-oss#1195

) Closes quickwit-oss#831

…1228) This work by introducing a new API method in the Directory trait. The user needs to explicitely call this method. (In particular, once before a commmit) Closes quickwit-oss#1225

- Using Option for fieldnorm readers. - Fixing the ratio computer. max_doc and not num_doc.

codecov-commenter · 2021-12-10T01:52:07Z

Codecov Report

Merging #1001 (dae66e5) into main (c503c6e) will increase coverage by 0.09%.
The diff coverage is 97.96%.

@@            Coverage Diff             @@
##             main    #1001      +/-   ##
==========================================
+ Coverage   94.04%   94.14%   +0.09%     
==========================================
  Files         205      206       +1     
  Lines       34525    34806     +281     
==========================================
+ Hits        32470    32767     +297     
+ Misses       2055     2039      -16

Impacted Files	Coverage Δ
src/directory/directory.rs	`94.11% <ø> (ø)`
src/fastfield/multivalued/mod.rs	`96.96% <ø> (ø)`
src/schema/flags.rs	`56.25% <ø> (ø)`
src/schema/text_options.rs	`95.12% <50.00%> (-4.88%)`	⬇️
src/indexer/index_writer.rs	`97.58% <80.00%> (-0.09%)`	⬇️
src/schema/schema.rs	`97.92% <81.25%> (-0.48%)`	⬇️
src/directory/mmap_directory.rs	`90.81% <90.90%> (+0.32%)`	⬆️
src/indexer/merger.rs	`98.89% <97.05%> (-0.05%)`	⬇️
src/collector/facet_collector.rs	`98.28% <100.00%> (ø)`
src/core/index_meta.rs	`92.89% <100.00%> (ø)`
... and 50 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c503c6e...dae66e5. Read the comment docs.

fulmicoton · 2021-12-10T12:21:43Z

This PR was merged as branch issue/922b

lpouget changed the title ~~Add a NORMED options on field~~ Add a normed options on fields Mar 29, 2021

lpouget mentioned this pull request Apr 1, 2021

Make it possible to index a field without fieldnorms #922

Closed

fulmicoton self-requested a review May 19, 2021 09:19