HF Tokenizer: get charspans

[andreabrduque]

Description

This PR extends the hugging face Java/Rust interface. I added a method to return the original char spans for each token result, together with the encoding object.

This function is useful in Named Entity Recognition tasks (NER), specifically to reconstruct entities together, since the tokens retrieved from the tokenization lose whitespace information.

[zachgk]

This looks great @andreabrduque!

It looks like you are getting caught on our checkstyle:

Error: eckstyle] [ERROR] /home/runner/work/djl/djl/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/CharSpan.java:17: Missing a Javadoc comment. [MissingJavadocType]

https://github.com/deepjavalibrary/djl/runs/6082365027?check_suite_focus=true#step:11:3779

[codecov-commenter]

Codecov Report

Merging #1584 (9d035f5) into master (bb5073f) will decrease coverage by 1.22%.
The diff coverage is 61.59%.

@@             Coverage Diff              @@
##             master    #1584      +/-   ##
============================================
- Coverage     72.08%   70.86%   -1.23%     
- Complexity     5126     5397     +271     
============================================
  Files           473      504      +31     
  Lines         21970    23606    +1636     
  Branches       2351     2567     +216     
============================================
+ Hits          15838    16729     +891     
- Misses         4925     5595     +670     
- Partials       1207     1282      +75     

Impacted Files	Coverage Δ
api/src/main/java/ai/djl/modality/cv/Image.java	69.23% <ø> (-4.11%)	⬇️
...i/djl/modality/cv/translator/BigGANTranslator.java	21.42% <ø> (-5.24%)	⬇️
...odality/cv/translator/BigGANTranslatorFactory.java	33.33% <0.00%> (+8.33%)	⬆️
...nslator/InstanceSegmentationTranslatorFactory.java	14.28% <0.00%> (-3.90%)	⬇️
.../modality/cv/translator/YoloTranslatorFactory.java	8.33% <0.00%> (-1.67%)	⬇️
...i/djl/modality/cv/translator/YoloV5Translator.java	5.69% <0.00%> (ø)
...odality/cv/translator/YoloV5TranslatorFactory.java	8.33% <0.00%> (-1.67%)	⬇️
...pi/src/main/java/ai/djl/ndarray/BytesSupplier.java	54.54% <0.00%> (-12.13%)	⬇️
...ain/java/ai/djl/ndarray/index/dim/NDIndexPick.java	100.00% <ø> (ø)
api/src/main/java/ai/djl/nn/Blocks.java	75.00% <0.00%> (-25.00%)	⬇️
... and 230 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06997ce...9d035f5. Read the comment docs.