Consider confidence values in 'ensure_consistent_bilou_tagging'

[tabergma]

Proposed changes:
The method ensure_consistent_bilou_tagging now also considers the confidence values of the predicted tags
when updating the BIlOU tags.

closes #6039

Status (please check what you already did):

• added some tests for the functionality
• updated the documentation
• updated the changelog (please check changelog for instructions)
• reformat files using black (please check Readme for instructions)

[Ghostvv]

not sure how troublesome to implement it, but is it possible to output original tags during rasa test as well?

[tabergma]

You mean the tags before they got updated? mmhh... not that easy, as the tags are converted into proper entities and I am not sure how you would add the "old" tags in there.

[Ghostvv]

Let's create an issue for it to discuss it. So that this PR is not blocked

[tabergma]

#6647

[Ghostvv]

Could you please add the description of this substitution process to docs, please?

[tabergma]

Where should it go? We do not have a section for it as far as i know.

[Ghostvv]

hm, it seems that we don't have a place where we describe bilou tagging scheme. I think we should add it some where

let's add docs as part of this PR

[tabergma]

@erohmensing tagging you as a review as I added a section about BILOU tagging to the docs. As the information about NLU Training Data was already included in Training Data Format I removed it and kept just the entity related stuff. Let me know if that does not work and the section should be moved somewhere else.

[Ghostvv]

the part about BILOU tags looks good.
I wouldn't remove nlu-training-data.mdx as part of this PR. To avoid confusion, let's do it in a separate PR

[tabergma]

followed the suggestion of @Ghostvv and added the page about nlu training data again and just added the bilou section to that page

[erohmensing]

👍

[tabergma]

Applied all review comments, thanks for the comments!

@erohmensing or @Ghostvv can someone please approve so that we can merge this PR? Thanks.

[tabergma]

oh no, just realised that we will not be able to merge as a lot of tests will all fail due to the convert issue