Abbreviations: remove dot if word is not an abbreviation

[Terrtia]

• Add abbreviation set
• remove . if word is not an abbreviation

[pierotofy]

Thanks for the PR !

I'm not sure I'd want to start introducing such specific tokenization rules and additional pickle files. One of the goals of the projects is to keep the logic simple (and fast). Texts with more specific tokenization rules could be handled outside of lexilang (by performing tokenization, then calling lexilang individually on the tokens)?

[Terrtia]

How should we handle abbreviations ?

Punctuation needs to be removed to check if a word is in a dictionary. However, lexilang utilizes punctuation to distinguish Bulgarian and Hungarian languages, which is why I didn't removed dot characters in my previous PR.

I think It's interesting to detect languages through abbreviations. I use lexilang to identify languages in chat messages, which frequently include language-specific abbreviations.

[pierotofy]

One possible approach could be client applications could replace abbreviations with their full word equivalent before sending the text to lexilang for detection?

[Terrtia]

I've removed all abbreviations and dot characters."

[pierotofy]

Looks good, thanks!!

[Terrtia]

Thanks !
Could you please create a new release on PyPI ?