Dataset preparation code

[shashwath94]

Hey @hadyelsahar ,
Great work/repo! Thanks for that.
I wanted to ask you whether the code to create the *.ids files (data preprocessing) is included in the repo? I'm having a hard time finding it.

[hadyelsahar]

Thanks, no it is not in the repo. The reason is because the code was hard to follow and tweak.
I wanted to rewrite it again using sentence-piece, FastText and Spacy so it becomes clean and simple to reapply on any dataset but I got involved in some other stuff.

if u want to just generate the ids (the vocab file) this should be 3-4lines using sentence piece