Change tokenize cache to prevent memory leak caused by inifinite cache#19

Merged

azu merged 1 commit intotextlint-ja:masterfrom k-tahiro:use-tokenize-function

Dec 10, 2021

+2-6

Commits on Dec 9, 2021

Use tokenize function instead of getTokenizer to use cache and prevent memory leak caused by inifinite cache.
k-tahiro
committed