Skip to content

A hierarchical character-word neural network for language identification

License

Notifications You must be signed in to change notification settings

ajaech/twitter_langid

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 

Repository files navigation

twitter_langid

For more information please see our paper Hierarchical Character-Word Models for Language Identification.

Getting Started

To train a model run the command

python langid.py path/to/outdir

For some simple visualization of a trained model run

python langid.py --mode=debug path/to/outdir

To evaluate a trained model run

python langid.py --mode=eval path/to/outdir

Data

The data directory holds an example input file created from Wikipedia sentence fragments. The file is saved in tab separated format. We partition the data according to the last digit of the id number in the data file. Separate lines are used for training, validation, and testing.

About

A hierarchical character-word neural network for language identification

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages