Parallelize nltk CoreNLP parser in simple way

[ajratner]

Emphasis on simple- this is not going to be an optimal preprocessing setup either way, we just want to make it a bit better through simple means that don't require any additional installs, configs, etc.

[ajratner]

See branch multicore

[chrismre]

Why not use python's multiprocessing or something? Also, we could potentially put in some small hadoop/spark connector (if people wanted to hit AWS?) @netj @raphaelhoffmann @alldefector

[alldefector]

With the current NN-SR model and the notebook docs, the java process seems to take 200MB - 900MB of memory during parsing (as opposed to 4GB with an old model). So spawning #core processes should be fine for a typical laptop.

A few months ago we also had a simple HTTP service wrapper for the parser:
https://github.com/HazyResearch/bazaar/blob/master/parser/src/main/scala/com/clearcut/nlp/Server.scala

[alldefector]

http://www.nltk.org/_modules/nltk/parse/stanford.html

Looks like the nltk wrapper is actually using some old model that probably has a throughput of 1 sent / sec, as opposed to the SR / NN models that are 100 sents / sec...

The StanfordNeuralDependencyParser class ought to address that.
http://nlp.stanford.edu/software/nndep.shtml

[ajratner]

Yeah I actually did set up a simple queue based multiprocessing version on
the 'multicore' branch (see comment on the issue), but think there's still
a bug; either way someone could work from that

The switch to SR / NN was what made the huge difference in our normal
pipeline when we did that, so this is probably the easiest gain to get...
On Sun, Feb 28, 2016 at 9:31 AM alldefector notifications@github.com
wrote:

http://www.nltk.org/_modules/nltk/parse/stanford.html

Looks like the nltk wrapper is actually using some old model that probably
has a throughput of 1 sent / sec, as opposed to the SR / NN models that are
100 sents / sec...

—
Reply to this email directly or view it on GitHub
#2 (comment).

[chrismre]

Let's definitely make that change!

On Sun, Feb 28, 2016 at 9:36 AM Alex Ratner notifications@github.com
wrote:

Yeah I actually did set up a simple queue based multiprocessing version on
the 'multicore' branch (see comment on the issue), but think there's still
a bug; either way someone could work from that

The switch to SR / NN was what made the huge difference in our normal
pipeline when we did that, so this is probably the easiest gain to get...
On Sun, Feb 28, 2016 at 9:31 AM alldefector notifications@github.com
wrote:

http://www.nltk.org/_modules/nltk/parse/stanford.html

Looks like the nltk wrapper is actually using some old model that
probably
has a throughput of 1 sent / sec, as opposed to the SR / NN models that
are
100 sents / sec...

—
Reply to this email directly or view it on GitHub
<#2 (comment)
.

—
Reply to this email directly or view it on GitHub
#2 (comment).

[ajratner]

Subset of #228