Missing data in indexed model on large data set; yields much lower counts than unindexed model on the same data with the same parameters!

[proycon]

As reported by Pavel Vondřička, something fishy is going on in the computation of an indexed model on a large dataset (8.5GB compressed):

Indexed:

$ colibri-patternmodeller -l 1 -t 1 -f gigacorpus.colibri.dat                                                        
Loading corpus data...
Training model on  gigacorpus.colibri.dat
Training patternmodel, occurrence threshold: 1
Counting *all* n-grams (occurrence threshold=1)
 Found 2562104 ngrams... computing total word types prior to pruning...2562104...pruned 0...total kept: 2562104
Sorting all indices...

Unindexed (these are the correct):

$ colibri-patternmodeller -u -l 1 -t 1 -f gigacorpus.colibri.dat
Training unindexed model on  gigacorpus.colibri.dat
Training patternmodel, occurrence threshold: 1
Counting *all* n-grams (occurrence threshold=1)
 Found 11459477 ngrams... computing total word types prior to pruning...11459477...pruned 0...total kept: 11459477

The encoded corpus file has been verified to be fine (i.e. it decodes properly):

yes, I tried decoding the corpus back and it had a different size, but there was the whole contents - it seems that just some (white)spaces got lost, which is understandable. Anyway, the corpus wasn’t clipped.

I did some tests and the problem does NOT reproduce on a small text (counts are equal there as expected), which also explains why it isn't caught by our automated tests. So the cause is not yet clear and further debugging is needed.

[proycon]

I enabled debug mode on the test data; the indexed model finishes prematurely (but without error) after processing line 18995401:

   (container ready)
        5 ngrams in line
                Adding @18995401:0 n=1 category=1
                Adding @18995401:1 n=1 category=1
                Adding @18995401:2 n=1 category=1
                Adding @18995401:3 n=1 category=1
                Adding @18995401:4 n=1 category=1
 Found 2562104 ngrams... computing total word types prior to pruning...2562104...pruned 0...total kept: 2562104
Sorting all indices...
Writing model to tst
Generating desired views..

The unindexed model continues to at least line 122777961 (and then the harddisk is full ;) )

[proycon]

The problem is the IndexedCorpus model loaded as reverse index, it seems to be clipped at exactly that sentence count, investigating further...

[proycon]

Found the problem, unsigned int is too small to store the corpus size.

[proycon]

Fixed and released in v2.4.10