memory error in get_bmus , np.unravel_index(activation_map.argmin(axis=1),

[passionato]

I am getting this memory error on a very small dataset:

Time for epoch 1: 0.0257 3% [== ]
Time for epoch 2: 0.02197 7% [==== ]
Time for epoch 3: 0.02456 11% [====== ]
Time for epoch 4: 0.02257 15% [======== ]
Time for epoch 5: 0.02368 19% [========== ]
Time for epoch 6: 0.02047 23% [============ ]
Time for epoch 7: 0.02465 26% [============== ]
Time for epoch 8: 0.02163 30% [================ ]
Time for epoch 9: 0.02215 34% [================== ]
Time for epoch 10: 0.02218 38% [==================== ]
Time for epoch 11: 0.02229 42% [====================== ]
Time for epoch 12: 0.02191 46% [======================== ]
Time for epoch 13: 0.02268 50% [========================== ]
Time for epoch 14: 0.02197 53% [=========================== ]
Time for epoch 15: 0.02284 57% [============================= ]
Time for epoch 16: 0.0237 61% [=============================== ]
Time for epoch 17: 0.0237 65% [================================= ]
Time for epoch 18: 0.01965 69% [=================================== ]
Time for epoch 19: 0.02176 73% [===================================== ]
Time for epoch 20: 0.02183 76% [======================================= ]
Time for epoch 21: 0.02405 80% [========================================= ]
Time for epoch 22: 0.02133 84% [=========================================== ]
Time for epoch 23: 0.02266 88% [============================================= ]
Time for epoch 24: 0.0214 92% [=============================================== ]
Time for epoch 25: 0.02141 96% [================================================= ]
Time for epoch 26: 0.0215 100% [===================================================]

/some/other/place/Python/2.7.13/lib/python2.7/site-packages/matplotlib-2.0.0-py2.7-linux-x86_64.egg/matplotlib/font_manager.py:1297: UserWarning: findfont: Font family [u'sans-serif'] not found. Falling back to DejaVu Sans
(prop.get_family(), self.defaultFamily[fontext]))
Traceback (most recent call last):
File "python-module/Project.py", line 89, in
MyDocs.calc(Debug)
File "/some/place/python-module/Doc.py", line 358, in calculate_map
my_bmus = som.get_bmus(som_state)
File "/some/other/place/Python/2.7.13/lib/python2.7/site-packages/somoclu-1.7.3-py2.7-linux-x86_64.egg/somoclu/train.py", line 582, in get_bmus
Y, X = np.unravel_index(activation_map.argmin(axis=1),
MemoryError

However my machine is powerful enough to handle this:

myuser@ubuntu: map]$ lscpu | egrep 'Thread|Core|Socket|^CPU\('
CPU(s):                64
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             4

[peterwittek]

It would help a lot if you gave an mcve. Thanks.

[kurceliana]

I got the same error message when running somoclu on a medium sized data set with 90GB RAM.
Upon running a memory profiler, I found that with me the error wasn't in the get_bmus() function, but rather with the get_surface_state(), which is called right before to generate the activation map needed for the get_bmus() function.
So the memory error occurs when cdist from scipy.spatial.distance is called:

am = cdist(self.codebook.reshape((self.codebook.shape[0] *
                                          self.codebook.shape[1],
                                          self.codebook.shape[2])),
                   d,  'euclidean').T

Since I couldn't find a solution for that problem except for using smaller data sets/ map dimensions I build a little workaround to get it running on bigger data sets for now (even though the runtime is much higher now, it hasn't killed my memory yet):

The above mentioned code has to be commented out and replaced by:

codebookReshape = self.codebook.reshape(self.codebook.shape[0] * self.codebook.shape[1], self.codebook.shape[2])

teile = np.array_split(d, 200, axis=0)

sum = np.empty((0, (self._n_columns * self._n_rows)), dtype="float64")

for teil in teile:
    sum = np.concatenate((sum, (cdist((teil), codebookReshape, 'euclidean'))), axis=0)
am = sum

Replace the 200 with a higher number to increase runtime (but carefully, if it is too high the memory error might occur again).

[peterwittek]

This is a good solution. Would you mind creating a PR to replace the single call to cdist? Thanks.