Incremental Memory

[XSeaty]

I add logger.info(memmory_use) in the iterate function which is called in HGRU4REC.fit and get the log below. Is this normal?

other information

I run this code with CPU.
I run this code on my own dataset (about 28 million interaction)
I notice this because the process is out of memory.

training log of the first epoch

2018-09-29 19:57:07,118: main: INFO: Training started
2018-09-29 19:57:38,527: hgru4rec: INFO: Epoch Begin!
2018-09-29 19:57:38,530: hgru4rec: INFO: 227.51(GB) memory left.
2018-09-29 19:57:38,600: hgru4rec: INFO: 227.51(GB) memory left.
..........
2018-09-29 20:01:47,200: hgru4rec: INFO: 209.16(GB) memory left.
..........
2018-09-29 20:05:31,439: hgru4rec: INFO: 191.81(GB) memory left.
..........
2018-09-29 20:08:45,621: hgru4rec: INFO: 177.09(GB) memory left.
..........
2018-09-29 20:11:37,108: hgru4rec: INFO: 164.1(GB) memory left.
..........
2018-09-29 20:14:05,733: hgru4rec: INFO: 152.86(GB) memory left.
..........
2018-09-29 20:16:07,679: hgru4rec: INFO: 143.68(GB) memory left.
..........
2018-09-29 20:17:50,016: hgru4rec: INFO: 136.02(GB) memory left.
..........
2018-09-29 20:19:15,243: hgru4rec: INFO: 129.7(GB) memory left.
..........
2018-09-29 20:20:31,730: hgru4rec: INFO: 124.03(GB) memory left.
..........
2018-09-29 20:22:54,691: hgru4rec: INFO: 114.76(GB) memory left.
..........
2018-09-29 20:26:05,653: hgru4rec: INFO: 106.84(GB) memory left.
..........
2018-09-29 20:28:25,798: hgru4rec: INFO: 96.48(GB) memory left.
..........
2018-09-29 20:31:37,315: hgru4rec: INFO: 82.28(GB) memory left.
....
2018-09-29 20:32:39,487: hgru4rec: INFO: Epoch 0 - train cost: 0.8193

[mquad]

Hi, sorry for being quite late on this...
What's the number of distinct items in your dataset?
Input to hidden and hidden to output matrices are the ones that use the most memory, so I'd start checking those.

[XSeaty]

@mquad In order to avoid out of memory, I downsample the dataset. And now the data has 27854 distinct number of items, 22068 distinct number of users and 4,867,558 number of samples.

[mquad]

Nice. Extremely large item sets will likely benefit of an additional embedding layer in input to reduce the memory consumption. Also, if you like, check out the main GRU4Rec repo (https://github.com/hidasib/GRU4Rec) for some updates on speed and mem. consumption that will likely apply to this code base as well.