Low metric values during both training and testing #9

soudabeh1000 · 2021-01-18T03:00:37Z

Hi,
I tried to test the model based on the pretrained model, but the value for the metrics is very low (as follows). I am not sure if I missed sth.

Loading existing Word2Vec model (Glove.6B.300d)
model-eur-akde-29.pth loaded
model-EUR-CNN-40.pth loaded
k_trn: 3954
k_tst: 3946
Reranking: 200 Iteration: 120/121 (99.2%) Loss: 0.00000 0.138070.13806557655334473
Tst Prec@1,3,5: [0.31358344113842174, 0.27684346701164514, 0.22675291073738868] Tst NDCG@1,3,5: [0.31358344113842174, 0.3239735490417916, 0.3486589907540996]

The same issue happens when I try to train the model (using EUR_Cap_grad.py). The only change I made was to change the batch size ("tr_batch_size": 16,) (since I am using one GPU and with batch size more than this I will get cuda-out-of-memory error). Any help would be appreciated.

Loading existing Word2Vec model (Glove.6B.300d)
model-EUR-CNN-40.pth loaded
0.001
Iteration: 724/725 (99.9%) Loss: 0.17726 0.401590.001
Iteration: 724/725 (99.9%) Loss: 0.11516 0.402710.001
Iteration: 724/725 (99.9%) Loss: 0.08906 0.399420.001
Iteration: 724/725 (99.9%) Loss: 0.07137 0.396240.001
Iteration: 724/725 (99.9%) Loss: 0.06250 0.402290.001
Iteration: 724/725 (99.9%) Loss: 0.04446 0.407690.001
Iteration: 724/725 (99.9%) Loss: 0.02744 0.407160.001
Iteration: 724/725 (99.9%) Loss: 0.02550 0.448430.001
Iteration: 724/725 (99.9%) Loss: 0.01622 0.409140.001
Iteration: 724/725 (99.9%) Loss: 0.01856 0.406260.001
Iteration: 724/725 (99.9%) Loss: 0.02143 0.407910.001
Iteration: 724/725 (99.9%) Loss: 0.01301 0.415560.001
Iteration: 724/725 (99.9%) Loss: 0.01624 0.411990.001
Iteration: 724/725 (99.9%) Loss: 0.01391 0.420290.001
Iteration: 724/725 (99.9%) Loss: 0.02283 0.401170.001
Iteration: 724/725 (99.9%) Loss: 0.01784 0.404300.001
Iteration: 724/725 (99.9%) Loss: 0.01874 0.396960.001
Iteration: 724/725 (99.9%) Loss: 0.01363 0.407790.001
Iteration: 724/725 (99.9%) Loss: 0.01652 0.398930.001
Iteration: 724/725 (99.9%) Loss: 0.01428 0.405640.00095
Iteration: 724/725 (99.9%) Loss: 0.01355 0.40836k_trn: 3954
k_tst: 3946
Epoch: 21 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.043910.04391336441040039
Tst Prec@1,3,5: [0.39353169469598964, 0.27848210435532844, 0.21930142302716957] Tst NDCG@1,3,5: [0.39353169469598964, 0.3588545342449854, 0.3867303767271317]
0.0009025
Iteration: 724/725 (99.9%) Loss: 0.01517 0.40230k_trn: 3954
k_tst: 3946
Epoch: 22 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.041040.04104208946228027
Tst Prec@1,3,5: [0.39922380336351876, 0.28089693833549245, 0.22033635187581077] Tst NDCG@1,3,5: [0.39922380336351876, 0.3645233558394241, 0.3909089661763729]
0.000857375
Iteration: 724/725 (99.9%) Loss: 0.01023 0.41681k_trn: 3954
k_tst: 3946
Epoch: 23 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.040320.04032182693481445
Tst Prec@1,3,5: [0.4028460543337646, 0.2809831824062126, 0.21862871927555227] Tst NDCG@1,3,5: [0.4028460543337646, 0.3632745081571306, 0.3872756265587336]
0.0008145062499999999
Iteration: 724/725 (99.9%) Loss: 0.01041 0.39818k_trn: 3954
k_tst: 3946
Epoch: 24 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.039640.039637088775634766
Tst Prec@1,3,5: [0.3994825355756792, 0.2830530401034952, 0.22080206985769987] Tst NDCG@1,3,5: [0.3994825355756792, 0.3660985894692969, 0.3910490727145218]
0.0007737809374999998
Iteration: 724/725 (99.9%) Loss: 0.00935 0.41835k_trn: 3954
k_tst: 3946
Epoch: 25 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.041700.04170083999633789
Tst Prec@1,3,5: [0.39741267787839585, 0.2796895213454108, 0.21940491591203337] Tst NDCG@1,3,5: [0.39741267787839585, 0.36214576487819594, 0.38832568567943937]
0.0007350918906249997
Iteration: 724/725 (99.9%) Loss: 0.00873 0.40648k_trn: 3954
k_tst: 3946
Epoch: 26 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.044140.044144630432128906
Tst Prec@1,3,5: [0.4018111254851229, 0.2808969383354923, 0.21966364812419392] Tst NDCG@1,3,5: [0.4018111254851229, 0.3636910100293636, 0.38958924133553113]
0.0006983372960937497
Iteration: 724/725 (99.9%) Loss: 0.00772 0.41684k_trn: 3954
k_tst: 3946
Epoch: 27 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.042820.04282426834106445
Tst Prec@1,3,5: [0.39534282018111255, 0.27925830099180965, 0.21904269081500896] Tst NDCG@1,3,5: [0.39534282018111255, 0.3597501180621696, 0.38599941467426513]
0.0006634204312890621
Iteration: 724/725 (99.9%) Loss: 0.00672 0.41296k_trn: 3954
k_tst: 3946
Epoch: 28 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.044380.04437899589538574
Tst Prec@1,3,5: [0.4, 0.28262181974989453, 0.2205433376455394] Tst NDCG@1,3,5: [0.4, 0.36555716477262373, 0.3910988410631433]
0.000630249409724609
Iteration: 724/725 (99.9%) Loss: 0.00784 0.40683k_trn: 3954
k_tst: 3946
Epoch: 29 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.041670.041669368743896484
Tst Prec@1,3,5: [0.40439844760672705, 0.28210435532557404, 0.22018111254851438] Tst NDCG@1,3,5: [0.40439844760672705, 0.3663117906146262, 0.3916956773619885]
0.0005987369392383785
Iteration: 724/725 (99.9%) Loss: 0.00492 0.41107

andyweizhao · 2021-01-19T02:34:07Z

Hello @soudabeh1000,

Yes, batch size is very sensitive to the final results generated by CapsuleNet (also by XML_CNN if I remember correctly). Also, it would better to modify batch size and learning rate simultaneously. Taken together, you could run XLM_CNN with a batch size of 16 to find a good learning rate in the first place. Next, you run CapsuleNet with this learning rate. Hope this can help.

soudabeh1000 · 2021-01-19T02:42:11Z

Thanks a lot for your reply. But what about testing the pretrained model? For testing, I used your pretrained model and the batch size for testing part is already 16 in your implementation. But for that also I got low values for the metrics: (is the pretrained model uploaded correctly? or do I miss sth? )
Thanks again

Loading existing Word2Vec model (Glove.6B.300d)
model-eur-akde-29.pth loaded
model-EUR-CNN-40.pth loaded
k_trn: 3954
k_tst: 3946
Reranking: 200 Iteration: 120/121 (99.2%) Loss: 0.00000 0.138070.13806557655334473
Tst Prec@1,3,5: [0.31358344113842174, 0.27684346701164514, 0.22675291073738868] Tst NDCG@1,3,5: [0.31358344113842174, 0.3239735490417916, 0.3486589907540996]

andyweizhao · 2021-01-20T21:14:37Z

Hello @soudabeh1000,

I just ran the EUR_eval.py with a batch size of 16 on a single GPU (GTX-1070). See the results below:

CUDA_VISIBLE_DEVICES=0 python3 EUR_eval.py
{
"dataset": "eurlex_raw_text.p",
"vocab_size": 30001,
"vec_size": 300,
"sequence_length": 500,
"is_AKDE": true,
"num_epochs": 30,
"ts_batch_size": 16,
"learning_rate": 0.001,
"start_from": "save",
"num_compressed_capsule": 128,
"dim_capsule": 16,
"re_ranking": 200
}
Loading existing Word2Vec model (Glove.6B.300d)
model-eur-akde-29.pth loaded
model-EUR-CNN-40.pth loaded
k_trn: 3954
k_tst: 3946
Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.039020.03901934623718262
Tst Prec@1,3,5: [0.8023285899094438, 0.6476929711082351, 0.5258473479948228] Tst NDCG@1,3,5: [0.8023285899094438, 0.7158824620063444, 0.7061671319835849]

Your results surprise me... Did you modify the code?

soudabeh1000 · 2021-01-27T23:21:48Z

Hello
Thanks for running the code again. I am running it on Google colab (using its GPU). The only two changes Imade to the code are as follows (since I got errors for them):

I changed line 6 of Data_helper to: (It was for fixing this error : "ModuleNotFoundError: No module named 'cPickle'"; I could not install cPickle).

import _pickle as pickle

I added these two lines of code to data_helper: (it was for fixiing this error: "Resource stopwords not found.
Please use the NLTK Downloader to obtain the resource:").

import nltk
nltk.download('stopwords')

I downloaded the pretrained models and the data sets several times and ran the code several times, but still I get the same results as before. So, I would appreciate it if you could check it with the pretrained models and dataset that you have uploaded in your google drive. I really appreciate your help.

[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data] Package stopwords is already up-to-date!
{
"dataset": "eurlex_raw_text.p",
"vocab_size": 30001,
"vec_size": 300,
"sequence_length": 500,
"is_AKDE": true,
"num_epochs": 30,
"ts_batch_size": 16,
"learning_rate": 0.001,
"start_from": "save",
"num_compressed_capsule": 128,
"dim_capsule": 16,
"re_ranking": 200
}
Loading existing Word2Vec model (Glove.6B.300d)
model-eur-akde-29.pth loaded
model-EUR-CNN-40.pth loaded
k_trn: 3954
k_tst: 3946
Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.044000.044003963470458984
Tst Prec@1,3,5: [0.30038809831824065, 0.2718413109098778, 0.22395860284605684] Tst NDCG@1,3,5: [0.30038809831824065, 0.31743066683443355, 0.3451319008565912]

zhangxin9988 · 2021-07-15T03:02:27Z

Now the google drive cant be reached,so the dataset cant be downloaded

andyweizhao · 2021-07-15T10:18:13Z

Hello, The below GD link in README still works for me. Could you have a look again? https://drive.google.com/drive/folders/1gPYAMyYo4YLrmx_Egc9wjCqzWx15D7U8

…

On Thu, 15 Jul 2021 at 05:02, 张鑫 ***@***.***> wrote: Now the google drive cant be reached,so the dataset cant be downloaded — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYRSAYWZOTSMJYQPGMS7O3TXZFU3ANCNFSM4WGUKIQQ> .

zhangxin9988 · 2021-07-15T10:25:07Z

Hello, The below GD link in README still works for me. Could you have a look again? https://drive.google.com/drive/folders/1gPYAMyYo4YLrmx_Egc9wjCqzWx15D7U8
…
On Thu, 15 Jul 2021 at 05:02, 张鑫 @.***> wrote: Now the google drive cant be reached,so the dataset cant be downloaded — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYRSAYWZOTSMJYQPGMS7O3TXZFU3ANCNFSM4WGUKIQQ .

Thanks for your reply, the download link works now ,

luoshunchong · 2021-08-30T11:43:18Z

Hi,
I tried to test the model based on the pretrained model, but the value for the metrics is very low (as follows). I am not sure if I missed sth.

Loading existing Word2Vec model (Glove.6B.300d)
model-eur-akde-29.pth loaded
model-EUR-CNN-40.pth loaded
k_trn: 3954
k_tst: 3946
Reranking: 200 Iteration: 120/121 (99.2%) Loss: 0.00000 0.138070.13806557655334473
Tst Prec@1,3,5: [0.31358344113842174, 0.27684346701164514, 0.22675291073738868] Tst NDCG@1,3,5: [0.31358344113842174, 0.3239735490417916, 0.3486589907540996]

The same issue happens when I try to train the model (using EUR_Cap_grad.py). The only change I made was to change the batch size ("tr_batch_size": 16,) (since I am using one GPU and with batch size more than this I will get cuda-out-of-memory error). Any help would be appreciated.

Loading existing Word2Vec model (Glove.6B.300d)
model-EUR-CNN-40.pth loaded
0.001
Iteration: 724/725 (99.9%) Loss: 0.17726 0.401590.001
Iteration: 724/725 (99.9%) Loss: 0.11516 0.402710.001
Iteration: 724/725 (99.9%) Loss: 0.08906 0.399420.001
Iteration: 724/725 (99.9%) Loss: 0.07137 0.396240.001
Iteration: 724/725 (99.9%) Loss: 0.06250 0.402290.001
Iteration: 724/725 (99.9%) Loss: 0.04446 0.407690.001
Iteration: 724/725 (99.9%) Loss: 0.02744 0.407160.001
Iteration: 724/725 (99.9%) Loss: 0.02550 0.448430.001
Iteration: 724/725 (99.9%) Loss: 0.01622 0.409140.001
Iteration: 724/725 (99.9%) Loss: 0.01856 0.406260.001
Iteration: 724/725 (99.9%) Loss: 0.02143 0.407910.001
Iteration: 724/725 (99.9%) Loss: 0.01301 0.415560.001
Iteration: 724/725 (99.9%) Loss: 0.01624 0.411990.001
Iteration: 724/725 (99.9%) Loss: 0.01391 0.420290.001
Iteration: 724/725 (99.9%) Loss: 0.02283 0.401170.001
Iteration: 724/725 (99.9%) Loss: 0.01784 0.404300.001
Iteration: 724/725 (99.9%) Loss: 0.01874 0.396960.001
Iteration: 724/725 (99.9%) Loss: 0.01363 0.407790.001
Iteration: 724/725 (99.9%) Loss: 0.01652 0.398930.001
Iteration: 724/725 (99.9%) Loss: 0.01428 0.405640.00095
Iteration: 724/725 (99.9%) Loss: 0.01355 0.40836k_trn: 3954
k_tst: 3946
Epoch: 21 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.043910.04391336441040039
Tst Prec@1,3,5: [0.39353169469598964, 0.27848210435532844, 0.21930142302716957] Tst NDCG@1,3,5: [0.39353169469598964, 0.3588545342449854, 0.3867303767271317]
0.0009025
Iteration: 724/725 (99.9%) Loss: 0.01517 0.40230k_trn: 3954
k_tst: 3946
Epoch: 22 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.041040.04104208946228027
Tst Prec@1,3,5: [0.39922380336351876, 0.28089693833549245, 0.22033635187581077] Tst NDCG@1,3,5: [0.39922380336351876, 0.3645233558394241, 0.3909089661763729]
0.000857375
Iteration: 724/725 (99.9%) Loss: 0.01023 0.41681k_trn: 3954
k_tst: 3946
Epoch: 23 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.040320.04032182693481445
Tst Prec@1,3,5: [0.4028460543337646, 0.2809831824062126, 0.21862871927555227] Tst NDCG@1,3,5: [0.4028460543337646, 0.3632745081571306, 0.3872756265587336]
0.0008145062499999999
Iteration: 724/725 (99.9%) Loss: 0.01041 0.39818k_trn: 3954
k_tst: 3946
Epoch: 24 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.039640.039637088775634766
Tst Prec@1,3,5: [0.3994825355756792, 0.2830530401034952, 0.22080206985769987] Tst NDCG@1,3,5: [0.3994825355756792, 0.3660985894692969, 0.3910490727145218]
0.0007737809374999998
Iteration: 724/725 (99.9%) Loss: 0.00935 0.41835k_trn: 3954
k_tst: 3946
Epoch: 25 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.041700.04170083999633789
Tst Prec@1,3,5: [0.39741267787839585, 0.2796895213454108, 0.21940491591203337] Tst NDCG@1,3,5: [0.39741267787839585, 0.36214576487819594, 0.38832568567943937]
0.0007350918906249997
Iteration: 724/725 (99.9%) Loss: 0.00873 0.40648k_trn: 3954
k_tst: 3946
Epoch: 26 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.044140.044144630432128906
Tst Prec@1,3,5: [0.4018111254851229, 0.2808969383354923, 0.21966364812419392] Tst NDCG@1,3,5: [0.4018111254851229, 0.3636910100293636, 0.38958924133553113]
0.0006983372960937497
Iteration: 724/725 (99.9%) Loss: 0.00772 0.41684k_trn: 3954
k_tst: 3946
Epoch: 27 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.042820.04282426834106445
Tst Prec@1,3,5: [0.39534282018111255, 0.27925830099180965, 0.21904269081500896] Tst NDCG@1,3,5: [0.39534282018111255, 0.3597501180621696, 0.38599941467426513]
0.0006634204312890621
Iteration: 724/725 (99.9%) Loss: 0.00672 0.41296k_trn: 3954
k_tst: 3946
Epoch: 28 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.044380.04437899589538574
Tst Prec@1,3,5: [0.4, 0.28262181974989453, 0.2205433376455394] Tst NDCG@1,3,5: [0.4, 0.36555716477262373, 0.3910988410631433]
0.000630249409724609
Iteration: 724/725 (99.9%) Loss: 0.00784 0.40683k_trn: 3954
k_tst: 3946
Epoch: 29 Reranking: 200 Iteration: 241/242 (99.6%) Loss: 0.00000 0.041670.041669368743896484
Tst Prec@1,3,5: [0.40439844760672705, 0.28210435532557404, 0.22018111254851438] Tst NDCG@1,3,5: [0.40439844760672705, 0.3663117906146262, 0.3916956773619885]
0.0005987369392383785
Iteration: 724/725 (99.9%) Loss: 0.00492 0.41107

Hello! I have also encountered the same problem. Will you solve this problem? Can you tell me how you solve it?

andyweizhao · 2021-08-30T13:08:08Z

Sadly, I cannot reproduce the issue. Did you change the code and hyperparameters?

luoshunchong · 2021-08-30T13:24:37Z

Sadly, I cannot reproduce the issue. Did you change the code and hyperparameters?

Think you for your reply! I only changed two places in data_helpers.py. The following is the two changes:
1、"import pickle as pickle" replace "import cPickle as pickle",because I can not dowload the package of cPickle.
2、added these two lines of code to data_helper:
"import nltk
nltk.download('stopwords')
"
then,I run the "EUR_eval.py" in Gogle Colab(used its GPU). But I can't get the same result as you. I need your help. Thank you very much.

soudabeh1000 · 2021-08-30T14:46:44Z

When I ran it in Google Colab (using GPU) this problem occurred. However, when I ran the same code in AWS, I got the same metrics values reported. I do not know why Google Colab gives low values.

andyweizhao · 2021-08-30T15:07:39Z

I just ran EUR_eval.py on two different GPUs (Tesla P40 and GTX-1070) and received the same results as reported in the paper. The difference of GPUs appear to have little impact on the results. Sadly, I have little experience on Colab.

luoshunchong · 2021-08-31T02:56:15Z

When I ran it in Google Colab (using GPU) this problem occurred. However, when I ran the same code in AWS, I got the same metrics values reported. I do not know why Google Colab gives low values.
Sincerely thank you for your reply! I will try it again.

luoshunchong · 2021-08-31T02:56:48Z

I just ran EUR_eval.py on two different GPUs (Tesla P40 and GTX-1070) and received the same results as reported in the paper. The difference of GPUs appear to have little impact on the results. Sadly, I have little experience on Colab.

Sincerely thank you for your reply! I will try it again.

luoshunchong · 2021-08-31T08:38:05Z

When I ran it in Google Colab (using GPU) this problem occurred. However, when I ran the same code in AWS, I got the same metrics values reported. I do not know why Google Colab gives low values.

Hi! I am very sorry to bother you again. I try to ran EUR_eval.py on GTX-1080ti or AWS again,just change "import _pickle as pickle" replace "import cPickle as pickle",because I can not install the package of cPickle in python3.But this problem still occurred. I think it should be what I am wrong, I guess whether the deactivated word will affect? because I dowloaded the "nltk_data" from "https://github.com/nltk/nltk_data/tree/gh-pages" .Can you tell me how to do it in GTX-1080ti or AWS? Sincere thanks again.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Low metric values during both training and testing #9

Low metric values during both training and testing #9

soudabeh1000 commented Jan 18, 2021

andyweizhao commented Jan 19, 2021

soudabeh1000 commented Jan 19, 2021

andyweizhao commented Jan 20, 2021

soudabeh1000 commented Jan 27, 2021

zhangxin9988 commented Jul 15, 2021

andyweizhao commented Jul 15, 2021 via email

zhangxin9988 commented Jul 15, 2021

luoshunchong commented Aug 30, 2021

andyweizhao commented Aug 30, 2021

luoshunchong commented Aug 30, 2021

soudabeh1000 commented Aug 30, 2021

andyweizhao commented Aug 30, 2021

luoshunchong commented Aug 31, 2021

luoshunchong commented Aug 31, 2021

luoshunchong commented Aug 31, 2021

Low metric values during both training and testing #9

Low metric values during both training and testing #9

Comments

soudabeh1000 commented Jan 18, 2021

andyweizhao commented Jan 19, 2021

soudabeh1000 commented Jan 19, 2021

andyweizhao commented Jan 20, 2021

soudabeh1000 commented Jan 27, 2021

zhangxin9988 commented Jul 15, 2021

andyweizhao commented Jul 15, 2021 via email

zhangxin9988 commented Jul 15, 2021

luoshunchong commented Aug 30, 2021

andyweizhao commented Aug 30, 2021

luoshunchong commented Aug 30, 2021

soudabeh1000 commented Aug 30, 2021

andyweizhao commented Aug 30, 2021

luoshunchong commented Aug 31, 2021

luoshunchong commented Aug 31, 2021

luoshunchong commented Aug 31, 2021