-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_full_L31.txt
3317 lines (3317 loc) · 235 KB
/
HCQ_MSRVTT_full_L31.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 783.7493462562561 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 40.293457984924316 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 234.79337000846863 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 189.60598349571228 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch0.pth ...
Done in 1.450s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch0.pth ...
Done in 3.214s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_full_val/t2v_metrics/R1: 0.0
MSRVTT_full_val/t2v_metrics/R5: 0.6036217303822937
MSRVTT_full_val/t2v_metrics/R10: 2.6156941649899395
MSRVTT_full_val/t2v_metrics/R50: 11.267605633802816
MSRVTT_full_val/t2v_metrics/MedR: 258.0
MSRVTT_full_val/t2v_metrics/MeanR: 253.80080482897384
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_val/v2t_metrics/R1: 0.0
MSRVTT_full_val/v2t_metrics/R5: 0.4024144869215292
MSRVTT_full_val/v2t_metrics/R10: 1.0060362173038229
MSRVTT_full_val/v2t_metrics/R50: 9.25553319919517
MSRVTT_full_val/v2t_metrics/MedR: 249.0
MSRVTT_full_val/v2t_metrics/MeanR: 251.30382293762577
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_test/t2v_metrics/R1: 0.0
MSRVTT_full_test/t2v_metrics/R5: 0.13377926421404682
MSRVTT_full_test/t2v_metrics/R10: 0.3010033444816054
MSRVTT_full_test/t2v_metrics/R50: 1.37123745819398
MSRVTT_full_test/t2v_metrics/MedR: 1515.5
MSRVTT_full_test/t2v_metrics/MeanR: 1496.3177257525083
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_test/v2t_metrics/R1: 0.06688963210702341
MSRVTT_full_test/v2t_metrics/R5: 0.23411371237458195
MSRVTT_full_test/v2t_metrics/R10: 0.5016722408026756
MSRVTT_full_test/v2t_metrics/R50: 1.806020066889632
MSRVTT_full_test/v2t_metrics/MedR: 1530.5
MSRVTT_full_test/v2t_metrics/MeanR: 1509.763712374582
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.198793376346593
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 29.85628 (QuantReg: 22.60394) QuantErr: 22.60394 batch_time=26.43361
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 28.16539 (QuantReg: 22.60820) QuantErr: 22.60820 batch_time=0.94055
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 23.53183 (QuantReg: 22.66299) QuantErr: 22.66299 batch_time=0.93835
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 21.27540 (QuantReg: 22.65042) QuantErr: 22.65042 batch_time=0.93985
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 20.58664 (QuantReg: 22.63933) QuantErr: 22.63933 batch_time=0.95708
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 18.01616 (QuantReg: 22.62373) QuantErr: 22.62373 batch_time=0.96874
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 18.96465 (QuantReg: 22.63382) QuantErr: 22.63382 batch_time=1.03448
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 16.99486 (QuantReg: 22.61813) QuantErr: 22.61813 batch_time=0.94911
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 17.24226 (QuantReg: 22.61182) QuantErr: 22.61182 batch_time=0.94107
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 16.54098 (QuantReg: 22.63662) QuantErr: 22.63662 batch_time=0.92796
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 15.84844 (QuantReg: 22.62155) QuantErr: 22.62155 batch_time=1.07047
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 16.30754 (QuantReg: 22.64636) QuantErr: 22.64636 batch_time=0.92533
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 15.51964 (QuantReg: 22.64624) QuantErr: 22.64624 batch_time=0.93021
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 14.98874 (QuantReg: 22.62999) QuantErr: 22.62999 batch_time=0.94066
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 16.24261 (QuantReg: 22.66182) QuantErr: 22.66182 batch_time=0.95924
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 14.04982 (QuantReg: 22.62665) QuantErr: 22.62665 batch_time=0.95225
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 13.17538 (QuantReg: 22.65325) QuantErr: 22.65325 batch_time=0.94503
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 12.82270 (QuantReg: 22.62968) QuantErr: 22.62968 batch_time=0.95267
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 14.28625 (QuantReg: 22.64149) QuantErr: 22.64149 batch_time=0.96047
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 12.91745 (QuantReg: 22.62531) QuantErr: 22.62531 batch_time=0.92921
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 12.32088 (QuantReg: 22.63708) QuantErr: 22.63708 batch_time=1.06591
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 13.50362 (QuantReg: 22.66206) QuantErr: 22.66206 batch_time=0.94850
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 13.10004 (QuantReg: 22.64015) QuantErr: 22.64015 batch_time=0.96977
Train Epoch: 1 codebook_update_time=9.26384
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch1.pth ...
Done in 4.454s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch1.pth ...
Done in 8.464s
epoch : 1
loss : 16.80583275604248
quant_reg : 22.636447143554687
quant_err : 22.636447143554687
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_full_val/t2v_metrics/R1: 19.517102615694164
MSRVTT_full_val/t2v_metrics/R5: 52.51509054325956
MSRVTT_full_val/t2v_metrics/R10: 66.39839034205231
MSRVTT_full_val/t2v_metrics/R50: 94.36619718309859
MSRVTT_full_val/t2v_metrics/MedR: 5.0
MSRVTT_full_val/t2v_metrics/MeanR: 14.014084507042254
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 40.82745778513133
MSRVTT_full_val/v2t_metrics/R1: 21.52917505030181
MSRVTT_full_val/v2t_metrics/R5: 57.142857142857146
MSRVTT_full_val/v2t_metrics/R10: 73.2394366197183
MSRVTT_full_val/v2t_metrics/R50: 94.76861167002012
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 12.167002012072434
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 44.83096752301555
MSRVTT_full_test/t2v_metrics/R1: 6.321070234113712
MSRVTT_full_test/t2v_metrics/R5: 20.20066889632107
MSRVTT_full_test/t2v_metrics/R10: 31.103678929765888
MSRVTT_full_test/t2v_metrics/R50: 65.28428093645485
MSRVTT_full_test/t2v_metrics/MedR: 25.0
MSRVTT_full_test/t2v_metrics/MeanR: 72.93361204013378
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 15.836384662273998
MSRVTT_full_test/v2t_metrics/R1: 6.822742474916388
MSRVTT_full_test/v2t_metrics/R5: 22.842809364548494
MSRVTT_full_test/v2t_metrics/R10: 35.45150501672241
MSRVTT_full_test/v2t_metrics/R50: 71.17056856187291
MSRVTT_full_test/v2t_metrics/MedR: 20.0
MSRVTT_full_test/v2t_metrics/MeanR: 64.68595317725752
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 17.678594069831338
mnt_best : 15.836384662273998
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 12.46076 (QuantReg: 11.51471) QuantErr: 11.51471 batch_time=25.87999
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 11.80633 (QuantReg: 11.24625) QuantErr: 11.24625 batch_time=0.94133
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 12.41349 (QuantReg: 11.66208) QuantErr: 11.66208 batch_time=0.93808
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 12.18893 (QuantReg: 11.39221) QuantErr: 11.39221 batch_time=0.93809
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 13.85918 (QuantReg: 11.71934) QuantErr: 11.71934 batch_time=0.92404
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 12.90500 (QuantReg: 12.08940) QuantErr: 12.08940 batch_time=0.92647
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 11.54389 (QuantReg: 12.21020) QuantErr: 12.21020 batch_time=0.93617
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 10.63754 (QuantReg: 12.48762) QuantErr: 12.48762 batch_time=0.95914
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 10.86102 (QuantReg: 12.32217) QuantErr: 12.32217 batch_time=0.94296
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 11.03831 (QuantReg: 13.03038) QuantErr: 13.03038 batch_time=0.93096
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 11.46546 (QuantReg: 12.72777) QuantErr: 12.72777 batch_time=0.93808
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 11.22059 (QuantReg: 12.62041) QuantErr: 12.62041 batch_time=0.93250
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 11.68107 (QuantReg: 12.99370) QuantErr: 12.99370 batch_time=1.01178
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 11.23085 (QuantReg: 12.94661) QuantErr: 12.94661 batch_time=0.93151
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 11.86016 (QuantReg: 13.29474) QuantErr: 13.29474 batch_time=0.99936
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 11.08922 (QuantReg: 13.28110) QuantErr: 13.28110 batch_time=0.94287
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 11.48515 (QuantReg: 13.34982) QuantErr: 13.34982 batch_time=0.97396
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 10.84655 (QuantReg: 13.59734) QuantErr: 13.59734 batch_time=1.06479
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 11.15677 (QuantReg: 13.77226) QuantErr: 13.77226 batch_time=0.94178
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 11.69947 (QuantReg: 14.03492) QuantErr: 14.03492 batch_time=0.92739
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 9.41048 (QuantReg: 13.99015) QuantErr: 13.99015 batch_time=1.57939
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 10.12785 (QuantReg: 14.39501) QuantErr: 14.39501 batch_time=0.90378
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 9.48420 (QuantReg: 14.23674) QuantErr: 14.23674 batch_time=0.94150
Train Epoch: 2 codebook_update_time=8.57093
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch2.pth ...
Done in 6.225s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch2.pth ...
Done in 11.264s
removing stale ckpt [epoch 1] [took 0.10s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 11.15474493408203
quant_reg : 12.906782775878906
quant_err : 12.906782775878906
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_full_val/t2v_metrics/R1: 20.925553319919516
MSRVTT_full_val/t2v_metrics/R5: 58.14889336016097
MSRVTT_full_val/t2v_metrics/R10: 73.03822937625755
MSRVTT_full_val/t2v_metrics/R50: 96.17706237424547
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.048289738430583
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 44.626162955946256
MSRVTT_full_val/v2t_metrics/R1: 23.74245472837022
MSRVTT_full_val/v2t_metrics/R5: 62.374245472837025
MSRVTT_full_val/v2t_metrics/R10: 78.87323943661971
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.116700201207243
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 48.88251407416717
MSRVTT_full_test/t2v_metrics/R1: 8.327759197324415
MSRVTT_full_test/t2v_metrics/R5: 26.120401337792643
MSRVTT_full_test/t2v_metrics/R10: 38.36120401337793
MSRVTT_full_test/t2v_metrics/R50: 72.90969899665552
MSRVTT_full_test/t2v_metrics/MedR: 18.0
MSRVTT_full_test/t2v_metrics/MeanR: 56.48729096989967
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.28305700824993
MSRVTT_full_test/v2t_metrics/R1: 9.531772575250836
MSRVTT_full_test/v2t_metrics/R5: 29.565217391304348
MSRVTT_full_test/v2t_metrics/R10: 42.474916387959865
MSRVTT_full_test/v2t_metrics/R50: 77.25752508361204
MSRVTT_full_test/v2t_metrics/MedR: 15.0
MSRVTT_full_test/v2t_metrics/MeanR: 48.49464882943144
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.875069751198783
mnt_best : 20.28305700824993
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 10.28485 (QuantReg: 11.36856) QuantErr: 11.36856 batch_time=38.44897
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 10.60340 (QuantReg: 11.19858) QuantErr: 11.19858 batch_time=0.93350
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 8.97968 (QuantReg: 11.64148) QuantErr: 11.64148 batch_time=0.93179
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 10.16430 (QuantReg: 11.74909) QuantErr: 11.74909 batch_time=0.92098
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 10.13159 (QuantReg: 11.41943) QuantErr: 11.41943 batch_time=0.94586
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 9.79254 (QuantReg: 11.69704) QuantErr: 11.69704 batch_time=0.95774
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 8.59840 (QuantReg: 11.80574) QuantErr: 11.80574 batch_time=4.42912
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 9.86317 (QuantReg: 11.60047) QuantErr: 11.60047 batch_time=1.00536
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 9.01305 (QuantReg: 11.77048) QuantErr: 11.77048 batch_time=0.93221
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 8.34818 (QuantReg: 11.95612) QuantErr: 11.95612 batch_time=0.92309
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 8.30414 (QuantReg: 12.14237) QuantErr: 12.14237 batch_time=0.96396
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 10.13243 (QuantReg: 12.40245) QuantErr: 12.40245 batch_time=0.93130
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 9.38241 (QuantReg: 12.28151) QuantErr: 12.28151 batch_time=1.05297
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 9.89315 (QuantReg: 12.33606) QuantErr: 12.33606 batch_time=0.93686
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 8.07359 (QuantReg: 11.77141) QuantErr: 11.77141 batch_time=1.41074
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 10.63847 (QuantReg: 12.19812) QuantErr: 12.19812 batch_time=0.95629
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 7.81556 (QuantReg: 12.55212) QuantErr: 12.55212 batch_time=0.97361
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 9.47619 (QuantReg: 12.28999) QuantErr: 12.28999 batch_time=0.92486
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 8.71000 (QuantReg: 12.39090) QuantErr: 12.39090 batch_time=2.37640
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 8.57723 (QuantReg: 12.05551) QuantErr: 12.05551 batch_time=0.94594
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 8.08948 (QuantReg: 12.93508) QuantErr: 12.93508 batch_time=1.27328
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 8.85210 (QuantReg: 13.09573) QuantErr: 13.09573 batch_time=0.93395
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 8.76218 (QuantReg: 12.56716) QuantErr: 12.56716 batch_time=1.01723
Train Epoch: 3 codebook_update_time=8.20792
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch3.pth ...
Done in 5.737s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch3.pth ...
Done in 11.167s
removing stale ckpt [epoch 2] [took 0.00s]
epoch : 3
loss : 9.297818017959594
quant_reg : 12.058822063446044
quant_err : 12.058822063446044
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_full_val/t2v_metrics/R1: 24.748490945674043
MSRVTT_full_val/t2v_metrics/R5: 61.56941649899397
MSRVTT_full_val/t2v_metrics/R10: 74.84909456740442
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.245472837022133
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.49535064817711
MSRVTT_full_val/v2t_metrics/R1: 28.37022132796781
MSRVTT_full_val/v2t_metrics/R5: 66.80080482897384
MSRVTT_full_val/v2t_metrics/R10: 80.88531187122736
MSRVTT_full_val/v2t_metrics/R50: 96.98189134808852
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.82092555331992
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 53.51859387195801
MSRVTT_full_test/t2v_metrics/R1: 8.762541806020067
MSRVTT_full_test/t2v_metrics/R5: 27.090301003344482
MSRVTT_full_test/t2v_metrics/R10: 39.79933110367893
MSRVTT_full_test/t2v_metrics/R50: 74.44816053511705
MSRVTT_full_test/t2v_metrics/MedR: 17.0
MSRVTT_full_test/t2v_metrics/MeanR: 55.30836120401338
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.140077196493788
MSRVTT_full_test/v2t_metrics/R1: 9.698996655518394
MSRVTT_full_test/v2t_metrics/R5: 31.304347826086957
MSRVTT_full_test/v2t_metrics/R10: 45.08361204013378
MSRVTT_full_test/v2t_metrics/R50: 78.49498327759197
MSRVTT_full_test/v2t_metrics/MedR: 13.0
MSRVTT_full_test/v2t_metrics/MeanR: 45.15183946488294
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.921223638908884
mnt_best : 21.140077196493788
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 8.88870 (QuantReg: 11.68915) QuantErr: 11.68915 batch_time=31.72590
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 9.08758 (QuantReg: 11.37098) QuantErr: 11.37098 batch_time=0.94924
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 9.50341 (QuantReg: 11.66935) QuantErr: 11.66935 batch_time=0.93257
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 7.92442 (QuantReg: 11.82254) QuantErr: 11.82254 batch_time=0.92035
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 7.74036 (QuantReg: 11.55809) QuantErr: 11.55809 batch_time=0.92694
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 8.43716 (QuantReg: 11.32563) QuantErr: 11.32563 batch_time=0.93502
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 8.83025 (QuantReg: 11.20307) QuantErr: 11.20307 batch_time=0.95206
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 7.75223 (QuantReg: 11.78610) QuantErr: 11.78610 batch_time=0.98903
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 8.74451 (QuantReg: 12.18289) QuantErr: 12.18289 batch_time=0.95016
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 9.02558 (QuantReg: 11.84477) QuantErr: 11.84477 batch_time=0.92359
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 7.74698 (QuantReg: 12.21038) QuantErr: 12.21038 batch_time=0.92752
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 8.66899 (QuantReg: 12.20769) QuantErr: 12.20769 batch_time=0.99492
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 8.08785 (QuantReg: 12.30983) QuantErr: 12.30983 batch_time=1.23424
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 7.31034 (QuantReg: 12.02400) QuantErr: 12.02400 batch_time=0.97920
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 8.29275 (QuantReg: 12.10734) QuantErr: 12.10734 batch_time=0.93319
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 7.79074 (QuantReg: 12.37785) QuantErr: 12.37785 batch_time=0.91813
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 8.98257 (QuantReg: 12.17898) QuantErr: 12.17898 batch_time=0.99331
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 7.51643 (QuantReg: 12.31481) QuantErr: 12.31481 batch_time=0.95921
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 7.32247 (QuantReg: 12.30629) QuantErr: 12.30629 batch_time=0.93463
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 7.19476 (QuantReg: 12.60331) QuantErr: 12.60331 batch_time=0.94371
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 7.58622 (QuantReg: 12.23652) QuantErr: 12.23652 batch_time=1.04023
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 7.82099 (QuantReg: 12.30073) QuantErr: 12.30073 batch_time=0.95645
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 6.56511 (QuantReg: 12.43386) QuantErr: 12.43386 batch_time=0.95388
Train Epoch: 4 codebook_update_time=8.05347
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch4.pth ...
Done in 18.386s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch4.pth ...
Done in 23.706s
removing stale ckpt [epoch 3] [took 0.00s]
epoch : 4
loss : 8.197911178588868
quant_reg : 11.983840110778809
quant_err : 11.983840110778809
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_full_val/t2v_metrics/R1: 28.571428571428573
MSRVTT_full_val/t2v_metrics/R5: 59.758551307847085
MSRVTT_full_val/t2v_metrics/R10: 75.85513078470825
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.845070422535212
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.594773874268064
MSRVTT_full_val/v2t_metrics/R1: 27.96780684104628
MSRVTT_full_val/v2t_metrics/R5: 69.01408450704226
MSRVTT_full_val/v2t_metrics/R10: 82.09255533199195
MSRVTT_full_val/v2t_metrics/R50: 96.579476861167
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.593561368209254
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.11279574493674
MSRVTT_full_test/t2v_metrics/R1: 9.531772575250836
MSRVTT_full_test/t2v_metrics/R5: 28.561872909698998
MSRVTT_full_test/t2v_metrics/R10: 41.60535117056856
MSRVTT_full_test/t2v_metrics/R50: 75.55183946488295
MSRVTT_full_test/t2v_metrics/MedR: 15.0
MSRVTT_full_test/t2v_metrics/MeanR: 51.29397993311037
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.457936456791245
MSRVTT_full_test/v2t_metrics/R1: 11.33779264214047
MSRVTT_full_test/v2t_metrics/R5: 33.01003344481605
MSRVTT_full_test/v2t_metrics/R10: 47.290969899665555
MSRVTT_full_test/v2t_metrics/R50: 80.5685618729097
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 40.67725752508361
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.060589283986047
mnt_best : 22.457936456791245
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 6.69148 (QuantReg: 11.78841) QuantErr: 11.78841 batch_time=40.91391
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 6.79195 (QuantReg: 12.02190) QuantErr: 12.02190 batch_time=0.95895
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 7.80327 (QuantReg: 11.90531) QuantErr: 11.90531 batch_time=0.98866
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 8.45246 (QuantReg: 11.65757) QuantErr: 11.65757 batch_time=0.95129
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 6.95036 (QuantReg: 11.89446) QuantErr: 11.89446 batch_time=0.97665
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 7.43941 (QuantReg: 11.76554) QuantErr: 11.76554 batch_time=0.98842
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 7.15383 (QuantReg: 12.09964) QuantErr: 12.09964 batch_time=0.93884
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 7.78507 (QuantReg: 11.91415) QuantErr: 11.91415 batch_time=0.95496
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 7.04285 (QuantReg: 12.00290) QuantErr: 12.00290 batch_time=0.96830
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 8.14923 (QuantReg: 11.57001) QuantErr: 11.57001 batch_time=0.95594
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 7.55481 (QuantReg: 11.99553) QuantErr: 11.99553 batch_time=0.95649
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 7.02070 (QuantReg: 11.92030) QuantErr: 11.92030 batch_time=0.94717
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 7.17545 (QuantReg: 12.18764) QuantErr: 12.18764 batch_time=1.01151
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 8.30066 (QuantReg: 12.05200) QuantErr: 12.05200 batch_time=0.96890
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 8.68805 (QuantReg: 11.97790) QuantErr: 11.97790 batch_time=0.96018
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 7.23069 (QuantReg: 12.02607) QuantErr: 12.02607 batch_time=0.94984
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 8.15121 (QuantReg: 12.30891) QuantErr: 12.30891 batch_time=1.76056
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 6.32168 (QuantReg: 12.11809) QuantErr: 12.11809 batch_time=1.38570
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 6.90049 (QuantReg: 12.36112) QuantErr: 12.36112 batch_time=0.94901
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 7.06618 (QuantReg: 11.96558) QuantErr: 11.96558 batch_time=0.94481
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 8.14231 (QuantReg: 12.46871) QuantErr: 12.46871 batch_time=0.96670
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 6.97863 (QuantReg: 12.67154) QuantErr: 12.67154 batch_time=0.94582
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 6.76935 (QuantReg: 12.34438) QuantErr: 12.34438 batch_time=0.96968
Train Epoch: 5 codebook_update_time=9.05216
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch5.pth ...
Done in 5.025s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch5.pth ...
Done in 10.325s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 7.391795722961426
quant_reg : 12.089642528533936
quant_err : 12.089642528533936
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_full_val/t2v_metrics/R1: 29.175050301810867
MSRVTT_full_val/t2v_metrics/R5: 63.38028169014085
MSRVTT_full_val/t2v_metrics/R10: 77.66599597585513
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.138832997987928
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.36794897803871
MSRVTT_full_val/v2t_metrics/R1: 33.40040241448692
MSRVTT_full_val/v2t_metrics/R5: 71.42857142857143
MSRVTT_full_val/v2t_metrics/R10: 82.49496981891348
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.559356136820925
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.167943599586756
MSRVTT_full_test/t2v_metrics/R1: 10.936454849498327
MSRVTT_full_test/t2v_metrics/R5: 31.40468227424749
MSRVTT_full_test/t2v_metrics/R10: 44.94983277591973
MSRVTT_full_test/t2v_metrics/R50: 78.69565217391305
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 44.936454849498325
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.900019252577064
MSRVTT_full_test/v2t_metrics/R1: 13.043478260869565
MSRVTT_full_test/v2t_metrics/R5: 36.92307692307692
MSRVTT_full_test/v2t_metrics/R10: 52.274247491638796
MSRVTT_full_test/v2t_metrics/R50: 83.47826086956522
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 35.111371237458194
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.308462154648804
mnt_best : 24.900019252577064
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 6.98431 (QuantReg: 12.26950) QuantErr: 12.26950 batch_time=33.62255
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 7.03732 (QuantReg: 12.11269) QuantErr: 12.11269 batch_time=1.17675
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 7.43979 (QuantReg: 11.71992) QuantErr: 11.71992 batch_time=0.93180
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 7.02288 (QuantReg: 11.87834) QuantErr: 11.87834 batch_time=0.99703
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 6.32184 (QuantReg: 11.96925) QuantErr: 11.96925 batch_time=0.96652
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 6.89756 (QuantReg: 12.29637) QuantErr: 12.29637 batch_time=0.93994
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 7.15624 (QuantReg: 12.38030) QuantErr: 12.38030 batch_time=2.60803
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 7.46775 (QuantReg: 12.02514) QuantErr: 12.02514 batch_time=0.95497
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 6.19514 (QuantReg: 12.40714) QuantErr: 12.40714 batch_time=0.94792
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 6.66330 (QuantReg: 12.48373) QuantErr: 12.48373 batch_time=0.97355
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 7.10341 (QuantReg: 11.95303) QuantErr: 11.95303 batch_time=0.93951
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 6.34701 (QuantReg: 12.23911) QuantErr: 12.23911 batch_time=0.93256
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 6.84662 (QuantReg: 12.51592) QuantErr: 12.51592 batch_time=0.96033
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 6.62805 (QuantReg: 12.15021) QuantErr: 12.15021 batch_time=0.96319
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 6.57726 (QuantReg: 12.19938) QuantErr: 12.19938 batch_time=0.94145
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 6.17619 (QuantReg: 12.26457) QuantErr: 12.26457 batch_time=0.93852
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 6.96382 (QuantReg: 12.49274) QuantErr: 12.49274 batch_time=0.92538
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 7.93138 (QuantReg: 12.59462) QuantErr: 12.59462 batch_time=0.93914
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 6.34804 (QuantReg: 11.79365) QuantErr: 11.79365 batch_time=0.92880
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 6.27937 (QuantReg: 12.54109) QuantErr: 12.54109 batch_time=0.92200
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 7.12010 (QuantReg: 12.46006) QuantErr: 12.46006 batch_time=0.92743
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 6.52232 (QuantReg: 12.36808) QuantErr: 12.36808 batch_time=0.95839
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 5.44758 (QuantReg: 12.28151) QuantErr: 12.28151 batch_time=0.91302
Train Epoch: 6 codebook_update_time=8.90493
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch6.pth ...
Done in 3.955s
removing stale ckpt [epoch 5] [took 0.01s]
epoch : 6
loss : 6.788179193496704
quant_reg : 12.183882175445557
quant_err : 12.183882175445557
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_full_val/t2v_metrics/R1: 27.364185110663986
MSRVTT_full_val/t2v_metrics/R5: 62.57545271629779
MSRVTT_full_val/t2v_metrics/R10: 75.45271629778672
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.82092555331992
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.55379926442329
MSRVTT_full_val/v2t_metrics/R1: 31.388329979879277
MSRVTT_full_val/v2t_metrics/R5: 69.21529175050301
MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.99195171026157
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.74532035598913
MSRVTT_full_test/t2v_metrics/R1: 9.899665551839465
MSRVTT_full_test/t2v_metrics/R5: 30.80267558528428
MSRVTT_full_test/t2v_metrics/R10: 44.81605351170568
MSRVTT_full_test/t2v_metrics/R50: 77.85953177257525
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 46.67224080267559
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.90823549056372
MSRVTT_full_test/v2t_metrics/R1: 12.307692307692308
MSRVTT_full_test/v2t_metrics/R5: 36.35451505016722
MSRVTT_full_test/v2t_metrics/R10: 50.50167224080268
MSRVTT_full_test/v2t_metrics/R50: 83.87959866220736
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 36.003010033444816
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.271373944009415
mnt_best : 24.900019252577064
not_improved_count: 1
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 5.33375 (QuantReg: 11.80667) QuantErr: 11.80667 batch_time=29.32580
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 6.61448 (QuantReg: 11.92832) QuantErr: 11.92832 batch_time=0.94240
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 7.18155 (QuantReg: 12.10066) QuantErr: 12.10066 batch_time=0.94497
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 6.88590 (QuantReg: 11.86663) QuantErr: 11.86663 batch_time=0.99366
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 7.13037 (QuantReg: 12.33940) QuantErr: 12.33940 batch_time=0.96476
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 6.62878 (QuantReg: 12.24736) QuantErr: 12.24736 batch_time=0.96031
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 7.65967 (QuantReg: 11.97242) QuantErr: 11.97242 batch_time=0.95470
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 6.47416 (QuantReg: 12.10569) QuantErr: 12.10569 batch_time=0.95234
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 7.68984 (QuantReg: 12.29870) QuantErr: 12.29870 batch_time=0.93227
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 7.38220 (QuantReg: 12.60205) QuantErr: 12.60205 batch_time=6.75758
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 5.93979 (QuantReg: 12.22334) QuantErr: 12.22334 batch_time=0.93303
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 6.24477 (QuantReg: 12.36218) QuantErr: 12.36218 batch_time=0.94394
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 5.77142 (QuantReg: 11.93115) QuantErr: 11.93115 batch_time=0.95102
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 5.73271 (QuantReg: 12.30020) QuantErr: 12.30020 batch_time=0.93977
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 5.24741 (QuantReg: 11.96067) QuantErr: 11.96067 batch_time=0.93523
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 6.44026 (QuantReg: 12.35112) QuantErr: 12.35112 batch_time=0.92330
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 5.85765 (QuantReg: 12.55864) QuantErr: 12.55864 batch_time=0.95583
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 7.53576 (QuantReg: 11.77634) QuantErr: 11.77634 batch_time=0.93860
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 6.79243 (QuantReg: 12.22569) QuantErr: 12.22569 batch_time=2.80854
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 7.55811 (QuantReg: 12.03611) QuantErr: 12.03611 batch_time=0.99369
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 6.29172 (QuantReg: 12.63498) QuantErr: 12.63498 batch_time=0.97472
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 5.16673 (QuantReg: 12.75867) QuantErr: 12.75867 batch_time=0.92837
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 6.93426 (QuantReg: 12.44558) QuantErr: 12.44558 batch_time=0.96636
Train Epoch: 7 codebook_update_time=8.98615
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch7.pth ...
Done in 3.992s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 6.265350616455078
quant_reg : 12.233642261505127
quant_err : 12.233642261505127
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_full_val/t2v_metrics/R1: 28.973843058350102
MSRVTT_full_val/t2v_metrics/R5: 62.374245472837025
MSRVTT_full_val/t2v_metrics/R10: 78.47082494969818
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.112676056338028
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.14826641528122
MSRVTT_full_val/v2t_metrics/R1: 35.814889336016094
MSRVTT_full_val/v2t_metrics/R5: 71.62977867203219
MSRVTT_full_val/v2t_metrics/R10: 85.91549295774648
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.9476861167002015
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.4054656263384
MSRVTT_full_test/t2v_metrics/R1: 10.367892976588628
MSRVTT_full_test/t2v_metrics/R5: 31.638795986622075
MSRVTT_full_test/t2v_metrics/R10: 45.1505016722408
MSRVTT_full_test/t2v_metrics/R50: 77.79264214046823
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 47.874581939799334
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.557887823885387
MSRVTT_full_test/v2t_metrics/R1: 12.54180602006689
MSRVTT_full_test/v2t_metrics/R5: 37.02341137123746
MSRVTT_full_test/v2t_metrics/R10: 51.30434782608695
MSRVTT_full_test/v2t_metrics/R50: 83.84615384615384
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 36.93929765886288
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.773778388989758
mnt_best : 24.900019252577064
not_improved_count: 2
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 6.72838 (QuantReg: 12.01827) QuantErr: 12.01827 batch_time=36.28811
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 6.12785 (QuantReg: 11.77076) QuantErr: 11.77076 batch_time=0.92809
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 6.63109 (QuantReg: 12.09980) QuantErr: 12.09980 batch_time=0.95764
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 5.91525 (QuantReg: 12.49012) QuantErr: 12.49012 batch_time=0.94888
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 6.42605 (QuantReg: 11.84849) QuantErr: 11.84849 batch_time=0.94603
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 6.38814 (QuantReg: 11.92345) QuantErr: 11.92345 batch_time=0.95619
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 7.12810 (QuantReg: 12.12666) QuantErr: 12.12666 batch_time=0.97749
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 5.44793 (QuantReg: 12.43743) QuantErr: 12.43743 batch_time=1.30343
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 5.88677 (QuantReg: 12.64684) QuantErr: 12.64684 batch_time=0.93011
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 6.55965 (QuantReg: 12.07725) QuantErr: 12.07725 batch_time=0.96066
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 5.44932 (QuantReg: 12.34081) QuantErr: 12.34081 batch_time=0.95450
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 6.19955 (QuantReg: 12.42908) QuantErr: 12.42908 batch_time=0.93954
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 6.41231 (QuantReg: 11.94412) QuantErr: 11.94412 batch_time=0.95676
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 5.50562 (QuantReg: 12.44926) QuantErr: 12.44926 batch_time=0.99441
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 5.37933 (QuantReg: 12.47451) QuantErr: 12.47451 batch_time=1.52615
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 5.12037 (QuantReg: 12.20220) QuantErr: 12.20220 batch_time=0.95090
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 5.33794 (QuantReg: 12.51379) QuantErr: 12.51379 batch_time=0.92645
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 4.70401 (QuantReg: 12.50134) QuantErr: 12.50134 batch_time=0.93906
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 6.41329 (QuantReg: 12.60028) QuantErr: 12.60028 batch_time=0.92115
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 4.97298 (QuantReg: 12.58084) QuantErr: 12.58084 batch_time=0.92233
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 4.86934 (QuantReg: 12.36290) QuantErr: 12.36290 batch_time=0.93068
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 6.80838 (QuantReg: 12.17968) QuantErr: 12.17968 batch_time=0.90596
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 5.19275 (QuantReg: 12.72093) QuantErr: 12.72093 batch_time=0.93623
Train Epoch: 8 codebook_update_time=7.96803
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch8.pth ...
Done in 16.957s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch8.pth ...
Done in 20.736s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 5.833094860076904
quant_reg : 12.283239761352538
quant_err : 12.283239761352538
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_full_val/t2v_metrics/R1: 30.985915492957748
MSRVTT_full_val/t2v_metrics/R5: 65.99597585513078
MSRVTT_full_val/t2v_metrics/R10: 78.67203219315896
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.943661971830986
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.38770316157045
MSRVTT_full_val/v2t_metrics/R1: 34.40643863179074
MSRVTT_full_val/v2t_metrics/R5: 73.44064386317908
MSRVTT_full_val/v2t_metrics/R10: 85.91549295774648
MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.933601609657948
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.10111944033106
MSRVTT_full_test/t2v_metrics/R1: 11.839464882943144
MSRVTT_full_test/t2v_metrics/R5: 32.97658862876254
MSRVTT_full_test/t2v_metrics/R10: 46.32107023411371
MSRVTT_full_test/t2v_metrics/R50: 78.8628762541806
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.6056856187291
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.248558722550193
MSRVTT_full_test/v2t_metrics/R1: 12.709030100334449
MSRVTT_full_test/v2t_metrics/R5: 37.22408026755853
MSRVTT_full_test/v2t_metrics/R10: 52.77591973244147
MSRVTT_full_test/v2t_metrics/R50: 83.34448160535118
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 37.0066889632107
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.227436902916526
mnt_best : 26.248558722550193
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 6.15022 (QuantReg: 11.91281) QuantErr: 11.91281 batch_time=37.44750
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 6.64332 (QuantReg: 12.06240) QuantErr: 12.06240 batch_time=0.92359
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 4.93222 (QuantReg: 11.87147) QuantErr: 11.87147 batch_time=0.92526
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 5.95843 (QuantReg: 12.63659) QuantErr: 12.63659 batch_time=0.92185
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 4.88875 (QuantReg: 12.67270) QuantErr: 12.67270 batch_time=0.93529
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 5.64185 (QuantReg: 12.50600) QuantErr: 12.50600 batch_time=0.91881
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 5.19340 (QuantReg: 12.03386) QuantErr: 12.03386 batch_time=0.95153
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 4.86604 (QuantReg: 12.35439) QuantErr: 12.35439 batch_time=0.92817
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 5.52142 (QuantReg: 12.39419) QuantErr: 12.39419 batch_time=1.02218
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 4.48837 (QuantReg: 12.43983) QuantErr: 12.43983 batch_time=0.92123
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 5.26939 (QuantReg: 12.33506) QuantErr: 12.33506 batch_time=1.04872
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 4.28921 (QuantReg: 12.47800) QuantErr: 12.47800 batch_time=1.17512
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 5.41368 (QuantReg: 12.79118) QuantErr: 12.79118 batch_time=0.97898
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 5.17302 (QuantReg: 12.36283) QuantErr: 12.36283 batch_time=0.91830
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 5.45759 (QuantReg: 12.26813) QuantErr: 12.26813 batch_time=0.94205
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 5.45694 (QuantReg: 12.25916) QuantErr: 12.25916 batch_time=0.94731
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 5.32852 (QuantReg: 12.65204) QuantErr: 12.65204 batch_time=0.94558
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 5.79780 (QuantReg: 12.57668) QuantErr: 12.57668 batch_time=1.28553
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 5.82396 (QuantReg: 12.65683) QuantErr: 12.65683 batch_time=2.86520
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 5.30407 (QuantReg: 12.31036) QuantErr: 12.31036 batch_time=1.14848
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 4.83579 (QuantReg: 12.51051) QuantErr: 12.51051 batch_time=0.94673
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 4.85341 (QuantReg: 12.86381) QuantErr: 12.86381 batch_time=0.94591
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 4.54832 (QuantReg: 12.47462) QuantErr: 12.47462 batch_time=0.93800
Train Epoch: 9 codebook_update_time=8.64358
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch9.pth ...
Done in 4.166s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 5.491644438743592
quant_reg : 12.377650302886963
quant_err : 12.377650302886963
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_full_val/t2v_metrics/R1: 29.77867203219316
MSRVTT_full_val/t2v_metrics/R5: 65.79476861167002
MSRVTT_full_val/t2v_metrics/R10: 80.0804828973843
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.231388329979879
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.93546880753877
MSRVTT_full_val/v2t_metrics/R1: 35.814889336016094
MSRVTT_full_val/v2t_metrics/R5: 75.0503018108652
MSRVTT_full_val/v2t_metrics/R10: 86.31790744466801
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.301810865191147
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.447703052216234
MSRVTT_full_test/t2v_metrics/R1: 10.903010033444817
MSRVTT_full_test/t2v_metrics/R5: 33.74581939799331
MSRVTT_full_test/t2v_metrics/R10: 47.357859531772576
MSRVTT_full_test/t2v_metrics/R50: 80.43478260869566
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 43.814046822742476
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.925042970181163
MSRVTT_full_test/v2t_metrics/R1: 13.979933110367893
MSRVTT_full_test/v2t_metrics/R5: 39.331103678929765
MSRVTT_full_test/v2t_metrics/R10: 53.7123745819398
MSRVTT_full_test/v2t_metrics/R50: 85.21739130434783
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 33.51120401337793
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.910440428064756
mnt_best : 26.248558722550193
not_improved_count: 1
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 5.19620 (QuantReg: 12.17664) QuantErr: 12.17664 batch_time=36.28912
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 6.10183 (QuantReg: 12.13098) QuantErr: 12.13098 batch_time=1.00533
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 5.67020 (QuantReg: 12.18241) QuantErr: 12.18241 batch_time=0.98708
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 5.59008 (QuantReg: 11.99827) QuantErr: 11.99827 batch_time=0.93376
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 4.15243 (QuantReg: 12.42102) QuantErr: 12.42102 batch_time=0.96182
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 4.80762 (QuantReg: 12.11273) QuantErr: 12.11273 batch_time=0.97890
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 4.32431 (QuantReg: 12.45482) QuantErr: 12.45482 batch_time=0.92361
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 5.00669 (QuantReg: 12.48437) QuantErr: 12.48437 batch_time=0.91819
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 5.22699 (QuantReg: 12.00070) QuantErr: 12.00070 batch_time=0.92431
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 4.12731 (QuantReg: 12.57689) QuantErr: 12.57689 batch_time=0.92330
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 5.73116 (QuantReg: 12.45136) QuantErr: 12.45136 batch_time=0.93185
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 5.51769 (QuantReg: 12.61541) QuantErr: 12.61541 batch_time=0.92741
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 4.93486 (QuantReg: 12.42756) QuantErr: 12.42756 batch_time=0.95683
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 5.31265 (QuantReg: 12.53603) QuantErr: 12.53603 batch_time=0.93476
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 5.30275 (QuantReg: 12.79820) QuantErr: 12.79820 batch_time=0.92874
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 4.23334 (QuantReg: 12.36100) QuantErr: 12.36100 batch_time=0.93247
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 5.33364 (QuantReg: 12.69955) QuantErr: 12.69955 batch_time=0.94110
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 5.39591 (QuantReg: 12.41162) QuantErr: 12.41162 batch_time=0.94113
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 4.49475 (QuantReg: 12.32463) QuantErr: 12.32463 batch_time=0.93345
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 4.88830 (QuantReg: 12.72626) QuantErr: 12.72626 batch_time=0.99286
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 5.80755 (QuantReg: 12.74810) QuantErr: 12.74810 batch_time=0.93735
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 4.93204 (QuantReg: 12.45013) QuantErr: 12.45013 batch_time=0.95747
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 4.10438 (QuantReg: 12.51867) QuantErr: 12.51867 batch_time=0.96727
Train Epoch: 10 codebook_update_time=8.75240
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch10.pth ...
Done in 4.024s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch10.pth ...
Done in 7.855s
removing stale ckpt [epoch 9] [took 0.01s]
epoch : 10
loss : 5.181815181732178
quant_reg : 12.444166889190674
quant_err : 12.444166889190674
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
MSRVTT_full_val/t2v_metrics/R5: 65.19114688128772
MSRVTT_full_val/t2v_metrics/R10: 78.87323943661971
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.400402414486923
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.256373398125135
MSRVTT_full_val/v2t_metrics/R1: 33.80281690140845
MSRVTT_full_val/v2t_metrics/R5: 72.83702213279678
MSRVTT_full_val/v2t_metrics/R10: 84.90945674044265
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.798792756539235
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.349952950749255
MSRVTT_full_test/t2v_metrics/R1: 11.638795986622073
MSRVTT_full_test/t2v_metrics/R5: 34.046822742474916
MSRVTT_full_test/t2v_metrics/R10: 47.15719063545151
MSRVTT_full_test/t2v_metrics/R50: 79.96655518394648
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 42.80367892976589
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.536533503973864
MSRVTT_full_test/v2t_metrics/R1: 14.080267558528428
MSRVTT_full_test/v2t_metrics/R5: 39.063545150501675
MSRVTT_full_test/v2t_metrics/R10: 52.97658862876254
MSRVTT_full_test/v2t_metrics/R50: 84.74916387959867
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 34.872240802675584
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.77198534704316
mnt_best : 26.536533503973864
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 5.06506 (QuantReg: 12.54698) QuantErr: 12.54698 batch_time=31.99147
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 4.14136 (QuantReg: 12.52534) QuantErr: 12.52534 batch_time=0.92633
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 4.61154 (QuantReg: 11.94664) QuantErr: 11.94664 batch_time=0.95680
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 4.45381 (QuantReg: 12.44482) QuantErr: 12.44482 batch_time=0.94580
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 4.92147 (QuantReg: 12.62867) QuantErr: 12.62867 batch_time=0.92703
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 5.45479 (QuantReg: 12.47040) QuantErr: 12.47040 batch_time=0.92578
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 4.66048 (QuantReg: 12.58807) QuantErr: 12.58807 batch_time=0.94934
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 5.04467 (QuantReg: 12.17999) QuantErr: 12.17999 batch_time=0.92321
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 5.81486 (QuantReg: 12.54550) QuantErr: 12.54550 batch_time=0.92154
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 6.33022 (QuantReg: 12.10997) QuantErr: 12.10997 batch_time=1.21835
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 4.31568 (QuantReg: 12.54102) QuantErr: 12.54102 batch_time=0.92760
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 5.34693 (QuantReg: 12.24563) QuantErr: 12.24563 batch_time=0.99530
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 4.94030 (QuantReg: 12.55965) QuantErr: 12.55965 batch_time=0.92083
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 4.77627 (QuantReg: 12.80532) QuantErr: 12.80532 batch_time=5.47978
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 4.66894 (QuantReg: 12.38932) QuantErr: 12.38932 batch_time=0.98063
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 4.47154 (QuantReg: 12.45611) QuantErr: 12.45611 batch_time=0.94879
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 4.52229 (QuantReg: 12.46845) QuantErr: 12.46845 batch_time=0.94548
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 5.13164 (QuantReg: 12.50118) QuantErr: 12.50118 batch_time=0.92276
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 3.83352 (QuantReg: 12.54675) QuantErr: 12.54675 batch_time=0.93243
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 4.68174 (QuantReg: 12.73378) QuantErr: 12.73378 batch_time=0.92741
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 4.26513 (QuantReg: 12.75307) QuantErr: 12.75307 batch_time=0.92906
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 4.07984 (QuantReg: 12.59661) QuantErr: 12.59661 batch_time=1.03894
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 4.47629 (QuantReg: 12.57062) QuantErr: 12.57062 batch_time=0.93030
Train Epoch: 11 codebook_update_time=7.76841
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch11.pth ...
Done in 16.682s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch11.pth ...
Done in 20.571s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 4.943440024375915
quant_reg : 12.513592372894287
quant_err : 12.513592372894287
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_full_val/t2v_metrics/R1: 30.382293762575454
MSRVTT_full_val/t2v_metrics/R5: 65.99597585513078
MSRVTT_full_val/t2v_metrics/R10: 78.87323943661971
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.086519114688128
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.07824187804044
MSRVTT_full_val/v2t_metrics/R1: 35.2112676056338
MSRVTT_full_val/v2t_metrics/R5: 74.64788732394366
MSRVTT_full_val/v2t_metrics/R10: 86.51911468812877
MSRVTT_full_val/v2t_metrics/R50: 98.79275653923541
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.935613682092555
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.038485493583515
MSRVTT_full_test/t2v_metrics/R1: 11.939799331103679
MSRVTT_full_test/t2v_metrics/R5: 34.147157190635454
MSRVTT_full_test/t2v_metrics/R10: 46.82274247491639
MSRVTT_full_test/t2v_metrics/R50: 80.2675585284281
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 43.276923076923076
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.726134087182814
MSRVTT_full_test/v2t_metrics/R1: 14.31438127090301
MSRVTT_full_test/v2t_metrics/R5: 39.63210702341137
MSRVTT_full_test/v2t_metrics/R10: 55.11705685618729
MSRVTT_full_test/v2t_metrics/R50: 85.38461538461539
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 31.558528428093645
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.504209531072686
mnt_best : 26.726134087182814
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 4.08621 (QuantReg: 12.01544) QuantErr: 12.01544 batch_time=29.62167
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 5.72658 (QuantReg: 12.19113) QuantErr: 12.19113 batch_time=0.93791
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 5.28946 (QuantReg: 12.43170) QuantErr: 12.43170 batch_time=0.97593
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 4.80429 (QuantReg: 12.18436) QuantErr: 12.18436 batch_time=0.94529
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 5.68627 (QuantReg: 12.46093) QuantErr: 12.46093 batch_time=0.96160
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 4.36038 (QuantReg: 12.35120) QuantErr: 12.35120 batch_time=0.96165
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 5.34329 (QuantReg: 12.16811) QuantErr: 12.16811 batch_time=0.94186
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 4.11050 (QuantReg: 12.54319) QuantErr: 12.54319 batch_time=0.92453
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 4.00151 (QuantReg: 12.78749) QuantErr: 12.78749 batch_time=1.47454
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 6.06365 (QuantReg: 12.28859) QuantErr: 12.28859 batch_time=1.79561
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 4.25091 (QuantReg: 12.71080) QuantErr: 12.71080 batch_time=0.95116
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 4.90655 (QuantReg: 12.50045) QuantErr: 12.50045 batch_time=0.90217
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 5.93467 (QuantReg: 12.45353) QuantErr: 12.45353 batch_time=0.92585
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 5.25392 (QuantReg: 12.40184) QuantErr: 12.40184 batch_time=0.93846
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 4.91180 (QuantReg: 12.40241) QuantErr: 12.40241 batch_time=0.94946
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 5.69892 (QuantReg: 12.50576) QuantErr: 12.50576 batch_time=0.92320
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 4.14914 (QuantReg: 12.35705) QuantErr: 12.35705 batch_time=0.94245
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 5.05122 (QuantReg: 12.60361) QuantErr: 12.60361 batch_time=0.91746
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 4.06864 (QuantReg: 12.33344) QuantErr: 12.33344 batch_time=1.05627
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 3.36055 (QuantReg: 12.49888) QuantErr: 12.49888 batch_time=0.93849
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 4.97796 (QuantReg: 12.74389) QuantErr: 12.74389 batch_time=1.56788
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 4.30402 (QuantReg: 12.39863) QuantErr: 12.39863 batch_time=1.16268
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 5.13295 (QuantReg: 12.68499) QuantErr: 12.68499 batch_time=0.95409
Train Epoch: 12 codebook_update_time=8.97935
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch12.pth ...
Done in 3.985s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch12.pth ...
Done in 8.010s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 4.766584445953369
quant_reg : 12.486056941986083
quant_err : 12.486056941986083
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_full_val/t2v_metrics/R1: 29.175050301810867
MSRVTT_full_val/t2v_metrics/R5: 64.38631790744466
MSRVTT_full_val/t2v_metrics/R10: 79.07444668008048
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.29979879275654
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.959895754905
MSRVTT_full_val/v2t_metrics/R1: 35.814889336016094
MSRVTT_full_val/v2t_metrics/R5: 74.64788732394366
MSRVTT_full_val/v2t_metrics/R10: 87.12273641851107
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.078470824949698
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.527728319862916
MSRVTT_full_test/t2v_metrics/R1: 11.806020066889632
MSRVTT_full_test/t2v_metrics/R5: 34.34782608695652
MSRVTT_full_test/t2v_metrics/R10: 48.22742474916388
MSRVTT_full_test/t2v_metrics/R50: 79.29765886287625
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 43.96923076923077
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.942151968465247
MSRVTT_full_test/v2t_metrics/R1: 15.351170568561873
MSRVTT_full_test/v2t_metrics/R5: 39.197324414715716
MSRVTT_full_test/v2t_metrics/R10: 54.51505016722408
MSRVTT_full_test/v2t_metrics/R50: 85.35117056856187
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 32.206020066889636
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.01140816008145
mnt_best : 26.942151968465247
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 5.53858 (QuantReg: 12.25043) QuantErr: 12.25043 batch_time=36.79826
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 5.57314 (QuantReg: 12.05701) QuantErr: 12.05701 batch_time=1.04553
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 4.74559 (QuantReg: 12.25715) QuantErr: 12.25715 batch_time=0.91552
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 5.04068 (QuantReg: 12.31306) QuantErr: 12.31306 batch_time=0.95057
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 6.43007 (QuantReg: 11.94934) QuantErr: 11.94934 batch_time=0.95062
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 5.74033 (QuantReg: 12.41300) QuantErr: 12.41300 batch_time=0.92066
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 4.29965 (QuantReg: 12.39179) QuantErr: 12.39179 batch_time=1.81990
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 5.57928 (QuantReg: 12.81143) QuantErr: 12.81143 batch_time=0.92856
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 4.56147 (QuantReg: 12.68671) QuantErr: 12.68671 batch_time=1.02574
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 4.63700 (QuantReg: 12.54486) QuantErr: 12.54486 batch_time=0.99597
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 6.18089 (QuantReg: 12.62291) QuantErr: 12.62291 batch_time=0.93095
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 5.07045 (QuantReg: 12.60911) QuantErr: 12.60911 batch_time=0.94468
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 3.87768 (QuantReg: 12.49783) QuantErr: 12.49783 batch_time=0.93994
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 5.22709 (QuantReg: 12.42730) QuantErr: 12.42730 batch_time=0.92912
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 3.85152 (QuantReg: 12.69192) QuantErr: 12.69192 batch_time=0.93936
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 4.80758 (QuantReg: 12.61007) QuantErr: 12.61007 batch_time=0.92154
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 4.34170 (QuantReg: 12.69626) QuantErr: 12.69626 batch_time=0.93024
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 3.64123 (QuantReg: 12.54328) QuantErr: 12.54328 batch_time=0.95060
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 3.75914 (QuantReg: 12.90935) QuantErr: 12.90935 batch_time=3.45386
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 5.67845 (QuantReg: 12.66538) QuantErr: 12.66538 batch_time=0.94843
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 4.94301 (QuantReg: 12.56732) QuantErr: 12.56732 batch_time=0.92391
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 4.32036 (QuantReg: 12.77960) QuantErr: 12.77960 batch_time=0.95530
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 4.40297 (QuantReg: 12.95953) QuantErr: 12.95953 batch_time=0.91891
Train Epoch: 13 codebook_update_time=8.39343
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch13.pth ...
Done in 3.961s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch13.pth ...
Done in 7.969s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 4.597233952522278
quant_reg : 12.57221310043335
quant_err : 12.57221310043335
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
MSRVTT_full_val/t2v_metrics/R5: 66.19718309859155
MSRVTT_full_val/t2v_metrics/R10: 79.67806841046277
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.3158953722334
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.71038405742207
MSRVTT_full_val/v2t_metrics/R1: 35.010060362173036
MSRVTT_full_val/v2t_metrics/R5: 74.44668008048289
MSRVTT_full_val/v2t_metrics/R10: 86.9215291750503
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.217303822937626
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.961435393985035
MSRVTT_full_test/t2v_metrics/R1: 12.073578595317725
MSRVTT_full_test/t2v_metrics/R5: 34.280936454849495
MSRVTT_full_test/t2v_metrics/R10: 48.16053511705686
MSRVTT_full_test/t2v_metrics/R50: 80.2675585284281
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 44.71438127090301
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.113983752518088
MSRVTT_full_test/v2t_metrics/R1: 14.949832775919733
MSRVTT_full_test/v2t_metrics/R5: 40.936454849498325
MSRVTT_full_test/v2t_metrics/R10: 54.74916387959866
MSRVTT_full_test/v2t_metrics/R50: 85.88628762541806
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 32.39397993311037
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.238489494147835
mnt_best : 27.113983752518088
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 4.18321 (QuantReg: 12.61503) QuantErr: 12.61503 batch_time=30.70846
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 4.34374 (QuantReg: 12.55787) QuantErr: 12.55787 batch_time=0.93159
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 5.53234 (QuantReg: 12.57137) QuantErr: 12.57137 batch_time=0.96716
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 4.07511 (QuantReg: 12.52374) QuantErr: 12.52374 batch_time=0.94863
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 5.25944 (QuantReg: 12.53706) QuantErr: 12.53706 batch_time=0.93494
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 3.98499 (QuantReg: 12.85023) QuantErr: 12.85023 batch_time=0.98985
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 4.06028 (QuantReg: 12.60535) QuantErr: 12.60535 batch_time=0.92848
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 4.89611 (QuantReg: 12.53024) QuantErr: 12.53024 batch_time=0.95794
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 4.51865 (QuantReg: 12.58549) QuantErr: 12.58549 batch_time=0.95298
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 4.01629 (QuantReg: 12.55242) QuantErr: 12.55242 batch_time=0.95371
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 4.51318 (QuantReg: 12.91124) QuantErr: 12.91124 batch_time=0.96216
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 4.73008 (QuantReg: 12.79242) QuantErr: 12.79242 batch_time=0.91277
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 4.21684 (QuantReg: 12.79474) QuantErr: 12.79474 batch_time=0.93477
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 3.77348 (QuantReg: 12.72577) QuantErr: 12.72577 batch_time=1.29397
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 4.18338 (QuantReg: 12.60168) QuantErr: 12.60168 batch_time=0.92915
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 4.93646 (QuantReg: 12.55016) QuantErr: 12.55016 batch_time=0.95854
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 5.80845 (QuantReg: 12.64253) QuantErr: 12.64253 batch_time=0.94810
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 3.74870 (QuantReg: 12.69537) QuantErr: 12.69537 batch_time=0.94096
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 4.87714 (QuantReg: 12.90978) QuantErr: 12.90978 batch_time=0.95843
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 4.62157 (QuantReg: 13.14879) QuantErr: 13.14879 batch_time=1.05123
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 3.88547 (QuantReg: 12.69516) QuantErr: 12.69516 batch_time=0.99425
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 3.87122 (QuantReg: 12.85442) QuantErr: 12.85442 batch_time=1.07072
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 4.73973 (QuantReg: 12.64051) QuantErr: 12.64051 batch_time=0.98269
Train Epoch: 14 codebook_update_time=8.55275
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch14.pth ...
Done in 4.946s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch14.pth ...
Done in 9.734s
removing stale ckpt [epoch 13] [took 0.01s]
epoch : 14
loss : 4.351657031059265
quant_reg : 12.674668968200683
quant_err : 12.674668968200683
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_full_val/t2v_metrics/R1: 29.979879275653925
MSRVTT_full_val/t2v_metrics/R5: 68.00804828973843
MSRVTT_full_val/t2v_metrics/R10: 81.48893360160966
MSRVTT_full_val/t2v_metrics/R50: 98.59154929577464
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.501006036217304
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.97470934587136
MSRVTT_full_val/v2t_metrics/R1: 35.010060362173036
MSRVTT_full_val/v2t_metrics/R5: 75.25150905432595
MSRVTT_full_val/v2t_metrics/R10: 86.72032193158954
MSRVTT_full_val/v2t_metrics/R50: 98.79275653923541
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.0241448692152915
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.13308580383543
MSRVTT_full_test/t2v_metrics/R1: 12.341137123745819
MSRVTT_full_test/t2v_metrics/R5: 35.51839464882943
MSRVTT_full_test/t2v_metrics/R10: 49.46488294314381
MSRVTT_full_test/t2v_metrics/R50: 81.80602006688963
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 40.592976588628765
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.884861856553243
MSRVTT_full_test/v2t_metrics/R1: 14.882943143812708
MSRVTT_full_test/v2t_metrics/R5: 41.83946488294314
MSRVTT_full_test/v2t_metrics/R10: 57.391304347826086
MSRVTT_full_test/v2t_metrics/R50: 86.32107023411372
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 31.680602006688964
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.938742344533715
mnt_best : 27.884861856553243
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 5.04878 (QuantReg: 12.50628) QuantErr: 12.50628 batch_time=31.98899
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 3.57446 (QuantReg: 12.54799) QuantErr: 12.54799 batch_time=0.97522
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 4.64760 (QuantReg: 11.79791) QuantErr: 11.79791 batch_time=0.95307
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 4.00087 (QuantReg: 12.48689) QuantErr: 12.48689 batch_time=0.98661
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 4.45568 (QuantReg: 12.49492) QuantErr: 12.49492 batch_time=1.90651
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 3.98762 (QuantReg: 12.49505) QuantErr: 12.49505 batch_time=0.95453
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 4.98318 (QuantReg: 12.32763) QuantErr: 12.32763 batch_time=1.91488
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 3.38377 (QuantReg: 12.49215) QuantErr: 12.49215 batch_time=0.93000
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 4.21686 (QuantReg: 12.72703) QuantErr: 12.72703 batch_time=0.99162
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 3.86768 (QuantReg: 12.75177) QuantErr: 12.75177 batch_time=1.29674
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 4.88497 (QuantReg: 12.40245) QuantErr: 12.40245 batch_time=1.00068
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 4.59677 (QuantReg: 12.58688) QuantErr: 12.58688 batch_time=0.92120
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 4.41302 (QuantReg: 12.33710) QuantErr: 12.33710 batch_time=0.92702
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 4.13455 (QuantReg: 12.83620) QuantErr: 12.83620 batch_time=1.04996
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 4.83016 (QuantReg: 12.63107) QuantErr: 12.63107 batch_time=0.94107
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 4.17596 (QuantReg: 12.89760) QuantErr: 12.89760 batch_time=0.91952
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 3.65632 (QuantReg: 12.65151) QuantErr: 12.65151 batch_time=0.95254
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 3.83211 (QuantReg: 12.45059) QuantErr: 12.45059 batch_time=0.93109
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 3.72594 (QuantReg: 12.68264) QuantErr: 12.68264 batch_time=1.01194
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 4.72612 (QuantReg: 12.62761) QuantErr: 12.62761 batch_time=0.95113
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 5.14468 (QuantReg: 12.77897) QuantErr: 12.77897 batch_time=0.95056
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 4.37039 (QuantReg: 12.59895) QuantErr: 12.59895 batch_time=0.92612
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 4.38240 (QuantReg: 12.56903) QuantErr: 12.56903 batch_time=0.92823
Train Epoch: 15 codebook_update_time=9.23750
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L31/checkpoint-epoch15.pth ...
Done in 15.236s
removing stale ckpt [epoch 14] [took 0.00s]
epoch : 15
loss : 4.252684841156006
quant_reg : 12.622312129974365
quant_err : 12.622312129974365
learning_rate : 2.4383748955776477e-05