-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_t0.03.txt
2597 lines (2597 loc) · 194 KB
/
HCQ_MSRVTT_1kA_t0.03.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 974.5945658683777 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 87.90278005599976 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 63.44787096977234 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch0.pth ...
Done in 1.836s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch0.pth ...
Done in 4.031s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 4.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 486.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 496.278
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 1.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 6.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 509.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 503.537
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.5192494101851104
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 10.02876 (QuantReg: 22.49989) QuantErr: 22.49989 batch_time=32.58669
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.02484 (QuantReg: 22.41601) QuantErr: 22.41601 batch_time=0.54696
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.71158 (QuantReg: 22.53222) QuantErr: 22.53222 batch_time=0.56557
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.39254 (QuantReg: 22.49412) QuantErr: 22.49412 batch_time=0.52268
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.48074 (QuantReg: 22.53743) QuantErr: 22.53743 batch_time=0.57367
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 5.80397 (QuantReg: 22.53082) QuantErr: 22.53082 batch_time=0.58207
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.86857 (QuantReg: 22.50875) QuantErr: 22.50875 batch_time=0.62730
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 4.93898 (QuantReg: 22.52799) QuantErr: 22.52799 batch_time=0.55227
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.38216 (QuantReg: 22.52645) QuantErr: 22.52645 batch_time=0.59547
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.29039 (QuantReg: 22.52378) QuantErr: 22.52378 batch_time=0.53073
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 4.72509 (QuantReg: 22.54294) QuantErr: 22.54294 batch_time=0.54195
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 4.66611 (QuantReg: 22.56499) QuantErr: 22.56499 batch_time=0.59734
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.02625 (QuantReg: 22.53927) QuantErr: 22.53927 batch_time=3.84415
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.20085 (QuantReg: 22.57690) QuantErr: 22.57690 batch_time=0.57553
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.69292 (QuantReg: 22.52621) QuantErr: 22.52621 batch_time=0.56940
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.05884 (QuantReg: 22.55178) QuantErr: 22.55178 batch_time=0.60380
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.79348 (QuantReg: 22.52709) QuantErr: 22.52709 batch_time=0.83168
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.55102 (QuantReg: 22.51549) QuantErr: 22.51549 batch_time=0.58941
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.14414 (QuantReg: 22.51702) QuantErr: 22.51702 batch_time=0.53123
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.41579 (QuantReg: 22.56558) QuantErr: 22.56558 batch_time=0.55037
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 3.56396 (QuantReg: 22.60607) QuantErr: 22.60607 batch_time=0.53757
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.08674 (QuantReg: 22.57864) QuantErr: 22.57864 batch_time=0.53112
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.72783 (QuantReg: 22.61997) QuantErr: 22.61997 batch_time=0.56769
Train Epoch: 1 codebook_update_time=2.34507
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch1.pth ...
Done in 4.390s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch1.pth ...
Done in 16.115s
epoch : 1
loss : 5.365193214416504
quant_reg : 22.532890480041505
quant_err : 22.532890480041505
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 9.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 31.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 46.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 78.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 12.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 43.075
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.453195662947756
MSRVTT_jsfusion_test/v2t_metrics/R1: 10.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 32.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 46.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 78.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 12.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 40.0315
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.09479953422108
mnt_best : 24.453195662947756
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.15948 (QuantReg: 9.01939) QuantErr: 9.01939 batch_time=39.54104
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 3.90411 (QuantReg: 8.98873) QuantErr: 8.98873 batch_time=0.56201
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.04051 (QuantReg: 9.68902) QuantErr: 9.68902 batch_time=0.52638
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.80909 (QuantReg: 9.58694) QuantErr: 9.58694 batch_time=0.57870
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.88675 (QuantReg: 10.31475) QuantErr: 10.31475 batch_time=0.62533
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.82945 (QuantReg: 10.18469) QuantErr: 10.18469 batch_time=0.52743
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.64430 (QuantReg: 9.99830) QuantErr: 9.99830 batch_time=0.52518
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.74978 (QuantReg: 10.43515) QuantErr: 10.43515 batch_time=0.52352
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.63979 (QuantReg: 10.70087) QuantErr: 10.70087 batch_time=0.81807
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.25690 (QuantReg: 10.08051) QuantErr: 10.08051 batch_time=0.55641
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.68227 (QuantReg: 10.37638) QuantErr: 10.37638 batch_time=0.52734
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.10885 (QuantReg: 10.76155) QuantErr: 10.76155 batch_time=0.55080
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.48231 (QuantReg: 10.85149) QuantErr: 10.85149 batch_time=0.51740
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 4.00083 (QuantReg: 10.86836) QuantErr: 10.86836 batch_time=0.58952
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.87478 (QuantReg: 11.01954) QuantErr: 11.01954 batch_time=0.61262
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.41617 (QuantReg: 11.19603) QuantErr: 11.19603 batch_time=0.68369
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.75723 (QuantReg: 11.05264) QuantErr: 11.05264 batch_time=0.56341
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.51556 (QuantReg: 11.51578) QuantErr: 11.51578 batch_time=0.57777
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.24760 (QuantReg: 11.75384) QuantErr: 11.75384 batch_time=0.54603
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.23895 (QuantReg: 11.88455) QuantErr: 11.88455 batch_time=0.53621
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.40373 (QuantReg: 11.77323) QuantErr: 11.77323 batch_time=0.50470
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.53068 (QuantReg: 11.61518) QuantErr: 11.61518 batch_time=0.56016
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.67777 (QuantReg: 11.68677) QuantErr: 11.68677 batch_time=5.20641
Train Epoch: 2 codebook_update_time=1.83275
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch2.pth ...
Done in 12.701s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch2.pth ...
Done in 17.133s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 3.6479547595977784
quant_reg : 10.694982028961181
quant_err : 10.694982028961181
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 11.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 37.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 52.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 82.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 36.695
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.609824326315852
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 39.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 53.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 82.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 32.8465
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.55621238539634
mnt_best : 28.609824326315852
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.20546 (QuantReg: 9.53543) QuantErr: 9.53543 batch_time=38.77961
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.07566 (QuantReg: 9.95069) QuantErr: 9.95069 batch_time=0.51554
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.42723 (QuantReg: 9.87097) QuantErr: 9.87097 batch_time=0.58399
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 2.84642 (QuantReg: 10.07668) QuantErr: 10.07668 batch_time=0.54514
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 2.85343 (QuantReg: 10.37559) QuantErr: 10.37559 batch_time=0.57817
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 2.42143 (QuantReg: 10.43814) QuantErr: 10.43814 batch_time=0.51946
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.65707 (QuantReg: 10.29533) QuantErr: 10.29533 batch_time=0.53576
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.46102 (QuantReg: 10.59333) QuantErr: 10.59333 batch_time=0.57251
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.38909 (QuantReg: 10.20278) QuantErr: 10.20278 batch_time=0.57181
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 2.55980 (QuantReg: 10.65861) QuantErr: 10.65861 batch_time=0.54250
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.46119 (QuantReg: 10.54830) QuantErr: 10.54830 batch_time=0.57097
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.04103 (QuantReg: 10.44193) QuantErr: 10.44193 batch_time=0.58587
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 2.78629 (QuantReg: 10.52778) QuantErr: 10.52778 batch_time=0.54315
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.29568 (QuantReg: 10.51325) QuantErr: 10.51325 batch_time=0.75789
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.44907 (QuantReg: 10.71007) QuantErr: 10.71007 batch_time=0.51326
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.73783 (QuantReg: 11.01702) QuantErr: 11.01702 batch_time=0.55896
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.20051 (QuantReg: 11.09759) QuantErr: 11.09759 batch_time=0.55871
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 2.76483 (QuantReg: 11.06096) QuantErr: 11.06096 batch_time=0.50683
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 2.87171 (QuantReg: 10.85855) QuantErr: 10.85855 batch_time=0.58984
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 2.95508 (QuantReg: 11.19415) QuantErr: 11.19415 batch_time=4.47558
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 2.98337 (QuantReg: 11.17523) QuantErr: 11.17523 batch_time=0.57523
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.95431 (QuantReg: 11.18961) QuantErr: 11.18961 batch_time=0.52919
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.10481 (QuantReg: 11.12600) QuantErr: 11.12600 batch_time=0.51761
Train Epoch: 3 codebook_update_time=1.92719
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch3.pth ...
Done in 4.298s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch3.pth ...
Done in 8.483s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 3.078625859260559
quant_reg : 10.618713760375977
quant_err : 10.618713760375977
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 41.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 55.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.546
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.085669660685646
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 41.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 57.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 30.2825
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.06873637742349
mnt_best : 33.085669660685646
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 2.91472 (QuantReg: 10.22027) QuantErr: 10.22027 batch_time=35.23962
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.73816 (QuantReg: 10.55529) QuantErr: 10.55529 batch_time=2.95171
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.65411 (QuantReg: 10.60437) QuantErr: 10.60437 batch_time=1.09941
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.56435 (QuantReg: 10.80359) QuantErr: 10.80359 batch_time=0.52944
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.63032 (QuantReg: 11.03859) QuantErr: 11.03859 batch_time=0.54423
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.85089 (QuantReg: 10.73528) QuantErr: 10.73528 batch_time=0.53849
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 2.52292 (QuantReg: 10.60240) QuantErr: 10.60240 batch_time=1.79437
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.72803 (QuantReg: 10.77089) QuantErr: 10.77089 batch_time=0.58800
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.97776 (QuantReg: 10.61660) QuantErr: 10.61660 batch_time=0.55457
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.36617 (QuantReg: 11.06787) QuantErr: 11.06787 batch_time=0.57564
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.73273 (QuantReg: 11.65799) QuantErr: 11.65799 batch_time=0.55675
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.46418 (QuantReg: 11.64611) QuantErr: 11.64611 batch_time=0.59025
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.95755 (QuantReg: 11.17359) QuantErr: 11.17359 batch_time=1.02115
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.58087 (QuantReg: 11.39120) QuantErr: 11.39120 batch_time=1.43249
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.54330 (QuantReg: 11.37361) QuantErr: 11.37361 batch_time=0.51879
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.78453 (QuantReg: 11.30855) QuantErr: 11.30855 batch_time=0.56583
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.68865 (QuantReg: 11.23495) QuantErr: 11.23495 batch_time=0.59132
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.61322 (QuantReg: 11.38949) QuantErr: 11.38949 batch_time=0.55206
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.43275 (QuantReg: 11.62392) QuantErr: 11.62392 batch_time=0.55156
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.30748 (QuantReg: 11.57031) QuantErr: 11.57031 batch_time=1.07580
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.30041 (QuantReg: 11.85154) QuantErr: 11.85154 batch_time=0.57325
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.12594 (QuantReg: 12.10512) QuantErr: 12.10512 batch_time=0.53210
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.70719 (QuantReg: 11.67930) QuantErr: 11.67930 batch_time=0.53677
Train Epoch: 4 codebook_update_time=2.23408
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch4.pth ...
Done in 11.351s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch4.pth ...
Done in 15.449s
removing stale ckpt [epoch 3] [took 0.01s]
epoch : 4
loss : 2.7689069385528566
quant_reg : 11.086144584655761
quant_err : 11.086144584655761
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.958
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.66765110199053
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 44.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 30.1165
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.415266803878396
mnt_best : 35.66765110199053
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.82022 (QuantReg: 10.71519) QuantErr: 10.71519 batch_time=35.89126
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 3.08755 (QuantReg: 11.03462) QuantErr: 11.03462 batch_time=0.52235
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.08917 (QuantReg: 11.04524) QuantErr: 11.04524 batch_time=0.56189
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.41003 (QuantReg: 11.10707) QuantErr: 11.10707 batch_time=0.57086
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.59800 (QuantReg: 11.20139) QuantErr: 11.20139 batch_time=0.51765
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.24213 (QuantReg: 11.29139) QuantErr: 11.29139 batch_time=0.55152
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.59154 (QuantReg: 11.11478) QuantErr: 11.11478 batch_time=0.51206
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.97436 (QuantReg: 10.70343) QuantErr: 10.70343 batch_time=0.50950
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.44099 (QuantReg: 11.47832) QuantErr: 11.47832 batch_time=0.56748
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.47591 (QuantReg: 11.13460) QuantErr: 11.13460 batch_time=1.22114
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.54874 (QuantReg: 11.51268) QuantErr: 11.51268 batch_time=0.53850
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.58132 (QuantReg: 11.29262) QuantErr: 11.29262 batch_time=0.57277
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.42397 (QuantReg: 11.73921) QuantErr: 11.73921 batch_time=0.54611
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.07607 (QuantReg: 11.45949) QuantErr: 11.45949 batch_time=0.53312
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.76115 (QuantReg: 11.49100) QuantErr: 11.49100 batch_time=0.56306
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.30961 (QuantReg: 11.74481) QuantErr: 11.74481 batch_time=0.96866
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.58854 (QuantReg: 11.45506) QuantErr: 11.45506 batch_time=0.60561
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.48894 (QuantReg: 11.58608) QuantErr: 11.58608 batch_time=0.51244
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.54435 (QuantReg: 11.84856) QuantErr: 11.84856 batch_time=1.52292
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.49249 (QuantReg: 11.74698) QuantErr: 11.74698 batch_time=0.54468
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.75506 (QuantReg: 11.72796) QuantErr: 11.72796 batch_time=0.50715
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.30029 (QuantReg: 11.76689) QuantErr: 11.76689 batch_time=0.57996
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.77683 (QuantReg: 11.80853) QuantErr: 11.80853 batch_time=0.56209
Train Epoch: 5 codebook_update_time=2.18563
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch5.pth ...
Done in 4.092s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 2.49213094329834
quant_reg : 11.47167614364624
quant_err : 11.47167614364624
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.574
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.45639451954613
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.8905
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.45839608181488
mnt_best : 35.66765110199053
not_improved_count: 1
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.62073 (QuantReg: 11.01402) QuantErr: 11.01402 batch_time=36.76708
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.63222 (QuantReg: 10.89770) QuantErr: 10.89770 batch_time=0.59714
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.40122 (QuantReg: 11.11276) QuantErr: 11.11276 batch_time=0.51079
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.34581 (QuantReg: 11.29353) QuantErr: 11.29353 batch_time=0.52478
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.16314 (QuantReg: 11.23720) QuantErr: 11.23720 batch_time=0.53330
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.14548 (QuantReg: 11.54247) QuantErr: 11.54247 batch_time=0.53507
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.69151 (QuantReg: 11.80203) QuantErr: 11.80203 batch_time=0.52602
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.35417 (QuantReg: 11.85767) QuantErr: 11.85767 batch_time=0.51763
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.34696 (QuantReg: 11.79883) QuantErr: 11.79883 batch_time=0.51486
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 1.99615 (QuantReg: 11.55888) QuantErr: 11.55888 batch_time=0.54658
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.17716 (QuantReg: 12.05142) QuantErr: 12.05142 batch_time=0.50402
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.21294 (QuantReg: 11.48416) QuantErr: 11.48416 batch_time=0.54908
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.12656 (QuantReg: 11.98695) QuantErr: 11.98695 batch_time=0.51451
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.18414 (QuantReg: 11.46620) QuantErr: 11.46620 batch_time=0.56674
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.48107 (QuantReg: 12.08366) QuantErr: 12.08366 batch_time=0.57102
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.32993 (QuantReg: 11.87683) QuantErr: 11.87683 batch_time=0.52454
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.12433 (QuantReg: 12.20901) QuantErr: 12.20901 batch_time=0.57461
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.44163 (QuantReg: 11.90924) QuantErr: 11.90924 batch_time=0.57434
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.55282 (QuantReg: 12.12537) QuantErr: 12.12537 batch_time=0.54566
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.40019 (QuantReg: 12.09247) QuantErr: 12.09247 batch_time=0.59841
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.16869 (QuantReg: 12.43141) QuantErr: 12.43141 batch_time=0.53680
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.01935 (QuantReg: 12.04833) QuantErr: 12.04833 batch_time=0.61289
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 1.93225 (QuantReg: 12.24376) QuantErr: 12.24376 batch_time=0.56480
Train Epoch: 6 codebook_update_time=1.96785
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch6.pth ...
Done in 7.882s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch6.pth ...
Done in 12.407s
removing stale ckpt [epoch 5] [took 0.01s]
epoch : 6
loss : 2.3039336981773375
quant_reg : 11.816265438079833
quant_err : 11.816265438079833
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.971
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.34068017259891
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.3655
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.037598498179875
mnt_best : 37.34068017259891
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.32304 (QuantReg: 11.38595) QuantErr: 11.38595 batch_time=36.55312
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.32813 (QuantReg: 11.76931) QuantErr: 11.76931 batch_time=0.54239
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.07772 (QuantReg: 11.26183) QuantErr: 11.26183 batch_time=0.57973
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 1.87193 (QuantReg: 11.80820) QuantErr: 11.80820 batch_time=0.55405
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.00332 (QuantReg: 11.79894) QuantErr: 11.79894 batch_time=0.54201
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.08098 (QuantReg: 12.18635) QuantErr: 12.18635 batch_time=0.51569
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.21801 (QuantReg: 11.80910) QuantErr: 11.80910 batch_time=0.57894
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.07873 (QuantReg: 11.89858) QuantErr: 11.89858 batch_time=0.53553
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.16845 (QuantReg: 12.23880) QuantErr: 12.23880 batch_time=1.60733
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 1.78231 (QuantReg: 12.03247) QuantErr: 12.03247 batch_time=0.53623
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.63163 (QuantReg: 11.90342) QuantErr: 11.90342 batch_time=0.53223
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.04459 (QuantReg: 12.22876) QuantErr: 12.22876 batch_time=0.51668
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.03950 (QuantReg: 12.26177) QuantErr: 12.26177 batch_time=0.57657
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.23844 (QuantReg: 12.20332) QuantErr: 12.20332 batch_time=0.54846
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 1.56490 (QuantReg: 12.05758) QuantErr: 12.05758 batch_time=0.50817
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.19063 (QuantReg: 12.36848) QuantErr: 12.36848 batch_time=0.54485
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.71944 (QuantReg: 12.60380) QuantErr: 12.60380 batch_time=0.62334
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.31399 (QuantReg: 12.26746) QuantErr: 12.26746 batch_time=0.56850
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.61262 (QuantReg: 12.51473) QuantErr: 12.51473 batch_time=0.73434
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.48342 (QuantReg: 12.72585) QuantErr: 12.72585 batch_time=0.86154
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.01662 (QuantReg: 12.63237) QuantErr: 12.63237 batch_time=0.51484
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.09251 (QuantReg: 12.94096) QuantErr: 12.94096 batch_time=0.51125
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.16048 (QuantReg: 12.61533) QuantErr: 12.61533 batch_time=0.58742
Train Epoch: 7 codebook_update_time=1.78637
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch7.pth ...
Done in 4.456s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch7.pth ...
Done in 8.801s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 2.1160386476516724
quant_reg : 12.179861724853515
quant_err : 12.179861724853515
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.494
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.80359418246868
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.1965
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.93717689735291
mnt_best : 37.80359418246868
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.27425 (QuantReg: 12.02553) QuantErr: 12.02553 batch_time=35.02597
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.39587 (QuantReg: 12.08929) QuantErr: 12.08929 batch_time=0.55437
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.53578 (QuantReg: 12.16206) QuantErr: 12.16206 batch_time=4.01041
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.01781 (QuantReg: 11.88329) QuantErr: 11.88329 batch_time=0.52609
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.05025 (QuantReg: 12.40365) QuantErr: 12.40365 batch_time=0.53604
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 1.96983 (QuantReg: 12.08589) QuantErr: 12.08589 batch_time=0.56083
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 1.99371 (QuantReg: 12.42113) QuantErr: 12.42113 batch_time=0.57749
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.23675 (QuantReg: 12.55848) QuantErr: 12.55848 batch_time=0.54684
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.27441 (QuantReg: 12.34791) QuantErr: 12.34791 batch_time=0.51942
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.44005 (QuantReg: 12.49910) QuantErr: 12.49910 batch_time=0.55133
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.31231 (QuantReg: 12.49706) QuantErr: 12.49706 batch_time=1.03875
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.04941 (QuantReg: 12.34741) QuantErr: 12.34741 batch_time=0.50096
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.20833 (QuantReg: 12.40288) QuantErr: 12.40288 batch_time=0.51941
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.29362 (QuantReg: 12.87201) QuantErr: 12.87201 batch_time=0.54432
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.01464 (QuantReg: 12.87212) QuantErr: 12.87212 batch_time=0.60717
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.76018 (QuantReg: 12.23674) QuantErr: 12.23674 batch_time=0.57290
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 1.86714 (QuantReg: 12.42446) QuantErr: 12.42446 batch_time=0.57117
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.61638 (QuantReg: 12.47894) QuantErr: 12.47894 batch_time=0.56310
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.12674 (QuantReg: 12.50910) QuantErr: 12.50910 batch_time=0.56661
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.15788 (QuantReg: 12.25735) QuantErr: 12.25735 batch_time=2.04117
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.13896 (QuantReg: 12.68444) QuantErr: 12.68444 batch_time=0.57952
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.56086 (QuantReg: 12.76606) QuantErr: 12.76606 batch_time=0.54923
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.13084 (QuantReg: 12.78443) QuantErr: 12.78443 batch_time=0.85770
Train Epoch: 8 codebook_update_time=2.08814
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch8.pth ...
Done in 4.304s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch8.pth ...
Done in 8.647s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 2.0277540111541748
quant_reg : 12.439874855041504
quant_err : 12.439874855041504
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.419
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.072724265521686
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.7
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.24025847198873
mnt_best : 39.072724265521686
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 1.66497 (QuantReg: 12.73631) QuantErr: 12.73631 batch_time=36.96553
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 2.10859 (QuantReg: 12.39280) QuantErr: 12.39280 batch_time=0.55532
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 1.64059 (QuantReg: 12.30936) QuantErr: 12.30936 batch_time=0.58717
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.10116 (QuantReg: 12.68623) QuantErr: 12.68623 batch_time=0.57019
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 1.91623 (QuantReg: 12.75368) QuantErr: 12.75368 batch_time=0.55402
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.63554 (QuantReg: 12.78493) QuantErr: 12.78493 batch_time=0.55276
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.63013 (QuantReg: 12.99095) QuantErr: 12.99095 batch_time=0.57482
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.88760 (QuantReg: 12.82943) QuantErr: 12.82943 batch_time=0.52108
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 1.96522 (QuantReg: 12.31941) QuantErr: 12.31941 batch_time=0.53443
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.81930 (QuantReg: 12.77526) QuantErr: 12.77526 batch_time=0.55033
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 1.89122 (QuantReg: 12.57127) QuantErr: 12.57127 batch_time=0.54524
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.58609 (QuantReg: 12.69921) QuantErr: 12.69921 batch_time=0.55073
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.93962 (QuantReg: 12.62312) QuantErr: 12.62312 batch_time=0.57839
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.76967 (QuantReg: 12.80263) QuantErr: 12.80263 batch_time=0.55752
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 1.98790 (QuantReg: 12.43595) QuantErr: 12.43595 batch_time=0.52331
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.62637 (QuantReg: 13.16329) QuantErr: 13.16329 batch_time=0.51669
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.99571 (QuantReg: 13.13731) QuantErr: 13.13731 batch_time=0.55473
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.31479 (QuantReg: 12.98221) QuantErr: 12.98221 batch_time=0.59082
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.83845 (QuantReg: 12.78280) QuantErr: 12.78280 batch_time=0.56060
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.94772 (QuantReg: 12.83359) QuantErr: 12.83359 batch_time=0.58386
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 1.62711 (QuantReg: 13.14478) QuantErr: 13.14478 batch_time=0.55044
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.76429 (QuantReg: 12.58054) QuantErr: 12.58054 batch_time=0.55695
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.75685 (QuantReg: 12.53851) QuantErr: 12.53851 batch_time=0.51949
Train Epoch: 9 codebook_update_time=1.93809
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch9.pth ...
Done in 6.343s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch9.pth ...
Done in 11.307s
removing stale ckpt [epoch 8] [took 0.14s]
epoch : 9
loss : 1.8858279175758361
quant_reg : 12.679026329040527
quant_err : 12.679026329040527
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.195
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.90413961987165
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.379
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.89689110895558
mnt_best : 39.90413961987165
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.83536 (QuantReg: 12.70021) QuantErr: 12.70021 batch_time=38.36347
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.83956 (QuantReg: 12.53177) QuantErr: 12.53177 batch_time=0.54514
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.22245 (QuantReg: 12.90121) QuantErr: 12.90121 batch_time=0.57356
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.84333 (QuantReg: 12.40102) QuantErr: 12.40102 batch_time=0.54592
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.50370 (QuantReg: 12.52585) QuantErr: 12.52585 batch_time=0.55053
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 1.82718 (QuantReg: 12.83668) QuantErr: 12.83668 batch_time=0.56123
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 2.30870 (QuantReg: 12.23475) QuantErr: 12.23475 batch_time=0.53404
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.91640 (QuantReg: 12.71163) QuantErr: 12.71163 batch_time=0.59947
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.89934 (QuantReg: 12.49147) QuantErr: 12.49147 batch_time=0.53122
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 1.90205 (QuantReg: 12.75770) QuantErr: 12.75770 batch_time=0.61291
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.56679 (QuantReg: 12.75045) QuantErr: 12.75045 batch_time=0.56679
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 1.79643 (QuantReg: 12.95070) QuantErr: 12.95070 batch_time=0.52563
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.88890 (QuantReg: 12.63148) QuantErr: 12.63148 batch_time=2.36601
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 1.36613 (QuantReg: 13.34169) QuantErr: 13.34169 batch_time=0.58636
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.79354 (QuantReg: 12.66322) QuantErr: 12.66322 batch_time=0.55372
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.99482 (QuantReg: 13.12137) QuantErr: 13.12137 batch_time=0.56936
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 1.89505 (QuantReg: 12.93789) QuantErr: 12.93789 batch_time=0.54223
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.66087 (QuantReg: 13.18338) QuantErr: 13.18338 batch_time=0.53478
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.34416 (QuantReg: 13.16018) QuantErr: 13.16018 batch_time=0.56131
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.43864 (QuantReg: 13.52051) QuantErr: 13.52051 batch_time=2.47101
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.78009 (QuantReg: 12.96513) QuantErr: 12.96513 batch_time=0.55855
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 1.96358 (QuantReg: 12.96768) QuantErr: 12.96768 batch_time=0.60246
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.95551 (QuantReg: 13.28050) QuantErr: 13.28050 batch_time=0.54156
Train Epoch: 10 codebook_update_time=2.11356
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch10.pth ...
Done in 19.957s
removing stale ckpt [epoch 9] [took 0.22s]
epoch : 10
loss : 1.786734960079193
quant_reg : 12.883144535064698
quant_err : 12.883144535064698
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.442
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.674754214065665
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.9315
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.01297427979136
mnt_best : 39.90413961987165
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.70539 (QuantReg: 12.60551) QuantErr: 12.60551 batch_time=37.81386
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 2.01646 (QuantReg: 12.52336) QuantErr: 12.52336 batch_time=0.50691
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 2.13373 (QuantReg: 12.82700) QuantErr: 12.82700 batch_time=0.58958
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 1.85542 (QuantReg: 12.97635) QuantErr: 12.97635 batch_time=0.51572
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 2.00358 (QuantReg: 12.85028) QuantErr: 12.85028 batch_time=0.60305
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 2.28389 (QuantReg: 13.56890) QuantErr: 13.56890 batch_time=0.53621
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.51295 (QuantReg: 13.13612) QuantErr: 13.13612 batch_time=0.54031
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.96856 (QuantReg: 12.67210) QuantErr: 12.67210 batch_time=0.53309
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.63807 (QuantReg: 13.08327) QuantErr: 13.08327 batch_time=0.62876
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.60363 (QuantReg: 13.31747) QuantErr: 13.31747 batch_time=0.54200
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.46718 (QuantReg: 13.27476) QuantErr: 13.27476 batch_time=0.55468
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.51049 (QuantReg: 13.03330) QuantErr: 13.03330 batch_time=0.61087
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.95212 (QuantReg: 13.07162) QuantErr: 13.07162 batch_time=0.57307
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.67777 (QuantReg: 13.15534) QuantErr: 13.15534 batch_time=0.60981
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.35537 (QuantReg: 13.37844) QuantErr: 13.37844 batch_time=0.53759
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.20181 (QuantReg: 12.86350) QuantErr: 12.86350 batch_time=0.52003
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.60148 (QuantReg: 13.40428) QuantErr: 13.40428 batch_time=0.55271
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.63646 (QuantReg: 13.53162) QuantErr: 13.53162 batch_time=0.61080
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.75089 (QuantReg: 12.92929) QuantErr: 12.92929 batch_time=0.57052
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.59376 (QuantReg: 13.35964) QuantErr: 13.35964 batch_time=0.52033
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.68100 (QuantReg: 13.16617) QuantErr: 13.16617 batch_time=0.58324
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.71937 (QuantReg: 13.46618) QuantErr: 13.46618 batch_time=0.55167
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.64961 (QuantReg: 13.57228) QuantErr: 13.57228 batch_time=0.55008
Train Epoch: 11 codebook_update_time=1.89409
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch11.pth ...
Done in 5.939s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch11.pth ...
Done in 10.997s
removing stale ckpt [epoch 10] [took 0.21s]
epoch : 11
loss : 1.700813175201416
quant_reg : 13.151041416168212
quant_err : 13.151041416168212
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.144
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.932494108873875
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.1745
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.3541482499114
mnt_best : 40.932494108873875
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.53445 (QuantReg: 13.01768) QuantErr: 13.01768 batch_time=35.69703
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.61916 (QuantReg: 13.16354) QuantErr: 13.16354 batch_time=0.51212
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.73734 (QuantReg: 13.05319) QuantErr: 13.05319 batch_time=0.52837
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.94119 (QuantReg: 13.20445) QuantErr: 13.20445 batch_time=0.51996
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.55926 (QuantReg: 13.11301) QuantErr: 13.11301 batch_time=0.52876
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.61051 (QuantReg: 13.30202) QuantErr: 13.30202 batch_time=0.58337
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.65524 (QuantReg: 13.49623) QuantErr: 13.49623 batch_time=0.52819
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.77327 (QuantReg: 12.91685) QuantErr: 12.91685 batch_time=1.21759
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.73012 (QuantReg: 13.21727) QuantErr: 13.21727 batch_time=0.56185
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.45931 (QuantReg: 13.33764) QuantErr: 13.33764 batch_time=0.52760
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.50981 (QuantReg: 13.25101) QuantErr: 13.25101 batch_time=0.54439
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.40621 (QuantReg: 13.44068) QuantErr: 13.44068 batch_time=0.51108
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.48024 (QuantReg: 13.10738) QuantErr: 13.10738 batch_time=0.58516
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.60111 (QuantReg: 13.15454) QuantErr: 13.15454 batch_time=0.54235
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.97413 (QuantReg: 13.76062) QuantErr: 13.76062 batch_time=0.64146
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.54988 (QuantReg: 13.49907) QuantErr: 13.49907 batch_time=0.55800
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.79121 (QuantReg: 13.57594) QuantErr: 13.57594 batch_time=0.61103
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.27645 (QuantReg: 13.72522) QuantErr: 13.72522 batch_time=0.53724
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.85535 (QuantReg: 13.38056) QuantErr: 13.38056 batch_time=0.55190
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.33610 (QuantReg: 13.70567) QuantErr: 13.70567 batch_time=0.55182
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.26620 (QuantReg: 13.86208) QuantErr: 13.86208 batch_time=0.66088
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.72263 (QuantReg: 13.65953) QuantErr: 13.65953 batch_time=0.55849
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.46698 (QuantReg: 13.32895) QuantErr: 13.32895 batch_time=0.55228
Train Epoch: 12 codebook_update_time=1.89137
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch12.pth ...
Done in 5.316s
removing stale ckpt [epoch 11] [took 0.09s]
epoch : 12
loss : 1.6315488271713257
quant_reg : 13.299734371185302
quant_err : 13.299734371185302
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.827
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.86923967589396
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.023
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.05104607686846
mnt_best : 40.932494108873875
not_improved_count: 1
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.70500 (QuantReg: 12.91316) QuantErr: 12.91316 batch_time=36.17541
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.13531 (QuantReg: 13.67617) QuantErr: 13.67617 batch_time=0.57943
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.55083 (QuantReg: 13.26554) QuantErr: 13.26554 batch_time=0.58028
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.26453 (QuantReg: 13.59634) QuantErr: 13.59634 batch_time=0.56105
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.72978 (QuantReg: 13.54667) QuantErr: 13.54667 batch_time=0.54747
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.72887 (QuantReg: 13.08155) QuantErr: 13.08155 batch_time=0.53712
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.61747 (QuantReg: 13.71927) QuantErr: 13.71927 batch_time=5.99802
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.68305 (QuantReg: 13.76252) QuantErr: 13.76252 batch_time=0.58123
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.55235 (QuantReg: 13.70761) QuantErr: 13.70761 batch_time=0.53310
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.39758 (QuantReg: 13.50022) QuantErr: 13.50022 batch_time=0.53404
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.48009 (QuantReg: 13.56337) QuantErr: 13.56337 batch_time=0.52934
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.31939 (QuantReg: 13.73066) QuantErr: 13.73066 batch_time=0.54275
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.61343 (QuantReg: 13.34625) QuantErr: 13.34625 batch_time=0.58141
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.59275 (QuantReg: 13.62960) QuantErr: 13.62960 batch_time=0.56703
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.28837 (QuantReg: 13.67068) QuantErr: 13.67068 batch_time=0.54496
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.84870 (QuantReg: 13.47408) QuantErr: 13.47408 batch_time=0.55620
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.51005 (QuantReg: 13.77838) QuantErr: 13.77838 batch_time=0.52368
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.69052 (QuantReg: 13.49655) QuantErr: 13.49655 batch_time=0.58206
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.34874 (QuantReg: 13.69231) QuantErr: 13.69231 batch_time=0.52737
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.73655 (QuantReg: 13.50677) QuantErr: 13.50677 batch_time=0.55775
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.24038 (QuantReg: 13.98494) QuantErr: 13.98494 batch_time=0.51673
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.56166 (QuantReg: 13.66071) QuantErr: 13.66071 batch_time=0.54823
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.80718 (QuantReg: 13.53470) QuantErr: 13.53470 batch_time=0.58618
Train Epoch: 13 codebook_update_time=1.92827
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch13.pth ...
Done in 13.922s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch13.pth ...
Done in 19.775s
removing stale ckpt [epoch 12] [took 0.04s]
epoch : 13
loss : 1.5761362500190734
quant_reg : 13.515619247436524
quant_err : 13.515619247436524
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.643
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.19818463739063
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.4
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.98640830045152
mnt_best : 41.19818463739063
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.52009 (QuantReg: 13.61273) QuantErr: 13.61273 batch_time=35.88498
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.43019 (QuantReg: 13.42814) QuantErr: 13.42814 batch_time=0.61070
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.39790 (QuantReg: 13.59758) QuantErr: 13.59758 batch_time=0.56741
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.31997 (QuantReg: 13.64278) QuantErr: 13.64278 batch_time=0.57292
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.37733 (QuantReg: 13.62203) QuantErr: 13.62203 batch_time=0.75169
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.87899 (QuantReg: 13.20510) QuantErr: 13.20510 batch_time=0.57904
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.81115 (QuantReg: 13.58936) QuantErr: 13.58936 batch_time=0.52923
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.38216 (QuantReg: 13.52147) QuantErr: 13.52147 batch_time=0.53059
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.37040 (QuantReg: 13.21872) QuantErr: 13.21872 batch_time=0.56585
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.21307 (QuantReg: 13.59072) QuantErr: 13.59072 batch_time=0.55295
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.01638 (QuantReg: 13.47875) QuantErr: 13.47875 batch_time=0.59408
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.30608 (QuantReg: 13.70394) QuantErr: 13.70394 batch_time=0.57124
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.26445 (QuantReg: 13.85737) QuantErr: 13.85737 batch_time=0.55129
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.79142 (QuantReg: 13.38852) QuantErr: 13.38852 batch_time=0.63286
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.33578 (QuantReg: 13.71864) QuantErr: 13.71864 batch_time=0.56775
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.68312 (QuantReg: 13.58455) QuantErr: 13.58455 batch_time=0.53115
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.44554 (QuantReg: 13.89783) QuantErr: 13.89783 batch_time=0.55373
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.72181 (QuantReg: 13.63371) QuantErr: 13.63371 batch_time=0.55515
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.42007 (QuantReg: 13.90423) QuantErr: 13.90423 batch_time=0.56036
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.49241 (QuantReg: 13.51384) QuantErr: 13.51384 batch_time=2.42561
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.62587 (QuantReg: 13.58747) QuantErr: 13.58747 batch_time=0.51131
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.29473 (QuantReg: 14.39055) QuantErr: 14.39055 batch_time=0.53654
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.20252 (QuantReg: 14.05593) QuantErr: 14.05593 batch_time=0.51589
Train Epoch: 14 codebook_update_time=1.80481
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch14.pth ...
Done in 5.063s
removing stale ckpt [epoch 13] [took 0.02s]
epoch : 14
loss : 1.5027634556293488
quant_reg : 13.64565140914917
quant_err : 13.64565140914917
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.799
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.69725847672348
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.587
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.64141339077035
mnt_best : 41.19818463739063
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.34774 (QuantReg: 13.62905) QuantErr: 13.62905 batch_time=38.23665
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.05461 (QuantReg: 13.70122) QuantErr: 13.70122 batch_time=0.52268
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.46680 (QuantReg: 13.74487) QuantErr: 13.74487 batch_time=0.56074
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.30899 (QuantReg: 13.40222) QuantErr: 13.40222 batch_time=0.56247
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.72009 (QuantReg: 13.72574) QuantErr: 13.72574 batch_time=0.57811
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.64707 (QuantReg: 13.70954) QuantErr: 13.70954 batch_time=0.55647
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.69769 (QuantReg: 14.10766) QuantErr: 14.10766 batch_time=0.50683
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.21158 (QuantReg: 13.72325) QuantErr: 13.72325 batch_time=0.54997
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.81709 (QuantReg: 13.99048) QuantErr: 13.99048 batch_time=0.56787
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.29388 (QuantReg: 13.93869) QuantErr: 13.93869 batch_time=0.56459
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.73020 (QuantReg: 13.84831) QuantErr: 13.84831 batch_time=0.59608
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.23055 (QuantReg: 14.14728) QuantErr: 14.14728 batch_time=0.53909
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.54012 (QuantReg: 13.92504) QuantErr: 13.92504 batch_time=0.55154
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.43096 (QuantReg: 13.85972) QuantErr: 13.85972 batch_time=4.85674
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.50951 (QuantReg: 13.89357) QuantErr: 13.89357 batch_time=0.59436
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.44039 (QuantReg: 13.79960) QuantErr: 13.79960 batch_time=0.57761
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.30456 (QuantReg: 13.98984) QuantErr: 13.98984 batch_time=0.57089
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.57456 (QuantReg: 13.81552) QuantErr: 13.81552 batch_time=0.52419
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.73278 (QuantReg: 13.61856) QuantErr: 13.61856 batch_time=0.51676
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.23021 (QuantReg: 14.08494) QuantErr: 14.08494 batch_time=0.64938
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.27067 (QuantReg: 14.00265) QuantErr: 14.00265 batch_time=0.53881
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.59847 (QuantReg: 14.13858) QuantErr: 14.13858 batch_time=0.53097
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.29506 (QuantReg: 13.96204) QuantErr: 13.96204 batch_time=0.57143
Train Epoch: 15 codebook_update_time=1.78004
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch15.pth ...
Done in 4.826s
removing stale ckpt [epoch 14] [took 0.53s]
epoch : 15
loss : 1.4367010972499847
quant_reg : 13.824031967163085
quant_err : 13.824031967163085
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 90.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.68
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.077819681278584
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.54
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.15028361208397
mnt_best : 41.19818463739063
not_improved_count: 2
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.58314 (QuantReg: 13.57404) QuantErr: 13.57404 batch_time=33.22275
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.45882 (QuantReg: 13.45728) QuantErr: 13.45728 batch_time=0.51296
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.29190 (QuantReg: 13.66893) QuantErr: 13.66893 batch_time=1.06857
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 2.06734 (QuantReg: 13.30089) QuantErr: 13.30089 batch_time=0.52975
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.51636 (QuantReg: 14.17410) QuantErr: 14.17410 batch_time=0.57903
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.50781 (QuantReg: 14.04478) QuantErr: 14.04478 batch_time=0.55496
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.12108 (QuantReg: 14.08356) QuantErr: 14.08356 batch_time=0.54944
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.40914 (QuantReg: 13.49253) QuantErr: 13.49253 batch_time=0.55451
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.37045 (QuantReg: 13.77872) QuantErr: 13.77872 batch_time=0.55813
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.27423 (QuantReg: 14.07961) QuantErr: 14.07961 batch_time=0.52672
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.27409 (QuantReg: 14.05866) QuantErr: 14.05866 batch_time=0.61304
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.52296 (QuantReg: 13.95644) QuantErr: 13.95644 batch_time=0.56475
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.29619 (QuantReg: 13.88102) QuantErr: 13.88102 batch_time=0.58825
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.74360 (QuantReg: 13.87797) QuantErr: 13.87797 batch_time=0.65194
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.69343 (QuantReg: 14.13528) QuantErr: 14.13528 batch_time=0.54291
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.52658 (QuantReg: 14.12181) QuantErr: 14.12181 batch_time=0.57392
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.15566 (QuantReg: 14.33250) QuantErr: 14.33250 batch_time=0.58649
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.52734 (QuantReg: 14.04809) QuantErr: 14.04809 batch_time=0.54215
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.36897 (QuantReg: 14.22539) QuantErr: 14.22539 batch_time=2.78761
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.16164 (QuantReg: 14.39582) QuantErr: 14.39582 batch_time=0.51857
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.39936 (QuantReg: 13.99905) QuantErr: 13.99905 batch_time=1.64025
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.67899 (QuantReg: 13.73922) QuantErr: 13.73922 batch_time=0.54271
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.40495 (QuantReg: 13.91766) QuantErr: 13.91766 batch_time=0.52330
Train Epoch: 16 codebook_update_time=2.26603
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch16.pth ...
Done in 4.742s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch16.pth ...
Done in 9.252s
removing stale ckpt [epoch 15] [took 0.01s]
epoch : 16
loss : 1.440385494709015
quant_reg : 13.906950862884521
quant_err : 13.906950862884521
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 53.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.098
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.38222525399104
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.8975
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.35992713194375
mnt_best : 42.38222525399104
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.10318 (QuantReg: 14.06154) QuantErr: 14.06154 batch_time=43.94632
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.44537 (QuantReg: 13.42159) QuantErr: 13.42159 batch_time=0.52695
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.26150 (QuantReg: 13.71315) QuantErr: 13.71315 batch_time=0.58274
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 1.88307 (QuantReg: 14.04290) QuantErr: 14.04290 batch_time=0.64313
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.45130 (QuantReg: 13.41956) QuantErr: 13.41956 batch_time=0.57848
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.35015 (QuantReg: 13.99237) QuantErr: 13.99237 batch_time=0.53146
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.68745 (QuantReg: 14.13094) QuantErr: 14.13094 batch_time=0.57348
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.07520 (QuantReg: 13.94429) QuantErr: 13.94429 batch_time=0.53608
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.34414 (QuantReg: 13.75468) QuantErr: 13.75468 batch_time=0.58503
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.48135 (QuantReg: 13.92606) QuantErr: 13.92606 batch_time=0.56384
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.48128 (QuantReg: 14.10293) QuantErr: 14.10293 batch_time=0.55707
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.45398 (QuantReg: 13.90475) QuantErr: 13.90475 batch_time=0.54316
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.52957 (QuantReg: 13.91396) QuantErr: 13.91396 batch_time=0.51601
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.26917 (QuantReg: 14.21767) QuantErr: 14.21767 batch_time=0.56706
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.79160 (QuantReg: 13.65192) QuantErr: 13.65192 batch_time=0.53871
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.30078 (QuantReg: 14.02083) QuantErr: 14.02083 batch_time=0.54303
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.48624 (QuantReg: 14.26959) QuantErr: 14.26959 batch_time=0.50642
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.18070 (QuantReg: 14.42859) QuantErr: 14.42859 batch_time=0.55715
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.36447 (QuantReg: 13.71043) QuantErr: 13.71043 batch_time=0.56660
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.24516 (QuantReg: 14.32525) QuantErr: 14.32525 batch_time=0.53595
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.38712 (QuantReg: 13.88755) QuantErr: 13.88755 batch_time=0.53142
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.46635 (QuantReg: 14.47150) QuantErr: 14.47150 batch_time=0.57443
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.13654 (QuantReg: 14.06585) QuantErr: 14.06585 batch_time=0.54962
Train Epoch: 17 codebook_update_time=2.33719
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch17.pth ...
Done in 13.690s
removing stale ckpt [epoch 16] [took 0.02s]
epoch : 17
loss : 1.3733231134414672
quant_reg : 14.036452915191651
quant_err : 14.036452915191651
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.444
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.35237770582959
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.169
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.96746682889879
mnt_best : 42.38222525399104
not_improved_count: 1
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.36115 (QuantReg: 13.87973) QuantErr: 13.87973 batch_time=37.66380
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.27297 (QuantReg: 13.94315) QuantErr: 13.94315 batch_time=0.56498
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.21384 (QuantReg: 14.08607) QuantErr: 14.08607 batch_time=0.50116
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.14658 (QuantReg: 14.25710) QuantErr: 14.25710 batch_time=0.58928
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.38269 (QuantReg: 13.75348) QuantErr: 13.75348 batch_time=0.76755
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.08297 (QuantReg: 14.40557) QuantErr: 14.40557 batch_time=0.55168
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 0.97129 (QuantReg: 14.63864) QuantErr: 14.63864 batch_time=0.51070
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.09058 (QuantReg: 14.40231) QuantErr: 14.40231 batch_time=0.53525
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.63919 (QuantReg: 14.46728) QuantErr: 14.46728 batch_time=0.51116
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.23934 (QuantReg: 14.10114) QuantErr: 14.10114 batch_time=0.50575
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.55970 (QuantReg: 14.14750) QuantErr: 14.14750 batch_time=0.51051
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.34807 (QuantReg: 14.30202) QuantErr: 14.30202 batch_time=0.50460
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.41596 (QuantReg: 14.16864) QuantErr: 14.16864 batch_time=0.53856
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.54455 (QuantReg: 14.09618) QuantErr: 14.09618 batch_time=0.66397
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.50009 (QuantReg: 14.00326) QuantErr: 14.00326 batch_time=0.52205
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.37992 (QuantReg: 14.66383) QuantErr: 14.66383 batch_time=0.51160
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.22281 (QuantReg: 14.09934) QuantErr: 14.09934 batch_time=0.50520
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.10700 (QuantReg: 14.26558) QuantErr: 14.26558 batch_time=0.54364
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.26243 (QuantReg: 14.41823) QuantErr: 14.41823 batch_time=0.50729
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 0.84940 (QuantReg: 14.36943) QuantErr: 14.36943 batch_time=0.58243
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.72632 (QuantReg: 13.98377) QuantErr: 13.98377 batch_time=0.54946
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.18673 (QuantReg: 14.28427) QuantErr: 14.28427 batch_time=0.52406
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.21755 (QuantReg: 14.41697) QuantErr: 14.41697 batch_time=0.51905
Train Epoch: 18 codebook_update_time=1.92627
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch18.pth ...
Done in 5.850s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch18.pth ...
Done in 12.888s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 1.3159167428016663
quant_reg : 14.230369262695312
quant_err : 14.230369262695312
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 55.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 4.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.412
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.700739675698614
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.1475
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.60635533567132
mnt_best : 42.700739675698614
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.09609 (QuantReg: 14.33232) QuantErr: 14.33232 batch_time=37.22356
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.18782 (QuantReg: 14.35553) QuantErr: 14.35553 batch_time=0.53641
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.50137 (QuantReg: 13.90294) QuantErr: 13.90294 batch_time=0.54680
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.53360 (QuantReg: 14.19654) QuantErr: 14.19654 batch_time=0.51470
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.18148 (QuantReg: 14.18224) QuantErr: 14.18224 batch_time=0.51570
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 0.90561 (QuantReg: 14.41300) QuantErr: 14.41300 batch_time=0.52624
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.02796 (QuantReg: 14.43344) QuantErr: 14.43344 batch_time=0.63417
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.07931 (QuantReg: 14.09113) QuantErr: 14.09113 batch_time=0.56510
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.05884 (QuantReg: 14.34923) QuantErr: 14.34923 batch_time=0.55866
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.49406 (QuantReg: 14.00695) QuantErr: 14.00695 batch_time=0.55881
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.25403 (QuantReg: 14.12390) QuantErr: 14.12390 batch_time=0.51526
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.02231 (QuantReg: 14.56426) QuantErr: 14.56426 batch_time=0.54529
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.32635 (QuantReg: 14.64159) QuantErr: 14.64159 batch_time=0.54533
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.32766 (QuantReg: 14.38091) QuantErr: 14.38091 batch_time=0.57037
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.63556 (QuantReg: 14.12589) QuantErr: 14.12589 batch_time=0.57997
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 0.92388 (QuantReg: 14.37493) QuantErr: 14.37493 batch_time=0.54440
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.11152 (QuantReg: 14.27488) QuantErr: 14.27488 batch_time=0.54803
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.39320 (QuantReg: 14.27303) QuantErr: 14.27303 batch_time=0.54278
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.38031 (QuantReg: 13.94229) QuantErr: 13.94229 batch_time=0.52451
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.28175 (QuantReg: 14.36359) QuantErr: 14.36359 batch_time=0.57867
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.34441 (QuantReg: 14.45797) QuantErr: 14.45797 batch_time=0.54996
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.77459 (QuantReg: 14.21695) QuantErr: 14.21695 batch_time=0.60101
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.36965 (QuantReg: 14.72274) QuantErr: 14.72274 batch_time=0.55442
Train Epoch: 19 codebook_update_time=2.08406
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch19.pth ...
Done in 4.733s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.03/checkpoint-epoch19.pth ...
Done in 9.845s
removing stale ckpt [epoch 18] [took 0.01s]
epoch : 19
loss : 1.2927898151874542
quant_reg : 14.31296494293213
quant_err : 14.31296494293213
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 55.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 67.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 4.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.231