-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_LSMDC.txt
2591 lines (2591 loc) · 189 KB
/
HCQ_LSMDC.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC
Preparing the dataloaders ...
Loading dataset LSMDC_full_trainval in ram ...
Finish loading dataset LSMDC_full_trainval in ram, taking 9626.645081043243 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 30.45574450492859 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 25.286539793014526 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch0.pth ...
Done in 2.153s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch0.pth ...
Done in 3.709s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
LSMDC_full_test/t2v_metrics/R1: 0.0
LSMDC_full_test/t2v_metrics/R5: 0.9
LSMDC_full_test/t2v_metrics/R10: 1.6
LSMDC_full_test/t2v_metrics/R50: 4.4
LSMDC_full_test/t2v_metrics/MedR: 508.5
LSMDC_full_test/t2v_metrics/MeanR: 502.992
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
LSMDC_full_test/v2t_metrics/R1: 0.0
LSMDC_full_test/v2t_metrics/R5: 0.3
LSMDC_full_test/v2t_metrics/R10: 0.9
LSMDC_full_test/v2t_metrics/R50: 5.1
LSMDC_full_test/v2t_metrics/MedR: 510.0
LSMDC_full_test/v2t_metrics/MeanR: 501.125
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.80646 (QuantReg: 22.49566) QuantErr: 22.49566 batch_time=19.25055
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.20387 (QuantReg: 22.72755) QuantErr: 22.72755 batch_time=0.50281
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 8.74745 (QuantReg: 22.77307) QuantErr: 22.77307 batch_time=0.50368
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 8.14334 (QuantReg: 22.66275) QuantErr: 22.66275 batch_time=0.49668
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 7.83894 (QuantReg: 22.67912) QuantErr: 22.67912 batch_time=0.50762
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 7.45089 (QuantReg: 22.71301) QuantErr: 22.71301 batch_time=0.48692
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 7.27044 (QuantReg: 22.72690) QuantErr: 22.72690 batch_time=2.61348
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 6.97407 (QuantReg: 22.66125) QuantErr: 22.66125 batch_time=0.48854
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 7.03312 (QuantReg: 22.65001) QuantErr: 22.65001 batch_time=0.50360
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 6.74469 (QuantReg: 22.59913) QuantErr: 22.59913 batch_time=0.50422
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 7.19390 (QuantReg: 22.64696) QuantErr: 22.64696 batch_time=0.51954
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 6.19752 (QuantReg: 22.70680) QuantErr: 22.70680 batch_time=0.51320
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 6.68776 (QuantReg: 22.67888) QuantErr: 22.67888 batch_time=0.84427
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 7.03601 (QuantReg: 22.59605) QuantErr: 22.59605 batch_time=0.50133
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 6.54894 (QuantReg: 22.61918) QuantErr: 22.61918 batch_time=0.49710
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 6.14300 (QuantReg: 22.68196) QuantErr: 22.68196 batch_time=0.55703
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 6.95971 (QuantReg: 22.63796) QuantErr: 22.63796 batch_time=0.51786
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 6.33332 (QuantReg: 22.61819) QuantErr: 22.61819 batch_time=0.51938
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 6.14834 (QuantReg: 22.66921) QuantErr: 22.66921 batch_time=0.50571
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 6.42348 (QuantReg: 22.66271) QuantErr: 22.66271 batch_time=0.50745
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 5.79571 (QuantReg: 22.68092) QuantErr: 22.68092 batch_time=3.35326
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 5.79859 (QuantReg: 22.67046) QuantErr: 22.67046 batch_time=0.55392
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 7.03164 (QuantReg: 22.69639) QuantErr: 22.69639 batch_time=0.49138
Train Epoch: 1 codebook_update_time=2.10017
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch1.pth ...
Done in 4.592s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch1.pth ...
Done in 10.061s
epoch : 1
loss : 7.0317460117340085
quant_reg : 22.67053665161133
quant_err : 22.67053665161133
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
LSMDC_full_test/t2v_metrics/R1: 8.0
LSMDC_full_test/t2v_metrics/R5: 21.1
LSMDC_full_test/t2v_metrics/R10: 28.7
LSMDC_full_test/t2v_metrics/R50: 55.9
LSMDC_full_test/t2v_metrics/MedR: 38.5
LSMDC_full_test/t2v_metrics/MeanR: 99.301
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 16.92069171723653
LSMDC_full_test/v2t_metrics/R1: 6.2
LSMDC_full_test/v2t_metrics/R5: 17.8
LSMDC_full_test/v2t_metrics/R10: 26.0
LSMDC_full_test/v2t_metrics/R50: 55.6
LSMDC_full_test/v2t_metrics/MedR: 40.0
LSMDC_full_test/v2t_metrics/MeanR: 104.335
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 14.210030603842524
mnt_best : 16.92069171723653
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 5.94041 (QuantReg: 10.61254) QuantErr: 10.61254 batch_time=17.09697
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 6.09503 (QuantReg: 10.60899) QuantErr: 10.60899 batch_time=0.54193
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 6.18389 (QuantReg: 11.30963) QuantErr: 11.30963 batch_time=0.49578
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 5.99357 (QuantReg: 11.31604) QuantErr: 11.31604 batch_time=0.55017
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 6.22044 (QuantReg: 11.03336) QuantErr: 11.03336 batch_time=0.50218
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 6.56701 (QuantReg: 11.74750) QuantErr: 11.74750 batch_time=0.50915
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 6.24266 (QuantReg: 11.41699) QuantErr: 11.41699 batch_time=0.51590
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 5.42951 (QuantReg: 11.99984) QuantErr: 11.99984 batch_time=0.52541
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 5.87866 (QuantReg: 12.29715) QuantErr: 12.29715 batch_time=1.86614
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.85040 (QuantReg: 12.94896) QuantErr: 12.94896 batch_time=0.51306
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 5.47221 (QuantReg: 11.97905) QuantErr: 11.97905 batch_time=0.58911
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 5.70562 (QuantReg: 12.49326) QuantErr: 12.49326 batch_time=0.50389
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 5.92470 (QuantReg: 12.28568) QuantErr: 12.28568 batch_time=0.51794
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 5.79068 (QuantReg: 12.66051) QuantErr: 12.66051 batch_time=4.15701
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 5.78870 (QuantReg: 12.62403) QuantErr: 12.62403 batch_time=0.54459
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 5.36084 (QuantReg: 12.98852) QuantErr: 12.98852 batch_time=0.48838
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 5.27421 (QuantReg: 12.80945) QuantErr: 12.80945 batch_time=0.49559
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 6.21328 (QuantReg: 13.46010) QuantErr: 13.46010 batch_time=0.49531
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 6.23980 (QuantReg: 13.59974) QuantErr: 13.59974 batch_time=1.85647
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 5.57788 (QuantReg: 13.77868) QuantErr: 13.77868 batch_time=0.52128
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 5.33652 (QuantReg: 13.50353) QuantErr: 13.50353 batch_time=0.49846
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 5.79423 (QuantReg: 13.99748) QuantErr: 13.99748 batch_time=0.56345
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 5.71768 (QuantReg: 13.33311) QuantErr: 13.33311 batch_time=0.50283
Train Epoch: 2 codebook_update_time=1.90644
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch2.pth ...
Done in 16.926s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch2.pth ...
Done in 21.783s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.14s]
epoch : 2
loss : 5.7412131690979
quant_reg : 12.453678661346435
quant_err : 12.453678661346435
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
LSMDC_full_test/t2v_metrics/R1: 9.2
LSMDC_full_test/t2v_metrics/R5: 22.8
LSMDC_full_test/t2v_metrics/R10: 31.0
LSMDC_full_test/t2v_metrics/R50: 60.9
LSMDC_full_test/t2v_metrics/MedR: 30.0
LSMDC_full_test/t2v_metrics/MeanR: 86.179
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 18.66500552111338
LSMDC_full_test/v2t_metrics/R1: 7.9
LSMDC_full_test/v2t_metrics/R5: 20.8
LSMDC_full_test/v2t_metrics/R10: 30.8
LSMDC_full_test/v2t_metrics/R50: 59.6
LSMDC_full_test/v2t_metrics/MedR: 31.0
LSMDC_full_test/v2t_metrics/MeanR: 89.432
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 17.169080922692775
mnt_best : 18.66500552111338
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 5.76047 (QuantReg: 11.85055) QuantErr: 11.85055 batch_time=24.04843
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 5.17457 (QuantReg: 11.46447) QuantErr: 11.46447 batch_time=0.50802
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 5.59755 (QuantReg: 11.60199) QuantErr: 11.60199 batch_time=0.49175
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 5.18392 (QuantReg: 11.99460) QuantErr: 11.99460 batch_time=0.51319
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 5.14654 (QuantReg: 12.35239) QuantErr: 12.35239 batch_time=0.51496
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 5.22591 (QuantReg: 12.04302) QuantErr: 12.04302 batch_time=0.53086
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 5.15792 (QuantReg: 12.41278) QuantErr: 12.41278 batch_time=0.49343
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 5.77841 (QuantReg: 11.87966) QuantErr: 11.87966 batch_time=0.51581
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 5.24834 (QuantReg: 12.25544) QuantErr: 12.25544 batch_time=0.48186
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 5.01408 (QuantReg: 12.86727) QuantErr: 12.86727 batch_time=0.52097
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 5.17553 (QuantReg: 12.26240) QuantErr: 12.26240 batch_time=0.71629
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 5.06093 (QuantReg: 12.55836) QuantErr: 12.55836 batch_time=0.51360
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 5.12269 (QuantReg: 12.27846) QuantErr: 12.27846 batch_time=0.49791
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 5.34002 (QuantReg: 12.74949) QuantErr: 12.74949 batch_time=0.51267
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 4.78409 (QuantReg: 12.77183) QuantErr: 12.77183 batch_time=0.51233
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 5.26673 (QuantReg: 12.49954) QuantErr: 12.49954 batch_time=0.51594
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 4.92385 (QuantReg: 13.15568) QuantErr: 13.15568 batch_time=0.49632
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 5.39124 (QuantReg: 12.60345) QuantErr: 12.60345 batch_time=0.50748
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 4.82348 (QuantReg: 13.25797) QuantErr: 13.25797 batch_time=0.49443
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 5.22930 (QuantReg: 13.17685) QuantErr: 13.17685 batch_time=0.49296
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 4.92529 (QuantReg: 12.72368) QuantErr: 12.72368 batch_time=0.51638
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 4.97187 (QuantReg: 13.42447) QuantErr: 13.42447 batch_time=0.49138
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 5.00476 (QuantReg: 13.16537) QuantErr: 13.16537 batch_time=0.53007
Train Epoch: 3 codebook_update_time=1.64866
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch3.pth ...
Done in 3.890s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch3.pth ...
Done in 7.642s
removing stale ckpt [epoch 2] [took 0.00s]
epoch : 3
loss : 5.237949201583862
quant_reg : 12.472835514068603
quant_err : 12.472835514068603
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
LSMDC_full_test/t2v_metrics/R1: 8.9
LSMDC_full_test/t2v_metrics/R5: 24.7
LSMDC_full_test/t2v_metrics/R10: 34.1
LSMDC_full_test/t2v_metrics/R50: 62.2
LSMDC_full_test/t2v_metrics/MedR: 26.0
LSMDC_full_test/t2v_metrics/MeanR: 80.12
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.57103436992235
LSMDC_full_test/v2t_metrics/R1: 9.8
LSMDC_full_test/v2t_metrics/R5: 24.1
LSMDC_full_test/v2t_metrics/R10: 33.3
LSMDC_full_test/v2t_metrics/R50: 61.8
LSMDC_full_test/v2t_metrics/MedR: 27.0
LSMDC_full_test/v2t_metrics/MeanR: 86.065
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 19.886687560299414
mnt_best : 19.57103436992235
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 5.24468 (QuantReg: 11.83405) QuantErr: 11.83405 batch_time=27.05408
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 4.92738 (QuantReg: 12.34734) QuantErr: 12.34734 batch_time=0.50003
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 5.29920 (QuantReg: 12.25455) QuantErr: 12.25455 batch_time=1.90780
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 4.92167 (QuantReg: 12.42153) QuantErr: 12.42153 batch_time=0.73878
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 4.44591 (QuantReg: 12.80142) QuantErr: 12.80142 batch_time=0.52364
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 4.46657 (QuantReg: 12.64261) QuantErr: 12.64261 batch_time=0.61087
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 5.04618 (QuantReg: 12.61732) QuantErr: 12.61732 batch_time=0.50022
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 5.30351 (QuantReg: 12.55042) QuantErr: 12.55042 batch_time=0.50206
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 4.37882 (QuantReg: 12.34480) QuantErr: 12.34480 batch_time=0.51883
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 4.88971 (QuantReg: 12.15495) QuantErr: 12.15495 batch_time=0.51448
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 5.54922 (QuantReg: 12.29326) QuantErr: 12.29326 batch_time=0.49717
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 4.30216 (QuantReg: 12.65736) QuantErr: 12.65736 batch_time=0.48752
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 4.98024 (QuantReg: 12.62256) QuantErr: 12.62256 batch_time=0.48271
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 5.04439 (QuantReg: 12.55535) QuantErr: 12.55535 batch_time=0.56222
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 5.09492 (QuantReg: 12.56550) QuantErr: 12.56550 batch_time=0.52394
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 5.26438 (QuantReg: 12.65230) QuantErr: 12.65230 batch_time=0.50324
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 4.61131 (QuantReg: 12.85573) QuantErr: 12.85573 batch_time=0.50811
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 5.35108 (QuantReg: 12.99085) QuantErr: 12.99085 batch_time=0.49056
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 4.39204 (QuantReg: 13.44945) QuantErr: 13.44945 batch_time=0.52590
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 4.53527 (QuantReg: 13.12844) QuantErr: 13.12844 batch_time=0.55288
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 3.78907 (QuantReg: 13.43728) QuantErr: 13.43728 batch_time=0.53691
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 4.53584 (QuantReg: 13.56426) QuantErr: 13.56426 batch_time=0.49506
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 4.74266 (QuantReg: 13.27479) QuantErr: 13.27479 batch_time=0.50100
Train Epoch: 4 codebook_update_time=1.94111
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch4.pth ...
Done in 10.940s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch4.pth ...
Done in 25.091s
removing stale ckpt [epoch 3] [took 0.00s]
epoch : 4
loss : 4.875932681083679
quant_reg : 12.71564012145996
quant_err : 12.71564012145996
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
LSMDC_full_test/t2v_metrics/R1: 11.7
LSMDC_full_test/t2v_metrics/R5: 25.9
LSMDC_full_test/t2v_metrics/R10: 35.0
LSMDC_full_test/t2v_metrics/R50: 64.1
LSMDC_full_test/t2v_metrics/MedR: 25.0
LSMDC_full_test/t2v_metrics/MeanR: 79.466
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.971070791232027
LSMDC_full_test/v2t_metrics/R1: 10.5
LSMDC_full_test/v2t_metrics/R5: 26.6
LSMDC_full_test/v2t_metrics/R10: 36.8
LSMDC_full_test/v2t_metrics/R50: 62.6
LSMDC_full_test/v2t_metrics/MedR: 24.0
LSMDC_full_test/v2t_metrics/MeanR: 79.215
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.742338429768463
mnt_best : 21.971070791232027
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 5.62856 (QuantReg: 12.30854) QuantErr: 12.30854 batch_time=21.54201
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 4.41379 (QuantReg: 12.85020) QuantErr: 12.85020 batch_time=0.53399
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 4.68511 (QuantReg: 12.91639) QuantErr: 12.91639 batch_time=0.48625
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 5.03775 (QuantReg: 12.89013) QuantErr: 12.89013 batch_time=0.53448
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 4.80664 (QuantReg: 13.26204) QuantErr: 13.26204 batch_time=0.47951
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 4.22795 (QuantReg: 13.24580) QuantErr: 13.24580 batch_time=0.48971
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 4.52282 (QuantReg: 13.14964) QuantErr: 13.14964 batch_time=1.70338
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 4.95592 (QuantReg: 13.19013) QuantErr: 13.19013 batch_time=0.48557
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 5.33553 (QuantReg: 13.02886) QuantErr: 13.02886 batch_time=0.56013
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 4.97937 (QuantReg: 13.07985) QuantErr: 13.07985 batch_time=0.48614
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 4.74659 (QuantReg: 13.03334) QuantErr: 13.03334 batch_time=0.51959
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 4.52162 (QuantReg: 13.28604) QuantErr: 13.28604 batch_time=0.55891
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 4.28170 (QuantReg: 13.25134) QuantErr: 13.25134 batch_time=1.92181
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 4.49210 (QuantReg: 13.40897) QuantErr: 13.40897 batch_time=1.16914
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 4.85025 (QuantReg: 13.46471) QuantErr: 13.46471 batch_time=0.52410
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 4.84591 (QuantReg: 13.10037) QuantErr: 13.10037 batch_time=0.48372
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 4.15968 (QuantReg: 13.54168) QuantErr: 13.54168 batch_time=0.60675
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 4.25948 (QuantReg: 13.38039) QuantErr: 13.38039 batch_time=0.49452
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 4.23840 (QuantReg: 13.48424) QuantErr: 13.48424 batch_time=0.49668
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 4.47719 (QuantReg: 13.37124) QuantErr: 13.37124 batch_time=0.47800
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 4.15238 (QuantReg: 13.41493) QuantErr: 13.41493 batch_time=1.16066
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 4.34722 (QuantReg: 13.95414) QuantErr: 13.95414 batch_time=0.51089
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 3.86533 (QuantReg: 13.81402) QuantErr: 13.81402 batch_time=0.56689
Train Epoch: 5 codebook_update_time=1.68494
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch5.pth ...
Done in 5.629s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch5.pth ...
Done in 10.747s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 4.541494495391846
quant_reg : 13.264000789642335
quant_err : 13.264000789642335
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
LSMDC_full_test/t2v_metrics/R1: 10.6
LSMDC_full_test/t2v_metrics/R5: 27.6
LSMDC_full_test/t2v_metrics/R10: 37.0
LSMDC_full_test/t2v_metrics/R50: 64.8
LSMDC_full_test/t2v_metrics/MedR: 22.0
LSMDC_full_test/t2v_metrics/MeanR: 76.491
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.121040818582063
LSMDC_full_test/v2t_metrics/R1: 11.4
LSMDC_full_test/v2t_metrics/R5: 27.3
LSMDC_full_test/v2t_metrics/R10: 36.3
LSMDC_full_test/v2t_metrics/R50: 64.1
LSMDC_full_test/v2t_metrics/MedR: 25.0
LSMDC_full_test/v2t_metrics/MeanR: 77.117
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.438373584544998
mnt_best : 22.121040818582063
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 4.98788 (QuantReg: 13.07576) QuantErr: 13.07576 batch_time=20.60166
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 4.42090 (QuantReg: 13.16578) QuantErr: 13.16578 batch_time=0.49693
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 4.72534 (QuantReg: 13.21066) QuantErr: 13.21066 batch_time=0.49115
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 4.17931 (QuantReg: 13.48193) QuantErr: 13.48193 batch_time=0.49056
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 4.18375 (QuantReg: 13.53833) QuantErr: 13.53833 batch_time=0.48313
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 4.51513 (QuantReg: 13.22451) QuantErr: 13.22451 batch_time=0.49034
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 4.24964 (QuantReg: 13.35549) QuantErr: 13.35549 batch_time=0.48663
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 4.26356 (QuantReg: 13.53818) QuantErr: 13.53818 batch_time=0.52116
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 4.82779 (QuantReg: 13.16201) QuantErr: 13.16201 batch_time=0.48648
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 4.59746 (QuantReg: 13.59044) QuantErr: 13.59044 batch_time=0.47670
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 3.87482 (QuantReg: 13.75482) QuantErr: 13.75482 batch_time=0.50752
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 4.29189 (QuantReg: 13.47815) QuantErr: 13.47815 batch_time=0.50218
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 4.22504 (QuantReg: 13.84182) QuantErr: 13.84182 batch_time=0.48890
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 5.00504 (QuantReg: 13.51904) QuantErr: 13.51904 batch_time=0.49354
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 3.99273 (QuantReg: 13.95034) QuantErr: 13.95034 batch_time=0.49343
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 3.59310 (QuantReg: 13.69462) QuantErr: 13.69462 batch_time=0.53494
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 3.92653 (QuantReg: 13.95626) QuantErr: 13.95626 batch_time=0.48574
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 4.17916 (QuantReg: 14.06710) QuantErr: 14.06710 batch_time=0.49674
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 4.79924 (QuantReg: 13.59501) QuantErr: 13.59501 batch_time=0.53755
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 4.20154 (QuantReg: 13.98911) QuantErr: 13.98911 batch_time=1.49560
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 3.97964 (QuantReg: 14.11820) QuantErr: 14.11820 batch_time=1.76959
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 4.83651 (QuantReg: 13.43325) QuantErr: 13.43325 batch_time=0.48385
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 4.49985 (QuantReg: 13.82336) QuantErr: 13.82336 batch_time=0.48919
Train Epoch: 6 codebook_update_time=1.62484
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch6.pth ...
Done in 5.800s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch6.pth ...
Done in 10.772s
removing stale ckpt [epoch 5] [took 0.05s]
epoch : 6
loss : 4.344158237457275
quant_reg : 13.631624481201172
quant_err : 13.631624481201172
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
LSMDC_full_test/t2v_metrics/R1: 11.0
LSMDC_full_test/t2v_metrics/R5: 29.2
LSMDC_full_test/t2v_metrics/R10: 38.2
LSMDC_full_test/t2v_metrics/R50: 66.2
LSMDC_full_test/t2v_metrics/MedR: 22.0
LSMDC_full_test/t2v_metrics/MeanR: 73.566
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.064619789339012
LSMDC_full_test/v2t_metrics/R1: 11.5
LSMDC_full_test/v2t_metrics/R5: 28.3
LSMDC_full_test/v2t_metrics/R10: 36.9
LSMDC_full_test/v2t_metrics/R50: 64.7
LSMDC_full_test/v2t_metrics/MedR: 22.0
LSMDC_full_test/v2t_metrics/MeanR: 78.032
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.900073733418825
mnt_best : 23.064619789339012
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 4.34433 (QuantReg: 13.71051) QuantErr: 13.71051 batch_time=22.05860
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 4.00687 (QuantReg: 13.80888) QuantErr: 13.80888 batch_time=0.51961
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 4.02090 (QuantReg: 13.52396) QuantErr: 13.52396 batch_time=2.10427
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 3.94221 (QuantReg: 13.62283) QuantErr: 13.62283 batch_time=0.49860
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 4.60728 (QuantReg: 13.75600) QuantErr: 13.75600 batch_time=0.49065
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 4.05903 (QuantReg: 13.84675) QuantErr: 13.84675 batch_time=0.49805
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 4.17738 (QuantReg: 13.66180) QuantErr: 13.66180 batch_time=0.49607
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 4.50574 (QuantReg: 13.70996) QuantErr: 13.70996 batch_time=0.50001
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 3.95048 (QuantReg: 13.59104) QuantErr: 13.59104 batch_time=0.50026
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 4.09419 (QuantReg: 13.86123) QuantErr: 13.86123 batch_time=0.57805
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 4.04285 (QuantReg: 14.00081) QuantErr: 14.00081 batch_time=0.48381
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 4.37784 (QuantReg: 13.70425) QuantErr: 13.70425 batch_time=0.50589
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 3.99443 (QuantReg: 13.87494) QuantErr: 13.87494 batch_time=0.49228
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 3.81320 (QuantReg: 13.93103) QuantErr: 13.93103 batch_time=2.36875
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 4.00652 (QuantReg: 14.23179) QuantErr: 14.23179 batch_time=0.52881
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 3.76549 (QuantReg: 14.07976) QuantErr: 14.07976 batch_time=0.53662
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 4.20494 (QuantReg: 13.91564) QuantErr: 13.91564 batch_time=0.48932
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 4.12478 (QuantReg: 14.22445) QuantErr: 14.22445 batch_time=0.49167
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 3.79486 (QuantReg: 14.01488) QuantErr: 14.01488 batch_time=0.52597
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 3.87255 (QuantReg: 14.17650) QuantErr: 14.17650 batch_time=0.50024
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 3.76308 (QuantReg: 14.18180) QuantErr: 14.18180 batch_time=0.49405
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 3.85727 (QuantReg: 14.45146) QuantErr: 14.45146 batch_time=0.52105
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 3.83237 (QuantReg: 14.30319) QuantErr: 14.30319 batch_time=0.48934
Train Epoch: 7 codebook_update_time=1.89272
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch7.pth ...
Done in 4.736s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch7.pth ...
Done in 8.927s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 4.108590788841248
quant_reg : 13.929740890502929
quant_err : 13.929740890502929
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
LSMDC_full_test/t2v_metrics/R1: 11.6
LSMDC_full_test/t2v_metrics/R5: 29.9
LSMDC_full_test/t2v_metrics/R10: 39.3
LSMDC_full_test/t2v_metrics/R50: 66.1
LSMDC_full_test/t2v_metrics/MedR: 20.0
LSMDC_full_test/t2v_metrics/MeanR: 70.896
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.88767651880651
LSMDC_full_test/v2t_metrics/R1: 11.9
LSMDC_full_test/v2t_metrics/R5: 29.7
LSMDC_full_test/v2t_metrics/R10: 39.5
LSMDC_full_test/v2t_metrics/R50: 66.2
LSMDC_full_test/v2t_metrics/MedR: 20.0
LSMDC_full_test/v2t_metrics/MeanR: 69.952
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.078725852641853
mnt_best : 23.88767651880651
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 4.02636 (QuantReg: 13.84004) QuantErr: 13.84004 batch_time=23.62820
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 3.82903 (QuantReg: 14.01167) QuantErr: 14.01167 batch_time=0.68211
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 4.05562 (QuantReg: 14.32215) QuantErr: 14.32215 batch_time=0.49648
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 4.47502 (QuantReg: 14.02694) QuantErr: 14.02694 batch_time=0.51399
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 3.60997 (QuantReg: 14.03046) QuantErr: 14.03046 batch_time=0.52889
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 4.01205 (QuantReg: 14.14976) QuantErr: 14.14976 batch_time=0.51889
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 3.83322 (QuantReg: 13.86267) QuantErr: 13.86267 batch_time=0.52136
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 4.32325 (QuantReg: 14.24657) QuantErr: 14.24657 batch_time=0.49255
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 3.67781 (QuantReg: 14.28468) QuantErr: 14.28468 batch_time=0.52552
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 3.92709 (QuantReg: 13.94063) QuantErr: 13.94063 batch_time=0.52217
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 4.32643 (QuantReg: 14.11018) QuantErr: 14.11018 batch_time=0.49559
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 3.82351 (QuantReg: 14.31007) QuantErr: 14.31007 batch_time=0.60578
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 3.81310 (QuantReg: 14.33179) QuantErr: 14.33179 batch_time=0.51076
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 3.55121 (QuantReg: 14.66822) QuantErr: 14.66822 batch_time=0.56190
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 4.14633 (QuantReg: 14.39167) QuantErr: 14.39167 batch_time=0.52501
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 3.65605 (QuantReg: 14.22293) QuantErr: 14.22293 batch_time=0.49946
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 3.88911 (QuantReg: 14.43343) QuantErr: 14.43343 batch_time=0.50048
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 3.70964 (QuantReg: 14.34420) QuantErr: 14.34420 batch_time=0.52185
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 3.80310 (QuantReg: 14.17896) QuantErr: 14.17896 batch_time=0.49583
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 4.06557 (QuantReg: 14.26658) QuantErr: 14.26658 batch_time=0.52978
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 3.63568 (QuantReg: 14.50715) QuantErr: 14.50715 batch_time=0.55528
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 3.61395 (QuantReg: 14.57050) QuantErr: 14.57050 batch_time=0.50862
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 3.81020 (QuantReg: 14.31740) QuantErr: 14.31740 batch_time=0.52667
Train Epoch: 8 codebook_update_time=1.65586
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch8.pth ...
Done in 5.809s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 3.9208778266906736
quant_reg : 14.196909938812256
quant_err : 14.196909938812256
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
LSMDC_full_test/t2v_metrics/R1: 12.1
LSMDC_full_test/t2v_metrics/R5: 29.4
LSMDC_full_test/t2v_metrics/R10: 37.8
LSMDC_full_test/t2v_metrics/R50: 66.1
LSMDC_full_test/t2v_metrics/MedR: 23.0
LSMDC_full_test/t2v_metrics/MeanR: 72.768
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.77979831304413
LSMDC_full_test/v2t_metrics/R1: 11.8
LSMDC_full_test/v2t_metrics/R5: 29.4
LSMDC_full_test/v2t_metrics/R10: 38.0
LSMDC_full_test/v2t_metrics/R50: 64.7
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 73.308
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.623141143183286
mnt_best : 23.88767651880651
not_improved_count: 1
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 3.82574 (QuantReg: 13.67512) QuantErr: 13.67512 batch_time=20.91374
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 3.89387 (QuantReg: 14.28181) QuantErr: 14.28181 batch_time=0.49129
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 3.57512 (QuantReg: 14.34366) QuantErr: 14.34366 batch_time=0.70712
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 4.22766 (QuantReg: 14.49150) QuantErr: 14.49150 batch_time=0.49599
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 4.09153 (QuantReg: 14.23469) QuantErr: 14.23469 batch_time=0.50011
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 3.46709 (QuantReg: 14.87689) QuantErr: 14.87689 batch_time=0.49216
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 3.75796 (QuantReg: 14.26065) QuantErr: 14.26065 batch_time=0.81672
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 4.18075 (QuantReg: 13.92644) QuantErr: 13.92644 batch_time=0.48164
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 3.68453 (QuantReg: 14.11707) QuantErr: 14.11707 batch_time=0.49140
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 3.53200 (QuantReg: 14.45447) QuantErr: 14.45447 batch_time=1.00112
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 4.26756 (QuantReg: 14.31398) QuantErr: 14.31398 batch_time=0.49213
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 3.66823 (QuantReg: 14.46457) QuantErr: 14.46457 batch_time=0.48272
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 3.47624 (QuantReg: 14.46646) QuantErr: 14.46646 batch_time=0.52103
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 3.64168 (QuantReg: 14.67084) QuantErr: 14.67084 batch_time=0.58159
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 3.70481 (QuantReg: 14.57283) QuantErr: 14.57283 batch_time=0.51357
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 3.84021 (QuantReg: 14.84067) QuantErr: 14.84067 batch_time=0.50624
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 3.44820 (QuantReg: 14.75933) QuantErr: 14.75933 batch_time=0.51339
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 3.46284 (QuantReg: 14.09201) QuantErr: 14.09201 batch_time=0.50276
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 3.75694 (QuantReg: 14.42306) QuantErr: 14.42306 batch_time=0.50718
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 3.62269 (QuantReg: 14.81061) QuantErr: 14.81061 batch_time=0.52329
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 4.24608 (QuantReg: 14.72643) QuantErr: 14.72643 batch_time=0.53217
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 4.29558 (QuantReg: 14.69179) QuantErr: 14.69179 batch_time=0.49251
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 3.70320 (QuantReg: 14.40913) QuantErr: 14.40913 batch_time=0.49959
Train Epoch: 9 codebook_update_time=1.64047
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch9.pth ...
Done in 5.077s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch9.pth ...
Done in 9.320s
removing stale ckpt [epoch 8] [took 0.02s]
epoch : 9
loss : 3.738044695854187
quant_reg : 14.476089488983154
quant_err : 14.476089488983154
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
LSMDC_full_test/t2v_metrics/R1: 12.3
LSMDC_full_test/t2v_metrics/R5: 29.7
LSMDC_full_test/t2v_metrics/R10: 38.5
LSMDC_full_test/t2v_metrics/R50: 66.9
LSMDC_full_test/t2v_metrics/MedR: 20.0
LSMDC_full_test/t2v_metrics/MeanR: 69.516
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.13834165886024
LSMDC_full_test/v2t_metrics/R1: 13.0
LSMDC_full_test/v2t_metrics/R5: 30.5
LSMDC_full_test/v2t_metrics/R10: 39.5
LSMDC_full_test/v2t_metrics/R50: 66.9
LSMDC_full_test/v2t_metrics/MedR: 20.0
LSMDC_full_test/v2t_metrics/MeanR: 68.053
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.019584653647332
mnt_best : 24.13834165886024
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 3.57195 (QuantReg: 14.49104) QuantErr: 14.49104 batch_time=23.93646
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 3.48344 (QuantReg: 13.97122) QuantErr: 13.97122 batch_time=0.48706
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 3.59164 (QuantReg: 14.23781) QuantErr: 14.23781 batch_time=0.51732
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 3.44207 (QuantReg: 14.42399) QuantErr: 14.42399 batch_time=0.51438
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 3.83450 (QuantReg: 14.52170) QuantErr: 14.52170 batch_time=1.50587
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 3.27452 (QuantReg: 14.71473) QuantErr: 14.71473 batch_time=0.49216
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 3.70993 (QuantReg: 14.41140) QuantErr: 14.41140 batch_time=0.49235
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 3.52475 (QuantReg: 14.46597) QuantErr: 14.46597 batch_time=0.48659
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 3.58651 (QuantReg: 14.82762) QuantErr: 14.82762 batch_time=0.50135
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 3.79491 (QuantReg: 14.71527) QuantErr: 14.71527 batch_time=0.49785
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 3.53341 (QuantReg: 14.76471) QuantErr: 14.76471 batch_time=1.12088
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 3.60708 (QuantReg: 14.58728) QuantErr: 14.58728 batch_time=0.50026
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 3.30650 (QuantReg: 14.67471) QuantErr: 14.67471 batch_time=0.51159
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 3.55336 (QuantReg: 14.71909) QuantErr: 14.71909 batch_time=0.49996
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 4.01628 (QuantReg: 14.76700) QuantErr: 14.76700 batch_time=0.48458
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 3.76233 (QuantReg: 15.10110) QuantErr: 15.10110 batch_time=0.48565
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 3.70337 (QuantReg: 14.51776) QuantErr: 14.51776 batch_time=0.49997
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 3.68811 (QuantReg: 14.97330) QuantErr: 14.97330 batch_time=0.51058
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 3.23377 (QuantReg: 14.79965) QuantErr: 14.79965 batch_time=0.51893
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 3.29394 (QuantReg: 14.80492) QuantErr: 14.80492 batch_time=1.63795
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 3.20356 (QuantReg: 14.90326) QuantErr: 14.90326 batch_time=0.50849
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 4.14896 (QuantReg: 14.79976) QuantErr: 14.79976 batch_time=0.58283
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 3.65206 (QuantReg: 14.73933) QuantErr: 14.73933 batch_time=0.52795
Train Epoch: 10 codebook_update_time=1.65006
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch10.pth ...
Done in 6.265s
removing stale ckpt [epoch 9] [took 0.01s]
epoch : 10
loss : 3.584129936218262
quant_reg : 14.668781867980957
quant_err : 14.668781867980957
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
LSMDC_full_test/t2v_metrics/R1: 12.1
LSMDC_full_test/t2v_metrics/R5: 29.8
LSMDC_full_test/t2v_metrics/R10: 38.0
LSMDC_full_test/t2v_metrics/R50: 68.5
LSMDC_full_test/t2v_metrics/MedR: 20.0
LSMDC_full_test/t2v_metrics/MeanR: 70.087
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.92921271658576
LSMDC_full_test/v2t_metrics/R1: 11.0
LSMDC_full_test/v2t_metrics/R5: 29.6
LSMDC_full_test/v2t_metrics/R10: 37.8
LSMDC_full_test/v2t_metrics/R50: 66.4
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 68.265
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.088305769179335
mnt_best : 24.13834165886024
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 3.03870 (QuantReg: 14.91508) QuantErr: 14.91508 batch_time=22.53547
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 3.57801 (QuantReg: 14.76150) QuantErr: 14.76150 batch_time=0.51221
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 3.51044 (QuantReg: 14.49371) QuantErr: 14.49371 batch_time=0.52088
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 3.48824 (QuantReg: 14.64496) QuantErr: 14.64496 batch_time=0.49916
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 3.53000 (QuantReg: 14.67780) QuantErr: 14.67780 batch_time=0.51981
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 4.08270 (QuantReg: 14.71171) QuantErr: 14.71171 batch_time=0.51309
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 3.65252 (QuantReg: 14.75002) QuantErr: 14.75002 batch_time=0.49450
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 3.56780 (QuantReg: 14.87922) QuantErr: 14.87922 batch_time=0.49497
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 3.58116 (QuantReg: 14.85802) QuantErr: 14.85802 batch_time=0.49277
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 2.95225 (QuantReg: 14.97419) QuantErr: 14.97419 batch_time=0.49616
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 3.61277 (QuantReg: 15.05061) QuantErr: 15.05061 batch_time=0.48561
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 2.87328 (QuantReg: 14.98964) QuantErr: 14.98964 batch_time=0.61459
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 3.64890 (QuantReg: 14.95701) QuantErr: 14.95701 batch_time=0.48830
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 2.89007 (QuantReg: 14.83210) QuantErr: 14.83210 batch_time=0.51046
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 3.43028 (QuantReg: 14.75998) QuantErr: 14.75998 batch_time=0.50932
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 3.94033 (QuantReg: 14.88355) QuantErr: 14.88355 batch_time=0.49914
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 2.97370 (QuantReg: 15.08792) QuantErr: 15.08792 batch_time=0.50018
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 3.26142 (QuantReg: 14.96029) QuantErr: 14.96029 batch_time=0.49420
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 3.50686 (QuantReg: 14.77231) QuantErr: 14.77231 batch_time=0.50096
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 3.65520 (QuantReg: 14.93796) QuantErr: 14.93796 batch_time=0.52224
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 3.48898 (QuantReg: 14.59676) QuantErr: 14.59676 batch_time=0.48987
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 2.98301 (QuantReg: 15.03550) QuantErr: 15.03550 batch_time=0.52788
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 3.48546 (QuantReg: 14.71025) QuantErr: 14.71025 batch_time=0.49953
Train Epoch: 11 codebook_update_time=1.67083
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch11.pth ...
Done in 3.901s
removing stale ckpt [epoch 10] [took 0.01s]
epoch : 11
loss : 3.4240048780441286
quant_reg : 14.883848529815674
quant_err : 14.883848529815674
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
LSMDC_full_test/t2v_metrics/R1: 11.2
LSMDC_full_test/t2v_metrics/R5: 29.4
LSMDC_full_test/t2v_metrics/R10: 39.6
LSMDC_full_test/t2v_metrics/R50: 68.3
LSMDC_full_test/t2v_metrics/MedR: 21.0
LSMDC_full_test/t2v_metrics/MeanR: 71.055
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.537130377505253
LSMDC_full_test/v2t_metrics/R1: 11.7
LSMDC_full_test/v2t_metrics/R5: 29.1
LSMDC_full_test/v2t_metrics/R10: 39.7
LSMDC_full_test/v2t_metrics/R50: 66.7
LSMDC_full_test/v2t_metrics/MedR: 20.75
LSMDC_full_test/v2t_metrics/MeanR: 70.779
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.820806018065813
mnt_best : 24.13834165886024
not_improved_count: 2
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 3.39334 (QuantReg: 14.62989) QuantErr: 14.62989 batch_time=24.40161
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 3.04843 (QuantReg: 14.93742) QuantErr: 14.93742 batch_time=0.48351
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 3.77386 (QuantReg: 14.68280) QuantErr: 14.68280 batch_time=0.48862
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 2.90997 (QuantReg: 14.79715) QuantErr: 14.79715 batch_time=0.51369
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 2.94118 (QuantReg: 15.09708) QuantErr: 15.09708 batch_time=0.61757
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 3.04404 (QuantReg: 15.08082) QuantErr: 15.08082 batch_time=0.49068
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 3.46063 (QuantReg: 15.18704) QuantErr: 15.18704 batch_time=0.48731
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 3.38394 (QuantReg: 14.73355) QuantErr: 14.73355 batch_time=0.49046
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 3.27863 (QuantReg: 15.02370) QuantErr: 15.02370 batch_time=0.49583
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 3.18253 (QuantReg: 15.09444) QuantErr: 15.09444 batch_time=0.50253
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 3.35276 (QuantReg: 14.95709) QuantErr: 14.95709 batch_time=0.49941
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 3.24307 (QuantReg: 15.05900) QuantErr: 15.05900 batch_time=0.49988
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 3.66593 (QuantReg: 14.92819) QuantErr: 14.92819 batch_time=0.49378
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 3.69162 (QuantReg: 15.03204) QuantErr: 15.03204 batch_time=0.56152
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 3.36411 (QuantReg: 14.79167) QuantErr: 14.79167 batch_time=0.53237
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 3.33641 (QuantReg: 14.96424) QuantErr: 14.96424 batch_time=0.49840
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 2.56019 (QuantReg: 15.09340) QuantErr: 15.09340 batch_time=0.49597
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 3.63342 (QuantReg: 14.77622) QuantErr: 14.77622 batch_time=0.52191
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 2.87333 (QuantReg: 15.06111) QuantErr: 15.06111 batch_time=0.52942
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 3.58858 (QuantReg: 14.88951) QuantErr: 14.88951 batch_time=1.69826
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 3.20071 (QuantReg: 15.19884) QuantErr: 15.19884 batch_time=2.59348
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 3.03633 (QuantReg: 15.17500) QuantErr: 15.17500 batch_time=0.51569
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 3.27422 (QuantReg: 15.09743) QuantErr: 15.09743 batch_time=0.49764
Train Epoch: 12 codebook_update_time=1.75478
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch12.pth ...
Done in 4.219s
removing stale ckpt [epoch 11] [took 0.01s]
epoch : 12
loss : 3.2981453971862793
quant_reg : 14.971224960327149
quant_err : 14.971224960327149
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
LSMDC_full_test/t2v_metrics/R1: 12.0
LSMDC_full_test/t2v_metrics/R5: 29.5
LSMDC_full_test/t2v_metrics/R10: 39.7
LSMDC_full_test/t2v_metrics/R50: 68.2
LSMDC_full_test/t2v_metrics/MedR: 18.5
LSMDC_full_test/t2v_metrics/MeanR: 71.116
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.13225595412838
LSMDC_full_test/v2t_metrics/R1: 10.9
LSMDC_full_test/v2t_metrics/R5: 30.6
LSMDC_full_test/v2t_metrics/R10: 40.2
LSMDC_full_test/v2t_metrics/R50: 66.7
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 67.9335
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.756985129146212
mnt_best : 24.13834165886024
not_improved_count: 3
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 2.81405 (QuantReg: 14.93278) QuantErr: 14.93278 batch_time=22.63615
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 3.31316 (QuantReg: 14.85714) QuantErr: 14.85714 batch_time=0.51347
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 3.25977 (QuantReg: 15.25751) QuantErr: 15.25751 batch_time=0.96141
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 3.08393 (QuantReg: 15.15457) QuantErr: 15.15457 batch_time=0.49548
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 2.87767 (QuantReg: 15.16733) QuantErr: 15.16733 batch_time=0.48826
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 3.48499 (QuantReg: 15.03884) QuantErr: 15.03884 batch_time=0.55151
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 3.11655 (QuantReg: 15.13236) QuantErr: 15.13236 batch_time=0.48630
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 3.25605 (QuantReg: 15.20572) QuantErr: 15.20572 batch_time=0.49910
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 2.75344 (QuantReg: 15.25057) QuantErr: 15.25057 batch_time=0.48123
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 3.43456 (QuantReg: 15.09769) QuantErr: 15.09769 batch_time=0.49645
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 3.30539 (QuantReg: 15.14015) QuantErr: 15.14015 batch_time=0.52966
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 2.97147 (QuantReg: 15.51633) QuantErr: 15.51633 batch_time=0.48698
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 3.41453 (QuantReg: 14.96736) QuantErr: 14.96736 batch_time=0.48847
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 3.40049 (QuantReg: 15.06511) QuantErr: 15.06511 batch_time=0.48420
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 3.21059 (QuantReg: 15.39115) QuantErr: 15.39115 batch_time=0.48809
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 3.01519 (QuantReg: 15.48291) QuantErr: 15.48291 batch_time=0.49724
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 2.97666 (QuantReg: 15.03642) QuantErr: 15.03642 batch_time=0.49401
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 3.41428 (QuantReg: 15.15256) QuantErr: 15.15256 batch_time=0.52726
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 3.16880 (QuantReg: 15.43320) QuantErr: 15.43320 batch_time=0.53338
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 3.03117 (QuantReg: 14.97570) QuantErr: 14.97570 batch_time=0.47944
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 3.26240 (QuantReg: 15.00532) QuantErr: 15.00532 batch_time=0.48539
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 3.11828 (QuantReg: 15.28825) QuantErr: 15.28825 batch_time=0.49538
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 3.02764 (QuantReg: 15.08301) QuantErr: 15.08301 batch_time=0.61623
Train Epoch: 13 codebook_update_time=2.04388
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch13.pth ...
Done in 6.908s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch13.pth ...
Done in 11.826s
removing stale ckpt [epoch 12] [took 0.05s]
epoch : 13
loss : 3.1613081245422365
quant_reg : 15.152067691802978
quant_err : 15.152067691802978
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
LSMDC_full_test/t2v_metrics/R1: 12.8
LSMDC_full_test/t2v_metrics/R5: 31.5
LSMDC_full_test/t2v_metrics/R10: 40.7
LSMDC_full_test/t2v_metrics/R50: 68.3
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 70.714
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.41196865002143
LSMDC_full_test/v2t_metrics/R1: 12.0
LSMDC_full_test/v2t_metrics/R5: 30.9
LSMDC_full_test/v2t_metrics/R10: 39.7
LSMDC_full_test/v2t_metrics/R50: 68.1
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 68.792
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.508124474770273
mnt_best : 25.41196865002143
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 3.22980 (QuantReg: 14.91242) QuantErr: 14.91242 batch_time=23.68376
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 2.97709 (QuantReg: 14.99692) QuantErr: 14.99692 batch_time=0.50138
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 3.37969 (QuantReg: 15.00765) QuantErr: 15.00765 batch_time=0.49316
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 2.82685 (QuantReg: 15.13013) QuantErr: 15.13013 batch_time=0.50064
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 2.83071 (QuantReg: 15.31273) QuantErr: 15.31273 batch_time=0.50825
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 3.02401 (QuantReg: 15.16787) QuantErr: 15.16787 batch_time=0.49491
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 2.95299 (QuantReg: 15.06216) QuantErr: 15.06216 batch_time=0.54622
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 3.76156 (QuantReg: 14.99871) QuantErr: 14.99871 batch_time=0.51176
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 2.98420 (QuantReg: 15.07226) QuantErr: 15.07226 batch_time=0.49230
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 3.26040 (QuantReg: 15.30614) QuantErr: 15.30614 batch_time=0.49165
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.91072 (QuantReg: 15.23295) QuantErr: 15.23295 batch_time=0.48412
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 3.33807 (QuantReg: 15.38572) QuantErr: 15.38572 batch_time=0.49571
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 3.40388 (QuantReg: 15.14029) QuantErr: 15.14029 batch_time=0.52737
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 3.07514 (QuantReg: 15.33007) QuantErr: 15.33007 batch_time=0.49602
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 3.23942 (QuantReg: 15.31456) QuantErr: 15.31456 batch_time=1.80299
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 3.17297 (QuantReg: 15.18296) QuantErr: 15.18296 batch_time=0.52309
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 2.97576 (QuantReg: 15.36908) QuantErr: 15.36908 batch_time=0.49355
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 3.34976 (QuantReg: 15.46246) QuantErr: 15.46246 batch_time=0.60940
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 3.45531 (QuantReg: 15.14432) QuantErr: 15.14432 batch_time=0.48592
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 3.21881 (QuantReg: 15.36954) QuantErr: 15.36954 batch_time=0.49645
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 2.82124 (QuantReg: 15.43226) QuantErr: 15.43226 batch_time=0.48802
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 2.82955 (QuantReg: 15.50684) QuantErr: 15.50684 batch_time=0.48255
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 2.94055 (QuantReg: 15.42179) QuantErr: 15.42179 batch_time=0.49041
Train Epoch: 14 codebook_update_time=1.77485
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch14.pth ...
Done in 6.489s
removing stale ckpt [epoch 13] [took 0.01s]
epoch : 14
loss : 3.0871253747940064
quant_reg : 15.2322801322937
quant_err : 15.2322801322937
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
LSMDC_full_test/t2v_metrics/R1: 11.1
LSMDC_full_test/t2v_metrics/R5: 32.2
LSMDC_full_test/t2v_metrics/R10: 42.1
LSMDC_full_test/t2v_metrics/R50: 68.1
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 71.68
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.688061018068613
LSMDC_full_test/v2t_metrics/R1: 12.5
LSMDC_full_test/v2t_metrics/R5: 32.6
LSMDC_full_test/v2t_metrics/R10: 41.7
LSMDC_full_test/v2t_metrics/R50: 67.5
LSMDC_full_test/v2t_metrics/MedR: 17.0
LSMDC_full_test/v2t_metrics/MeanR: 70.896
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.709160133598253
mnt_best : 25.41196865002143
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 2.32802 (QuantReg: 15.17485) QuantErr: 15.17485 batch_time=29.95373
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 2.73866 (QuantReg: 14.95268) QuantErr: 14.95268 batch_time=0.53002
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 3.02678 (QuantReg: 15.31756) QuantErr: 15.31756 batch_time=0.51077
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 3.01722 (QuantReg: 15.35511) QuantErr: 15.35511 batch_time=0.51015
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 2.53022 (QuantReg: 15.47388) QuantErr: 15.47388 batch_time=0.51180
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 3.17832 (QuantReg: 15.48273) QuantErr: 15.48273 batch_time=0.53570
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 2.40631 (QuantReg: 15.53064) QuantErr: 15.53064 batch_time=0.51291
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 2.78840 (QuantReg: 15.33989) QuantErr: 15.33989 batch_time=0.51119
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 3.43631 (QuantReg: 15.46420) QuantErr: 15.46420 batch_time=0.52298
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 3.23721 (QuantReg: 15.18345) QuantErr: 15.18345 batch_time=0.52792
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 2.67958 (QuantReg: 15.35156) QuantErr: 15.35156 batch_time=0.51765
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 2.78583 (QuantReg: 15.25363) QuantErr: 15.25363 batch_time=0.65450
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 2.91551 (QuantReg: 15.49558) QuantErr: 15.49558 batch_time=0.52815
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 3.26142 (QuantReg: 15.47423) QuantErr: 15.47423 batch_time=1.34514
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 2.74287 (QuantReg: 15.28409) QuantErr: 15.28409 batch_time=0.49232
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 2.90092 (QuantReg: 15.58489) QuantErr: 15.58489 batch_time=0.48660
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 3.22994 (QuantReg: 15.36506) QuantErr: 15.36506 batch_time=0.51920
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 3.20629 (QuantReg: 15.40315) QuantErr: 15.40315 batch_time=0.62134
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 2.95116 (QuantReg: 15.42104) QuantErr: 15.42104 batch_time=0.48775
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 2.76790 (QuantReg: 15.30691) QuantErr: 15.30691 batch_time=0.52093
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 2.95578 (QuantReg: 15.63961) QuantErr: 15.63961 batch_time=0.51574
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 2.96307 (QuantReg: 15.58809) QuantErr: 15.58809 batch_time=0.52925
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 2.95814 (QuantReg: 15.53874) QuantErr: 15.53874 batch_time=0.50771
Train Epoch: 15 codebook_update_time=1.69771
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch15.pth ...
Done in 4.138s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch15.pth ...
Done in 9.250s
removing stale ckpt [epoch 14] [took 0.00s]
epoch : 15
loss : 2.9701788778305054
quant_reg : 15.355252708435058
quant_err : 15.355252708435058
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
LSMDC_full_test/t2v_metrics/R1: 12.7
LSMDC_full_test/t2v_metrics/R5: 32.3
LSMDC_full_test/t2v_metrics/R10: 42.5
LSMDC_full_test/t2v_metrics/R50: 68.6
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 69.281
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.92975367456968
LSMDC_full_test/v2t_metrics/R1: 12.8
LSMDC_full_test/v2t_metrics/R5: 32.4
LSMDC_full_test/v2t_metrics/R10: 41.1
LSMDC_full_test/v2t_metrics/R50: 66.4
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 69.151
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.73547966980102
mnt_best : 25.92975367456968
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 2.91978 (QuantReg: 15.42098) QuantErr: 15.42098 batch_time=24.78299
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 2.58198 (QuantReg: 15.46780) QuantErr: 15.46780 batch_time=0.57871
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 2.48403 (QuantReg: 15.37534) QuantErr: 15.37534 batch_time=4.04799
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 3.25653 (QuantReg: 15.28411) QuantErr: 15.28411 batch_time=0.50859
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 3.23543 (QuantReg: 15.33045) QuantErr: 15.33045 batch_time=0.50715
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 2.91642 (QuantReg: 15.60405) QuantErr: 15.60405 batch_time=0.52717
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 2.72453 (QuantReg: 15.36668) QuantErr: 15.36668 batch_time=0.52023
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 2.94009 (QuantReg: 15.31731) QuantErr: 15.31731 batch_time=0.54102
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 2.68504 (QuantReg: 15.50475) QuantErr: 15.50475 batch_time=0.50426
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 3.07324 (QuantReg: 15.52563) QuantErr: 15.52563 batch_time=0.51864
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 3.13908 (QuantReg: 15.28142) QuantErr: 15.28142 batch_time=0.48852
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 3.20652 (QuantReg: 15.20188) QuantErr: 15.20188 batch_time=0.50676
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 2.89634 (QuantReg: 15.46936) QuantErr: 15.46936 batch_time=1.49239
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 3.15866 (QuantReg: 15.50217) QuantErr: 15.50217 batch_time=1.01393
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 2.48426 (QuantReg: 15.49042) QuantErr: 15.49042 batch_time=0.51289
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 2.78046 (QuantReg: 15.70833) QuantErr: 15.70833 batch_time=0.52220
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 2.85922 (QuantReg: 15.29770) QuantErr: 15.29770 batch_time=0.53733
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 2.97415 (QuantReg: 15.56251) QuantErr: 15.56251 batch_time=0.55193
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 2.85193 (QuantReg: 15.54869) QuantErr: 15.54869 batch_time=0.53519
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 3.06110 (QuantReg: 15.57310) QuantErr: 15.57310 batch_time=0.55970
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 2.60441 (QuantReg: 15.75712) QuantErr: 15.75712 batch_time=0.52388
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 2.73582 (QuantReg: 15.66502) QuantErr: 15.66502 batch_time=0.50304
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 2.77391 (QuantReg: 15.68834) QuantErr: 15.68834 batch_time=0.50714
Train Epoch: 16 codebook_update_time=1.67877
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch16.pth ...
Done in 5.028s
removing stale ckpt [epoch 15] [took 0.00s]
epoch : 16
loss : 2.8806168384552002
quant_reg : 15.463501258850098
quant_err : 15.463501258850098
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
LSMDC_full_test/t2v_metrics/R1: 12.8
LSMDC_full_test/t2v_metrics/R5: 31.7
LSMDC_full_test/t2v_metrics/R10: 42.1
LSMDC_full_test/t2v_metrics/R50: 68.0
LSMDC_full_test/t2v_metrics/MedR: 16.0
LSMDC_full_test/t2v_metrics/MeanR: 70.876
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.754341053400786
LSMDC_full_test/v2t_metrics/R1: 11.5
LSMDC_full_test/v2t_metrics/R5: 30.6
LSMDC_full_test/v2t_metrics/R10: 39.9
LSMDC_full_test/v2t_metrics/R50: 66.7
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 71.476
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.12481847250813
mnt_best : 25.92975367456968
not_improved_count: 1
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 2.82544 (QuantReg: 15.30355) QuantErr: 15.30355 batch_time=29.75651
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 3.35902 (QuantReg: 15.40180) QuantErr: 15.40180 batch_time=0.49505
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 2.99976 (QuantReg: 15.65120) QuantErr: 15.65120 batch_time=0.49172
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 2.55468 (QuantReg: 15.62197) QuantErr: 15.62197 batch_time=0.51321
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 3.07742 (QuantReg: 15.35294) QuantErr: 15.35294 batch_time=0.48799
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 2.71046 (QuantReg: 15.36494) QuantErr: 15.36494 batch_time=0.50197
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 2.50781 (QuantReg: 15.58271) QuantErr: 15.58271 batch_time=0.50569
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 2.30532 (QuantReg: 15.62879) QuantErr: 15.62879 batch_time=0.51020
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 2.98055 (QuantReg: 15.41893) QuantErr: 15.41893 batch_time=0.49569
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 2.65220 (QuantReg: 15.72803) QuantErr: 15.72803 batch_time=0.54409
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 2.26245 (QuantReg: 15.81507) QuantErr: 15.81507 batch_time=0.51043
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 2.84224 (QuantReg: 15.51343) QuantErr: 15.51343 batch_time=0.50133
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 2.59502 (QuantReg: 15.55627) QuantErr: 15.55627 batch_time=0.49310
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 2.84946 (QuantReg: 15.46775) QuantErr: 15.46775 batch_time=0.53746
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 2.88640 (QuantReg: 15.42031) QuantErr: 15.42031 batch_time=0.52535
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 2.42102 (QuantReg: 15.67105) QuantErr: 15.67105 batch_time=0.50172
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 2.84704 (QuantReg: 15.51297) QuantErr: 15.51297 batch_time=0.49605
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 2.98901 (QuantReg: 15.55447) QuantErr: 15.55447 batch_time=0.49717
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 2.55873 (QuantReg: 15.80661) QuantErr: 15.80661 batch_time=0.49741
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 3.05358 (QuantReg: 15.43497) QuantErr: 15.43497 batch_time=0.50247
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 2.70734 (QuantReg: 15.62945) QuantErr: 15.62945 batch_time=0.49552
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 3.24838 (QuantReg: 15.52886) QuantErr: 15.52886 batch_time=0.49053
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 2.46368 (QuantReg: 15.67032) QuantErr: 15.67032 batch_time=0.48557
Train Epoch: 17 codebook_update_time=1.62130
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch17.pth ...
Done in 4.052s
removing stale ckpt [epoch 16] [took 0.01s]
epoch : 17
loss : 2.774734208106995
quant_reg : 15.519091297149659
quant_err : 15.519091297149659
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
LSMDC_full_test/t2v_metrics/R1: 13.6
LSMDC_full_test/t2v_metrics/R5: 30.7
LSMDC_full_test/t2v_metrics/R10: 41.2
LSMDC_full_test/t2v_metrics/R50: 67.2
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 73.369
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.81417004986908
LSMDC_full_test/v2t_metrics/R1: 12.2
LSMDC_full_test/v2t_metrics/R5: 30.7
LSMDC_full_test/v2t_metrics/R10: 41.3
LSMDC_full_test/v2t_metrics/R50: 67.5
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 71.11
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.916254178744794
mnt_best : 25.92975367456968
not_improved_count: 2
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 2.81828 (QuantReg: 15.40072) QuantErr: 15.40072 batch_time=21.77964
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 2.51631 (QuantReg: 15.64215) QuantErr: 15.64215 batch_time=0.51975
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 2.53215 (QuantReg: 15.47678) QuantErr: 15.47678 batch_time=0.53772
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 2.58940 (QuantReg: 15.55015) QuantErr: 15.55015 batch_time=0.49387
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 2.54186 (QuantReg: 15.64844) QuantErr: 15.64844 batch_time=0.55500
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 2.72422 (QuantReg: 15.58881) QuantErr: 15.58881 batch_time=0.49598
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 2.57232 (QuantReg: 15.61292) QuantErr: 15.61292 batch_time=5.96768
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 2.88813 (QuantReg: 15.74625) QuantErr: 15.74625 batch_time=0.49878
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 3.14547 (QuantReg: 15.47215) QuantErr: 15.47215 batch_time=0.51618
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 2.58166 (QuantReg: 15.58800) QuantErr: 15.58800 batch_time=0.49872
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 2.60722 (QuantReg: 15.71329) QuantErr: 15.71329 batch_time=0.51322
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 2.54975 (QuantReg: 15.79955) QuantErr: 15.79955 batch_time=0.49965
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 2.39661 (QuantReg: 15.55825) QuantErr: 15.55825 batch_time=0.50091
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 2.67312 (QuantReg: 15.57818) QuantErr: 15.57818 batch_time=0.54323
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 2.49201 (QuantReg: 15.67777) QuantErr: 15.67777 batch_time=0.50990
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 3.05878 (QuantReg: 15.63169) QuantErr: 15.63169 batch_time=0.50643
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 2.74236 (QuantReg: 15.66126) QuantErr: 15.66126 batch_time=0.50463
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 2.78933 (QuantReg: 15.54911) QuantErr: 15.54911 batch_time=0.55751
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 2.43994 (QuantReg: 15.72536) QuantErr: 15.72536 batch_time=0.49903
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 2.69263 (QuantReg: 15.69522) QuantErr: 15.69522 batch_time=0.50120
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 3.34024 (QuantReg: 15.66333) QuantErr: 15.66333 batch_time=0.48832
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 2.90443 (QuantReg: 15.66675) QuantErr: 15.66675 batch_time=0.51089
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 2.78251 (QuantReg: 15.42438) QuantErr: 15.42438 batch_time=0.49217
Train Epoch: 18 codebook_update_time=1.70059
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch18.pth ...
Done in 4.893s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 2.6787100219726563
quant_reg : 15.605230697631836
quant_err : 15.605230697631836
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
LSMDC_full_test/t2v_metrics/R1: 12.3
LSMDC_full_test/t2v_metrics/R5: 31.5
LSMDC_full_test/t2v_metrics/R10: 41.3
LSMDC_full_test/t2v_metrics/R50: 67.8
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 70.436
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.19930553641765
LSMDC_full_test/v2t_metrics/R1: 12.5
LSMDC_full_test/v2t_metrics/R5: 32.1
LSMDC_full_test/v2t_metrics/R10: 40.6
LSMDC_full_test/v2t_metrics/R50: 68.0
LSMDC_full_test/v2t_metrics/MedR: 17.0
LSMDC_full_test/v2t_metrics/MeanR: 69.711
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.35013985583164
mnt_best : 25.92975367456968
not_improved_count: 3
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 2.23869 (QuantReg: 15.59028) QuantErr: 15.59028 batch_time=25.15831
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 2.88702 (QuantReg: 15.50405) QuantErr: 15.50405 batch_time=0.52884
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 2.98372 (QuantReg: 15.55759) QuantErr: 15.55759 batch_time=0.52368
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 2.72585 (QuantReg: 15.58763) QuantErr: 15.58763 batch_time=0.53116
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 2.34105 (QuantReg: 15.65405) QuantErr: 15.65405 batch_time=0.51111
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 3.12431 (QuantReg: 15.72095) QuantErr: 15.72095 batch_time=0.48788
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 2.80681 (QuantReg: 15.63144) QuantErr: 15.63144 batch_time=0.48920
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 2.82685 (QuantReg: 15.71789) QuantErr: 15.71789 batch_time=0.50545
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 2.46535 (QuantReg: 15.69223) QuantErr: 15.69223 batch_time=0.52714
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 2.67690 (QuantReg: 15.67342) QuantErr: 15.67342 batch_time=0.50174
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 2.76294 (QuantReg: 15.55523) QuantErr: 15.55523 batch_time=0.54106
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 2.69873 (QuantReg: 15.53086) QuantErr: 15.53086 batch_time=0.52139
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 2.36648 (QuantReg: 15.87694) QuantErr: 15.87694 batch_time=0.48880
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 2.76472 (QuantReg: 15.53306) QuantErr: 15.53306 batch_time=3.22231
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 2.83441 (QuantReg: 15.90073) QuantErr: 15.90073 batch_time=0.50192
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 2.50906 (QuantReg: 16.03325) QuantErr: 16.03325 batch_time=0.51269
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 2.53539 (QuantReg: 15.71817) QuantErr: 15.71817 batch_time=0.50109
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 2.51921 (QuantReg: 15.68431) QuantErr: 15.68431 batch_time=0.51115
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 2.23887 (QuantReg: 15.92842) QuantErr: 15.92842 batch_time=2.68426
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 2.69459 (QuantReg: 15.68806) QuantErr: 15.68806 batch_time=0.50615
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 2.60923 (QuantReg: 15.61248) QuantErr: 15.61248 batch_time=0.56356
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 2.49781 (QuantReg: 15.80331) QuantErr: 15.80331 batch_time=0.50023
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 2.23375 (QuantReg: 15.73761) QuantErr: 15.73761 batch_time=0.51300
Train Epoch: 19 codebook_update_time=2.05499
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC/checkpoint-epoch19.pth ...
Done in 17.801s
removing stale ckpt [epoch 18] [took 0.05s]
epoch : 19
loss : 2.6269238166809084
quant_reg : 15.68517105102539
quant_err : 15.68517105102539
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
LSMDC_full_test/t2v_metrics/R1: 12.0
LSMDC_full_test/t2v_metrics/R5: 31.8
LSMDC_full_test/t2v_metrics/R10: 41.6
LSMDC_full_test/t2v_metrics/R50: 68.2
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 70.597
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.132396277959582
LSMDC_full_test/v2t_metrics/R1: 11.9
LSMDC_full_test/v2t_metrics/R5: 31.5
LSMDC_full_test/v2t_metrics/R10: 41.1
LSMDC_full_test/v2t_metrics/R50: 68.0
LSMDC_full_test/v2t_metrics/MedR: 18.5