-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_LSMDC_M16.txt
2595 lines (2595 loc) · 189 KB
/
HCQ_LSMDC_M16.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16
Preparing the dataloaders ...
Loading dataset LSMDC_full_trainval in ram ...
Finish loading dataset LSMDC_full_trainval in ram, taking 6888.030732631683 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 135.2057557106018 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 133.85634183883667 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch0.pth ...
Done in 2.301s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch0.pth ...
Done in 4.086s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
LSMDC_full_test/t2v_metrics/R1: 0.1
LSMDC_full_test/t2v_metrics/R5: 0.5
LSMDC_full_test/t2v_metrics/R10: 1.0
LSMDC_full_test/t2v_metrics/R50: 5.2
LSMDC_full_test/t2v_metrics/MedR: 502.5
LSMDC_full_test/t2v_metrics/MeanR: 502.101
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.3684031498640387
LSMDC_full_test/v2t_metrics/R1: 0.1
LSMDC_full_test/v2t_metrics/R5: 0.4
LSMDC_full_test/v2t_metrics/R10: 1.0
LSMDC_full_test/v2t_metrics/R50: 5.0
LSMDC_full_test/v2t_metrics/MedR: 500.5
LSMDC_full_test/v2t_metrics/MeanR: 504.556
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.3419951893353394
mnt_best : 0.3684031498640387
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.80813 (QuantReg: 16.60020) QuantErr: 16.60020 batch_time=18.20107
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.23511 (QuantReg: 16.67810) QuantErr: 16.67810 batch_time=0.90251
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 8.76913 (QuantReg: 16.78547) QuantErr: 16.78547 batch_time=0.43609
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 8.04045 (QuantReg: 16.70878) QuantErr: 16.70878 batch_time=0.46313
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 7.82351 (QuantReg: 16.73412) QuantErr: 16.73412 batch_time=0.45716
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 7.42084 (QuantReg: 16.73405) QuantErr: 16.73405 batch_time=0.43350
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 7.25053 (QuantReg: 16.75078) QuantErr: 16.75078 batch_time=0.44789
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 6.93105 (QuantReg: 16.73678) QuantErr: 16.73678 batch_time=0.46055
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 6.95985 (QuantReg: 16.71000) QuantErr: 16.71000 batch_time=0.45532
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 6.76522 (QuantReg: 16.71279) QuantErr: 16.71279 batch_time=0.45071
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 7.25570 (QuantReg: 16.70739) QuantErr: 16.70739 batch_time=0.44356
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 6.15134 (QuantReg: 16.71145) QuantErr: 16.71145 batch_time=0.46603
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 6.61384 (QuantReg: 16.68544) QuantErr: 16.68544 batch_time=0.44095
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 6.97149 (QuantReg: 16.71037) QuantErr: 16.71037 batch_time=3.70542
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 6.50945 (QuantReg: 16.71835) QuantErr: 16.71835 batch_time=0.47087
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 6.13913 (QuantReg: 16.73113) QuantErr: 16.73113 batch_time=0.45113
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 6.92151 (QuantReg: 16.72342) QuantErr: 16.72342 batch_time=0.43872
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 6.31329 (QuantReg: 16.68822) QuantErr: 16.68822 batch_time=0.45352
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 6.04838 (QuantReg: 16.73128) QuantErr: 16.73128 batch_time=0.45909
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 6.43167 (QuantReg: 16.71347) QuantErr: 16.71347 batch_time=0.48534
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 5.78127 (QuantReg: 16.71377) QuantErr: 16.71377 batch_time=0.99296
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 5.79281 (QuantReg: 16.71746) QuantErr: 16.71746 batch_time=0.44619
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 6.96205 (QuantReg: 16.70837) QuantErr: 16.70837 batch_time=0.45252
Train Epoch: 1 codebook_update_time=1.31078
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch1.pth ...
Done in 6.317s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch1.pth ...
Done in 11.308s
epoch : 1
loss : 7.000573362350464
quant_reg : 16.713659873962403
quant_err : 16.713659873962403
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
LSMDC_full_test/t2v_metrics/R1: 6.9
LSMDC_full_test/t2v_metrics/R5: 19.2
LSMDC_full_test/t2v_metrics/R10: 26.7
LSMDC_full_test/t2v_metrics/R50: 53.5
LSMDC_full_test/t2v_metrics/MedR: 40.0
LSMDC_full_test/t2v_metrics/MeanR: 105.077
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 15.236569380944509
LSMDC_full_test/v2t_metrics/R1: 5.7
LSMDC_full_test/v2t_metrics/R5: 16.7
LSMDC_full_test/v2t_metrics/R10: 24.5
LSMDC_full_test/v2t_metrics/R50: 54.2
LSMDC_full_test/v2t_metrics/MedR: 42.0
LSMDC_full_test/v2t_metrics/MeanR: 107.612
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 13.261290957168043
mnt_best : 15.236569380944509
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 5.88743 (QuantReg: 6.49538) QuantErr: 6.49538 batch_time=23.10187
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 6.05093 (QuantReg: 6.39893) QuantErr: 6.39893 batch_time=0.49437
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 6.14830 (QuantReg: 6.80723) QuantErr: 6.80723 batch_time=0.43611
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 5.91374 (QuantReg: 6.85566) QuantErr: 6.85566 batch_time=0.44630
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 6.22207 (QuantReg: 6.52403) QuantErr: 6.52403 batch_time=0.44451
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 6.53468 (QuantReg: 7.11671) QuantErr: 7.11671 batch_time=0.46668
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 6.25592 (QuantReg: 6.86123) QuantErr: 6.86123 batch_time=3.73259
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 5.38881 (QuantReg: 7.20683) QuantErr: 7.20683 batch_time=0.48664
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 5.84300 (QuantReg: 7.47165) QuantErr: 7.47165 batch_time=0.46332
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.85582 (QuantReg: 7.85799) QuantErr: 7.85799 batch_time=0.44623
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 5.41230 (QuantReg: 7.35883) QuantErr: 7.35883 batch_time=0.45864
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 5.68491 (QuantReg: 7.57092) QuantErr: 7.57092 batch_time=0.44809
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 5.89715 (QuantReg: 7.40820) QuantErr: 7.40820 batch_time=0.43927
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 5.81691 (QuantReg: 7.73598) QuantErr: 7.73598 batch_time=0.44840
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 5.87690 (QuantReg: 7.82400) QuantErr: 7.82400 batch_time=0.48746
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 5.37332 (QuantReg: 8.06760) QuantErr: 8.06760 batch_time=0.44713
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 5.27356 (QuantReg: 7.78609) QuantErr: 7.78609 batch_time=0.46527
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 6.19162 (QuantReg: 8.33595) QuantErr: 8.33595 batch_time=0.48867
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 6.22588 (QuantReg: 8.29670) QuantErr: 8.29670 batch_time=0.47005
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 5.52882 (QuantReg: 8.36941) QuantErr: 8.36941 batch_time=0.49377
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 5.29403 (QuantReg: 8.29875) QuantErr: 8.29875 batch_time=1.05806
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 5.74681 (QuantReg: 8.82193) QuantErr: 8.82193 batch_time=0.45138
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 5.70693 (QuantReg: 8.23384) QuantErr: 8.23384 batch_time=0.46849
Train Epoch: 2 codebook_update_time=0.96108
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch2.pth ...
Done in 6.457s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch2.pth ...
Done in 12.216s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.04s]
epoch : 2
loss : 5.717106925964355
quant_reg : 7.602819591522217
quant_err : 7.602819591522217
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
LSMDC_full_test/t2v_metrics/R1: 8.1
LSMDC_full_test/t2v_metrics/R5: 20.2
LSMDC_full_test/t2v_metrics/R10: 28.1
LSMDC_full_test/t2v_metrics/R50: 58.7
LSMDC_full_test/t2v_metrics/MedR: 33.0
LSMDC_full_test/t2v_metrics/MeanR: 92.905
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 16.62828920918468
LSMDC_full_test/v2t_metrics/R1: 7.1
LSMDC_full_test/v2t_metrics/R5: 19.2
LSMDC_full_test/v2t_metrics/R10: 28.9
LSMDC_full_test/v2t_metrics/R50: 56.2
LSMDC_full_test/v2t_metrics/MedR: 38.0
LSMDC_full_test/v2t_metrics/MeanR: 95.186
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 15.79376990927839
mnt_best : 16.62828920918468
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 5.68815 (QuantReg: 7.36868) QuantErr: 7.36868 batch_time=21.68868
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 5.07569 (QuantReg: 7.19173) QuantErr: 7.19173 batch_time=0.45111
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 5.56694 (QuantReg: 7.16687) QuantErr: 7.16687 batch_time=0.44199
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 5.13218 (QuantReg: 7.38348) QuantErr: 7.38348 batch_time=0.48782
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 5.12229 (QuantReg: 7.60393) QuantErr: 7.60393 batch_time=0.45182
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 5.19731 (QuantReg: 7.35706) QuantErr: 7.35706 batch_time=0.44317
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 5.11533 (QuantReg: 7.83664) QuantErr: 7.83664 batch_time=0.45933
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 5.71197 (QuantReg: 7.43404) QuantErr: 7.43404 batch_time=0.44217
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 5.25756 (QuantReg: 7.60197) QuantErr: 7.60197 batch_time=0.44941
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 5.00012 (QuantReg: 7.96150) QuantErr: 7.96150 batch_time=0.47071
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 5.15210 (QuantReg: 7.61969) QuantErr: 7.61969 batch_time=0.44485
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 5.03294 (QuantReg: 7.75224) QuantErr: 7.75224 batch_time=0.50004
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 5.12249 (QuantReg: 7.59288) QuantErr: 7.59288 batch_time=0.44047
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 5.35070 (QuantReg: 7.84923) QuantErr: 7.84923 batch_time=0.45983
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 4.78238 (QuantReg: 7.89198) QuantErr: 7.89198 batch_time=0.46746
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 5.23313 (QuantReg: 7.82489) QuantErr: 7.82489 batch_time=0.44342
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 4.92784 (QuantReg: 8.24293) QuantErr: 8.24293 batch_time=0.44822
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 5.36509 (QuantReg: 7.87507) QuantErr: 7.87507 batch_time=0.48853
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 4.80463 (QuantReg: 8.32699) QuantErr: 8.32699 batch_time=5.13555
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 5.29440 (QuantReg: 8.24912) QuantErr: 8.24912 batch_time=0.46038
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 4.86281 (QuantReg: 7.95062) QuantErr: 7.95062 batch_time=0.44005
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 4.90890 (QuantReg: 8.41081) QuantErr: 8.41081 batch_time=0.45323
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 4.93700 (QuantReg: 8.21226) QuantErr: 8.21226 batch_time=0.54202
Train Epoch: 3 codebook_update_time=1.00422
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch3.pth ...
Done in 4.351s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch3.pth ...
Done in 10.448s
removing stale ckpt [epoch 2] [took 0.02s]
epoch : 3
loss : 5.210936080932617
quant_reg : 7.751526786804199
quant_err : 7.751526786804199
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
LSMDC_full_test/t2v_metrics/R1: 8.7
LSMDC_full_test/t2v_metrics/R5: 21.6
LSMDC_full_test/t2v_metrics/R10: 30.5
LSMDC_full_test/t2v_metrics/R50: 60.6
LSMDC_full_test/t2v_metrics/MedR: 31.0
LSMDC_full_test/t2v_metrics/MeanR: 84.99
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 17.896067715016112
LSMDC_full_test/v2t_metrics/R1: 7.6
LSMDC_full_test/v2t_metrics/R5: 22.2
LSMDC_full_test/v2t_metrics/R10: 30.7
LSMDC_full_test/v2t_metrics/R50: 58.9
LSMDC_full_test/v2t_metrics/MedR: 33.0
LSMDC_full_test/v2t_metrics/MeanR: 90.133
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 17.302212732218724
mnt_best : 17.896067715016112
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 5.25445 (QuantReg: 7.41508) QuantErr: 7.41508 batch_time=22.51966
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 4.93641 (QuantReg: 7.74656) QuantErr: 7.74656 batch_time=0.46798
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 5.21480 (QuantReg: 7.80516) QuantErr: 7.80516 batch_time=0.87759
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 4.89903 (QuantReg: 7.80707) QuantErr: 7.80707 batch_time=0.48978
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 4.44482 (QuantReg: 8.07814) QuantErr: 8.07814 batch_time=0.44226
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 4.45259 (QuantReg: 7.87737) QuantErr: 7.87737 batch_time=0.45081
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 4.91288 (QuantReg: 7.97359) QuantErr: 7.97359 batch_time=0.87049
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 5.22983 (QuantReg: 7.89100) QuantErr: 7.89100 batch_time=0.45054
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 4.35648 (QuantReg: 7.70114) QuantErr: 7.70114 batch_time=0.44177
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 4.89024 (QuantReg: 7.61102) QuantErr: 7.61102 batch_time=0.45078
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 5.51251 (QuantReg: 7.70697) QuantErr: 7.70697 batch_time=0.44574
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 4.30665 (QuantReg: 7.92575) QuantErr: 7.92575 batch_time=0.44366
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 4.98278 (QuantReg: 7.86965) QuantErr: 7.86965 batch_time=0.44660
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 5.05132 (QuantReg: 7.87337) QuantErr: 7.87337 batch_time=0.55576
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 5.08486 (QuantReg: 7.84504) QuantErr: 7.84504 batch_time=0.44877
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 5.28050 (QuantReg: 8.03465) QuantErr: 8.03465 batch_time=0.45200
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 4.56714 (QuantReg: 8.01540) QuantErr: 8.01540 batch_time=0.43720
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 5.32211 (QuantReg: 8.12245) QuantErr: 8.12245 batch_time=0.44492
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 4.38575 (QuantReg: 8.46876) QuantErr: 8.46876 batch_time=0.43174
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 4.49552 (QuantReg: 8.13517) QuantErr: 8.13517 batch_time=0.46796
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 3.76879 (QuantReg: 8.32564) QuantErr: 8.32564 batch_time=0.45421
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 4.47401 (QuantReg: 8.44999) QuantErr: 8.44999 batch_time=0.44756
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 4.72603 (QuantReg: 8.38128) QuantErr: 8.38128 batch_time=0.48104
Train Epoch: 4 codebook_update_time=1.00372
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch4.pth ...
Done in 5.201s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch4.pth ...
Done in 10.335s
removing stale ckpt [epoch 3] [took 0.21s]
epoch : 4
loss : 4.851342294692993
quant_reg : 7.973060361862182
quant_err : 7.973060361862182
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
LSMDC_full_test/t2v_metrics/R1: 9.6
LSMDC_full_test/t2v_metrics/R5: 23.0
LSMDC_full_test/t2v_metrics/R10: 32.7
LSMDC_full_test/t2v_metrics/R50: 61.3
LSMDC_full_test/t2v_metrics/MedR: 29.0
LSMDC_full_test/t2v_metrics/MeanR: 85.717
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.32779336577184
LSMDC_full_test/v2t_metrics/R1: 8.0
LSMDC_full_test/v2t_metrics/R5: 22.8
LSMDC_full_test/v2t_metrics/R10: 31.3
LSMDC_full_test/v2t_metrics/R50: 60.3
LSMDC_full_test/v2t_metrics/MedR: 29.0
LSMDC_full_test/v2t_metrics/MeanR: 84.8775
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 17.87268181911679
mnt_best : 19.32779336577184
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 5.57839 (QuantReg: 7.68633) QuantErr: 7.68633 batch_time=20.76483
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 4.37124 (QuantReg: 8.06041) QuantErr: 8.06041 batch_time=0.45729
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 4.65974 (QuantReg: 8.07576) QuantErr: 8.07576 batch_time=0.44600
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 5.00905 (QuantReg: 8.09737) QuantErr: 8.09737 batch_time=0.44862
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 4.74048 (QuantReg: 8.40351) QuantErr: 8.40351 batch_time=0.44514
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 4.22274 (QuantReg: 8.38735) QuantErr: 8.38735 batch_time=0.44375
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 4.52215 (QuantReg: 8.25353) QuantErr: 8.25353 batch_time=0.44396
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 4.95398 (QuantReg: 8.35333) QuantErr: 8.35333 batch_time=0.48368
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 5.20216 (QuantReg: 8.31112) QuantErr: 8.31112 batch_time=0.45121
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 4.95051 (QuantReg: 8.23996) QuantErr: 8.23996 batch_time=0.48646
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 4.75036 (QuantReg: 8.14888) QuantErr: 8.14888 batch_time=0.46813
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 4.52281 (QuantReg: 8.39878) QuantErr: 8.39878 batch_time=0.48977
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 4.23655 (QuantReg: 8.34173) QuantErr: 8.34173 batch_time=0.45895
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 4.45114 (QuantReg: 8.54931) QuantErr: 8.54931 batch_time=1.37315
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 4.87717 (QuantReg: 8.50009) QuantErr: 8.50009 batch_time=2.15069
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 4.84122 (QuantReg: 8.31745) QuantErr: 8.31745 batch_time=0.45582
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 4.13270 (QuantReg: 8.60820) QuantErr: 8.60820 batch_time=0.46260
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 4.23391 (QuantReg: 8.54260) QuantErr: 8.54260 batch_time=0.44866
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 4.25949 (QuantReg: 8.57550) QuantErr: 8.57550 batch_time=0.45510
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 4.42217 (QuantReg: 8.40551) QuantErr: 8.40551 batch_time=1.63622
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 4.11235 (QuantReg: 8.52475) QuantErr: 8.52475 batch_time=0.46014
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 4.33139 (QuantReg: 8.81980) QuantErr: 8.81980 batch_time=0.44048
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 3.80499 (QuantReg: 8.75303) QuantErr: 8.75303 batch_time=0.47180
Train Epoch: 5 codebook_update_time=1.07633
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch5.pth ...
Done in 4.521s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch5.pth ...
Done in 9.990s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 4.5177671909332275
quant_reg : 8.391044189453124
quant_err : 8.391044189453124
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
LSMDC_full_test/t2v_metrics/R1: 10.4
LSMDC_full_test/t2v_metrics/R5: 25.1
LSMDC_full_test/t2v_metrics/R10: 33.5
LSMDC_full_test/t2v_metrics/R50: 61.9
LSMDC_full_test/t2v_metrics/MedR: 25.0
LSMDC_full_test/t2v_metrics/MeanR: 80.818
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.60237506784811
LSMDC_full_test/v2t_metrics/R1: 9.0
LSMDC_full_test/v2t_metrics/R5: 25.2
LSMDC_full_test/v2t_metrics/R10: 34.3
LSMDC_full_test/v2t_metrics/R50: 61.2
LSMDC_full_test/v2t_metrics/MedR: 27.0
LSMDC_full_test/v2t_metrics/MeanR: 81.467
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 19.814314717329435
mnt_best : 20.60237506784811
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 4.87517 (QuantReg: 8.31013) QuantErr: 8.31013 batch_time=16.49701
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 4.45743 (QuantReg: 8.37688) QuantErr: 8.37688 batch_time=1.75927
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 4.71660 (QuantReg: 8.41461) QuantErr: 8.41461 batch_time=0.52196
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 4.17345 (QuantReg: 8.56005) QuantErr: 8.56005 batch_time=0.56361
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 4.15126 (QuantReg: 8.60410) QuantErr: 8.60410 batch_time=0.54110
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 4.47279 (QuantReg: 8.34987) QuantErr: 8.34987 batch_time=0.48126
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 4.26265 (QuantReg: 8.42287) QuantErr: 8.42287 batch_time=0.49206
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 4.27063 (QuantReg: 8.60718) QuantErr: 8.60718 batch_time=0.45567
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 4.88965 (QuantReg: 8.37517) QuantErr: 8.37517 batch_time=0.45330
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 4.51956 (QuantReg: 8.63547) QuantErr: 8.63547 batch_time=0.45921
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 3.90147 (QuantReg: 8.73513) QuantErr: 8.73513 batch_time=0.43561
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 4.29055 (QuantReg: 8.52868) QuantErr: 8.52868 batch_time=0.47516
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 4.20752 (QuantReg: 8.88189) QuantErr: 8.88189 batch_time=0.45832
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 4.93187 (QuantReg: 8.61882) QuantErr: 8.61882 batch_time=0.57388
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 3.98490 (QuantReg: 8.88018) QuantErr: 8.88018 batch_time=0.44270
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 3.55746 (QuantReg: 8.72691) QuantErr: 8.72691 batch_time=0.55246
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 3.92641 (QuantReg: 8.86756) QuantErr: 8.86756 batch_time=0.44763
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 4.17111 (QuantReg: 9.05954) QuantErr: 9.05954 batch_time=0.43706
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 4.84330 (QuantReg: 8.56766) QuantErr: 8.56766 batch_time=0.45111
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 4.23096 (QuantReg: 8.78454) QuantErr: 8.78454 batch_time=0.43191
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 3.95406 (QuantReg: 8.98522) QuantErr: 8.98522 batch_time=0.44267
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 4.82011 (QuantReg: 8.47509) QuantErr: 8.47509 batch_time=1.27016
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 4.46337 (QuantReg: 8.76305) QuantErr: 8.76305 batch_time=0.85399
Train Epoch: 6 codebook_update_time=0.97281
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch6.pth ...
Done in 4.718s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch6.pth ...
Done in 8.970s
removing stale ckpt [epoch 5] [took 0.01s]
epoch : 6
loss : 4.325583485603333
quant_reg : 8.655608528137208
quant_err : 8.655608528137208
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
LSMDC_full_test/t2v_metrics/R1: 10.3
LSMDC_full_test/t2v_metrics/R5: 25.1
LSMDC_full_test/t2v_metrics/R10: 34.7
LSMDC_full_test/t2v_metrics/R50: 62.8
LSMDC_full_test/t2v_metrics/MedR: 24.0
LSMDC_full_test/t2v_metrics/MeanR: 78.791
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.778465601280267
LSMDC_full_test/v2t_metrics/R1: 9.5
LSMDC_full_test/v2t_metrics/R5: 25.0
LSMDC_full_test/v2t_metrics/R10: 33.2
LSMDC_full_test/v2t_metrics/R50: 61.9
LSMDC_full_test/v2t_metrics/MedR: 27.0
LSMDC_full_test/v2t_metrics/MeanR: 84.328
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 19.903703762525467
mnt_best : 20.778465601280267
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 4.27217 (QuantReg: 8.71106) QuantErr: 8.71106 batch_time=17.89161
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 3.98079 (QuantReg: 8.79245) QuantErr: 8.79245 batch_time=0.44543
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 4.04631 (QuantReg: 8.66875) QuantErr: 8.66875 batch_time=0.44796
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 3.92168 (QuantReg: 8.72384) QuantErr: 8.72384 batch_time=0.44405
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 4.57243 (QuantReg: 8.76772) QuantErr: 8.76772 batch_time=0.44417
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 4.01230 (QuantReg: 8.84206) QuantErr: 8.84206 batch_time=0.44704
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 4.16819 (QuantReg: 8.74328) QuantErr: 8.74328 batch_time=0.69376
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 4.49653 (QuantReg: 8.69528) QuantErr: 8.69528 batch_time=0.45620
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 3.99568 (QuantReg: 8.63845) QuantErr: 8.63845 batch_time=0.45503
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 4.07636 (QuantReg: 8.82223) QuantErr: 8.82223 batch_time=0.47163
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 3.98293 (QuantReg: 8.85756) QuantErr: 8.85756 batch_time=0.46582
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 4.36265 (QuantReg: 8.64992) QuantErr: 8.64992 batch_time=0.46955
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 3.95735 (QuantReg: 8.81306) QuantErr: 8.81306 batch_time=0.45449
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 3.80620 (QuantReg: 8.88463) QuantErr: 8.88463 batch_time=0.43203
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 3.87020 (QuantReg: 9.11216) QuantErr: 9.11216 batch_time=0.44183
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 3.74933 (QuantReg: 8.98969) QuantErr: 8.98969 batch_time=0.44032
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 4.23145 (QuantReg: 8.85579) QuantErr: 8.85579 batch_time=0.43813
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 4.09613 (QuantReg: 9.00907) QuantErr: 9.00907 batch_time=0.45032
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 3.76415 (QuantReg: 8.85251) QuantErr: 8.85251 batch_time=0.45221
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 3.83590 (QuantReg: 9.04399) QuantErr: 9.04399 batch_time=0.44413
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 3.70531 (QuantReg: 9.03404) QuantErr: 9.03404 batch_time=0.48577
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 3.80366 (QuantReg: 9.21252) QuantErr: 9.21252 batch_time=0.43863
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 3.88262 (QuantReg: 9.17271) QuantErr: 9.17271 batch_time=0.44588
Train Epoch: 7 codebook_update_time=0.97537
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch7.pth ...
Done in 14.946s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch7.pth ...
Done in 18.844s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 4.093128994941711
quant_reg : 8.884630176544189
quant_err : 8.884630176544189
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
LSMDC_full_test/t2v_metrics/R1: 10.6
LSMDC_full_test/t2v_metrics/R5: 26.6
LSMDC_full_test/t2v_metrics/R10: 35.7
LSMDC_full_test/t2v_metrics/R50: 63.7
LSMDC_full_test/t2v_metrics/MedR: 23.0
LSMDC_full_test/t2v_metrics/MeanR: 76.755
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.591620549442688
LSMDC_full_test/v2t_metrics/R1: 10.2
LSMDC_full_test/v2t_metrics/R5: 27.0
LSMDC_full_test/v2t_metrics/R10: 35.6
LSMDC_full_test/v2t_metrics/R50: 64.9
LSMDC_full_test/v2t_metrics/MedR: 23.5
LSMDC_full_test/v2t_metrics/MeanR: 75.875
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.40283538871361
mnt_best : 21.591620549442688
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 4.02503 (QuantReg: 8.90068) QuantErr: 8.90068 batch_time=19.63869
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 3.74321 (QuantReg: 8.97005) QuantErr: 8.97005 batch_time=0.44553
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 4.12001 (QuantReg: 9.21117) QuantErr: 9.21117 batch_time=0.44378
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 4.44422 (QuantReg: 9.03352) QuantErr: 9.03352 batch_time=0.47501
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 3.59150 (QuantReg: 8.95560) QuantErr: 8.95560 batch_time=0.44764
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 4.02183 (QuantReg: 9.14499) QuantErr: 9.14499 batch_time=0.46842
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 3.81012 (QuantReg: 8.89268) QuantErr: 8.89268 batch_time=0.46622
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 4.29228 (QuantReg: 9.13689) QuantErr: 9.13689 batch_time=0.67878
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 3.60996 (QuantReg: 9.23592) QuantErr: 9.23592 batch_time=0.47026
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 3.90563 (QuantReg: 8.90475) QuantErr: 8.90475 batch_time=0.45790
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 4.38930 (QuantReg: 9.03157) QuantErr: 9.03157 batch_time=0.44635
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 3.79139 (QuantReg: 9.15474) QuantErr: 9.15474 batch_time=0.46352
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 3.76597 (QuantReg: 9.17656) QuantErr: 9.17656 batch_time=1.79186
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 3.46704 (QuantReg: 9.40180) QuantErr: 9.40180 batch_time=0.79393
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 4.11457 (QuantReg: 9.23867) QuantErr: 9.23867 batch_time=0.48200
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 3.71788 (QuantReg: 9.15928) QuantErr: 9.15928 batch_time=0.44181
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 3.86718 (QuantReg: 9.26263) QuantErr: 9.26263 batch_time=0.44164
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 3.68707 (QuantReg: 9.16543) QuantErr: 9.16543 batch_time=0.44259
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 3.80844 (QuantReg: 8.99455) QuantErr: 8.99455 batch_time=0.45073
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 4.07120 (QuantReg: 9.14694) QuantErr: 9.14694 batch_time=1.72211
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 3.65729 (QuantReg: 9.28780) QuantErr: 9.28780 batch_time=0.43770
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 3.61612 (QuantReg: 9.36805) QuantErr: 9.36805 batch_time=0.48735
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 3.76437 (QuantReg: 9.16905) QuantErr: 9.16905 batch_time=0.48644
Train Epoch: 8 codebook_update_time=0.95504
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch8.pth ...
Done in 20.460s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch8.pth ...
Done in 25.144s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 3.9085370998382567
quant_reg : 9.08931160736084
quant_err : 9.08931160736084
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
LSMDC_full_test/t2v_metrics/R1: 10.6
LSMDC_full_test/t2v_metrics/R5: 27.4
LSMDC_full_test/t2v_metrics/R10: 35.7
LSMDC_full_test/t2v_metrics/R50: 63.6
LSMDC_full_test/t2v_metrics/MedR: 24.5
LSMDC_full_test/t2v_metrics/MeanR: 76.573
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.80594344581306
LSMDC_full_test/v2t_metrics/R1: 11.4
LSMDC_full_test/v2t_metrics/R5: 26.9
LSMDC_full_test/v2t_metrics/R10: 35.6
LSMDC_full_test/v2t_metrics/R50: 62.4
LSMDC_full_test/v2t_metrics/MedR: 24.0
LSMDC_full_test/v2t_metrics/MeanR: 78.896
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.183788180235894
mnt_best : 21.80594344581306
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 3.81232 (QuantReg: 8.71953) QuantErr: 8.71953 batch_time=27.36607
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 3.91829 (QuantReg: 9.09963) QuantErr: 9.09963 batch_time=0.45704
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 3.57479 (QuantReg: 9.22628) QuantErr: 9.22628 batch_time=1.77012
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 4.15988 (QuantReg: 9.40548) QuantErr: 9.40548 batch_time=0.44678
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 4.15752 (QuantReg: 9.17285) QuantErr: 9.17285 batch_time=0.44409
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 3.40946 (QuantReg: 9.65050) QuantErr: 9.65050 batch_time=0.48390
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 3.74175 (QuantReg: 9.11323) QuantErr: 9.11323 batch_time=0.46178
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 4.18967 (QuantReg: 8.86868) QuantErr: 8.86868 batch_time=0.47214
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 3.61913 (QuantReg: 9.09366) QuantErr: 9.09366 batch_time=0.47820
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 3.49847 (QuantReg: 9.25379) QuantErr: 9.25379 batch_time=0.47161
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 4.27660 (QuantReg: 9.15442) QuantErr: 9.15442 batch_time=0.46526
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 3.64869 (QuantReg: 9.28783) QuantErr: 9.28783 batch_time=0.45000
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 3.43065 (QuantReg: 9.34021) QuantErr: 9.34021 batch_time=0.46210
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 3.60344 (QuantReg: 9.50682) QuantErr: 9.50682 batch_time=0.45756
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 3.65814 (QuantReg: 9.37974) QuantErr: 9.37974 batch_time=0.45819
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 3.88352 (QuantReg: 9.48736) QuantErr: 9.48736 batch_time=0.45036
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 3.44085 (QuantReg: 9.41875) QuantErr: 9.41875 batch_time=0.73268
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 3.44160 (QuantReg: 9.01103) QuantErr: 9.01103 batch_time=0.46306
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 3.77665 (QuantReg: 9.27922) QuantErr: 9.27922 batch_time=0.44832
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 3.66429 (QuantReg: 9.56770) QuantErr: 9.56770 batch_time=0.45072
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 4.29488 (QuantReg: 9.53256) QuantErr: 9.53256 batch_time=0.45955
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 4.31395 (QuantReg: 9.50042) QuantErr: 9.50042 batch_time=0.44686
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 3.70165 (QuantReg: 9.28005) QuantErr: 9.28005 batch_time=0.45572
Train Epoch: 9 codebook_update_time=0.96333
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch9.pth ...
Done in 18.746s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch9.pth ...
Done in 22.957s
removing stale ckpt [epoch 8] [took 0.01s]
epoch : 9
loss : 3.7323485202789306
quant_reg : 9.30251868057251
quant_err : 9.30251868057251
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
LSMDC_full_test/t2v_metrics/R1: 10.7
LSMDC_full_test/t2v_metrics/R5: 27.3
LSMDC_full_test/t2v_metrics/R10: 36.0
LSMDC_full_test/t2v_metrics/R50: 64.8
LSMDC_full_test/t2v_metrics/MedR: 24.0
LSMDC_full_test/t2v_metrics/MeanR: 74.373
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.908684864646492
LSMDC_full_test/v2t_metrics/R1: 11.5
LSMDC_full_test/v2t_metrics/R5: 27.4
LSMDC_full_test/v2t_metrics/R10: 36.7
LSMDC_full_test/v2t_metrics/R50: 63.2
LSMDC_full_test/v2t_metrics/MedR: 23.0
LSMDC_full_test/v2t_metrics/MeanR: 75.046
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.61369285334514
mnt_best : 21.908684864646492
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 3.49843 (QuantReg: 9.33204) QuantErr: 9.33204 batch_time=21.78026
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 3.48832 (QuantReg: 9.04024) QuantErr: 9.04024 batch_time=0.44568
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 3.51047 (QuantReg: 9.22651) QuantErr: 9.22651 batch_time=0.44963
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 3.44978 (QuantReg: 9.29053) QuantErr: 9.29053 batch_time=0.45555
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 3.81193 (QuantReg: 9.37398) QuantErr: 9.37398 batch_time=0.44078
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 3.24229 (QuantReg: 9.50597) QuantErr: 9.50597 batch_time=0.44268
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 3.68831 (QuantReg: 9.21218) QuantErr: 9.21218 batch_time=0.45585
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 3.53461 (QuantReg: 9.27088) QuantErr: 9.27088 batch_time=0.43644
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 3.56240 (QuantReg: 9.54774) QuantErr: 9.54774 batch_time=2.59153
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 3.75134 (QuantReg: 9.58114) QuantErr: 9.58114 batch_time=0.44414
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 3.59451 (QuantReg: 9.45369) QuantErr: 9.45369 batch_time=0.45743
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 3.63416 (QuantReg: 9.36587) QuantErr: 9.36587 batch_time=0.44996
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 3.28522 (QuantReg: 9.41054) QuantErr: 9.41054 batch_time=0.46132
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 3.51970 (QuantReg: 9.51122) QuantErr: 9.51122 batch_time=2.98188
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 4.03921 (QuantReg: 9.53700) QuantErr: 9.53700 batch_time=0.44285
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 3.74627 (QuantReg: 9.74001) QuantErr: 9.74001 batch_time=0.51397
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 3.70155 (QuantReg: 9.32814) QuantErr: 9.32814 batch_time=0.45187
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 3.57363 (QuantReg: 9.64512) QuantErr: 9.64512 batch_time=0.44138
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 3.18852 (QuantReg: 9.54450) QuantErr: 9.54450 batch_time=1.13185
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 3.38101 (QuantReg: 9.52805) QuantErr: 9.52805 batch_time=0.49280
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 3.22793 (QuantReg: 9.60392) QuantErr: 9.60392 batch_time=0.47119
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 4.10826 (QuantReg: 9.62496) QuantErr: 9.62496 batch_time=0.47901
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 3.61813 (QuantReg: 9.46037) QuantErr: 9.46037 batch_time=0.47147
Train Epoch: 10 codebook_update_time=0.94987
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch10.pth ...
Done in 4.026s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch10.pth ...
Done in 7.917s
removing stale ckpt [epoch 9] [took 0.00s]
epoch : 10
loss : 3.5787005853652953
quant_reg : 9.454306610107421
quant_err : 9.454306610107421
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
LSMDC_full_test/t2v_metrics/R1: 12.1
LSMDC_full_test/t2v_metrics/R5: 27.2
LSMDC_full_test/t2v_metrics/R10: 36.4
LSMDC_full_test/t2v_metrics/R50: 65.2
LSMDC_full_test/t2v_metrics/MedR: 22.0
LSMDC_full_test/t2v_metrics/MeanR: 73.351
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.881538358103207
LSMDC_full_test/v2t_metrics/R1: 9.7
LSMDC_full_test/v2t_metrics/R5: 25.4
LSMDC_full_test/v2t_metrics/R10: 35.8
LSMDC_full_test/v2t_metrics/R50: 63.1
LSMDC_full_test/v2t_metrics/MedR: 23.0
LSMDC_full_test/v2t_metrics/MeanR: 74.483
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.661546540598835
mnt_best : 22.881538358103207
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 3.08450 (QuantReg: 9.64860) QuantErr: 9.64860 batch_time=18.18239
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 3.56059 (QuantReg: 9.57872) QuantErr: 9.57872 batch_time=0.44363
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 3.57552 (QuantReg: 9.27504) QuantErr: 9.27504 batch_time=0.49027
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 3.45355 (QuantReg: 9.42262) QuantErr: 9.42262 batch_time=0.43493
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 3.53020 (QuantReg: 9.50331) QuantErr: 9.50331 batch_time=0.78820
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 4.10223 (QuantReg: 9.51430) QuantErr: 9.51430 batch_time=0.47177
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 3.60314 (QuantReg: 9.57871) QuantErr: 9.57871 batch_time=0.46293
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 3.55599 (QuantReg: 9.59161) QuantErr: 9.59161 batch_time=0.50026
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 3.57284 (QuantReg: 9.59464) QuantErr: 9.59464 batch_time=0.48084
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 2.98766 (QuantReg: 9.69353) QuantErr: 9.69353 batch_time=0.43895
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 3.64564 (QuantReg: 9.74210) QuantErr: 9.74210 batch_time=0.46062
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 2.90328 (QuantReg: 9.72550) QuantErr: 9.72550 batch_time=0.48611
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 3.66389 (QuantReg: 9.68914) QuantErr: 9.68914 batch_time=0.45718
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 2.92773 (QuantReg: 9.58800) QuantErr: 9.58800 batch_time=0.44222
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 3.39896 (QuantReg: 9.50994) QuantErr: 9.50994 batch_time=0.43781
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 3.92227 (QuantReg: 9.63297) QuantErr: 9.63297 batch_time=0.73079
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 2.98835 (QuantReg: 9.80018) QuantErr: 9.80018 batch_time=0.45853
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 3.30559 (QuantReg: 9.66726) QuantErr: 9.66726 batch_time=0.47023
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 3.51355 (QuantReg: 9.48385) QuantErr: 9.48385 batch_time=2.20445
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 3.66253 (QuantReg: 9.68864) QuantErr: 9.68864 batch_time=0.47145
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 3.44636 (QuantReg: 9.40356) QuantErr: 9.40356 batch_time=0.45774
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 2.95677 (QuantReg: 9.73173) QuantErr: 9.73173 batch_time=0.45914
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 3.43730 (QuantReg: 9.54524) QuantErr: 9.54524 batch_time=0.45508
Train Epoch: 11 codebook_update_time=0.88501
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch11.pth ...
Done in 4.919s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 3.417562710762024
quant_reg : 9.626183433532715
quant_err : 9.626183433532715
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
LSMDC_full_test/t2v_metrics/R1: 10.1
LSMDC_full_test/t2v_metrics/R5: 27.4
LSMDC_full_test/t2v_metrics/R10: 36.7
LSMDC_full_test/t2v_metrics/R50: 64.4
LSMDC_full_test/t2v_metrics/MedR: 23.0
LSMDC_full_test/t2v_metrics/MeanR: 75.579
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.656054395361966
LSMDC_full_test/v2t_metrics/R1: 11.1
LSMDC_full_test/v2t_metrics/R5: 25.5
LSMDC_full_test/v2t_metrics/R10: 34.3
LSMDC_full_test/v2t_metrics/R50: 63.8
LSMDC_full_test/v2t_metrics/MedR: 24.75
LSMDC_full_test/v2t_metrics/MeanR: 77.546
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.333024219696398
mnt_best : 22.881538358103207
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 3.50819 (QuantReg: 9.45185) QuantErr: 9.45185 batch_time=18.31366
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 3.02950 (QuantReg: 9.67419) QuantErr: 9.67419 batch_time=0.45855
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 3.79575 (QuantReg: 9.47812) QuantErr: 9.47812 batch_time=1.83537
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 2.92187 (QuantReg: 9.48413) QuantErr: 9.48413 batch_time=0.44238
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 2.92235 (QuantReg: 9.76719) QuantErr: 9.76719 batch_time=0.48614
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 3.00653 (QuantReg: 9.72903) QuantErr: 9.72903 batch_time=0.79663
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 3.42632 (QuantReg: 9.91706) QuantErr: 9.91706 batch_time=0.82651
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 3.44598 (QuantReg: 9.47915) QuantErr: 9.47915 batch_time=0.43812
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 3.30585 (QuantReg: 9.72928) QuantErr: 9.72928 batch_time=0.44151
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 3.23678 (QuantReg: 9.71546) QuantErr: 9.71546 batch_time=0.43921
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 3.34743 (QuantReg: 9.65701) QuantErr: 9.65701 batch_time=0.44157
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 3.18694 (QuantReg: 9.68745) QuantErr: 9.68745 batch_time=0.44055
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 3.70572 (QuantReg: 9.61939) QuantErr: 9.61939 batch_time=0.44478
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 3.59876 (QuantReg: 9.74815) QuantErr: 9.74815 batch_time=0.48691
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 3.42354 (QuantReg: 9.55854) QuantErr: 9.55854 batch_time=1.07808
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 3.33564 (QuantReg: 9.64395) QuantErr: 9.64395 batch_time=0.44460
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 2.58232 (QuantReg: 9.74638) QuantErr: 9.74638 batch_time=0.44124
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 3.63395 (QuantReg: 9.60320) QuantErr: 9.60320 batch_time=0.48746
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 2.90026 (QuantReg: 9.76363) QuantErr: 9.76363 batch_time=1.00492
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 3.56096 (QuantReg: 9.64527) QuantErr: 9.64527 batch_time=0.44657
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 3.22262 (QuantReg: 9.90398) QuantErr: 9.90398 batch_time=0.44797
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 2.98980 (QuantReg: 9.79226) QuantErr: 9.79226 batch_time=0.48894
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 3.28636 (QuantReg: 9.70418) QuantErr: 9.70418 batch_time=0.44105
Train Epoch: 12 codebook_update_time=0.93617
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch12.pth ...
Done in 4.002s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch12.pth ...
Done in 8.028s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 3.3003182191848754
quant_reg : 9.675750621795654
quant_err : 9.675750621795654
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
LSMDC_full_test/t2v_metrics/R1: 12.1
LSMDC_full_test/t2v_metrics/R5: 28.4
LSMDC_full_test/t2v_metrics/R10: 37.2
LSMDC_full_test/t2v_metrics/R50: 66.3
LSMDC_full_test/t2v_metrics/MedR: 21.0
LSMDC_full_test/t2v_metrics/MeanR: 73.954
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.3820301806873
LSMDC_full_test/v2t_metrics/R1: 12.0
LSMDC_full_test/v2t_metrics/R5: 27.5
LSMDC_full_test/v2t_metrics/R10: 37.2
LSMDC_full_test/v2t_metrics/R50: 64.7
LSMDC_full_test/v2t_metrics/MedR: 23.0
LSMDC_full_test/v2t_metrics/MeanR: 73.113
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.068478961858812
mnt_best : 23.3820301806873
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 2.81047 (QuantReg: 9.58516) QuantErr: 9.58516 batch_time=19.64388
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 3.35855 (QuantReg: 9.57770) QuantErr: 9.57770 batch_time=0.44283
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 3.23608 (QuantReg: 9.88609) QuantErr: 9.88609 batch_time=0.44012
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 3.08053 (QuantReg: 9.76530) QuantErr: 9.76530 batch_time=0.44504
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 2.89246 (QuantReg: 9.82256) QuantErr: 9.82256 batch_time=0.65700
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 3.52526 (QuantReg: 9.75738) QuantErr: 9.75738 batch_time=0.57852
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 3.18107 (QuantReg: 9.76152) QuantErr: 9.76152 batch_time=0.43338
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 3.21026 (QuantReg: 9.84838) QuantErr: 9.84838 batch_time=0.43623
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 2.71934 (QuantReg: 9.91179) QuantErr: 9.91179 batch_time=0.43537
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 3.49204 (QuantReg: 9.80798) QuantErr: 9.80798 batch_time=0.43066
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 3.26781 (QuantReg: 9.79236) QuantErr: 9.79236 batch_time=0.43347
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 3.04063 (QuantReg: 10.10048) QuantErr: 10.10048 batch_time=0.49571
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 3.42238 (QuantReg: 9.71375) QuantErr: 9.71375 batch_time=0.47486
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 3.47607 (QuantReg: 9.74513) QuantErr: 9.74513 batch_time=0.49694
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 3.21110 (QuantReg: 9.96949) QuantErr: 9.96949 batch_time=0.48043
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 3.03585 (QuantReg: 10.09367) QuantErr: 10.09367 batch_time=0.44068
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 2.96655 (QuantReg: 9.70682) QuantErr: 9.70682 batch_time=0.45187
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 3.39146 (QuantReg: 9.85115) QuantErr: 9.85115 batch_time=0.44386
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 3.20363 (QuantReg: 10.04651) QuantErr: 10.04651 batch_time=0.47751
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 3.06502 (QuantReg: 9.64026) QuantErr: 9.64026 batch_time=0.43326
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 3.30602 (QuantReg: 9.77302) QuantErr: 9.77302 batch_time=0.45873
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 3.20574 (QuantReg: 9.95782) QuantErr: 9.95782 batch_time=0.48753
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 3.07543 (QuantReg: 9.77176) QuantErr: 9.77176 batch_time=0.43666
Train Epoch: 13 codebook_update_time=0.90649
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch13.pth ...
Done in 4.601s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 3.1646372900009156
quant_reg : 9.821142448425293
quant_err : 9.821142448425293
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
LSMDC_full_test/t2v_metrics/R1: 11.5
LSMDC_full_test/t2v_metrics/R5: 28.4
LSMDC_full_test/t2v_metrics/R10: 38.1
LSMDC_full_test/t2v_metrics/R50: 65.4
LSMDC_full_test/t2v_metrics/MedR: 21.0
LSMDC_full_test/t2v_metrics/MeanR: 72.611
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.172899886381632
LSMDC_full_test/v2t_metrics/R1: 10.7
LSMDC_full_test/v2t_metrics/R5: 28.0
LSMDC_full_test/v2t_metrics/R10: 38.1
LSMDC_full_test/v2t_metrics/R50: 65.7
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 73.9615
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.515880147428675
mnt_best : 23.3820301806873
not_improved_count: 1
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 3.26329 (QuantReg: 9.64610) QuantErr: 9.64610 batch_time=20.72617
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 3.03235 (QuantReg: 9.75688) QuantErr: 9.75688 batch_time=1.67453
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 3.41506 (QuantReg: 9.67713) QuantErr: 9.67713 batch_time=0.44102
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 2.80442 (QuantReg: 9.76401) QuantErr: 9.76401 batch_time=0.43697
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 2.82830 (QuantReg: 9.96508) QuantErr: 9.96508 batch_time=0.43455
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 2.94484 (QuantReg: 9.91731) QuantErr: 9.91731 batch_time=0.43606
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 2.91898 (QuantReg: 9.80770) QuantErr: 9.80770 batch_time=1.20014
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 3.72807 (QuantReg: 9.71377) QuantErr: 9.71377 batch_time=0.48489
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 2.92520 (QuantReg: 9.80297) QuantErr: 9.80297 batch_time=0.44988
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 3.34590 (QuantReg: 9.94029) QuantErr: 9.94029 batch_time=0.50985
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.96336 (QuantReg: 9.84003) QuantErr: 9.84003 batch_time=0.43845
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 3.36344 (QuantReg: 9.99435) QuantErr: 9.99435 batch_time=0.43767
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 3.40734 (QuantReg: 9.86572) QuantErr: 9.86572 batch_time=0.44353
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 3.12663 (QuantReg: 9.99314) QuantErr: 9.99314 batch_time=0.45723
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 3.27142 (QuantReg: 9.97890) QuantErr: 9.97890 batch_time=0.49651
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 3.20768 (QuantReg: 9.83748) QuantErr: 9.83748 batch_time=0.43805
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 2.97379 (QuantReg: 10.03696) QuantErr: 10.03696 batch_time=0.43869
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 3.41326 (QuantReg: 10.15220) QuantErr: 10.15220 batch_time=0.44021
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 3.38098 (QuantReg: 9.86057) QuantErr: 9.86057 batch_time=0.45934
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 3.24306 (QuantReg: 10.06587) QuantErr: 10.06587 batch_time=0.48361
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 2.93951 (QuantReg: 10.12242) QuantErr: 10.12242 batch_time=0.44130
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 2.89963 (QuantReg: 10.08784) QuantErr: 10.08784 batch_time=0.44160
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 2.99238 (QuantReg: 10.02321) QuantErr: 10.02321 batch_time=0.45425
Train Epoch: 14 codebook_update_time=0.93166
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch14.pth ...
Done in 3.988s
removing stale ckpt [epoch 13] [took 0.01s]
epoch : 14
loss : 3.097786684036255
quant_reg : 9.898029998779297
quant_err : 9.898029998779297
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
LSMDC_full_test/t2v_metrics/R1: 11.1
LSMDC_full_test/t2v_metrics/R5: 27.9
LSMDC_full_test/t2v_metrics/R10: 39.7
LSMDC_full_test/t2v_metrics/R50: 66.0
LSMDC_full_test/t2v_metrics/MedR: 19.5
LSMDC_full_test/t2v_metrics/MeanR: 74.279
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.080182024361715
LSMDC_full_test/v2t_metrics/R1: 10.9
LSMDC_full_test/v2t_metrics/R5: 28.5
LSMDC_full_test/v2t_metrics/R10: 38.7
LSMDC_full_test/v2t_metrics/R50: 65.4
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 75.3335
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.908365713489992
mnt_best : 23.3820301806873
not_improved_count: 2
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 2.32133 (QuantReg: 9.86727) QuantErr: 9.86727 batch_time=19.27666
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 2.75903 (QuantReg: 9.69505) QuantErr: 9.69505 batch_time=0.43404
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 3.07635 (QuantReg: 9.97828) QuantErr: 9.97828 batch_time=0.43941
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 3.03974 (QuantReg: 9.98312) QuantErr: 9.98312 batch_time=0.44790
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 2.50659 (QuantReg: 10.04853) QuantErr: 10.04853 batch_time=0.48129
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 3.16791 (QuantReg: 10.08130) QuantErr: 10.08130 batch_time=0.43848
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 2.43819 (QuantReg: 10.10612) QuantErr: 10.10612 batch_time=1.30750
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 2.83491 (QuantReg: 9.95075) QuantErr: 9.95075 batch_time=0.44909
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 3.36948 (QuantReg: 10.03804) QuantErr: 10.03804 batch_time=0.44147
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 3.22301 (QuantReg: 9.91371) QuantErr: 9.91371 batch_time=0.46205
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 2.73590 (QuantReg: 9.88324) QuantErr: 9.88324 batch_time=0.48706
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 2.81725 (QuantReg: 9.93471) QuantErr: 9.93471 batch_time=0.44473
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 2.93244 (QuantReg: 9.96526) QuantErr: 9.96526 batch_time=0.68780
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 3.31961 (QuantReg: 10.05706) QuantErr: 10.05706 batch_time=0.55362
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 2.68895 (QuantReg: 9.90734) QuantErr: 9.90734 batch_time=0.43726
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 2.93725 (QuantReg: 10.13732) QuantErr: 10.13732 batch_time=0.46823
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 3.24783 (QuantReg: 10.03126) QuantErr: 10.03126 batch_time=0.46634
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 3.19189 (QuantReg: 9.97518) QuantErr: 9.97518 batch_time=0.44644
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 2.93129 (QuantReg: 9.98955) QuantErr: 9.98955 batch_time=1.17699
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 2.80155 (QuantReg: 9.91967) QuantErr: 9.91967 batch_time=0.99928
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 3.01584 (QuantReg: 10.18684) QuantErr: 10.18684 batch_time=0.47136
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 2.97072 (QuantReg: 10.14388) QuantErr: 10.14388 batch_time=0.44533
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 3.05026 (QuantReg: 10.11332) QuantErr: 10.11332 batch_time=0.45321
Train Epoch: 15 codebook_update_time=0.93115
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch15.pth ...
Done in 4.789s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch15.pth ...
Done in 9.449s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 2.988892858505249
quant_reg : 9.975523712158203
quant_err : 9.975523712158203
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
LSMDC_full_test/t2v_metrics/R1: 11.5
LSMDC_full_test/t2v_metrics/R5: 29.7
LSMDC_full_test/t2v_metrics/R10: 39.2
LSMDC_full_test/t2v_metrics/R50: 66.4
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 72.561
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.745434394774012
LSMDC_full_test/v2t_metrics/R1: 10.7
LSMDC_full_test/v2t_metrics/R5: 29.2
LSMDC_full_test/v2t_metrics/R10: 37.7
LSMDC_full_test/v2t_metrics/R50: 65.1
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 73.432
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.75285992303599
mnt_best : 23.745434394774012
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 2.94470 (QuantReg: 10.03616) QuantErr: 10.03616 batch_time=18.54312
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 2.64424 (QuantReg: 10.02532) QuantErr: 10.02532 batch_time=0.46773
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 2.57899 (QuantReg: 9.96509) QuantErr: 9.96509 batch_time=0.48012
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 3.22720 (QuantReg: 9.89879) QuantErr: 9.89879 batch_time=0.61516
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 3.26603 (QuantReg: 9.98230) QuantErr: 9.98230 batch_time=0.52007
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 2.93551 (QuantReg: 10.13495) QuantErr: 10.13495 batch_time=0.48112
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 2.79569 (QuantReg: 9.97728) QuantErr: 9.97728 batch_time=2.03918
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 3.03166 (QuantReg: 9.94693) QuantErr: 9.94693 batch_time=0.46005
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 2.70493 (QuantReg: 10.05117) QuantErr: 10.05117 batch_time=0.44925
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 3.06379 (QuantReg: 10.25565) QuantErr: 10.25565 batch_time=0.46834
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 3.23758 (QuantReg: 9.89538) QuantErr: 9.89538 batch_time=0.45386
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 3.31837 (QuantReg: 9.89844) QuantErr: 9.89844 batch_time=0.45236
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 2.88390 (QuantReg: 10.11952) QuantErr: 10.11952 batch_time=5.17446
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 3.18970 (QuantReg: 10.12513) QuantErr: 10.12513 batch_time=0.45520
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 2.47760 (QuantReg: 10.10478) QuantErr: 10.10478 batch_time=0.47308
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 2.80284 (QuantReg: 10.22119) QuantErr: 10.22119 batch_time=0.45020
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 2.89638 (QuantReg: 9.97857) QuantErr: 9.97857 batch_time=0.44221
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 3.03949 (QuantReg: 10.20189) QuantErr: 10.20189 batch_time=0.43966
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 2.81863 (QuantReg: 10.10804) QuantErr: 10.10804 batch_time=0.46793
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 3.06401 (QuantReg: 10.21363) QuantErr: 10.21363 batch_time=0.44927
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 2.56235 (QuantReg: 10.28218) QuantErr: 10.28218 batch_time=0.45539
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 2.76752 (QuantReg: 10.18348) QuantErr: 10.18348 batch_time=0.46418
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 2.78661 (QuantReg: 10.25572) QuantErr: 10.25572 batch_time=0.46154
Train Epoch: 16 codebook_update_time=1.11643
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch16.pth ...
Done in 22.961s
removing stale ckpt [epoch 15] [took 0.01s]
epoch : 16
loss : 2.895274779319763
quant_reg : 10.070990058898925
quant_err : 10.070990058898925
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
LSMDC_full_test/t2v_metrics/R1: 11.1
LSMDC_full_test/t2v_metrics/R5: 28.7
LSMDC_full_test/t2v_metrics/R10: 38.1
LSMDC_full_test/t2v_metrics/R50: 65.9
LSMDC_full_test/t2v_metrics/MedR: 20.5
LSMDC_full_test/t2v_metrics/MeanR: 74.181
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.981407154093908
LSMDC_full_test/v2t_metrics/R1: 11.2
LSMDC_full_test/v2t_metrics/R5: 27.6
LSMDC_full_test/v2t_metrics/R10: 37.4
LSMDC_full_test/v2t_metrics/R50: 65.5
LSMDC_full_test/v2t_metrics/MedR: 23.0
LSMDC_full_test/v2t_metrics/MeanR: 76.275
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.611683728209357
mnt_best : 23.745434394774012
not_improved_count: 1
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 2.82049 (QuantReg: 9.99072) QuantErr: 9.99072 batch_time=25.09242
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 3.34063 (QuantReg: 10.04739) QuantErr: 10.04739 batch_time=0.46697
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 3.05337 (QuantReg: 10.22948) QuantErr: 10.22948 batch_time=0.44111
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 2.61842 (QuantReg: 10.08718) QuantErr: 10.08718 batch_time=0.45236
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 3.06310 (QuantReg: 9.92312) QuantErr: 9.92312 batch_time=0.46843
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 2.73176 (QuantReg: 9.96657) QuantErr: 9.96657 batch_time=0.45314
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 2.46959 (QuantReg: 10.16198) QuantErr: 10.16198 batch_time=0.44443
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 2.31009 (QuantReg: 10.14337) QuantErr: 10.14337 batch_time=0.44942
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 2.98439 (QuantReg: 10.11490) QuantErr: 10.11490 batch_time=0.58939
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 2.67246 (QuantReg: 10.31755) QuantErr: 10.31755 batch_time=0.47196
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 2.28784 (QuantReg: 10.26248) QuantErr: 10.26248 batch_time=0.49732
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 2.86456 (QuantReg: 10.05383) QuantErr: 10.05383 batch_time=0.44042
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 2.58437 (QuantReg: 10.06874) QuantErr: 10.06874 batch_time=0.45076
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 2.84701 (QuantReg: 10.11598) QuantErr: 10.11598 batch_time=0.44917
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 3.02370 (QuantReg: 10.06936) QuantErr: 10.06936 batch_time=0.45317
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 2.45247 (QuantReg: 10.20415) QuantErr: 10.20415 batch_time=0.47973
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 2.83382 (QuantReg: 10.02373) QuantErr: 10.02373 batch_time=0.44654
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 3.03362 (QuantReg: 10.12439) QuantErr: 10.12439 batch_time=0.46692
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 2.61608 (QuantReg: 10.28924) QuantErr: 10.28924 batch_time=1.94481
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 3.16694 (QuantReg: 10.10982) QuantErr: 10.10982 batch_time=0.49525
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 2.75198 (QuantReg: 10.19293) QuantErr: 10.19293 batch_time=0.62924
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 3.33597 (QuantReg: 10.16115) QuantErr: 10.16115 batch_time=0.46597
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 2.44297 (QuantReg: 10.27078) QuantErr: 10.27078 batch_time=0.45077
Train Epoch: 17 codebook_update_time=0.91807
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch17.pth ...
Done in 4.547s
removing stale ckpt [epoch 16] [took 0.01s]
epoch : 17
loss : 2.8018358602523805
quant_reg : 10.112594104766846
quant_err : 10.112594104766846
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
LSMDC_full_test/t2v_metrics/R1: 10.9
LSMDC_full_test/t2v_metrics/R5: 28.5
LSMDC_full_test/t2v_metrics/R10: 38.1
LSMDC_full_test/t2v_metrics/R50: 66.5
LSMDC_full_test/t2v_metrics/MedR: 20.0
LSMDC_full_test/t2v_metrics/MeanR: 76.789
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.789359069658616
LSMDC_full_test/v2t_metrics/R1: 11.9
LSMDC_full_test/v2t_metrics/R5: 27.9
LSMDC_full_test/v2t_metrics/R10: 38.0
LSMDC_full_test/v2t_metrics/R50: 66.1
LSMDC_full_test/v2t_metrics/MedR: 22.0
LSMDC_full_test/v2t_metrics/MeanR: 75.104
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.27974687080845
mnt_best : 23.745434394774012
not_improved_count: 2
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 2.81192 (QuantReg: 10.12034) QuantErr: 10.12034 batch_time=19.50405
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 2.59780 (QuantReg: 10.21333) QuantErr: 10.21333 batch_time=0.45849
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 2.48089 (QuantReg: 10.07666) QuantErr: 10.07666 batch_time=0.44616
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 2.64122 (QuantReg: 10.22268) QuantErr: 10.22268 batch_time=1.34059
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 2.71550 (QuantReg: 10.23039) QuantErr: 10.23039 batch_time=0.45154
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 2.70470 (QuantReg: 10.11109) QuantErr: 10.11109 batch_time=0.44145
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 2.57683 (QuantReg: 10.15337) QuantErr: 10.15337 batch_time=0.45536
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 2.94393 (QuantReg: 10.24636) QuantErr: 10.24636 batch_time=0.44402
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 3.19129 (QuantReg: 10.05849) QuantErr: 10.05849 batch_time=0.44715
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 2.58990 (QuantReg: 10.14654) QuantErr: 10.14654 batch_time=0.90590
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 2.61650 (QuantReg: 10.28132) QuantErr: 10.28132 batch_time=0.44706
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 2.57666 (QuantReg: 10.24963) QuantErr: 10.24963 batch_time=0.45784
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 2.42376 (QuantReg: 10.10896) QuantErr: 10.10896 batch_time=0.48388
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 2.69053 (QuantReg: 10.15129) QuantErr: 10.15129 batch_time=0.67600
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 2.58433 (QuantReg: 10.26006) QuantErr: 10.26006 batch_time=0.46190
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 3.10930 (QuantReg: 10.21195) QuantErr: 10.21195 batch_time=0.45040
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 2.83188 (QuantReg: 10.22104) QuantErr: 10.22104 batch_time=0.44852
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 2.80400 (QuantReg: 10.17972) QuantErr: 10.17972 batch_time=0.45057
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 2.51521 (QuantReg: 10.29625) QuantErr: 10.29625 batch_time=0.44765
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 2.73052 (QuantReg: 10.18403) QuantErr: 10.18403 batch_time=0.45088
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 3.34062 (QuantReg: 10.27829) QuantErr: 10.27829 batch_time=0.46614
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 2.98949 (QuantReg: 10.16280) QuantErr: 10.16280 batch_time=0.47261
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 2.84276 (QuantReg: 10.05401) QuantErr: 10.05401 batch_time=0.44596
Train Epoch: 18 codebook_update_time=0.97179
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch18.pth ...
Done in 4.006s
removing stale ckpt [epoch 17] [took 0.00s]
epoch : 18
loss : 2.7165291233062745
quant_reg : 10.17250288772583
quant_err : 10.17250288772583
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
LSMDC_full_test/t2v_metrics/R1: 11.3
LSMDC_full_test/t2v_metrics/R5: 28.2
LSMDC_full_test/t2v_metrics/R10: 38.3
LSMDC_full_test/t2v_metrics/R50: 66.5
LSMDC_full_test/t2v_metrics/MedR: 20.0
LSMDC_full_test/t2v_metrics/MeanR: 72.905
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.0237171857724
LSMDC_full_test/v2t_metrics/R1: 10.6
LSMDC_full_test/v2t_metrics/R5: 28.9
LSMDC_full_test/v2t_metrics/R10: 38.4
LSMDC_full_test/v2t_metrics/R50: 65.0
LSMDC_full_test/v2t_metrics/MedR: 23.0
LSMDC_full_test/v2t_metrics/MeanR: 74.424
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.742854737915213
mnt_best : 23.745434394774012
not_improved_count: 3
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 2.32798 (QuantReg: 10.09624) QuantErr: 10.09624 batch_time=19.33607
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 2.85928 (QuantReg: 10.03199) QuantErr: 10.03199 batch_time=0.44462
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 2.99205 (QuantReg: 10.15269) QuantErr: 10.15269 batch_time=0.43823
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 2.73924 (QuantReg: 10.08375) QuantErr: 10.08375 batch_time=0.46434
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 2.42023 (QuantReg: 10.16567) QuantErr: 10.16567 batch_time=0.47544
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 3.12280 (QuantReg: 10.30587) QuantErr: 10.30587 batch_time=0.45130
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 2.80352 (QuantReg: 10.18713) QuantErr: 10.18713 batch_time=0.44070
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 2.79610 (QuantReg: 10.26580) QuantErr: 10.26580 batch_time=0.44784
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 2.42836 (QuantReg: 10.33279) QuantErr: 10.33279 batch_time=0.46759
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 2.69903 (QuantReg: 10.28392) QuantErr: 10.28392 batch_time=0.44484
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 2.80880 (QuantReg: 10.18311) QuantErr: 10.18311 batch_time=0.44298
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 2.61764 (QuantReg: 10.09945) QuantErr: 10.09945 batch_time=0.43961
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 2.30636 (QuantReg: 10.38138) QuantErr: 10.38138 batch_time=0.42951
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 2.88821 (QuantReg: 10.14419) QuantErr: 10.14419 batch_time=4.28019
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 2.84678 (QuantReg: 10.32325) QuantErr: 10.32325 batch_time=0.44450
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 2.50878 (QuantReg: 10.51081) QuantErr: 10.51081 batch_time=0.45676
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 2.56104 (QuantReg: 10.19806) QuantErr: 10.19806 batch_time=0.44736
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 2.54895 (QuantReg: 10.22729) QuantErr: 10.22729 batch_time=0.45166
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 2.28225 (QuantReg: 10.41257) QuantErr: 10.41257 batch_time=0.44189
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 2.75990 (QuantReg: 10.19370) QuantErr: 10.19370 batch_time=0.44990
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 2.73145 (QuantReg: 10.20433) QuantErr: 10.20433 batch_time=1.15524
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 2.51312 (QuantReg: 10.36274) QuantErr: 10.36274 batch_time=0.49723
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 2.28302 (QuantReg: 10.27152) QuantErr: 10.27152 batch_time=0.50569
Train Epoch: 19 codebook_update_time=0.87775
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M16/checkpoint-epoch19.pth ...
Done in 5.089s
removing stale ckpt [epoch 18] [took 0.00s]
epoch : 19
loss : 2.6615102434158326
quant_reg : 10.231581840515137
quant_err : 10.231581840515137
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
LSMDC_full_test/t2v_metrics/R1: 11.7
LSMDC_full_test/t2v_metrics/R5: 27.4
LSMDC_full_test/t2v_metrics/R10: 38.6
LSMDC_full_test/t2v_metrics/R50: 66.1
LSMDC_full_test/t2v_metrics/MedR: 20.0
LSMDC_full_test/t2v_metrics/MeanR: 73.441
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.12994373972408
LSMDC_full_test/v2t_metrics/R1: 10.1