-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_LSMDC_t0.1.txt
2587 lines (2587 loc) · 190 KB
/
HCQ_LSMDC_t0.1.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1
Preparing the dataloaders ...
Loading dataset LSMDC_full_trainval in ram ...
Finish loading dataset LSMDC_full_trainval in ram, taking 4919.341999053955 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 143.25452399253845 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 90.70437288284302 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch0.pth ...
Done in 1.972s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch0.pth ...
Done in 4.632s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
LSMDC_full_test/t2v_metrics/R1: 0.0
LSMDC_full_test/t2v_metrics/R5: 0.9
LSMDC_full_test/t2v_metrics/R10: 1.6
LSMDC_full_test/t2v_metrics/R50: 4.4
LSMDC_full_test/t2v_metrics/MedR: 508.5
LSMDC_full_test/t2v_metrics/MeanR: 502.992
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
LSMDC_full_test/v2t_metrics/R1: 0.0
LSMDC_full_test/v2t_metrics/R5: 0.3
LSMDC_full_test/v2t_metrics/R10: 0.9
LSMDC_full_test/v2t_metrics/R50: 5.1
LSMDC_full_test/v2t_metrics/MedR: 510.0
LSMDC_full_test/v2t_metrics/MeanR: 501.125
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.73139 (QuantReg: 22.49545) QuantErr: 22.49545 batch_time=16.07998
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.12179 (QuantReg: 22.62909) QuantErr: 22.62909 batch_time=2.03713
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 8.68251 (QuantReg: 22.67187) QuantErr: 22.67187 batch_time=0.51471
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 8.19385 (QuantReg: 22.73547) QuantErr: 22.73547 batch_time=0.92654
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 7.91067 (QuantReg: 22.65860) QuantErr: 22.65860 batch_time=0.52168
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 7.55796 (QuantReg: 22.70231) QuantErr: 22.70231 batch_time=0.52820
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 7.39709 (QuantReg: 22.68485) QuantErr: 22.68485 batch_time=0.48128
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 7.08429 (QuantReg: 22.67240) QuantErr: 22.67240 batch_time=0.49520
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 7.13841 (QuantReg: 22.73050) QuantErr: 22.73050 batch_time=0.50665
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 6.83363 (QuantReg: 22.71068) QuantErr: 22.71068 batch_time=0.54250
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 7.34309 (QuantReg: 22.70517) QuantErr: 22.70517 batch_time=0.51776
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 6.34882 (QuantReg: 22.71003) QuantErr: 22.71003 batch_time=0.52412
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 6.92179 (QuantReg: 22.71148) QuantErr: 22.71148 batch_time=0.49786
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 7.15116 (QuantReg: 22.71646) QuantErr: 22.71646 batch_time=0.47536
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 6.61258 (QuantReg: 22.68718) QuantErr: 22.68718 batch_time=0.50476
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 6.31341 (QuantReg: 22.71490) QuantErr: 22.71490 batch_time=0.51928
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 7.06336 (QuantReg: 22.69262) QuantErr: 22.69262 batch_time=0.52385
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 6.48157 (QuantReg: 22.71298) QuantErr: 22.71298 batch_time=1.05690
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 6.36234 (QuantReg: 22.71904) QuantErr: 22.71904 batch_time=0.50982
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 6.54789 (QuantReg: 22.67059) QuantErr: 22.67059 batch_time=0.53590
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 6.11056 (QuantReg: 22.70102) QuantErr: 22.70102 batch_time=0.70438
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 6.00901 (QuantReg: 22.70154) QuantErr: 22.70154 batch_time=0.50147
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 7.10161 (QuantReg: 22.72392) QuantErr: 22.72392 batch_time=0.51037
Train Epoch: 1 codebook_update_time=1.88836
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch1.pth ...
Done in 5.846s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch1.pth ...
Done in 10.324s
epoch : 1
loss : 7.167079341888428
quant_reg : 22.69012572479248
quant_err : 22.69012572479248
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
LSMDC_full_test/t2v_metrics/R1: 7.2
LSMDC_full_test/t2v_metrics/R5: 18.2
LSMDC_full_test/t2v_metrics/R10: 26.5
LSMDC_full_test/t2v_metrics/R50: 55.4
LSMDC_full_test/t2v_metrics/MedR: 36.0
LSMDC_full_test/t2v_metrics/MeanR: 102.248
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 15.143162617377657
LSMDC_full_test/v2t_metrics/R1: 6.7
LSMDC_full_test/v2t_metrics/R5: 17.5
LSMDC_full_test/v2t_metrics/R10: 27.3
LSMDC_full_test/v2t_metrics/R50: 54.8
LSMDC_full_test/v2t_metrics/MedR: 41.0
LSMDC_full_test/v2t_metrics/MeanR: 105.293
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 14.737545744911575
mnt_best : 15.143162617377657
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 6.22672 (QuantReg: 12.47564) QuantErr: 12.47564 batch_time=25.63905
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 6.38777 (QuantReg: 12.34383) QuantErr: 12.34383 batch_time=1.51652
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 6.38566 (QuantReg: 12.71857) QuantErr: 12.71857 batch_time=0.48536
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 6.08297 (QuantReg: 12.97976) QuantErr: 12.97976 batch_time=0.50323
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 6.35407 (QuantReg: 12.38572) QuantErr: 12.38572 batch_time=0.51361
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 6.62963 (QuantReg: 12.85162) QuantErr: 12.85162 batch_time=0.59552
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 6.53339 (QuantReg: 12.93231) QuantErr: 12.93231 batch_time=0.53831
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 5.69531 (QuantReg: 13.19307) QuantErr: 13.19307 batch_time=0.50866
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 6.11591 (QuantReg: 13.53595) QuantErr: 13.53595 batch_time=0.48085
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 5.14833 (QuantReg: 13.46150) QuantErr: 13.46150 batch_time=0.53795
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 5.73153 (QuantReg: 13.42498) QuantErr: 13.42498 batch_time=0.49073
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 5.89168 (QuantReg: 13.54650) QuantErr: 13.54650 batch_time=0.49869
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 6.13475 (QuantReg: 13.59072) QuantErr: 13.59072 batch_time=0.49087
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 6.07610 (QuantReg: 13.56137) QuantErr: 13.56137 batch_time=0.48389
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 6.00843 (QuantReg: 13.79423) QuantErr: 13.79423 batch_time=0.52179
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 5.72569 (QuantReg: 14.23890) QuantErr: 14.23890 batch_time=0.86019
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 5.54976 (QuantReg: 13.63635) QuantErr: 13.63635 batch_time=0.51587
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 6.30949 (QuantReg: 14.18639) QuantErr: 14.18639 batch_time=0.53517
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 6.35706 (QuantReg: 14.60217) QuantErr: 14.60217 batch_time=0.48636
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 5.85537 (QuantReg: 14.54254) QuantErr: 14.54254 batch_time=0.53161
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 5.49593 (QuantReg: 14.43559) QuantErr: 14.43559 batch_time=1.26105
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 6.08249 (QuantReg: 14.73911) QuantErr: 14.73911 batch_time=0.53385
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 5.95714 (QuantReg: 14.57731) QuantErr: 14.57731 batch_time=0.50978
Train Epoch: 2 codebook_update_time=1.72042
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch2.pth ...
Done in 4.928s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch2.pth ...
Done in 10.317s
removing stale ckpt [epoch 1] [took 0.30s]
removing stale ckpt [epoch 0] [took 0.14s]
epoch : 2
loss : 5.9801371459960935
quant_reg : 13.60879600906372
quant_err : 13.60879600906372
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
LSMDC_full_test/t2v_metrics/R1: 7.8
LSMDC_full_test/t2v_metrics/R5: 20.3
LSMDC_full_test/t2v_metrics/R10: 30.0
LSMDC_full_test/t2v_metrics/R50: 58.6
LSMDC_full_test/t2v_metrics/MedR: 33.0
LSMDC_full_test/t2v_metrics/MeanR: 88.925
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 16.81011295878234
LSMDC_full_test/v2t_metrics/R1: 7.5
LSMDC_full_test/v2t_metrics/R5: 19.3
LSMDC_full_test/v2t_metrics/R10: 30.6
LSMDC_full_test/v2t_metrics/R50: 58.8
LSMDC_full_test/v2t_metrics/MedR: 33.0
LSMDC_full_test/v2t_metrics/MeanR: 91.361
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 16.42277966692525
mnt_best : 16.81011295878234
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 5.99545 (QuantReg: 12.49252) QuantErr: 12.49252 batch_time=22.13775
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 5.40023 (QuantReg: 12.23790) QuantErr: 12.23790 batch_time=0.85354
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 5.86470 (QuantReg: 12.33692) QuantErr: 12.33692 batch_time=0.55333
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 5.53682 (QuantReg: 12.59656) QuantErr: 12.59656 batch_time=0.48900
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 5.40190 (QuantReg: 12.30249) QuantErr: 12.30249 batch_time=0.50968
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 5.50908 (QuantReg: 12.26534) QuantErr: 12.26534 batch_time=0.49183
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 5.54639 (QuantReg: 12.57050) QuantErr: 12.57050 batch_time=0.49884
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 5.93483 (QuantReg: 12.47252) QuantErr: 12.47252 batch_time=3.41459
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 5.55860 (QuantReg: 12.43313) QuantErr: 12.43313 batch_time=0.50399
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 5.44225 (QuantReg: 12.79784) QuantErr: 12.79784 batch_time=0.50280
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 5.48718 (QuantReg: 12.51322) QuantErr: 12.51322 batch_time=0.49777
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 5.41857 (QuantReg: 12.82888) QuantErr: 12.82888 batch_time=0.50856
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 5.39000 (QuantReg: 12.90266) QuantErr: 12.90266 batch_time=0.49015
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 5.60934 (QuantReg: 12.92323) QuantErr: 12.92323 batch_time=0.49304
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 5.19229 (QuantReg: 12.73726) QuantErr: 12.73726 batch_time=0.51185
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 5.65382 (QuantReg: 12.73715) QuantErr: 12.73715 batch_time=0.51591
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 5.19997 (QuantReg: 13.08460) QuantErr: 13.08460 batch_time=0.50098
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 5.65486 (QuantReg: 12.83434) QuantErr: 12.83434 batch_time=0.50582
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 5.27031 (QuantReg: 13.07327) QuantErr: 13.07327 batch_time=0.52124
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 5.62089 (QuantReg: 12.97262) QuantErr: 12.97262 batch_time=0.51783
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 5.21822 (QuantReg: 12.87830) QuantErr: 12.87830 batch_time=0.50338
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 5.25688 (QuantReg: 13.03057) QuantErr: 13.03057 batch_time=0.51617
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 5.27571 (QuantReg: 12.94181) QuantErr: 12.94181 batch_time=0.49292
Train Epoch: 3 codebook_update_time=1.73632
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch3.pth ...
Done in 6.787s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch3.pth ...
Done in 11.919s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 5.532175567626953
quant_reg : 12.695390396118164
quant_err : 12.695390396118164
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
LSMDC_full_test/t2v_metrics/R1: 8.9
LSMDC_full_test/t2v_metrics/R5: 22.3
LSMDC_full_test/t2v_metrics/R10: 32.5
LSMDC_full_test/t2v_metrics/R50: 60.6
LSMDC_full_test/t2v_metrics/MedR: 27.5
LSMDC_full_test/t2v_metrics/MeanR: 84.472
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 18.61484439517126
LSMDC_full_test/v2t_metrics/R1: 9.1
LSMDC_full_test/v2t_metrics/R5: 24.4
LSMDC_full_test/v2t_metrics/R10: 31.7
LSMDC_full_test/v2t_metrics/R50: 59.5
LSMDC_full_test/v2t_metrics/MedR: 30.0
LSMDC_full_test/v2t_metrics/MeanR: 91.217
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 19.164470607454952
mnt_best : 18.61484439517126
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 5.42889 (QuantReg: 12.14582) QuantErr: 12.14582 batch_time=22.67789
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 5.27852 (QuantReg: 12.71689) QuantErr: 12.71689 batch_time=0.81660
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 5.47700 (QuantReg: 12.50300) QuantErr: 12.50300 batch_time=0.51309
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 5.34330 (QuantReg: 12.50327) QuantErr: 12.50327 batch_time=0.50271
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 4.80581 (QuantReg: 12.55760) QuantErr: 12.55760 batch_time=0.49661
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 4.78445 (QuantReg: 12.41708) QuantErr: 12.41708 batch_time=0.52195
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 5.31166 (QuantReg: 12.71120) QuantErr: 12.71120 batch_time=0.55472
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 5.61644 (QuantReg: 12.55605) QuantErr: 12.55605 batch_time=2.08172
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 4.77304 (QuantReg: 12.33940) QuantErr: 12.33940 batch_time=0.51018
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 4.98544 (QuantReg: 12.15667) QuantErr: 12.15667 batch_time=0.49851
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 5.76968 (QuantReg: 12.33981) QuantErr: 12.33981 batch_time=0.55518
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 4.59796 (QuantReg: 12.17267) QuantErr: 12.17267 batch_time=0.64858
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 5.20763 (QuantReg: 12.30483) QuantErr: 12.30483 batch_time=0.49410
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 5.38973 (QuantReg: 12.57637) QuantErr: 12.57637 batch_time=0.53063
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 5.33275 (QuantReg: 12.35904) QuantErr: 12.35904 batch_time=0.55110
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 5.56933 (QuantReg: 12.50950) QuantErr: 12.50950 batch_time=0.50632
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 5.10568 (QuantReg: 12.62333) QuantErr: 12.62333 batch_time=1.40851
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 5.75423 (QuantReg: 12.67622) QuantErr: 12.67622 batch_time=0.52378
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 4.88561 (QuantReg: 12.91457) QuantErr: 12.91457 batch_time=0.49438
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 4.89416 (QuantReg: 12.62071) QuantErr: 12.62071 batch_time=1.16750
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 4.25967 (QuantReg: 12.72224) QuantErr: 12.72224 batch_time=0.49630
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 4.78632 (QuantReg: 12.66602) QuantErr: 12.66602 batch_time=0.50598
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 5.14171 (QuantReg: 12.69688) QuantErr: 12.69688 batch_time=0.50139
Train Epoch: 4 codebook_update_time=1.95314
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch4.pth ...
Done in 4.964s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch4.pth ...
Done in 10.485s
removing stale ckpt [epoch 3] [took 0.02s]
epoch : 4
loss : 5.215415403366089
quant_reg : 12.545991271972657
quant_err : 12.545991271972657
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
LSMDC_full_test/t2v_metrics/R1: 9.7
LSMDC_full_test/t2v_metrics/R5: 23.8
LSMDC_full_test/t2v_metrics/R10: 33.8
LSMDC_full_test/t2v_metrics/R50: 61.7
LSMDC_full_test/t2v_metrics/MedR: 28.0
LSMDC_full_test/t2v_metrics/MeanR: 83.328
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.834524671668508
LSMDC_full_test/v2t_metrics/R1: 9.1
LSMDC_full_test/v2t_metrics/R5: 22.7
LSMDC_full_test/v2t_metrics/R10: 33.2
LSMDC_full_test/v2t_metrics/R50: 60.9
LSMDC_full_test/v2t_metrics/MedR: 28.0
LSMDC_full_test/v2t_metrics/MeanR: 86.709
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 18.999191101296827
mnt_best : 19.834524671668508
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 5.87134 (QuantReg: 12.37896) QuantErr: 12.37896 batch_time=28.45755
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 4.81100 (QuantReg: 12.52641) QuantErr: 12.52641 batch_time=0.50749
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 4.98637 (QuantReg: 12.20180) QuantErr: 12.20180 batch_time=0.53425
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 5.40295 (QuantReg: 12.33016) QuantErr: 12.33016 batch_time=0.50995
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 5.20125 (QuantReg: 12.74449) QuantErr: 12.74449 batch_time=0.56617
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 4.62500 (QuantReg: 12.71752) QuantErr: 12.71752 batch_time=0.52997
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 4.84529 (QuantReg: 12.58482) QuantErr: 12.58482 batch_time=0.49863
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 5.14947 (QuantReg: 12.54591) QuantErr: 12.54591 batch_time=0.51795
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 5.61665 (QuantReg: 12.46111) QuantErr: 12.46111 batch_time=0.50580
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 5.25169 (QuantReg: 12.28829) QuantErr: 12.28829 batch_time=0.51555
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 5.18682 (QuantReg: 12.36007) QuantErr: 12.36007 batch_time=0.52267
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 4.94457 (QuantReg: 12.50037) QuantErr: 12.50037 batch_time=0.50338
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 4.67096 (QuantReg: 12.54753) QuantErr: 12.54753 batch_time=0.49545
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 4.88349 (QuantReg: 12.78868) QuantErr: 12.78868 batch_time=0.52904
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 5.25332 (QuantReg: 12.74935) QuantErr: 12.74935 batch_time=0.55801
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 5.32265 (QuantReg: 12.53201) QuantErr: 12.53201 batch_time=0.51437
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 4.68946 (QuantReg: 12.75058) QuantErr: 12.75058 batch_time=0.51456
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 4.76672 (QuantReg: 12.80098) QuantErr: 12.80098 batch_time=0.50816
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 4.72300 (QuantReg: 12.51633) QuantErr: 12.51633 batch_time=0.52327
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 4.82824 (QuantReg: 12.42453) QuantErr: 12.42453 batch_time=0.54500
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 4.68904 (QuantReg: 12.55794) QuantErr: 12.55794 batch_time=0.50532
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 4.90148 (QuantReg: 12.62822) QuantErr: 12.62822 batch_time=0.49945
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 4.41941 (QuantReg: 12.75009) QuantErr: 12.75009 batch_time=0.50994
Train Epoch: 5 codebook_update_time=1.76434
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch5.pth ...
Done in 4.233s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch5.pth ...
Done in 9.576s
removing stale ckpt [epoch 4] [took 0.10s]
epoch : 5
loss : 4.962849361419678
quant_reg : 12.554430618286133
quant_err : 12.554430618286133
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
LSMDC_full_test/t2v_metrics/R1: 10.8
LSMDC_full_test/t2v_metrics/R5: 24.9
LSMDC_full_test/t2v_metrics/R10: 34.8
LSMDC_full_test/t2v_metrics/R50: 62.5
LSMDC_full_test/t2v_metrics/MedR: 28.0
LSMDC_full_test/t2v_metrics/MeanR: 83.67
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.07337597191706
LSMDC_full_test/v2t_metrics/R1: 10.3
LSMDC_full_test/v2t_metrics/R5: 25.3
LSMDC_full_test/v2t_metrics/R10: 34.2
LSMDC_full_test/v2t_metrics/R50: 61.6
LSMDC_full_test/v2t_metrics/MedR: 27.0
LSMDC_full_test/v2t_metrics/MeanR: 85.945
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.73295876697988
mnt_best : 21.07337597191706
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 5.37128 (QuantReg: 12.32107) QuantErr: 12.32107 batch_time=23.39301
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 4.91433 (QuantReg: 12.40976) QuantErr: 12.40976 batch_time=3.70367
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 5.12509 (QuantReg: 12.42857) QuantErr: 12.42857 batch_time=0.53024
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 4.64912 (QuantReg: 12.43305) QuantErr: 12.43305 batch_time=0.50825
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 4.60063 (QuantReg: 12.33496) QuantErr: 12.33496 batch_time=0.49782
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 4.97513 (QuantReg: 12.39675) QuantErr: 12.39675 batch_time=0.51955
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 4.62618 (QuantReg: 12.29262) QuantErr: 12.29262 batch_time=0.54147
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 4.79899 (QuantReg: 12.43854) QuantErr: 12.43854 batch_time=0.49862
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 5.21797 (QuantReg: 12.51645) QuantErr: 12.51645 batch_time=0.57790
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 5.03373 (QuantReg: 12.68490) QuantErr: 12.68490 batch_time=0.50245
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 4.43359 (QuantReg: 12.62999) QuantErr: 12.62999 batch_time=0.51031
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 4.83125 (QuantReg: 12.43114) QuantErr: 12.43114 batch_time=0.53245
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 4.77038 (QuantReg: 12.78275) QuantErr: 12.78275 batch_time=0.50812
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 5.41288 (QuantReg: 12.43934) QuantErr: 12.43934 batch_time=0.57143
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 4.65523 (QuantReg: 12.71020) QuantErr: 12.71020 batch_time=0.51351
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 4.08732 (QuantReg: 12.35453) QuantErr: 12.35453 batch_time=0.49894
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 4.53219 (QuantReg: 12.56487) QuantErr: 12.56487 batch_time=0.75954
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 4.76876 (QuantReg: 12.97543) QuantErr: 12.97543 batch_time=0.54818
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 5.12578 (QuantReg: 12.24171) QuantErr: 12.24171 batch_time=0.49580
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 4.71455 (QuantReg: 12.60629) QuantErr: 12.60629 batch_time=0.50966
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 4.53866 (QuantReg: 12.82738) QuantErr: 12.82738 batch_time=0.59146
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 5.17772 (QuantReg: 12.60758) QuantErr: 12.60758 batch_time=0.49739
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 5.02088 (QuantReg: 12.74262) QuantErr: 12.74262 batch_time=0.52076
Train Epoch: 6 codebook_update_time=1.99021
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch6.pth ...
Done in 5.618s
removing stale ckpt [epoch 5] [took 0.07s]
epoch : 6
loss : 4.809051475524902
quant_reg : 12.545920333862306
quant_err : 12.545920333862306
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
LSMDC_full_test/t2v_metrics/R1: 10.0
LSMDC_full_test/t2v_metrics/R5: 25.0
LSMDC_full_test/t2v_metrics/R10: 35.6
LSMDC_full_test/t2v_metrics/R50: 64.1
LSMDC_full_test/t2v_metrics/MedR: 24.0
LSMDC_full_test/t2v_metrics/MeanR: 79.459
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.723510980592607
LSMDC_full_test/v2t_metrics/R1: 10.9
LSMDC_full_test/v2t_metrics/R5: 24.7
LSMDC_full_test/v2t_metrics/R10: 34.8
LSMDC_full_test/v2t_metrics/R50: 62.5
LSMDC_full_test/v2t_metrics/MedR: 25.0
LSMDC_full_test/v2t_metrics/MeanR: 85.137
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.08147037165157
mnt_best : 21.07337597191706
not_improved_count: 1
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 4.88483 (QuantReg: 12.46880) QuantErr: 12.46880 batch_time=27.05304
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 4.64553 (QuantReg: 12.57535) QuantErr: 12.57535 batch_time=0.50601
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 4.45213 (QuantReg: 12.69930) QuantErr: 12.69930 batch_time=0.50637
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 4.47387 (QuantReg: 12.41919) QuantErr: 12.41919 batch_time=0.52684
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 5.09833 (QuantReg: 12.81213) QuantErr: 12.81213 batch_time=0.53819
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 4.69072 (QuantReg: 12.56485) QuantErr: 12.56485 batch_time=0.54529
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 4.65719 (QuantReg: 12.47593) QuantErr: 12.47593 batch_time=0.53250
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 4.83836 (QuantReg: 12.48731) QuantErr: 12.48731 batch_time=0.51040
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 4.54366 (QuantReg: 12.41058) QuantErr: 12.41058 batch_time=0.51494
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 4.52150 (QuantReg: 12.56904) QuantErr: 12.56904 batch_time=0.82317
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 4.47680 (QuantReg: 12.40552) QuantErr: 12.40552 batch_time=0.50416
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 4.66529 (QuantReg: 12.32360) QuantErr: 12.32360 batch_time=0.54058
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 4.61923 (QuantReg: 12.44427) QuantErr: 12.44427 batch_time=2.73720
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 4.42596 (QuantReg: 12.44870) QuantErr: 12.44870 batch_time=0.51368
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 4.53219 (QuantReg: 12.82615) QuantErr: 12.82615 batch_time=0.51227
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 4.27289 (QuantReg: 12.71424) QuantErr: 12.71424 batch_time=0.50893
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 4.64641 (QuantReg: 12.48559) QuantErr: 12.48559 batch_time=0.49049
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 4.70435 (QuantReg: 12.55319) QuantErr: 12.55319 batch_time=0.52877
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 4.45632 (QuantReg: 12.44065) QuantErr: 12.44065 batch_time=0.48309
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 4.45963 (QuantReg: 12.76456) QuantErr: 12.76456 batch_time=1.54632
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 4.38640 (QuantReg: 12.53337) QuantErr: 12.53337 batch_time=0.51484
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 4.42009 (QuantReg: 12.54744) QuantErr: 12.54744 batch_time=0.52789
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 4.40198 (QuantReg: 12.77962) QuantErr: 12.77962 batch_time=0.51336
Train Epoch: 7 codebook_update_time=1.71418
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch7.pth ...
Done in 4.167s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch7.pth ...
Done in 9.185s
removing stale ckpt [epoch 6] [took 0.03s]
epoch : 7
loss : 4.613537400245667
quant_reg : 12.576404769897461
quant_err : 12.576404769897461
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
LSMDC_full_test/t2v_metrics/R1: 10.6
LSMDC_full_test/t2v_metrics/R5: 27.3
LSMDC_full_test/t2v_metrics/R10: 37.1
LSMDC_full_test/t2v_metrics/R50: 65.5
LSMDC_full_test/t2v_metrics/MedR: 23.0
LSMDC_full_test/t2v_metrics/MeanR: 77.726
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.06043849425947
LSMDC_full_test/v2t_metrics/R1: 10.3
LSMDC_full_test/v2t_metrics/R5: 27.1
LSMDC_full_test/v2t_metrics/R10: 36.3
LSMDC_full_test/v2t_metrics/R50: 65.3
LSMDC_full_test/v2t_metrics/MedR: 22.0
LSMDC_full_test/v2t_metrics/MeanR: 78.981
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.639026239634035
mnt_best : 22.06043849425947
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 4.45899 (QuantReg: 12.49302) QuantErr: 12.49302 batch_time=21.63260
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 4.38253 (QuantReg: 12.52004) QuantErr: 12.52004 batch_time=0.50257
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 4.57998 (QuantReg: 12.78036) QuantErr: 12.78036 batch_time=0.50368
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 5.00613 (QuantReg: 12.72001) QuantErr: 12.72001 batch_time=1.00210
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 4.23989 (QuantReg: 12.52187) QuantErr: 12.52187 batch_time=0.50372
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 4.55679 (QuantReg: 12.83802) QuantErr: 12.83802 batch_time=0.49661
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 4.44935 (QuantReg: 12.50570) QuantErr: 12.50570 batch_time=1.03302
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 4.84499 (QuantReg: 12.61950) QuantErr: 12.61950 batch_time=0.48552
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 4.30990 (QuantReg: 12.68203) QuantErr: 12.68203 batch_time=1.07443
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 4.58155 (QuantReg: 12.38528) QuantErr: 12.38528 batch_time=0.50059
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 4.90038 (QuantReg: 12.74652) QuantErr: 12.74652 batch_time=0.50405
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 4.49351 (QuantReg: 12.74780) QuantErr: 12.74780 batch_time=0.53738
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 4.48023 (QuantReg: 12.69872) QuantErr: 12.69872 batch_time=0.49822
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 4.17304 (QuantReg: 12.94895) QuantErr: 12.94895 batch_time=0.55532
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 4.74939 (QuantReg: 12.66431) QuantErr: 12.66431 batch_time=0.54597
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 4.16569 (QuantReg: 12.40882) QuantErr: 12.40882 batch_time=0.53689
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 4.61307 (QuantReg: 12.82496) QuantErr: 12.82496 batch_time=0.54614
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 4.38045 (QuantReg: 12.70624) QuantErr: 12.70624 batch_time=0.49769
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 4.23596 (QuantReg: 12.37515) QuantErr: 12.37515 batch_time=0.54485
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 4.63077 (QuantReg: 12.71401) QuantErr: 12.71401 batch_time=0.50135
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 4.16625 (QuantReg: 12.62148) QuantErr: 12.62148 batch_time=0.50881
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 4.16784 (QuantReg: 12.77930) QuantErr: 12.77930 batch_time=0.51789
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 4.45329 (QuantReg: 12.56101) QuantErr: 12.56101 batch_time=0.50899
Train Epoch: 8 codebook_update_time=1.67298
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch8.pth ...
Done in 4.529s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch8.pth ...
Done in 9.264s
removing stale ckpt [epoch 7] [took 0.81s]
epoch : 8
loss : 4.471146016120911
quant_reg : 12.613680480957031
quant_err : 12.613680480957031
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
LSMDC_full_test/t2v_metrics/R1: 11.4
LSMDC_full_test/t2v_metrics/R5: 26.1
LSMDC_full_test/t2v_metrics/R10: 36.5
LSMDC_full_test/t2v_metrics/R50: 64.3
LSMDC_full_test/t2v_metrics/MedR: 23.0
LSMDC_full_test/t2v_metrics/MeanR: 78.039
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.145189843934784
LSMDC_full_test/v2t_metrics/R1: 10.7
LSMDC_full_test/v2t_metrics/R5: 26.2
LSMDC_full_test/v2t_metrics/R10: 36.5
LSMDC_full_test/v2t_metrics/R50: 63.1
LSMDC_full_test/v2t_metrics/MedR: 22.0
LSMDC_full_test/v2t_metrics/MeanR: 81.64
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.70997439055721
mnt_best : 22.145189843934784
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 4.42420 (QuantReg: 12.32506) QuantErr: 12.32506 batch_time=24.06271
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 4.47669 (QuantReg: 12.67559) QuantErr: 12.67559 batch_time=0.50612
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 4.30558 (QuantReg: 12.70356) QuantErr: 12.70356 batch_time=1.02226
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 4.73041 (QuantReg: 12.79778) QuantErr: 12.79778 batch_time=0.50603
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 4.64037 (QuantReg: 12.62723) QuantErr: 12.62723 batch_time=0.50833
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 4.02640 (QuantReg: 12.94512) QuantErr: 12.94512 batch_time=0.51014
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 4.22045 (QuantReg: 12.53065) QuantErr: 12.53065 batch_time=3.96039
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 4.80391 (QuantReg: 12.25290) QuantErr: 12.25290 batch_time=0.50944
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 4.27041 (QuantReg: 12.41626) QuantErr: 12.41626 batch_time=0.57562
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 4.10877 (QuantReg: 12.63947) QuantErr: 12.63947 batch_time=0.50987
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 4.94032 (QuantReg: 12.75589) QuantErr: 12.75589 batch_time=0.87388
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 4.45605 (QuantReg: 12.78541) QuantErr: 12.78541 batch_time=0.49262
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 4.11802 (QuantReg: 12.49858) QuantErr: 12.49858 batch_time=0.90965
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 4.34376 (QuantReg: 12.87824) QuantErr: 12.87824 batch_time=3.62168
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 4.29236 (QuantReg: 12.71894) QuantErr: 12.71894 batch_time=0.56696
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 4.47947 (QuantReg: 12.92018) QuantErr: 12.92018 batch_time=0.55987
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 4.02395 (QuantReg: 12.70639) QuantErr: 12.70639 batch_time=0.50327
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 4.17415 (QuantReg: 12.13044) QuantErr: 12.13044 batch_time=0.50999
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 4.31866 (QuantReg: 12.64545) QuantErr: 12.64545 batch_time=0.49482
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 4.13460 (QuantReg: 12.86618) QuantErr: 12.86618 batch_time=0.54797
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 4.85531 (QuantReg: 12.79830) QuantErr: 12.79830 batch_time=0.50881
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 4.84028 (QuantReg: 12.75533) QuantErr: 12.75533 batch_time=0.51860
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 4.25622 (QuantReg: 12.37959) QuantErr: 12.37959 batch_time=0.50208
Train Epoch: 9 codebook_update_time=1.72012
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch9.pth ...
Done in 5.701s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch9.pth ...
Done in 10.953s
removing stale ckpt [epoch 8] [took 0.02s]
epoch : 9
loss : 4.337491041183472
quant_reg : 12.658823467254638
quant_err : 12.658823467254638
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
LSMDC_full_test/t2v_metrics/R1: 11.9
LSMDC_full_test/t2v_metrics/R5: 26.1
LSMDC_full_test/t2v_metrics/R10: 36.5
LSMDC_full_test/t2v_metrics/R50: 65.1
LSMDC_full_test/t2v_metrics/MedR: 21.0
LSMDC_full_test/t2v_metrics/MeanR: 75.981
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.464328661988997
LSMDC_full_test/v2t_metrics/R1: 11.4
LSMDC_full_test/v2t_metrics/R5: 26.5
LSMDC_full_test/v2t_metrics/R10: 38.0
LSMDC_full_test/v2t_metrics/R50: 63.6
LSMDC_full_test/v2t_metrics/MedR: 23.0
LSMDC_full_test/v2t_metrics/MeanR: 77.922
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.558563486822468
mnt_best : 22.464328661988997
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 4.18592 (QuantReg: 12.55937) QuantErr: 12.55937 batch_time=22.65371
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 4.24462 (QuantReg: 12.26062) QuantErr: 12.26062 batch_time=5.38131
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 4.16282 (QuantReg: 12.59695) QuantErr: 12.59695 batch_time=0.58094
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 4.02359 (QuantReg: 12.57017) QuantErr: 12.57017 batch_time=0.50858
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 4.34144 (QuantReg: 12.75744) QuantErr: 12.75744 batch_time=0.50330
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 4.05369 (QuantReg: 12.78251) QuantErr: 12.78251 batch_time=0.50980
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 4.22712 (QuantReg: 12.22700) QuantErr: 12.22700 batch_time=1.03825
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 3.97452 (QuantReg: 12.50932) QuantErr: 12.50932 batch_time=0.54319
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 4.25949 (QuantReg: 12.77528) QuantErr: 12.77528 batch_time=0.54280
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 4.35956 (QuantReg: 12.87122) QuantErr: 12.87122 batch_time=0.51616
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 4.16832 (QuantReg: 12.68051) QuantErr: 12.68051 batch_time=0.50109
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 4.14055 (QuantReg: 12.53263) QuantErr: 12.53263 batch_time=0.53333
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 3.87255 (QuantReg: 12.53554) QuantErr: 12.53554 batch_time=0.55091
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 4.52610 (QuantReg: 12.86852) QuantErr: 12.86852 batch_time=0.52574
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 4.68984 (QuantReg: 12.93649) QuantErr: 12.93649 batch_time=0.55320
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 4.43319 (QuantReg: 12.88792) QuantErr: 12.88792 batch_time=0.51310
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 4.30214 (QuantReg: 12.54100) QuantErr: 12.54100 batch_time=0.49058
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 4.34828 (QuantReg: 12.82469) QuantErr: 12.82469 batch_time=0.52615
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 3.91324 (QuantReg: 12.75323) QuantErr: 12.75323 batch_time=0.55402
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 3.99382 (QuantReg: 12.60526) QuantErr: 12.60526 batch_time=0.52972
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 4.03387 (QuantReg: 12.77023) QuantErr: 12.77023 batch_time=0.50896
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 4.75852 (QuantReg: 12.79944) QuantErr: 12.79944 batch_time=0.50042
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 4.29973 (QuantReg: 12.62069) QuantErr: 12.62069 batch_time=0.50882
Train Epoch: 10 codebook_update_time=2.15890
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch10.pth ...
Done in 4.326s
removing stale ckpt [epoch 9] [took 0.07s]
epoch : 10
loss : 4.223619153976441
quant_reg : 12.674819744110108
quant_err : 12.674819744110108
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
LSMDC_full_test/t2v_metrics/R1: 11.7
LSMDC_full_test/t2v_metrics/R5: 25.9
LSMDC_full_test/t2v_metrics/R10: 36.6
LSMDC_full_test/t2v_metrics/R50: 65.8
LSMDC_full_test/t2v_metrics/MedR: 22.0
LSMDC_full_test/t2v_metrics/MeanR: 78.261
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.30089213319757
LSMDC_full_test/v2t_metrics/R1: 10.8
LSMDC_full_test/v2t_metrics/R5: 27.6
LSMDC_full_test/v2t_metrics/R10: 35.7
LSMDC_full_test/v2t_metrics/R50: 62.8
LSMDC_full_test/v2t_metrics/MedR: 24.0
LSMDC_full_test/v2t_metrics/MeanR: 78.779
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.995492189358163
mnt_best : 22.464328661988997
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 3.65118 (QuantReg: 12.69397) QuantErr: 12.69397 batch_time=22.18084
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 4.31207 (QuantReg: 12.75535) QuantErr: 12.75535 batch_time=0.51836
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 4.07630 (QuantReg: 12.46048) QuantErr: 12.46048 batch_time=0.67581
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 4.08948 (QuantReg: 12.51385) QuantErr: 12.51385 batch_time=1.23592
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 4.23452 (QuantReg: 12.50183) QuantErr: 12.50183 batch_time=0.49669
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 4.57853 (QuantReg: 12.75753) QuantErr: 12.75753 batch_time=0.51646
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 4.23722 (QuantReg: 12.66917) QuantErr: 12.66917 batch_time=0.54831
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 4.10929 (QuantReg: 12.73294) QuantErr: 12.73294 batch_time=0.56597
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 4.24642 (QuantReg: 12.75515) QuantErr: 12.75515 batch_time=0.50488
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 3.84624 (QuantReg: 12.99877) QuantErr: 12.99877 batch_time=0.50193
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 4.15586 (QuantReg: 12.72998) QuantErr: 12.72998 batch_time=0.53113
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 3.61882 (QuantReg: 12.70729) QuantErr: 12.70729 batch_time=0.51005
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 4.24899 (QuantReg: 12.64732) QuantErr: 12.64732 batch_time=0.50640
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 3.67921 (QuantReg: 12.66751) QuantErr: 12.66751 batch_time=0.52301
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 4.14433 (QuantReg: 12.33614) QuantErr: 12.33614 batch_time=0.50450
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 4.57142 (QuantReg: 12.71858) QuantErr: 12.71858 batch_time=0.53598
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 3.83891 (QuantReg: 12.67139) QuantErr: 12.67139 batch_time=0.89626
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 3.97081 (QuantReg: 12.57680) QuantErr: 12.57680 batch_time=0.55606
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 4.12245 (QuantReg: 12.56271) QuantErr: 12.56271 batch_time=0.48723
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 4.41858 (QuantReg: 12.75138) QuantErr: 12.75138 batch_time=2.22494
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 4.03430 (QuantReg: 12.42715) QuantErr: 12.42715 batch_time=0.63313
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 3.72856 (QuantReg: 12.68640) QuantErr: 12.68640 batch_time=0.49599
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 4.19450 (QuantReg: 12.51130) QuantErr: 12.51130 batch_time=0.51507
Train Epoch: 11 codebook_update_time=1.95792
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch11.pth ...
Done in 4.866s
removing stale ckpt [epoch 10] [took 0.02s]
epoch : 11
loss : 4.097312510490418
quant_reg : 12.684706638336182
quant_err : 12.684706638336182
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
LSMDC_full_test/t2v_metrics/R1: 10.5
LSMDC_full_test/t2v_metrics/R5: 26.6
LSMDC_full_test/t2v_metrics/R10: 36.1
LSMDC_full_test/t2v_metrics/R50: 65.1
LSMDC_full_test/t2v_metrics/MedR: 24.0
LSMDC_full_test/t2v_metrics/MeanR: 78.111
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.60359593767129
LSMDC_full_test/v2t_metrics/R1: 11.5
LSMDC_full_test/v2t_metrics/R5: 27.6
LSMDC_full_test/v2t_metrics/R10: 36.8
LSMDC_full_test/v2t_metrics/R50: 63.8
LSMDC_full_test/v2t_metrics/MedR: 22.0
LSMDC_full_test/v2t_metrics/MeanR: 80.889
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.689151083840333
mnt_best : 22.464328661988997
not_improved_count: 2
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 4.18112 (QuantReg: 12.48187) QuantErr: 12.48187 batch_time=26.63182
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 3.90145 (QuantReg: 12.68081) QuantErr: 12.68081 batch_time=0.52468
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 4.42897 (QuantReg: 12.62220) QuantErr: 12.62220 batch_time=0.49729
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 3.58419 (QuantReg: 12.34849) QuantErr: 12.34849 batch_time=0.50903
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 3.91851 (QuantReg: 12.73137) QuantErr: 12.73137 batch_time=0.49179
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 3.83465 (QuantReg: 12.76102) QuantErr: 12.76102 batch_time=0.51765
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 4.19215 (QuantReg: 12.80533) QuantErr: 12.80533 batch_time=0.49602
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 4.11901 (QuantReg: 12.49285) QuantErr: 12.49285 batch_time=0.54344
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 4.05032 (QuantReg: 12.78423) QuantErr: 12.78423 batch_time=0.52394
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 3.85871 (QuantReg: 12.56266) QuantErr: 12.56266 batch_time=0.49879
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 3.95781 (QuantReg: 12.60136) QuantErr: 12.60136 batch_time=0.50759
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 4.02735 (QuantReg: 12.77647) QuantErr: 12.77647 batch_time=0.54198
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 4.21786 (QuantReg: 12.70736) QuantErr: 12.70736 batch_time=0.51469
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 4.18541 (QuantReg: 12.80873) QuantErr: 12.80873 batch_time=0.61986
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 3.96180 (QuantReg: 12.64340) QuantErr: 12.64340 batch_time=0.53052
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 3.95754 (QuantReg: 12.68354) QuantErr: 12.68354 batch_time=0.51412
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 3.39764 (QuantReg: 12.61420) QuantErr: 12.61420 batch_time=0.53059
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 4.29005 (QuantReg: 12.51101) QuantErr: 12.51101 batch_time=0.56439
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 3.69233 (QuantReg: 12.66827) QuantErr: 12.66827 batch_time=0.50692
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 4.16949 (QuantReg: 12.56123) QuantErr: 12.56123 batch_time=0.52786
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 3.81054 (QuantReg: 12.74509) QuantErr: 12.74509 batch_time=0.50447
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 3.70414 (QuantReg: 12.72886) QuantErr: 12.72886 batch_time=0.48861
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 3.97017 (QuantReg: 12.58982) QuantErr: 12.58982 batch_time=0.68263
Train Epoch: 12 codebook_update_time=1.71774
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch12.pth ...
Done in 5.995s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch12.pth ...
Done in 11.080s
removing stale ckpt [epoch 11] [took 0.01s]
epoch : 12
loss : 4.008724123001099
quant_reg : 12.655554092407227
quant_err : 12.655554092407227
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
LSMDC_full_test/t2v_metrics/R1: 11.7
LSMDC_full_test/t2v_metrics/R5: 27.1
LSMDC_full_test/t2v_metrics/R10: 38.0
LSMDC_full_test/t2v_metrics/R50: 67.4
LSMDC_full_test/t2v_metrics/MedR: 21.0
LSMDC_full_test/t2v_metrics/MeanR: 77.708
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.92518855881001
LSMDC_full_test/v2t_metrics/R1: 11.6
LSMDC_full_test/v2t_metrics/R5: 27.2
LSMDC_full_test/v2t_metrics/R10: 38.5
LSMDC_full_test/v2t_metrics/R50: 63.8
LSMDC_full_test/v2t_metrics/MedR: 22.0
LSMDC_full_test/v2t_metrics/MeanR: 77.943
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.98771871113958
mnt_best : 22.92518855881001
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 3.65119 (QuantReg: 12.54759) QuantErr: 12.54759 batch_time=21.22650
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 4.09877 (QuantReg: 12.43585) QuantErr: 12.43585 batch_time=0.50467
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 4.01493 (QuantReg: 12.86849) QuantErr: 12.86849 batch_time=0.50501
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 3.64363 (QuantReg: 12.62622) QuantErr: 12.62622 batch_time=0.53731
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 3.65581 (QuantReg: 12.70967) QuantErr: 12.70967 batch_time=0.49437
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 4.24192 (QuantReg: 12.81067) QuantErr: 12.81067 batch_time=0.50298
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 3.92722 (QuantReg: 12.76066) QuantErr: 12.76066 batch_time=0.51486
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 3.78359 (QuantReg: 12.71482) QuantErr: 12.71482 batch_time=0.50555
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 3.64381 (QuantReg: 12.83953) QuantErr: 12.83953 batch_time=0.52769
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 4.12442 (QuantReg: 12.65970) QuantErr: 12.65970 batch_time=0.51436
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 3.88580 (QuantReg: 12.72146) QuantErr: 12.72146 batch_time=0.49818
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 3.81303 (QuantReg: 13.18569) QuantErr: 13.18569 batch_time=0.51262
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 4.15406 (QuantReg: 12.71184) QuantErr: 12.71184 batch_time=0.49506
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 4.17939 (QuantReg: 12.55111) QuantErr: 12.55111 batch_time=1.80202
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 3.79067 (QuantReg: 12.89882) QuantErr: 12.89882 batch_time=0.50740
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 3.72090 (QuantReg: 13.08471) QuantErr: 13.08471 batch_time=0.50192
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 3.60549 (QuantReg: 12.46357) QuantErr: 12.46357 batch_time=0.54637
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 4.12459 (QuantReg: 12.74512) QuantErr: 12.74512 batch_time=0.61479
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 3.98605 (QuantReg: 12.99883) QuantErr: 12.99883 batch_time=0.49635
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 3.80456 (QuantReg: 12.43778) QuantErr: 12.43778 batch_time=0.52062
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 4.01635 (QuantReg: 12.51521) QuantErr: 12.51521 batch_time=0.53170
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 3.84413 (QuantReg: 12.89175) QuantErr: 12.89175 batch_time=0.52639
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 3.71287 (QuantReg: 12.58679) QuantErr: 12.58679 batch_time=0.51602
Train Epoch: 13 codebook_update_time=1.87720
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch13.pth ...
Done in 19.705s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch13.pth ...
Done in 43.024s
removing stale ckpt [epoch 12] [took 0.03s]
epoch : 13
loss : 3.9041365432739257
quant_reg : 12.719718601226807
quant_err : 12.719718601226807
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
LSMDC_full_test/t2v_metrics/R1: 12.2
LSMDC_full_test/t2v_metrics/R5: 28.1
LSMDC_full_test/t2v_metrics/R10: 37.7
LSMDC_full_test/t2v_metrics/R50: 65.9
LSMDC_full_test/t2v_metrics/MedR: 21.0
LSMDC_full_test/t2v_metrics/MeanR: 76.391
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.467626465492383
LSMDC_full_test/v2t_metrics/R1: 11.5
LSMDC_full_test/v2t_metrics/R5: 28.5
LSMDC_full_test/v2t_metrics/R10: 38.8
LSMDC_full_test/v2t_metrics/R50: 65.0
LSMDC_full_test/v2t_metrics/MedR: 20.0
LSMDC_full_test/v2t_metrics/MeanR: 77.506
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.34128753758076
mnt_best : 23.467626465492383
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 4.07238 (QuantReg: 12.67378) QuantErr: 12.67378 batch_time=23.42081
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 3.68157 (QuantReg: 12.52543) QuantErr: 12.52543 batch_time=0.49924
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 4.06811 (QuantReg: 12.47474) QuantErr: 12.47474 batch_time=0.52018
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 3.66904 (QuantReg: 12.67008) QuantErr: 12.67008 batch_time=0.50812
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 3.65328 (QuantReg: 12.84928) QuantErr: 12.84928 batch_time=0.50229
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 3.70213 (QuantReg: 12.89494) QuantErr: 12.89494 batch_time=0.60819
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 3.67637 (QuantReg: 12.61000) QuantErr: 12.61000 batch_time=0.91369
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 4.42994 (QuantReg: 12.55031) QuantErr: 12.55031 batch_time=0.51927
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 3.80237 (QuantReg: 12.88140) QuantErr: 12.88140 batch_time=0.51595
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 3.89345 (QuantReg: 12.80528) QuantErr: 12.80528 batch_time=0.56041
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 3.62490 (QuantReg: 12.62563) QuantErr: 12.62563 batch_time=0.50310
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 4.03629 (QuantReg: 12.82449) QuantErr: 12.82449 batch_time=0.50554
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 4.24930 (QuantReg: 12.84979) QuantErr: 12.84979 batch_time=0.77355
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 3.74415 (QuantReg: 12.77281) QuantErr: 12.77281 batch_time=0.50664
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 4.05368 (QuantReg: 12.94893) QuantErr: 12.94893 batch_time=0.50227
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 3.95357 (QuantReg: 12.64007) QuantErr: 12.64007 batch_time=0.54382
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 3.61053 (QuantReg: 12.86448) QuantErr: 12.86448 batch_time=0.53176
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 4.05046 (QuantReg: 12.93210) QuantErr: 12.93210 batch_time=0.51926
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 4.20979 (QuantReg: 12.65102) QuantErr: 12.65102 batch_time=0.52268
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 4.06094 (QuantReg: 12.87580) QuantErr: 12.87580 batch_time=0.50864
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 3.72249 (QuantReg: 13.06040) QuantErr: 13.06040 batch_time=0.50083
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 3.66006 (QuantReg: 12.88359) QuantErr: 12.88359 batch_time=0.52448
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 3.77467 (QuantReg: 12.74222) QuantErr: 12.74222 batch_time=0.54295
Train Epoch: 14 codebook_update_time=1.72755
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch14.pth ...
Done in 7.763s
removing stale ckpt [epoch 13] [took 0.11s]
epoch : 14
loss : 3.8616334056854247
quant_reg : 12.714698627471924
quant_err : 12.714698627471924
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
LSMDC_full_test/t2v_metrics/R1: 11.8
LSMDC_full_test/t2v_metrics/R5: 28.0
LSMDC_full_test/t2v_metrics/R10: 39.0
LSMDC_full_test/t2v_metrics/R50: 66.1
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 78.278
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.44417107879048
LSMDC_full_test/v2t_metrics/R1: 11.7
LSMDC_full_test/v2t_metrics/R5: 29.3
LSMDC_full_test/v2t_metrics/R10: 38.7
LSMDC_full_test/v2t_metrics/R50: 64.4
LSMDC_full_test/v2t_metrics/MedR: 20.0
LSMDC_full_test/v2t_metrics/MeanR: 80.002
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.673082723402775
mnt_best : 23.467626465492383
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 3.31051 (QuantReg: 12.55565) QuantErr: 12.55565 batch_time=22.63474
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 3.61391 (QuantReg: 12.45829) QuantErr: 12.45829 batch_time=0.53252
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 3.80172 (QuantReg: 12.86683) QuantErr: 12.86683 batch_time=0.50482
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 3.93615 (QuantReg: 12.98325) QuantErr: 12.98325 batch_time=0.50879
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 3.32438 (QuantReg: 12.89656) QuantErr: 12.89656 batch_time=0.48635
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 4.06787 (QuantReg: 12.92741) QuantErr: 12.92741 batch_time=0.55432
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 3.24759 (QuantReg: 12.89566) QuantErr: 12.89566 batch_time=0.48597
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 3.66472 (QuantReg: 12.63462) QuantErr: 12.63462 batch_time=0.50269
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 3.97314 (QuantReg: 12.77380) QuantErr: 12.77380 batch_time=0.50326
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 3.98999 (QuantReg: 12.78815) QuantErr: 12.78815 batch_time=0.49626
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 3.51978 (QuantReg: 12.52929) QuantErr: 12.52929 batch_time=0.48917
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 3.65652 (QuantReg: 12.75427) QuantErr: 12.75427 batch_time=0.53490
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 3.71025 (QuantReg: 12.82542) QuantErr: 12.82542 batch_time=1.70152
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 3.88424 (QuantReg: 12.69018) QuantErr: 12.69018 batch_time=2.24938
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 3.46904 (QuantReg: 12.58990) QuantErr: 12.58990 batch_time=0.50701
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 3.79807 (QuantReg: 12.79717) QuantErr: 12.79717 batch_time=0.50288
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 4.07475 (QuantReg: 12.68867) QuantErr: 12.68867 batch_time=0.93413
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 3.81507 (QuantReg: 12.68522) QuantErr: 12.68522 batch_time=0.52160
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 3.91114 (QuantReg: 12.70833) QuantErr: 12.70833 batch_time=0.51050
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 3.53821 (QuantReg: 12.58032) QuantErr: 12.58032 batch_time=0.48537
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 3.83205 (QuantReg: 12.95678) QuantErr: 12.95678 batch_time=0.49582
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 3.71385 (QuantReg: 12.87653) QuantErr: 12.87653 batch_time=0.49748
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 3.75643 (QuantReg: 12.97521) QuantErr: 12.97521 batch_time=0.50175
Train Epoch: 15 codebook_update_time=2.37368
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch15.pth ...
Done in 4.123s
removing stale ckpt [epoch 14] [took 0.02s]
epoch : 15
loss : 3.7676404523849487
quant_reg : 12.725671249389649
quant_err : 12.725671249389649
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
LSMDC_full_test/t2v_metrics/R1: 11.2
LSMDC_full_test/t2v_metrics/R5: 28.2
LSMDC_full_test/t2v_metrics/R10: 38.6
LSMDC_full_test/t2v_metrics/R50: 67.0
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 77.245
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.015379757601124
LSMDC_full_test/v2t_metrics/R1: 11.5
LSMDC_full_test/v2t_metrics/R5: 29.2
LSMDC_full_test/v2t_metrics/R10: 38.2
LSMDC_full_test/v2t_metrics/R50: 64.5
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 79.105
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.40891860586064
mnt_best : 23.467626465492383
not_improved_count: 2
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 3.83076 (QuantReg: 12.80965) QuantErr: 12.80965 batch_time=28.38033
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 3.50594 (QuantReg: 12.66798) QuantErr: 12.66798 batch_time=0.49910
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 3.48211 (QuantReg: 12.46754) QuantErr: 12.46754 batch_time=0.54442
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 4.01141 (QuantReg: 12.61693) QuantErr: 12.61693 batch_time=0.49710
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 4.00049 (QuantReg: 12.73585) QuantErr: 12.73585 batch_time=0.49415
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 3.95481 (QuantReg: 12.89491) QuantErr: 12.89491 batch_time=0.49410
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 3.77547 (QuantReg: 12.66102) QuantErr: 12.66102 batch_time=0.53423
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 4.00518 (QuantReg: 12.78118) QuantErr: 12.78118 batch_time=0.49666
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 3.41355 (QuantReg: 12.72637) QuantErr: 12.72637 batch_time=0.50424
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 4.01956 (QuantReg: 12.92919) QuantErr: 12.92919 batch_time=0.48860
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 3.72021 (QuantReg: 12.51509) QuantErr: 12.51509 batch_time=0.53596
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 4.05606 (QuantReg: 12.69127) QuantErr: 12.69127 batch_time=0.52526
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 3.69730 (QuantReg: 12.76083) QuantErr: 12.76083 batch_time=0.53935
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 3.84461 (QuantReg: 12.87748) QuantErr: 12.87748 batch_time=0.55617
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 3.28579 (QuantReg: 12.72683) QuantErr: 12.72683 batch_time=0.50810
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 3.67451 (QuantReg: 12.87435) QuantErr: 12.87435 batch_time=0.50346
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 3.65481 (QuantReg: 12.60674) QuantErr: 12.60674 batch_time=0.49453
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 3.71482 (QuantReg: 12.95949) QuantErr: 12.95949 batch_time=0.49155
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 3.63372 (QuantReg: 12.69632) QuantErr: 12.69632 batch_time=0.49936
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 3.81087 (QuantReg: 12.98751) QuantErr: 12.98751 batch_time=0.99227
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 3.53072 (QuantReg: 12.98622) QuantErr: 12.98622 batch_time=0.50674
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 3.38036 (QuantReg: 12.84623) QuantErr: 12.84623 batch_time=0.53064
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 3.65299 (QuantReg: 12.86942) QuantErr: 12.86942 batch_time=0.51815
Train Epoch: 16 codebook_update_time=1.69831
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch16.pth ...
Done in 4.596s
removing stale ckpt [epoch 15] [took 0.01s]
epoch : 16
loss : 3.708818669319153
quant_reg : 12.760518962860107
quant_err : 12.760518962860107
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
LSMDC_full_test/t2v_metrics/R1: 11.8
LSMDC_full_test/t2v_metrics/R5: 28.1
LSMDC_full_test/t2v_metrics/R10: 38.9
LSMDC_full_test/t2v_metrics/R50: 66.0
LSMDC_full_test/t2v_metrics/MedR: 20.0
LSMDC_full_test/t2v_metrics/MeanR: 76.636
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.45196889586609
LSMDC_full_test/v2t_metrics/R1: 11.0
LSMDC_full_test/v2t_metrics/R5: 28.8
LSMDC_full_test/v2t_metrics/R10: 38.4
LSMDC_full_test/v2t_metrics/R50: 64.7
LSMDC_full_test/v2t_metrics/MedR: 20.0
LSMDC_full_test/v2t_metrics/MeanR: 79.623
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.99881531390137
mnt_best : 23.467626465492383
not_improved_count: 3
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 3.59748 (QuantReg: 12.71910) QuantErr: 12.71910 batch_time=24.60081
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 4.18558 (QuantReg: 12.80530) QuantErr: 12.80530 batch_time=0.50849
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 3.87242 (QuantReg: 12.84690) QuantErr: 12.84690 batch_time=0.49643
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 3.39620 (QuantReg: 12.74822) QuantErr: 12.74822 batch_time=1.00665
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 3.81450 (QuantReg: 12.59784) QuantErr: 12.59784 batch_time=0.50473
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 3.72105 (QuantReg: 12.76000) QuantErr: 12.76000 batch_time=0.57362
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 3.34027 (QuantReg: 12.83255) QuantErr: 12.83255 batch_time=0.77889
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 3.49264 (QuantReg: 12.82531) QuantErr: 12.82531 batch_time=0.50182
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 3.81228 (QuantReg: 12.78456) QuantErr: 12.78456 batch_time=0.52396
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 3.57462 (QuantReg: 13.03483) QuantErr: 13.03483 batch_time=0.50916
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 3.32910 (QuantReg: 12.85179) QuantErr: 12.85179 batch_time=0.53432
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 3.71878 (QuantReg: 12.65626) QuantErr: 12.65626 batch_time=0.55955
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 3.51746 (QuantReg: 12.70249) QuantErr: 12.70249 batch_time=1.42663
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 3.74859 (QuantReg: 12.70709) QuantErr: 12.70709 batch_time=0.55450
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 3.58906 (QuantReg: 12.70240) QuantErr: 12.70240 batch_time=0.51876
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 3.23600 (QuantReg: 12.89864) QuantErr: 12.89864 batch_time=0.53500
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 3.65956 (QuantReg: 12.52324) QuantErr: 12.52324 batch_time=0.50014
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 3.87362 (QuantReg: 12.77763) QuantErr: 12.77763 batch_time=0.59368
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 3.50601 (QuantReg: 12.93212) QuantErr: 12.93212 batch_time=0.53204
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 3.72949 (QuantReg: 12.60495) QuantErr: 12.60495 batch_time=0.50342
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 3.53015 (QuantReg: 12.86554) QuantErr: 12.86554 batch_time=0.50078
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 4.05605 (QuantReg: 12.90712) QuantErr: 12.90712 batch_time=0.52385
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 3.31773 (QuantReg: 12.86514) QuantErr: 12.86514 batch_time=0.58196
Train Epoch: 17 codebook_update_time=1.93085
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch17.pth ...
Done in 4.141s
removing stale ckpt [epoch 16] [took 0.00s]
epoch : 17
loss : 3.6391645803451538
quant_reg : 12.74675312423706
quant_err : 12.74675312423706
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
LSMDC_full_test/t2v_metrics/R1: 11.1
LSMDC_full_test/t2v_metrics/R5: 27.9
LSMDC_full_test/t2v_metrics/R10: 38.7
LSMDC_full_test/t2v_metrics/R50: 67.1
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 78.679
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.884743497695727
LSMDC_full_test/v2t_metrics/R1: 10.7
LSMDC_full_test/v2t_metrics/R5: 28.9
LSMDC_full_test/v2t_metrics/R10: 40.8
LSMDC_full_test/v2t_metrics/R50: 65.9
LSMDC_full_test/v2t_metrics/MedR: 20.0
LSMDC_full_test/v2t_metrics/MeanR: 79.419
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.27987234374562
mnt_best : 23.467626465492383
not_improved_count: 4
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 3.84805 (QuantReg: 12.81407) QuantErr: 12.81407 batch_time=23.75199
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 3.41747 (QuantReg: 12.94185) QuantErr: 12.94185 batch_time=1.27231
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 3.53571 (QuantReg: 12.86319) QuantErr: 12.86319 batch_time=0.50066
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 3.52760 (QuantReg: 12.77048) QuantErr: 12.77048 batch_time=1.13577
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 3.48711 (QuantReg: 12.87003) QuantErr: 12.87003 batch_time=0.50622
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 3.54627 (QuantReg: 12.80223) QuantErr: 12.80223 batch_time=0.51194
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 3.46107 (QuantReg: 12.71624) QuantErr: 12.71624 batch_time=2.82132
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 3.66100 (QuantReg: 12.84592) QuantErr: 12.84592 batch_time=0.50585
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 3.96050 (QuantReg: 12.56536) QuantErr: 12.56536 batch_time=0.60286
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 3.39401 (QuantReg: 12.76238) QuantErr: 12.76238 batch_time=0.51353
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 3.46565 (QuantReg: 12.84540) QuantErr: 12.84540 batch_time=0.52589
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 3.47172 (QuantReg: 12.88547) QuantErr: 12.88547 batch_time=0.51961
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 3.42291 (QuantReg: 12.68028) QuantErr: 12.68028 batch_time=0.52994
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 3.61950 (QuantReg: 12.69055) QuantErr: 12.69055 batch_time=0.52696
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 3.46841 (QuantReg: 12.90936) QuantErr: 12.90936 batch_time=1.65178
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 4.27146 (QuantReg: 12.86163) QuantErr: 12.86163 batch_time=0.79036
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 3.67658 (QuantReg: 12.88162) QuantErr: 12.88162 batch_time=0.50505
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 3.66323 (QuantReg: 12.81328) QuantErr: 12.81328 batch_time=0.54156
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 3.41552 (QuantReg: 12.88115) QuantErr: 12.88115 batch_time=0.49236
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 3.53439 (QuantReg: 12.76435) QuantErr: 12.76435 batch_time=0.64507
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 4.28888 (QuantReg: 12.87855) QuantErr: 12.87855 batch_time=0.55427
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 3.64708 (QuantReg: 12.76202) QuantErr: 12.76202 batch_time=0.52652
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 3.49626 (QuantReg: 12.38915) QuantErr: 12.38915 batch_time=0.52267
Train Epoch: 18 codebook_update_time=1.77613
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch18.pth ...
Done in 4.046s
removing stale ckpt [epoch 17] [took 0.02s]
epoch : 18
loss : 3.568705065727234
quant_reg : 12.771998008728028
quant_err : 12.771998008728028
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
LSMDC_full_test/t2v_metrics/R1: 11.3
LSMDC_full_test/t2v_metrics/R5: 28.4
LSMDC_full_test/t2v_metrics/R10: 39.0
LSMDC_full_test/t2v_metrics/R50: 66.9
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 76.65
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.217767800013917
LSMDC_full_test/v2t_metrics/R1: 12.5
LSMDC_full_test/v2t_metrics/R5: 29.3
LSMDC_full_test/v2t_metrics/R10: 39.1
LSMDC_full_test/v2t_metrics/R50: 66.2
LSMDC_full_test/v2t_metrics/MedR: 20.0
LSMDC_full_test/v2t_metrics/MeanR: 78.729
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.28388291512225
mnt_best : 23.467626465492383
not_improved_count: 5
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 3.16056 (QuantReg: 12.63927) QuantErr: 12.63927 batch_time=26.00566
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 3.73340 (QuantReg: 12.54248) QuantErr: 12.54248 batch_time=0.49984
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 3.87033 (QuantReg: 12.63551) QuantErr: 12.63551 batch_time=0.52126
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 3.49865 (QuantReg: 12.67744) QuantErr: 12.67744 batch_time=0.54849
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 3.40817 (QuantReg: 12.72872) QuantErr: 12.72872 batch_time=0.85527
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 3.91665 (QuantReg: 12.84876) QuantErr: 12.84876 batch_time=0.54766
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 3.62093 (QuantReg: 12.79744) QuantErr: 12.79744 batch_time=4.68015
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 3.54543 (QuantReg: 12.79161) QuantErr: 12.79161 batch_time=0.49880
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 3.16418 (QuantReg: 12.93278) QuantErr: 12.93278 batch_time=0.49823
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 3.55183 (QuantReg: 12.88541) QuantErr: 12.88541 batch_time=0.55237
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 3.68054 (QuantReg: 12.69621) QuantErr: 12.69621 batch_time=0.49971
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 3.49212 (QuantReg: 12.57175) QuantErr: 12.57175 batch_time=0.50794
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 3.30484 (QuantReg: 12.95784) QuantErr: 12.95784 batch_time=0.50831
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 3.49854 (QuantReg: 12.88468) QuantErr: 12.88468 batch_time=0.49716
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 3.72845 (QuantReg: 12.96455) QuantErr: 12.96455 batch_time=0.53758
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 3.42928 (QuantReg: 13.06378) QuantErr: 13.06378 batch_time=0.55064
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 3.43308 (QuantReg: 12.67937) QuantErr: 12.67937 batch_time=0.54733
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 3.42321 (QuantReg: 12.85424) QuantErr: 12.85424 batch_time=0.49961
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 3.26588 (QuantReg: 12.92267) QuantErr: 12.92267 batch_time=0.60525
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 3.65196 (QuantReg: 12.74992) QuantErr: 12.74992 batch_time=0.64532
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 3.29009 (QuantReg: 12.68063) QuantErr: 12.68063 batch_time=0.49544
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 3.47388 (QuantReg: 12.95838) QuantErr: 12.95838 batch_time=0.57777
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 3.24725 (QuantReg: 12.81534) QuantErr: 12.81534 batch_time=0.72250
Train Epoch: 19 codebook_update_time=1.68472
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch19.pth ...
Done in 4.750s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.1/checkpoint-epoch19.pth ...
Done in 9.356s
removing stale ckpt [epoch 18] [took 0.01s]
epoch : 19
loss : 3.5115742263793943
quant_reg : 12.79145115661621
quant_err : 12.79145115661621
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
LSMDC_full_test/t2v_metrics/R1: 11.9
LSMDC_full_test/t2v_metrics/R5: 30.0
LSMDC_full_test/t2v_metrics/R10: 39.1
LSMDC_full_test/t2v_metrics/R50: 67.5
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 75.416
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.07769956648204
LSMDC_full_test/v2t_metrics/R1: 12.0
LSMDC_full_test/v2t_metrics/R5: 29.8
LSMDC_full_test/v2t_metrics/R10: 40.8