Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

why my softmax loss echoing@9.5 , who can help me ? #132

Open
ghost opened this issue Dec 18, 2019 · 0 comments
Open

why my softmax loss echoing@9.5 , who can help me ? #132

ghost opened this issue Dec 18, 2019 · 0 comments

Comments

@ghost
Copy link

ghost commented Dec 18, 2019

net: "sphereface/sphereface_model.prototxt"

#文件总数:4686817 , 训练总数:4360447 , 验证总数:326370

test_iter: 5100 #326370 / 256
test_interval: 17033

base_lr: 0.05
lr_policy: "multistep"
gamma: 0.1

#17033
stepvalue: 34066
stepvalue: 85165
stepvalue: 136264
stepvalue: 170330
max_iter: 200000
iter_size: 2

display: 500
momentum: 0.9
weight_decay: 0.0005
snapshot: 2000
snapshot_prefix: "sphereface/modal"

solver_mode: GPU

I1218 09:44:23.435408 42442 solver.cpp:289] Solving SpherefaceNet-20
I1218 09:44:23.435416 42442 solver.cpp:290] Learning Rate Policy: multistep
I1218 09:44:23.438875 42442 solver.cpp:347] Iteration 0, Testing net (#0)
I1218 09:44:23.669739 42442 blocking_queue.cpp:49] Waiting for data
I1218 09:48:31.001149 42442 blocking_queue.cpp:49] Waiting for data
I1218 09:52:34.625572 42442 blocking_queue.cpp:49] Waiting for data
I1218 09:55:54.271853 42442 blocking_queue.cpp:49] Waiting for data
I1218 10:00:13.620533 42442 blocking_queue.cpp:49] Waiting for data
I1218 10:04:16.510815 42442 blocking_queue.cpp:49] Waiting for data
I1218 10:04:38.789021 42764 data_layer.cpp:73] Restarting data prefetching from start.
I1218 10:04:38.832135 42442 solver.cpp:414] Test net output #0: lambda = 11.936
I1218 10:04:38.832242 42442 solver.cpp:414] Test net output #1: softmax_loss = 11.2369 (* 1 = 11.2369 loss)
I1218 10:04:39.691632 42442 solver.cpp:239] Iteration 0 (-nan iter/s, 1216.21s/500 iters), loss = 9.59401
I1218 10:04:39.691682 42442 solver.cpp:258] Train net output #0: lambda = 806.452
I1218 10:04:39.691699 42442 solver.cpp:258] Train net output #1: softmax_loss = 9.58034 (* 1 = 9.58034 loss)
I1218 10:04:39.691731 42442 sgd_solver.cpp:112] Iteration 0, lr = 0.05
I1218 10:15:29.681690 42442 solver.cpp:239] Iteration 500 (0.769271 iter/s, 649.966s/500 iters), loss = 9.75478
I1218 10:15:29.681933 42442 solver.cpp:258] Train net output #0: lambda = 8.2481
I1218 10:15:29.681959 42442 solver.cpp:258] Train net output #1: softmax_loss = 9.59993 (* 1 = 9.59993 loss)
I1218 10:15:29.681974 42442 sgd_solver.cpp:112] Iteration 500, lr = 0.05
I1218 10:21:11.482547 42442 blocking_queue.cpp:49] Waiting for data
I1218 10:29:43.602144 42442 solver.cpp:239] Iteration 1000 (0.585557 iter/s, 853.888s/500 iters), loss = 9.61311
I1218 10:29:43.602360 42442 solver.cpp:258] Train net output #0: lambda = 5
I1218 10:29:43.602385 42442 solver.cpp:258] Train net output #1: softmax_loss = 9.30922 (* 1 = 9.30922 loss)
I1218 10:29:43.602398 42442 sgd_solver.cpp:112] Iteration 1000, lr = 0.05
I1218 10:36:50.166695 42442 blocking_queue.cpp:49] Waiting for data
I1218 10:44:11.976773 42442 solver.cpp:239] Iteration 1500 (0.57581 iter/s, 868.342s/500 iters), loss = 9.77459
I1218 10:44:11.977003 42442 solver.cpp:258] Train net output #0: lambda = 5
I1218 10:44:11.977030 42442 solver.cpp:258] Train net output #1: softmax_loss = 9.92095 (* 1 = 9.92095 loss)
I1218 10:44:11.977046 42442 sgd_solver.cpp:112] Iteration 1500, lr = 0.05
I1218 10:55:30.798585 42442 solver.cpp:464] Snapshotting to binary proto file sphereface/modal/sphereface_solver_iter_2000.caffemodel
I1218 10:55:32.176621 42442 sgd_solver.cpp:284] Snapshotting solver state to binary proto file sphereface/modal/sphereface_solver_iter_2000.solverstate
I1218 10:55:33.252632 42442 solver.cpp:239] Iteration 2000 (0.733945 iter/s, 681.25s/500 iters), loss = 9.62272
I1218 10:55:33.252786 42442 solver.cpp:258] Train net output #0: lambda = 5
I1218 10:55:33.252812 42442 solver.cpp:258] Train net output #1: softmax_loss = 9.62173 (* 1 = 9.62173 loss)
I1218 10:55:33.252830 42442 sgd_solver.cpp:112] Iteration 2000, lr = 0.05
I1218 11:02:54.046000 42442 solver.cpp:239] Iteration 2500 (1.13436 iter/s, 440.777s/500 iters), loss = 9.63188
I1218 11:02:54.046308 42442 solver.cpp:258] Train net output #0: lambda = 5
I1218 11:02:54.046353 42442 solver.cpp:258] Train net output #1: softmax_loss = 9.62981 (* 1 = 9.62981 loss)
I1218 11:02:54.046381 42442 sgd_solver.cpp:112] Iteration 2500, lr = 0.05

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

0 participants