Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

有关精度的问题 #135

Open
happy20200 opened this issue Apr 1, 2023 · 6 comments
Open

有关精度的问题 #135

happy20200 opened this issue Apr 1, 2023 · 6 comments

Comments

@happy20200
Copy link

在下面的参数配置下，为什么重新训练和剪枝之后的结果都达不到预期呢，请问是哪里除了问题

4-2
4-4
@VainF
Copy link
Owner

VainF commented Apr 1, 2023 •
edited
Loading

不好意思，请使用https://github.com/VainF/Torch-Pruning/blob/v1.0/torch_pruning/pruner/algorithms/group_norm_pruner.py这一个v1.0版本的pruner。

我刚才跑了一遍代码，同样发现稀疏训练阶段性能掉的厉害(VAL LOSS偏高)，造成这个现象的原因是后续更新里把bias也加入到稀疏训练中了。实际上bias稀疏存在问题，我们没有仔细测试就上传了，不好意思造成困扰。我们会尽快回退这部分。

v1.0的稀疏训练：

[01/03 18:46:51] cifar10-global-group_sl-resnet56 INFO: Regularizing...
[01/03 18:47:29] cifar10-global-group_sl-resnet56 INFO: Epoch 0/100, Acc=0.8880, Val Loss=0.3793, lr=0.0100
[01/03 18:48:08] cifar10-global-group_sl-resnet56 INFO: Epoch 1/100, Acc=0.8927, Val Loss=0.3641, lr=0.0100
[01/03 18:48:47] cifar10-global-group_sl-resnet56 INFO: Epoch 2/100, Acc=0.8776, Val Loss=0.3781, lr=0.0100
[01/03 18:49:24] cifar10-global-group_sl-resnet56 INFO: Epoch 3/100, Acc=0.8919, Val Loss=0.3406, lr=0.0100
[01/03 18:50:02] cifar10-global-group_sl-resnet56 INFO: Epoch 4/100, Acc=0.8530, Val Loss=0.4590, lr=0.0100
[01/03 18:50:40] cifar10-global-group_sl-resnet56 INFO: Epoch 5/100, Acc=0.8694, Val Loss=0.4012, lr=0.0100
[01/03 18:51:18] cifar10-global-group_sl-resnet56 INFO: Epoch 6/100, Acc=0.8795, Val Loss=0.3679, lr=0.0100
[01/03 18:51:56] cifar10-global-group_sl-resnet56 INFO: Epoch 7/100, Acc=0.8797, Val Loss=0.3591, lr=0.0100
[01/03 18:52:35] cifar10-global-group_sl-resnet56 INFO: Epoch 8/100, Acc=0.8406, Val Loss=0.4998, lr=0.0100
[01/03 18:53:14] cifar10-global-group_sl-resnet56 INFO: Epoch 9/100, Acc=0.8357, Val Loss=0.5032, lr=0.0100
[01/03 18:53:53] cifar10-global-group_sl-resnet56 INFO: Epoch 10/100, Acc=0.8506, Val Loss=0.4577, lr=0.0100
[01/03 18:54:31] cifar10-global-group_sl-resnet56 INFO: Epoch 11/100, Acc=0.8438, Val Loss=0.4718, lr=0.0100
[01/03 18:55:09] cifar10-global-group_sl-resnet56 INFO: Epoch 12/100, Acc=0.8605, Val Loss=0.4302, lr=0.0100
[01/03 18:55:48] cifar10-global-group_sl-resnet56 INFO: Epoch 13/100, Acc=0.8637, Val Loss=0.4081, lr=0.0100
[01/03 18:56:26] cifar10-global-group_sl-resnet56 INFO: Epoch 14/100, Acc=0.8389, Val Loss=0.4846, lr=0.0100
[01/03 18:57:04] cifar10-global-group_sl-resnet56 INFO: Epoch 15/100, Acc=0.8609, Val Loss=0.4332, lr=0.0100

v1.1的稀疏训练（包括bias）

[04/01 21:24:20] cifar10-global-group_sl-resnet56 INFO: Epoch 0/100, Acc=0.8937, Val Loss=0.3552, lr=0.0100
[04/01 21:24:54] cifar10-global-group_sl-resnet56 INFO: Epoch 1/100, Acc=0.8777, Val Loss=0.3756, lr=0.0100
[04/01 21:25:30] cifar10-global-group_sl-resnet56 INFO: Epoch 2/100, Acc=0.8775, Val Loss=0.3682, lr=0.0100
[04/01 21:26:06] cifar10-global-group_sl-resnet56 INFO: Epoch 3/100, Acc=0.8371, Val Loss=0.4986, lr=0.0100
[04/01 21:26:43] cifar10-global-group_sl-resnet56 INFO: Epoch 4/100, Acc=0.8186, Val Loss=0.5571, lr=0.0100
[04/01 21:27:19] cifar10-global-group_sl-resnet56 INFO: Epoch 5/100, Acc=0.8209, Val Loss=0.5196, lr=0.0100
[04/01 21:27:52] cifar10-global-group_sl-resnet56 INFO: Epoch 6/100, Acc=0.8145, Val Loss=0.5753, lr=0.0100
[04/01 21:28:28] cifar10-global-group_sl-resnet56 INFO: Epoch 7/100, Acc=0.8164, Val Loss=0.5387, lr=0.0100
[04/01 21:29:04] cifar10-global-group_sl-resnet56 INFO: Epoch 8/100, Acc=0.8078, Val Loss=0.5849, lr=0.0100
[04/01 21:29:40] cifar10-global-group_sl-resnet56 INFO: Epoch 9/100, Acc=0.8096, Val Loss=0.5848, lr=0.0100
[04/01 21:30:20] cifar10-global-group_sl-resnet56 INFO: Epoch 10/100, Acc=0.8095, Val Loss=0.5720, lr=0.0100
[04/01 21:30:57] cifar10-global-group_sl-resnet56 INFO: Epoch 11/100, Acc=0.7902, Val Loss=0.6302, lr=0.0100
[04/01 21:31:31] cifar10-global-group_sl-resnet56 INFO: Epoch 12/100, Acc=0.7458, Val Loss=0.7950, lr=0.0100
[04/01 21:32:05] cifar10-global-group_sl-resnet56 INFO: Epoch 13/100, Acc=0.7993, Val Loss=0.6104, lr=0.0100
[04/01 21:32:43] cifar10-global-group_sl-resnet56 INFO: Epoch 14/100, Acc=0.7557, Val Loss=0.7445, lr=0.0100
[04/01 21:33:16] cifar10-global-group_sl-resnet56 INFO: Epoch 15/100, Acc=0.7315, Val Loss=0.8837, lr=0.0100

v1.1 替换成v.1.0的pruner (看起来运行正常)

[04/01 21:46:14 cifar10-global-group_sl-resnet56]: Regularizing...
[04/01 21:46:44 cifar10-global-group_sl-resnet56]: Epoch 0/100, Acc=0.9058, Val Loss=0.3018, lr=0.0100
[04/01 21:47:14 cifar10-global-group_sl-resnet56]: Epoch 1/100, Acc=0.8964, Val Loss=0.3465, lr=0.0100
[04/01 21:47:47 cifar10-global-group_sl-resnet56]: Epoch 2/100, Acc=0.8854, Val Loss=0.3689, lr=0.0100
[04/01 21:48:18 cifar10-global-group_sl-resnet56]: Epoch 3/100, Acc=0.8735, Val Loss=0.3865, lr=0.0100
[04/01 21:48:48 cifar10-global-group_sl-resnet56]: Epoch 4/100, Acc=0.8287, Val Loss=0.5341, lr=0.0100
[04/01 21:49:19 cifar10-global-group_sl-resnet56]: Epoch 5/100, Acc=0.8805, Val Loss=0.3690, lr=0.0100
[04/01 21:49:49 cifar10-global-group_sl-resnet56]: Epoch 6/100, Acc=0.8173, Val Loss=0.5618, lr=0.0100
[04/01 21:50:19 cifar10-global-group_sl-resnet56]: Epoch 7/100, Acc=0.8476, Val Loss=0.4714, lr=0.0100
[04/01 21:50:49 cifar10-global-group_sl-resnet56]: Epoch 8/100, Acc=0.8610, Val Loss=0.4180, lr=0.0100
[04/01 21:51:20 cifar10-global-group_sl-resnet56]: Epoch 9/100, Acc=0.8491, Val Loss=0.4741, lr=0.0100
[04/01 21:51:53 cifar10-global-group_sl-resnet56]: Epoch 10/100, Acc=0.8335, Val Loss=0.5335, lr=0.0100
[04/01 21:52:25 cifar10-global-group_sl-resnet56]: Epoch 11/100, Acc=0.8414, Val Loss=0.4855, lr=0.0100
[04/01 21:52:56 cifar10-global-group_sl-resnet56]: Epoch 12/100, Acc=0.8024, Val Loss=0.6125, lr=0.0100
[04/01 21:53:26 cifar10-global-group_sl-resnet56]: Epoch 13/100, Acc=0.8316, Val Loss=0.5069, lr=0.0100
[04/01 21:53:56 cifar10-global-group_sl-resnet56]: Epoch 14/100, Acc=0.7901, Val Loss=0.6546, lr=0.0100
[04/01 21:54:26 cifar10-global-group_sl-resnet56]: Epoch 15/100, Acc=0.8363, Val Loss=0.4915, lr=0.0100
[04/01 21:54:56 cifar10-global-group_sl-resnet56]: Epoch 16/100, Acc=0.8415, Val Loss=0.4856, lr=0.0100
[04/01 21:55:26 cifar10-global-group_sl-resnet56]: Epoch 17/100, Acc=0.8730, Val Loss=0.3823, lr=0.0100
[04/01 21:55:58 cifar10-global-group_sl-resnet56]: Epoch 18/100, Acc=0.8692, Val Loss=0.3996, lr=0.0100
[04/01 21:56:31 cifar10-global-group_sl-resnet56]: Epoch 19/100, Acc=0.8654, Val Loss=0.4004, lr=0.0100
[04/01 21:57:04 cifar10-global-group_sl-resnet56]: Epoch 20/100, Acc=0.8447, Val Loss=0.4735, lr=0.0100
[04/01 21:57:36 cifar10-global-group_sl-resnet56]: Epoch 21/100, Acc=0.8495, Val Loss=0.4643, lr=0.0100
[04/01 21:58:08 cifar10-global-group_sl-resnet56]: Epoch 22/100, Acc=0.8583, Val Loss=0.4115, lr=0.0100
[04/01 21:58:38 cifar10-global-group_sl-resnet56]: Epoch 23/100, Acc=0.8610, Val Loss=0.4163, lr=0.0100
[04/01 21:59:12 cifar10-global-group_sl-resnet56]: Epoch 24/100, Acc=0.8563, Val Loss=0.4411, lr=0.0100
[04/01 21:59:43 cifar10-global-group_sl-resnet56]: Epoch 25/100, Acc=0.8142, Val Loss=0.5780, lr=0.0100
[04/01 22:00:15 cifar10-global-group_sl-resnet56]: Epoch 26/100, Acc=0.8499, Val Loss=0.4680, lr=0.0100
[04/01 22:00:49 cifar10-global-group_sl-resnet56]: Epoch 27/100, Acc=0.8386, Val Loss=0.4765, lr=0.0100
[04/01 22:01:21 cifar10-global-group_sl-resnet56]: Epoch 28/100, Acc=0.8652, Val Loss=0.4003, lr=0.0100
[04/01 22:01:52 cifar10-global-group_sl-resnet56]: Epoch 29/100, Acc=0.8467, Val Loss=0.4801, lr=0.0100
[04/01 22:02:22 cifar10-global-group_sl-resnet56]: Epoch 30/100, Acc=0.8673, Val Loss=0.4074, lr=0.0100
[04/01 22:02:54 cifar10-global-group_sl-resnet56]: Epoch 31/100, Acc=0.8353, Val Loss=0.5004, lr=0.0100
[04/01 22:03:24 cifar10-global-group_sl-resnet56]: Epoch 32/100, Acc=0.8279, Val Loss=0.5507, lr=0.0100
[04/01 22:03:54 cifar10-global-group_sl-resnet56]: Epoch 33/100, Acc=0.8079, Val Loss=0.6088, lr=0.0100
[04/01 22:04:24 cifar10-global-group_sl-resnet56]: Epoch 34/100, Acc=0.8345, Val Loss=0.5143, lr=0.0100
[04/01 22:04:57 cifar10-global-group_sl-resnet56]: Epoch 35/100, Acc=0.8464, Val Loss=0.4783, lr=0.0100
[04/01 22:05:27 cifar10-global-group_sl-resnet56]: Epoch 36/100, Acc=0.8650, Val Loss=0.4078, lr=0.0100
[04/01 22:05:57 cifar10-global-group_sl-resnet56]: Epoch 37/100, Acc=0.8253, Val Loss=0.5426, lr=0.0100
[04/01 22:06:30 cifar10-global-group_sl-resnet56]: Epoch 38/100, Acc=0.8294, Val Loss=0.5347, lr=0.0100
[04/01 22:07:01 cifar10-global-group_sl-resnet56]: Epoch 39/100, Acc=0.8607, Val Loss=0.4183, lr=0.0100
[04/01 22:07:31 cifar10-global-group_sl-resnet56]: Epoch 40/100, Acc=0.8491, Val Loss=0.4571, lr=0.0100
[04/01 22:08:04 cifar10-global-group_sl-resnet56]: Epoch 41/100, Acc=0.8073, Val Loss=0.5806, lr=0.0100
[04/01 22:08:40 cifar10-global-group_sl-resnet56]: Epoch 42/100, Acc=0.8501, Val Loss=0.4582, lr=0.0100
[04/01 22:09:10 cifar10-global-group_sl-resnet56]: Epoch 43/100, Acc=0.8587, Val Loss=0.4206, lr=0.0100
[04/01 22:09:39 cifar10-global-group_sl-resnet56]: Epoch 44/100, Acc=0.8384, Val Loss=0.4924, lr=0.0100
[04/01 22:10:13 cifar10-global-group_sl-resnet56]: Epoch 45/100, Acc=0.8545, Val Loss=0.4515, lr=0.0100
[04/01 22:11:02 cifar10-global-group_sl-resnet56]: Epoch 46/100, Acc=0.8625, Val Loss=0.4197, lr=0.0100
[04/01 22:11:33 cifar10-global-group_sl-resnet56]: Epoch 47/100, Acc=0.8513, Val Loss=0.4507, lr=0.0100
[04/01 22:12:05 cifar10-global-group_sl-resnet56]: Epoch 48/100, Acc=0.8529, Val Loss=0.4409, lr=0.0100
[04/01 22:12:36 cifar10-global-group_sl-resnet56]: Epoch 49/100, Acc=0.8245, Val Loss=0.5699, lr=0.0100
[04/01 22:13:06 cifar10-global-group_sl-resnet56]: Epoch 50/100, Acc=0.7740, Val Loss=0.7118, lr=0.0100
[04/01 22:13:37 cifar10-global-group_sl-resnet56]: Epoch 51/100, Acc=0.8499, Val Loss=0.4623, lr=0.0100
[04/01 22:14:13 cifar10-global-group_sl-resnet56]: Epoch 52/100, Acc=0.8347, Val Loss=0.5216, lr=0.0100
[04/01 22:14:44 cifar10-global-group_sl-resnet56]: Epoch 53/100, Acc=0.8342, Val Loss=0.4990, lr=0.0100
[04/01 22:15:15 cifar10-global-group_sl-resnet56]: Epoch 54/100, Acc=0.8388, Val Loss=0.5122, lr=0.0100
[04/01 22:15:44 cifar10-global-group_sl-resnet56]: Epoch 55/100, Acc=0.8560, Val Loss=0.4300, lr=0.0100
[04/01 22:16:14 cifar10-global-group_sl-resnet56]: Epoch 56/100, Acc=0.8552, Val Loss=0.4374, lr=0.0100
[04/01 22:16:45 cifar10-global-group_sl-resnet56]: Epoch 57/100, Acc=0.7804, Val Loss=0.6783, lr=0.0100
[04/01 22:17:18 cifar10-global-group_sl-resnet56]: Epoch 58/100, Acc=0.8266, Val Loss=0.5295, lr=0.0100
[04/01 22:17:49 cifar10-global-group_sl-resnet56]: Epoch 59/100, Acc=0.8637, Val Loss=0.4160, lr=0.0100
[04/01 22:18:22 cifar10-global-group_sl-resnet56]: Epoch 60/100, Acc=0.9178, Val Loss=0.2458, lr=0.0010
[04/01 22:18:55 cifar10-global-group_sl-resnet56]: Epoch 61/100, Acc=0.9251, Val Loss=0.2346, lr=0.0010
[04/01 22:19:25 cifar10-global-group_sl-resnet56]: Epoch 62/100, Acc=0.9232, Val Loss=0.2346, lr=0.0010

@VainF
Copy link
Owner

VainF commented Apr 1, 2023 •
edited
Loading

My results with the latest version.

[04/01 23:00:49 cifar10-global-group_sl-resnet56]: Params: 0.86 M => 0.45 M (52.18%)
[04/01 23:00:49 cifar10-global-group_sl-resnet56]: FLOPs: 127.12 M => 59.76 M (47.01%, 2.13X )
[04/01 23:00:49 cifar10-global-group_sl-resnet56]: Acc: 0.9353 => 0.9070
[04/01 23:00:49 cifar10-global-group_sl-resnet56]: Val Loss: 0.2391 => 0.3319
[04/01 23:00:49 cifar10-global-group_sl-resnet56]: Finetuning...

...

[04/01 23:35:14 cifar10-global-group_sl-resnet56]: Epoch 93/100, Acc=0.9378, Val Loss=0.2518, lr=0.0001
[04/01 23:35:27 cifar10-global-group_sl-resnet56]: Epoch 94/100, Acc=0.9386, Val Loss=0.2533, lr=0.0001
[04/01 23:35:44 cifar10-global-group_sl-resnet56]: Epoch 95/100, Acc=0.9376, Val Loss=0.2538, lr=0.0001
[04/01 23:36:00 cifar10-global-group_sl-resnet56]: Epoch 96/100, Acc=0.9375, Val Loss=0.2551, lr=0.0001
[04/01 23:36:30 cifar10-global-group_sl-resnet56]: Epoch 97/100, Acc=0.9377, Val Loss=0.2539, lr=0.0001
[04/01 23:36:58 cifar10-global-group_sl-resnet56]: Epoch 98/100, Acc=0.9385, Val Loss=0.2521, lr=0.0001
[04/01 23:37:12 cifar10-global-group_sl-resnet56]: Epoch 99/100, Acc=0.9375, Val Loss=0.2535, lr=0.0001
[04/01 23:37:12 cifar10-global-group_sl-resnet56]: Best Acc=0.9387

VainF added a commit that referenced this issue Apr 1, 2023
@VainF
[#135]: Fixed an accuracy issue in GroupNormPruner
f8a12fe
@happy20200
Copy link
Author

明白了，谢谢您的回复，我换成V1.0版本的代码试一试，不过这是影响剪枝后模型性能的原因对吗，那重新训练的精度也达不到预期，是什么原因啊

@VainF
Copy link
Owner

VainF commented Apr 2, 2023 •
edited
Loading

明白了，谢谢您的回复，我换成V1.0版本的代码试一试，不过这是影响剪枝后模型性能的原因对吗，那重新训练的精度也达不到预期，是什么原因啊

重新训练是指pretraining么？bias会同时影响regularization和finetuning两个阶段

@happy20200
Copy link
Author

happy20200 commented Apr 3, 2023 via email

@VainF
Copy link
Owner

VainF commented Apr 18, 2023 •
edited
Loading

您好，抱歉回复的比较迟，刚重新过了一遍pre-training => sparse learning => pruning => post-training的流程，好像并没有出现重新训练精度达不到预期的情况。

Pretraining

[04/18 16:27:23] cifar10-resnet56 INFO: Epoch 197/200, Acc=0.9359, Val Loss=0.2645, lr=0.0001
[04/18 16:28:52] cifar10-resnet56 INFO: Epoch 198/200, Acc=0.9359, Val Loss=0.2621, lr=0.0001
[04/18 16:30:24] cifar10-resnet56 INFO: Epoch 199/200, Acc=0.9363, Val Loss=0.2626, lr=0.0001
[04/18 16:30:24] cifar10-resnet56 INFO: Best Acc=0.9376

Sparse Learning & Pruning

[04/18 19:00:38] cifar10-global-group_sl-resnet56 INFO: Params: 0.86 M => 0.30 M (35.28%)
[04/18 19:00:38] cifar10-global-group_sl-resnet56 INFO: FLOPs: 127.12 M => 49.48 M (38.93%, 2.57X )
[04/18 19:00:38] cifar10-global-group_sl-resnet56 INFO: Acc: 0.9366 => 0.7992
[04/18 19:00:38] cifar10-global-group_sl-resnet56 INFO: Val Loss: 0.2264 => 0.7311
[04/18 19:00:38] cifar10-global-group_sl-resnet56 INFO: Finetuning...

Post-Training

[04/18 19:43:17] cifar10-global-group_sl-resnet56 INFO: Epoch 96/100, Acc=0.9363, Val Loss=0.2333, lr=0.0001
[04/18 19:43:33] cifar10-global-group_sl-resnet56 INFO: Epoch 97/100, Acc=0.9350, Val Loss=0.2335, lr=0.0001
[04/18 19:43:48] cifar10-global-group_sl-resnet56 INFO: Epoch 98/100, Acc=0.9361, Val Loss=0.2307, lr=0.0001
[04/18 19:44:03] cifar10-global-group_sl-resnet56 INFO: Epoch 99/100, Acc=0.9365, Val Loss=0.2326, lr=0.0001
[04/18 19:44:03] cifar10-global-group_sl-resnet56 INFO: Best Acc=0.9368

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@VainF @happy20200