ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0 #3408

firaterdem · 2023-07-26T17:54:41Z

问题确认 Search before asking

我已经查询历史issue(包括open与closed)，没有发现相似的bug。I have searched the open and closed issues and found no similar bug report.

Bug描述 Describe the Bug

In Google Colab Quick Start code is giving the following error message at the training part. How can I solve it?

2023-07-26 12:56:35 [INFO] [TRAIN] epoch: 8, iter: 500/1000, loss: 0.0543, lr: 0.005369, batch_cost: 0.1380, reader_cost: 0.04589, ips: 28.9826 samples/sec | ETA 00:01:09 2023-07-26 12:56:35 [INFO] Start evaluating (total_samples: 76, total_iters: 76)... Traceback (most recent call last): File "/content/drive/MyDrive/SmartPol_PaddleSeg/PaddleSeg/tools/train.py", line 195, in <module> main(args) File "/content/drive/MyDrive/SmartPol_PaddleSeg/PaddleSeg/tools/train.py", line 170, in main train( File "/usr/local/lib/python3.10/dist-packages/paddleseg/core/train.py", line 315, in train mean_iou, acc, _, _, _ = evaluate( File "/usr/local/lib/python3.10/dist-packages/paddleseg/core/val.py", line 161, in evaluate intersect_area, pred_area, label_area = metrics.calculate_area( File "/usr/local/lib/python3.10/dist-packages/paddleseg/utils/metrics.py", line 57, in calculate_area pred_area = paddle.concat(pred_area) File "/usr/local/lib/python3.10/dist-packages/paddle/tensor/manipulation.py", line 1121, in concat return _C_ops.concat(input, axis) ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0 [Hint: Expected axis >= -rank && axis < rank == true, but received axis >= -rank && axis < rank:0 != true:1.] (at ../paddle/phi/infermeta/multiary.cc:954)

复现环境 Environment

OS: Linux (Google Colab)
PaddlePaddle: 2.5.0
PaddleSeg: 2.8.0
Python: 3.10.6
CUDA: 11.8

Bug描述确认 Bug description confirmation

我确认已经提供了Bug复现步骤、代码改动说明、以及环境信息，确认问题是可以复现的。I confirm that the bug replication steps, code change instructions, and environment information have been provided, and the problem can be reproduced.

是否愿意提交PR？ Are you willing to submit a PR?

我愿意提交PR！I'd like to help by submitting a PR!

The text was updated successfully, but these errors were encountered:

Asthestarsfalll · 2023-07-27T07:49:40Z

You can attempt to fix the bug using this method, but finding the root cause of the issue will require additional effort.

firaterdem · 2023-07-28T19:09:46Z

You can attempt to fix the bug using this method, but finding the root cause of the issue will require additional effort.

@Asthestarsfalll I corrected the related part of the code ("/usr/local/lib/python3.10/dist-packages/paddleseg/utils/metrics.py") as suggested:

pred_area.append(paddle.sum(paddle.cast(pred_i, "int64")).unsqueeze(0))

But it is still giving the same error.

Asthestarsfalll · 2023-07-29T07:24:36Z

@firaterdem
Can you share the Colab notebook so that I can investigate the reason?

firaterdem · 2023-07-29T08:21:05Z

@firaterdem Can you share the Colab notebook so that I can investigate the reason?

@Asthestarsfalll sure, here is the notebook:

https://colab.research.google.com/drive/1Uu4eUHcDNA7LThE5KGexG5HdEZR-vB2t?usp=sharing

Asthestarsfalll · 2023-07-29T08:43:16Z

@firaterdem
I install paddleseg2.8 through source code instead of pip, It works well:

You can try this to install paddleseg

git clone https://github.com/PaddlePaddle/PaddleSeg
cd PaddleSeg
pip install -r requirements.txt
python setup.py install

firaterdem · 2023-07-29T19:44:48Z

@firaterdem I install paddleseg2.8 through source code instead of pip, It works well:

You can try this to install paddleseg
git clone https://github.com/PaddlePaddle/PaddleSeg
cd PaddleSeg
pip install -r requirements.txt
python setup.py install

@Asthestarsfalll Yes, installing paddleseg in this way solved the problem. Thank you very much!

ToddBear · 2023-08-07T06:55:45Z

以上回答已经充分解答了问题，如果有新的问题欢迎随时提交issue，或者在此条issue下继续回复～
我们开启了飞桨套件的ISSUE攻关活动，欢迎感兴趣的开发者参加：PaddlePaddle/PaddleOCR#10223

Jerry3062 · 2023-08-08T10:11:21Z

@firaterdem I install paddleseg2.8 through source code instead of pip, It works well:
You can try this to install paddleseg
git clone https://github.com/PaddlePaddle/PaddleSeg
cd PaddleSeg
pip install -r requirements.txt
python setup.py install
@Asthestarsfalll Yes, installing paddleseg in this way solved the problem. Thank you very much!

I have try this method, but not solve the problem. How can I solve this.

Asthestarsfalll · 2023-08-08T10:15:14Z

I have try this method, but not solve the problem. How can I solve this.

should give more details

Jerry3062 · 2023-08-09T02:30:31Z

I have try this method, but not solve the problem. How can I solve this.

should give more details

I change my code from gitee to github and solve the problem, thanks.

jason660519 · 2023-09-24T12:46:35Z

我也遇到同樣的問題，已經按照上面的步驟無論是重新安裝或downgrad版本重跑數變，都是一樣的問題:
ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0
[Hint: Expected axis >= -rank && axis < rank == true, but received axis >= -rank && axis < rank:0 != true:1.] (at ..\paddle\phi\infermeta\multiary.cc:961)

我是follow 官方教程:https://github.com/PaddlePaddle/PaddleSeg/blob/release/2.8/docs/quick_start_cn.md

I0924 22:43:01.913477 15680 eager_method.cc:140] Warning:: 0D Tensor cannot be used as 'Tensor.numpy()[0]' . In order to avoid this problem, 0D Tensor will be changed to 1D numpy currently, but it's not correct and will be removed in release 2.6. For Tensor contain only one element, Please modify 'Tensor.numpy()[0]' to 'float(Tensor)' as soon as possible, otherwise 'Tensor.numpy()[0]' will raise error in release 2.6.
2023-09-24 22:43:01 [INFO] [TRAIN] epoch: 8, iter: 500/1000, loss: 0.0397, lr: 0.005369, batch_cost: 0.1381, reader_cost: 0.04371, ips: 28.9607 samples/sec | ETA 00:01:09
2023-09-24 22:43:01 [INFO] Start evaluating (total_samples: 76, total_iters: 76)...
Traceback (most recent call last):
File "C:\Users\a0922\Desktop\paddlepaddle\PaddleSeg\tools\train.py", line 195, in
main(args)
File "C:\Users\a0922\Desktop\paddlepaddle\PaddleSeg\tools\train.py", line 170, in main
train(
File "C:\Users\a0922\Desktop\paddlepaddle\Lib\site-packages\paddleseg\core\train.py", line 315, in train
mean_iou, acc, _, _, _ = evaluate(
^^^^^^^^^
File "C:\Users\a0922\Desktop\paddlepaddle\Lib\site-packages\paddleseg\core\val.py", line 161, in evaluate
intersect_area, pred_area, label_area = metrics.calculate_area(
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\a0922\Desktop\paddlepaddle\Lib\site-packages\paddleseg\utils\metrics.py", line 57, in calculate_area
pred_area = paddle.concat(pred_area)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\a0922\Desktop\paddlepaddle\Lib\site-packages\paddle\tensor\manipulation.py", line 1121, in concat
return _C_ops.concat(input, axis)
^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0
[Hint: Expected axis >= -rank && axis < rank == true, but received axis >= -rank && axis < rank:0 != true:1.] (at ..\paddle\phi\infermeta\multiary.cc:961)

Asthestarsfalll · 2023-09-24T14:24:59Z

@jason660519

是按照这种方式安装的吗，请确认安装成功了，而不是使用的pypi源当中的paddleseg

git clone https://github.com/PaddlePaddle/PaddleSeg
cd PaddleSeg
pip install -r requirements.txt
python setup.py install

jason660519 · 2023-09-25T06:29:46Z

我是按照官方網站的教學:
https://github.com/PaddlePaddle/PaddleSeg/blob/release/2.8/docs/quick_start_cn.md
一步一步重複做了很多次，每次在第三步驟執行: python tools/train.py --config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml --save_interval 500 --do_eval --use_vdl --save_dir output 之後，都是一樣的結果。

我的硬體與安裝包如下:
OS:Windows 10, AMD CPU, NVIDIA RTX 3090 24G
GPU Compute Capability: 8.6 Driver API Version: 12.2, Runtime API Version: 12.0
cuDNN Version: 8.9.
PaddlePaddle works well on 1 GPU.
PaddlePaddle: 2.5.1
Paddleseg: 2.8.0

錯誤碼如下:

Warning:: 0D Tensor cannot be used as 'Tensor.numpy()[0]' . In order to avoid this problem, 0D Tensor will be changed to 1D numpy currently, but it's not correct and will be removed in release 2.6. For Tensor contain only one element, Please modify 'Tensor.numpy()[0]' to 'float(Tensor)' as soon as possible, otherwise 'Tensor.numpy()[0]' will raise error in release 2.6. (這一段的警告提示，應該無關訓練)

中止執行的時間點就是ve_interval 500的地方，
iter設500 iter就停在500(epoch約跑到:8)
iter設1000 iter就停在1000(epoch約跑到:17)
Iter設10000 iter就停在10000(epoch約跑到:?? 截稿前，還在跑....)

epoch: 8, iter: 500/1000, loss: 0.0478, lr: 0.005369, batch_cost: 0.1344, reader_cost: 0.05471, ips: 29.7619
samples/sec | ETA 00:01:07
2023-09-25 16:04:53 [INFO] Start evaluating (total_samples: 76, total_iters: 76)...
Traceback (most recent call last):
File "C:\Users\a0922\Desktop\paddlepaddle\paddleseg\tools\train.py", line 195, in
main(args)
File "C:\Users\a0922\Desktop\paddlepaddle\paddleseg\tools\train.py", line 170, in main
train(
File "C:\Users\a0922\Desktop\paddlepaddle\Lib\site-packages\paddleseg\core\train.py", line 315, in train
mean_iou, acc, _, _, _ = evaluate(
^^^^^^^^^
File "C:\Users\a0922\Desktop\paddlepaddle\Lib\site-packages\paddleseg\core\val.py", line 161, in evaluate
intersect_area, pred_area, label_area = metrics.calculate_area(
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\a0922\Desktop\paddlepaddle\Lib\site-packages\paddleseg\utils\metrics.py", line 57, in calculate_area
pred_area = paddle.concat(pred_area)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\a0922\Desktop\paddlepaddle\Lib\site-packages\paddle\tensor\manipulation.py", line 1121, in concat
return _C_ops.concat(input, axis)
^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0
[Hint: Expected axis >= -rank && axis < rank == true, but received axis >= -rank && axis < rank:0 != true:1.] (at ..\paddle\phi\infermeta\multiary.cc:961)

Asthestarsfalll · 2023-09-25T06:33:24Z

@jason660519 不要使用pip安装，确保本地的包是通过python setup.py安装上的

jason660519 · 2023-09-25T06:43:53Z

用 python setup.py 安裝不了，安裝到一半會跳出錯誤碼...

Asthestarsfalll · 2023-09-25T06:45:36Z

用 python setup.py 安裝不了，安裝到一半會跳出錯誤碼...

那就使用更低版本的paddle试试

jason660519 · 2023-09-25T07:04:54Z

(paddle2-4) PS C:\Users\a0922\Desktop\paddle2-4> python -m pip install paddlepaddle-gpu==2.4.2.post117 -f https://www.paddlepaddle.org.cn/whl/windows/mkl/avx/stable.html
Looking in links: https://www.paddlepaddle.org.cn/whl/windows/mkl/avx/stable.html
ERROR: Could not find a version that satisfies the requirement paddlepaddle-gpu==2.4.2.post117 (from versions: 2.5.1, 2.5.1.post102, 2.5.1.post112, 2.5.1.post116, 2.5.1.post117, 2.5.1.post120)
ERROR: No matching distribution found for paddlepaddle-gpu==2.4.2.post117
(paddle2-4) PS C:\Users\a0922\Desktop\paddle2-4>

PS C:\Users\a0922\Desktop\paddle2-4> python -m pip install paddlepaddle-gpu==2.3.2.post116 -f https://www.paddlepaddle.org.cn/whl/windows/mkl/avx/stable.html
Looking in links: https://www.paddlepaddle.org.cn/whl/windows/mkl/avx/stable.html
ERROR: Could not find a version that satisfies the requirement paddlepaddle-gpu==2.3.2.post116 (from versions: 2.5.1, 2.5.1.post102, 2.5.1.post112, 2.5.1.post116, 2.5.1.post117, 2.5.1.post120)
ERROR: No matching distribution found for paddlepaddle-gpu==2.3.2.post116

即使降了一代版本，官網回應satisfy的paddle版本全都2.5.1版本以上
2.4.2 版本，官網系統回應找不到該版本。
2.3.2 版本，官網系統回應找不到該版本。
2.4.2 CPU的版本雖可以安裝...但PaddleSeg會裝不了。
已經一個禮拜了...真想放棄!

Asthestarsfalll · 2023-09-25T08:34:52Z

@jason660519 请给出本地安装paddleseg的报错信息

yuzu16 · 2024-01-03T03:17:13Z

方式安装paddleseg解决了这个问题。非常

解决了吗

yuzu16 · 2024-01-03T08:55:54Z

@jason660519

是按照这种方式安装的吗，请确认安装成功了，而不是使用的pypi源使用的paddleseg
git clone https://github.com/PaddlePaddle/PaddleSeg
cd PaddleSeg
pip install -r requirements.txt
python setup.py install

这种方式安装的还是没有解决，

yuzu16 · 2024-01-03T13:53:17Z

我已经尝试过这个方法，但没有解决问题。我该如何解决这个问题。

应该提供更多细节

我也是源代码下载也是没解决问题版本是2.8.0 是因为版本问题吗？

FaberSober · 2024-02-12T13:57:05Z

跟着官网的步骤，同样遇到了这个问题。

Asthestarsfalll · 2024-03-05T12:54:46Z

跟着官网的步骤，同样遇到了这个问题。

使用更新的版本2.9呢？

pvinhd · 2024-04-02T13:50:18Z

I have try this method, but not solve the problem. How can I solve this.

should give more details

I change my code from gitee to github and solve the problem, thanks.

where you change this code

messy2333 · 2024-07-01T06:59:09Z

@firaterdem I install paddleseg2.8 through source code instead of pip, It works well:

You can try this to install paddleseg
git clone https://github.com/PaddlePaddle/PaddleSeg
cd PaddleSeg
pip install -r requirements.txt
python setup.py install

我按照您的方法重新装了paddleseg，还是报了这个错误，似乎不是metric的错误
Traceback (most recent call last):
File "/home/PaddleSeg/contrib/QualityInspector/tools/det/train.py", line 197, in
main()
File "/home/PaddleSeg/contrib/QualityInspector/tools/det/train.py", line 193, in main
run(FLAGS, cfg)
File "/home/PaddleSeg/contrib/QualityInspector/tools/det/train.py", line 146, in run
trainer.train(FLAGS.eval)
File "/usr/local/lib/python3.10/dist-packages/ppdet/engine/trainer.py", line 539, in train
outputs = model(data)
File "/usr/local/lib/python3.10/dist-packages/paddle/nn/layer/layers.py", line 1429, in call
return self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ppdet/modeling/architectures/meta_arch.py", line 60, in forward
out = self.get_loss()
File "/usr/local/lib/python3.10/dist-packages/ppdet/modeling/architectures/faster_rcnn.py", line 112, in get_ loss
rpn_loss, bbox_loss = self._forward()
File "/usr/local/lib/python3.10/dist-packages/ppdet/modeling/architectures/faster_rcnn.py", line 80, in _forw ard
rois, rois_num, rpn_loss = self.rpn_head(body_feats, self.inputs)
File "/usr/local/lib/python3.10/dist-packages/paddle/nn/layer/layers.py", line 1429, in call
return self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ppdet/modeling/proposal_generator/rpn_head.py", line 141, in fo rward
rois, rois_num = self._gen_proposal(scores, deltas, anchors, inputs)
File "/usr/local/lib/python3.10/dist-packages/ppdet/modeling/proposal_generator/rpn_head.py", line 234, in _g en_proposal
bs_rois_num_collect = paddle.concat(bs_rois_num_collect)
File "/usr/local/lib/python3.10/dist-packages/paddle/tensor/manipulation.py", line 1263, in concat
return _C_ops.concat(input, axis)
ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0

guajioao · 2024-09-14T09:52:30Z

@firaterdem I install paddleseg2.8 through source code instead of pip, It works well:
You can try this to install paddleseg
git clone https://github.com/PaddlePaddle/PaddleSeg
cd PaddleSeg
pip install -r requirements.txt
python setup.py install
我按照您的方法重新装了paddleseg，还是报了这个错误，似乎不是metric的错误 Traceback (most recent call last): File "/home/PaddleSeg/contrib/QualityInspector/tools/det/train.py", line 197, in main() File "/home/PaddleSeg/contrib/QualityInspector/tools/det/train.py", line 193, in main run(FLAGS, cfg) File "/home/PaddleSeg/contrib/QualityInspector/tools/det/train.py", line 146, in run trainer.train(FLAGS.eval) File "/usr/local/lib/python3.10/dist-packages/ppdet/engine/trainer.py", line 539, in train outputs = model(data) File "/usr/local/lib/python3.10/dist-packages/paddle/nn/layer/layers.py", line 1429, in call return self.forward(*inputs, **kwargs) File "/usr/local/lib/python3.10/dist-packages/ppdet/modeling/architectures/meta_arch.py", line 60, in forward out = self.get_loss() File "/usr/local/lib/python3.10/dist-packages/ppdet/modeling/architectures/faster_rcnn.py", line 112, in get_ loss rpn_loss, bbox_loss = self._forward() File "/usr/local/lib/python3.10/dist-packages/ppdet/modeling/architectures/faster_rcnn.py", line 80, in _forw ard rois, rois_num, rpn_loss = self.rpn_head(body_feats, self.inputs) File "/usr/local/lib/python3.10/dist-packages/paddle/nn/layer/layers.py", line 1429, in call return self.forward(*inputs, **kwargs) File "/usr/local/lib/python3.10/dist-packages/ppdet/modeling/proposal_generator/rpn_head.py", line 141, in fo rward rois, rois_num = self._gen_proposal(scores, deltas, anchors, inputs) File "/usr/local/lib/python3.10/dist-packages/ppdet/modeling/proposal_generator/rpn_head.py", line 234, in _g en_proposal bs_rois_num_collect = paddle.concat(bs_rois_num_collect) File "/usr/local/lib/python3.10/dist-packages/paddle/tensor/manipulation.py", line 1263, in concat return _C_ops.concat(input, axis) ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0

我调试了一下看到在metric.py第53-55行的paddle.sum(*)计算结果shape=[]，在for循环结束后进行concat就会报错。我将sum的结果用升了一维，之后就没再报错了。如果你也是这种情况可以试试
pred_area.append(paddle.sum(paddle.cast(pred_i, "int64")).unsqueeze(0))

TingquanGao · 2024-11-13T07:36:20Z

Thanks for this issue. As it has been inactive for a long time, we would close it. If you has any questions, please feel free to reopen or new issue, and we will follow up and resolve it.

firaterdem added the bug Something isn't working label Jul 26, 2023

ToddBear mentioned this issue Jul 28, 2023

CV套件建设专项活动 #3333

Closed

ToddBear assigned Asthestarsfalll Jul 28, 2023

firaterdem closed this as completed Jul 29, 2023

ToddBear reopened this Aug 7, 2023

Programrookie33 mentioned this issue Oct 11, 2023

paddleseg2.8根据官方文档跑默认的configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml配置报错 #3373

Closed

3 tasks

fly627600815 mentioned this issue Nov 2, 2023

按照官方PaddleSeg实战训练optic_disc_seg报错 #3556

Closed

1 task

Lee6384 mentioned this issue Jan 3, 2024

为什么训练时出现这种警告，特别长一串红色warnning #3353

Closed

3 tasks

paddle-bot bot assigned shiyutang Feb 7, 2024

TingquanGao closed this as completed Nov 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0 #3408

ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0 #3408

firaterdem commented Jul 26, 2023

Asthestarsfalll commented Jul 27, 2023

firaterdem commented Jul 28, 2023

Asthestarsfalll commented Jul 29, 2023

firaterdem commented Jul 29, 2023

Asthestarsfalll commented Jul 29, 2023

firaterdem commented Jul 29, 2023

ToddBear commented Aug 7, 2023

Jerry3062 commented Aug 8, 2023

Asthestarsfalll commented Aug 8, 2023

Jerry3062 commented Aug 9, 2023

jason660519 commented Sep 24, 2023

Asthestarsfalll commented Sep 24, 2023

jason660519 commented Sep 25, 2023

Asthestarsfalll commented Sep 25, 2023

jason660519 commented Sep 25, 2023

Asthestarsfalll commented Sep 25, 2023

jason660519 commented Sep 25, 2023 •

edited

Loading

Asthestarsfalll commented Sep 25, 2023

yuzu16 commented Jan 3, 2024

yuzu16 commented Jan 3, 2024

yuzu16 commented Jan 3, 2024

FaberSober commented Feb 12, 2024

Asthestarsfalll commented Mar 5, 2024

pvinhd commented Apr 2, 2024

messy2333 commented Jul 1, 2024

guajioao commented Sep 14, 2024

TingquanGao commented Nov 13, 2024

ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0 #3408

ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0 #3408

Comments

firaterdem commented Jul 26, 2023

问题确认 Search before asking

Bug描述 Describe the Bug

复现环境 Environment

Bug描述确认 Bug description confirmation

是否愿意提交PR？ Are you willing to submit a PR?

Asthestarsfalll commented Jul 27, 2023

firaterdem commented Jul 28, 2023

Asthestarsfalll commented Jul 29, 2023

firaterdem commented Jul 29, 2023

Asthestarsfalll commented Jul 29, 2023

firaterdem commented Jul 29, 2023

ToddBear commented Aug 7, 2023

Jerry3062 commented Aug 8, 2023

Asthestarsfalll commented Aug 8, 2023

Jerry3062 commented Aug 9, 2023

jason660519 commented Sep 24, 2023

Asthestarsfalll commented Sep 24, 2023

jason660519 commented Sep 25, 2023

Asthestarsfalll commented Sep 25, 2023

jason660519 commented Sep 25, 2023

Asthestarsfalll commented Sep 25, 2023

jason660519 commented Sep 25, 2023 • edited Loading

Asthestarsfalll commented Sep 25, 2023

yuzu16 commented Jan 3, 2024

yuzu16 commented Jan 3, 2024

yuzu16 commented Jan 3, 2024

FaberSober commented Feb 12, 2024

Asthestarsfalll commented Mar 5, 2024

pvinhd commented Apr 2, 2024

messy2333 commented Jul 1, 2024

guajioao commented Sep 14, 2024

TingquanGao commented Nov 13, 2024

jason660519 commented Sep 25, 2023 •

edited

Loading