bug: reqwest::Error when read a lot of files but with small open files

[BohuTANG]

Summary

A table with many segments(27912):

mysql> select * from fuse_snapshot('db7861', 't7861') limit 1\G;
*************************** 1. row ***************************
         snapshot_id: 5722b65baa494b88bab82554e567c534
   snapshot_location: 10/15/_ss/5722b65baa494b88bab82554e567c534_v1.json
      format_version: 1
previous_snapshot_id: 8f379fb9899740d5a69581e8cf9758ff
       segment_count: 27912
         block_count: 27912
           row_count: 872250000
  bytes_uncompressed: 1084373017914
    bytes_compressed: 340683903656
          index_size: 3100982328
           timestamp: 2022-09-26 05:12:40.955272

If I set open files to 1024:

root@bohutang-test:~/databend# ulimit -Sn 1024
root@bohutang-test:~/databend# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63801
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63801
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

The query:

mysql> select * from db7861.t7861 limit 1;

ERROR 1105 (HY000): Code: 4000, displayText = operation interrupted (object error: (op: stat, path: 10/15/_sg/37cdf21258a244dab988e75cd6d487cb_v1.json, source: sending request:  Custom { kind: Interrupted, error: reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("bohutang.oss-cn-shanghai-internal.aliyuncs.com")), port: None, path: "/10/15/_sg/37cdf21258a244dab988e75cd6d487cb_v1.json", query: None, fragment: None }, source: hyper::Error(Connect,

[BohuTANG]

We should warn or error a log when the retry happens, cc @Xuanwo
From the query log, I can not find any error about it.

[Xuanwo]

• Refactor databend's IO retry logic
• We should warn or error a log when the retry happens apache/opendal#719

[yufan022]

same problem
#8053

mysql> select count(1) from xx;
+------------+
| count(1)   |
+------------+
| 1032993806 |
+------------+
1 row in set (0.02 sec)
Read 1 rows, 1.00 B in 0.002 sec., 586.89 rows/sec., 586.89 B/sec.

mysql> select version();
+-----------------------------------------------------------------------------------------+
| version()                                                                               |
+-----------------------------------------------------------------------------------------+
| 8.0.26-v0.8.63-nightly-70052ca-simd(rust-1.66.0-nightly-2022-10-08T10:54:48.857705369Z) |
+-----------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

create table xx as select * from xx;

ERROR 1105 (HY000): Code: 4000, displayText = operation interrupted (object error: (op: write, path: 1/254442/_b/7b413ca024a743d4a06045a470ce22c2_v0.parquet, source: sending request:  Custom { kind: Interrupted, error: reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("s3.xx.amazonaws.com")), port: None, path: "/xx/1/254442/_b/7b413ca024a743d4a06045a470ce22c2_v0.parquet", query: None, fragment: None }, source: hyper

[BohuTANG]

Fixed by this commit 1e59548