metad service hang

[kikimo]

Please check the FAQ documentation before raising an issue

metad hang under high load(no response for any further request), and cannot be kill by SIGTERM(raft block on grabbing lock).

we can see from the log:

4 F0930 18:16:43.111698 117635 ThriftServer.cpp:585] Could not drain active requests within allotted deadline. Deadline value: 30 secs. Abort because undefined behavior is p    ossible. Underlying reasons could be either requests that have never terminated, long running requests, or long queues that could not be fully processed.

Describe the bug (must be provided)

A clear and concise description of what the bug is.

Your Environments (must be provided)

• OS: uname -a
• Compliler: g++ --version or clang++ --version
• CPU: lscpu
• Commit id (e.g. a3ffc7d8)

How To Reproduce(must be provided)

Steps to reproduce the behavior:

1. Step 1
2. Step 2
3. Step 3

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

Provide logs and configs, or any other context to trace the problem.

[liuyu85cn]

fixed in #3482