Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[docdb] yb-admin logging WARNING about "Failed to schedule invoking callback on response" #4131

Closed
bmatican opened this issue Apr 1, 2020 · 1 comment
Closed

[docdb] yb-admin logging WARNING about "Failed to schedule invoking callback on response" #4131

bmatican opened this issue Apr 1, 2020 · 1 comment
Assignees
@iSignal
Labels
area/docdb YugabyteDB core features priority/high High Priority

Comments

@bmatican
Copy link
Contributor

bmatican commented Apr 1, 2020 •
edited
Loading 

./bin/yb-admin --master_addresses=<masters> get_universe_config
I0401 19:37:02.099141 27746 mem_tracker.cc:249] MemTracker: hard memory limit is 3.145429 GB
I0401 19:37:02.099287 27746 mem_tracker.cc:251] MemTracker: soft memory limit is 2.673614 GB
Config:
version: 0
cluster_uuid: "20253f4b-8219-419d-881c-ada2cdd008fc"

W0401 19:37:02.107713 27751 outbound_call.cc:171] Failed to schedule invoking callback on response for 
request yb.master.MasterService.GetMasterRegistration to <MASTER_IP>: Aborted (yb/rpc/thread_pool.cc:249): Service is shutting down
@bmatican bmatican added the area/docdb YugabyteDB core features label Apr 1, 2020
@bmatican bmatican self-assigned this Apr 1, 2020
@kmuthukk kmuthukk added the priority/high High Priority label Jun 12, 2020
@kmuthukk
Copy link
Collaborator

kmuthukk commented Jun 12, 2020 •
edited
Loading

This harmless warning is confusing a lot of people. We should suppress it.

Example:

  1. https://yugabyte-db.slack.com/archives/CG0KQF0GG/p1591955638007400
Hello team,
While running ybadmin command,  

./bin/yb-admin -master_addresses MASTER_ADDRESSES list_all_masters 

get below error along with output

W0612 09:47:02.437229 66167 outbound_call.cc:171] \
Failed to schedule invoking callback on response for request yb.master.MasterService.GetMasterRegistration to MASTER-1-IP: \
Aborted (yb/rpc/thread_pool.cc:251): Service is shutting down
Can you please let me know as to whats causing this ?
  1. Another report:
I have 3 servers. The exact same line, does only work on 1 of them. 
On the other nodes i get "W0401 19:36:11.606232  5030 outbound_call.cc:171] \
Failed to schedule invoking callback on response for request yb.master.MasterService.GetMasterRegistration to IP_HERE: \
Aborted (yb/rpc/thread_pool.cc:249): Service is shutting down"

User42  2 months ago
Command: ./bin/yb-admin --master-addresses <ip:port,...> get_universe_config

@bmatican bmatican assigned iSignal and unassigned bmatican Jun 18, 2020
iSignal added a commit that referenced this issue Jun 19, 2020
@iSignal
[#4131] Suppress glogs below ERROR in yb-admin output
b689b91 
Summary: These glogs confuse users who expect to simply see the output if operations succeeded. See original issue.

Test Plan:
Run and verify glogs are not seen. Run with specified minloglevel and verify glogs are seen

New output:
```
build/latest/bin/yb-admin list_all_tablet_servers
Tablet Server UUID               RPC Host/Port Heartbeat delay Status   Reads/s  Writes/s Uptime   SST total size  SST uncomp size SST #files      Memory
f715e4b31b3e46a798124f1ffb64f71e 127.0.0.1:9100 0.97s           ALIVE    0.00     0.00     60       0 B             0 B             0               25.20 MB
```

With minloglevel specified:
```
build/latest/bin/yb-admin --minloglevel=0 list_all_tablet_servers
I0618 23:28:19.889179  9365 mem_tracker.cc:249] MemTracker: hard memory limit is 13.185853 GB
I0618 23:28:19.889333  9365 mem_tracker.cc:251] MemTracker: soft memory limit is 11.207974 GB
I0618 23:28:19.891425  9365 thread_pool.cc:166] Starting thread pool { name: yb-admin queue_limit: 10000 max_workers: 1024 }
I0618 23:28:19.893055  9365 client-internal.cc:1963] New master addresses: [localhost:7100]
W0618 23:28:19.897009  9367 dns_resolver.cc:149] Peer address 'localhost' resolves to [127.0.0.1, ::1] different addresses. Using 127.0.0.1
W0618 23:28:19.900408  9369 dns_resolver.cc:149] Peer address 'localhost' resolves to [127.0.0.1, ::1] different addresses. Using 127.0.0.1
Tablet Server UUID               RPC Host/Port Heartbeat delay Status   Reads/s  Writes/s Uptime   SST total size  SST uncomp size SST #files      Memory
f715e4b31b3e46a798124f1ffb64f71e 127.0.0.1:9100 0.61s           ALIVE    0.00     0.00     70       0 B             0 B             0               25.20 MB
```

Reviewers: hector, bogdan

Reviewed By: bogdan

Subscribers: kannan, ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D8706
deeps1991 pushed a commit to deeps1991/yugabyte-db that referenced this issue Jul 22, 2020
@iSignal @deeps1991
[yugabyte#4131] Suppress glogs below ERROR in yb-admin output
4b2cd46 
Summary: These glogs confuse users who expect to simply see the output if operations succeeded. See original issue.

Test Plan:
Run and verify glogs are not seen. Run with specified minloglevel and verify glogs are seen

New output:
```
build/latest/bin/yb-admin list_all_tablet_servers
Tablet Server UUID               RPC Host/Port Heartbeat delay Status   Reads/s  Writes/s Uptime   SST total size  SST uncomp size SST #files      Memory
f715e4b31b3e46a798124f1ffb64f71e 127.0.0.1:9100 0.97s           ALIVE    0.00     0.00     60       0 B             0 B             0               25.20 MB
```

With minloglevel specified:
```
build/latest/bin/yb-admin --minloglevel=0 list_all_tablet_servers
I0618 23:28:19.889179  9365 mem_tracker.cc:249] MemTracker: hard memory limit is 13.185853 GB
I0618 23:28:19.889333  9365 mem_tracker.cc:251] MemTracker: soft memory limit is 11.207974 GB
I0618 23:28:19.891425  9365 thread_pool.cc:166] Starting thread pool { name: yb-admin queue_limit: 10000 max_workers: 1024 }
I0618 23:28:19.893055  9365 client-internal.cc:1963] New master addresses: [localhost:7100]
W0618 23:28:19.897009  9367 dns_resolver.cc:149] Peer address 'localhost' resolves to [127.0.0.1, ::1] different addresses. Using 127.0.0.1
W0618 23:28:19.900408  9369 dns_resolver.cc:149] Peer address 'localhost' resolves to [127.0.0.1, ::1] different addresses. Using 127.0.0.1
Tablet Server UUID               RPC Host/Port Heartbeat delay Status   Reads/s  Writes/s Uptime   SST total size  SST uncomp size SST #files      Memory
f715e4b31b3e46a798124f1ffb64f71e 127.0.0.1:9100 0.61s           ALIVE    0.00     0.00     70       0 B             0 B             0               25.20 MB
```

Reviewers: hector, bogdan

Reviewed By: bogdan

Subscribers: kannan, ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D8706
@ryan-ally ryan-ally mentioned this issue Nov 27, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iSignal iSignal

Labels
area/docdb YugabyteDB core features priority/high High Priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bmatican @iSignal @kmuthukk