[service client] no indication if server crashes during or before processing of service call

[slessans]

Bug report

Required Info:

• Operating System:
• Mac OS X, Ubuntu 18.04
• Installation type:
• binaries
• Version or commit hash:
  • dashing
• DDS implementation:
  • Fast-RTPS
• Client library (if applicable):
  • rclpy

Steps to reproduce issue

Run this example server (which crashed when you send a request)

import rclpy
from example_interfaces.srv import AddTwoInts


def main(args=None):
    rclpy.init(args=args)

    node = rclpy.create_node('minimal_service')

    def add_two_ints_callback(request, response):
        response.sum = request.a + request.b
        node.get_logger().info('Incoming request\na: %d b: %d' % (request.a, request.b))
        exit(1)  # CRASH HERE
        return response

    srv = node.create_service(AddTwoInts, 'mock/add_two_ints', add_two_ints_callback)

    try:
        rclpy.spin(node)

    finally:
        node.destroy_service(srv)
        rclpy.shutdown()


if __name__ == '__main__':
    main()

Call this with rclpy client (or ros2 service call /mock/add_two_ints example_interfaces/srv/AddTwoInts '{a: 3, b: 34}').

Expected behavior

When the node crashes/dies, the client should get some sort of notification that the service failed.

Actual behavior

The client will wait for the response forever. I have not found any way to deal with this. This also happens if you call the service when it is not ready. This could be an issue because there is a very real race condition between checking client.service_is_ready() and actually making the call.

Additional information

Not sure if this is a bug, or just a feature that hasn't been implemented. This seems like a very likely failure to occur in practice and should be deal with.

If this is not a bug and just a missing feature, I am happy to work on it and send a PR if someone points me to where I should i get started with it.

[jacobperron]

I think this is a use-case for the deadline and/or liveliness quality of service settings (described in this pending PR). If your application requires a timely response from the service (deadline) and/or needs to ensure that the service is always available (liveliness) then you can set the appropriate QoS settings. Currently, deadline and liveliness QoS settings aren't supported for services or actions, but I expect there might be a follow-up to ros2/ros2#680 at some point.

In the meantime you might be able to use the client's service_is_ready() method as a workaround.
For example, after sending the request you can periodically check if service_is_ready() returns True and if not you might assume that the server has exited. I'm not sure if we really need to implement this in the CLI tool, but if you think it would be useful I'm happy to review a PR.

[slessans]

@jacobperron thanks for the response. Sorry for the confusion -- I agree it doesn't seem necessary in the CLI. I was trying to understand if the ability was present in the underlying library at all. Coming from higher level network programming, I am used to making requests where the callback handler is invoked with an error if the connection breaks or the peer dies.

My understanding is that liveliness is not implemented in fastrtps and deadlines are not implemented for service calls. So, am I correct in assuming the only way to detect this scenario is for the client to send a request and then wait a certain period of time and assume the server failed or dropped it, if it gets no response in this time period?

[jacobperron]

So, am I correct in assuming the only way to detect this scenario is for the client to send a request and then wait a certain period of time and assume the server failed or dropped it, if it gets no response in this time period?

I believe you are correct. Besides the timeout, you should also be able to use the client method, service_is_ready(), while you wait to detect if the server goes down.