Snap plugin 'connection is shut down' error message

[nanliu]

There's several plugin bugs reporting the error connection is shut down which seems related to the go rpc client. I'm listing them below so we can centrally track them all:

intelsdi-x/snap-plugin-collector-snmp#16
intelsdi-x/snap-plugin-collector-perfevents#24
intelsdi-x/snap-plugin-publisher-opentsdb#28

[nanliu]

Per update in opentsdb, we were able to reproduce this error by killing the publisher plugin several times. During normal Snap operation, when the plugin health check fails consecutively, it triggers a plugin restart. After the plugin restarts several times, it hits MaxPluginRestartCount and Snap no longer attempts to start the plugin. But if the task is configured with max-failure: 1, the task will continue to collect but does not publish any new data.

The current proposal to resolve this issue:

1. allow MaxPluginRestartCount be unlimited with -1.
2. improve logging so we notify users that max plugin restart have been reached.

Maybe we may want to reset the restart counter after successful health checks, so it behaves more like MaxPluginConsecutiveRestartCount.

For now, if you encounter this issue recompile and change the MaxPluginRestartCount to a large number in https://github.com/intelsdi-x/snap/blame/master/control/runner.go#L59 simply update the setting: MaxPluginRestarts per change completed in #1422.

cc: @jtlisi