Hot configuration reload

[HuKeping]

Hi there,

Recently, I'm working on the hot configuration reload on some of our projects. I wonder if Notary need this feature as well?

One scenario is: when in production, the log should not be set as debug, but when something bad happens, the Ops may want some more detail via log on the debug level. Since they couldn't stop the running process, they wish to have the feature of hot reloading.

[endophage]

I actually had this in a very early version and tore it out because we didn't need it and it added a lot of complexity. I think it would be great to add and if you'd like to take that on it would be appreciated.

[HuKeping]

I completely agree with you that it may cause a lot of unnecessary complexity especially when we want to hot reload everything.

I'll send you a draft some day later.

[HuKeping]

To implement this, maybe we could:

1. find a way to monitor the change of the config file (Update vendor spf13 #717)
2. when the change happen, try to parse the new config file
3. find out whether the changed configuration items are those we supporte to reload
4. replace the old with the new one

[endophage]

At a simpler level, it would be acceptable to send a signal to the running application to tell it to reload the configuration. I'd probably approach this in 3 steps, with the final one being optional:

1. Graceful shutdown. When sent a signal, the server will stop accepting connections and finish serving any active requests before shutting down. It can then be restarted manually with the new configuration.
2. Triggered hot restart. A signal is used to tell the application to reload it's configuration. The way this is generally implemented is the current instance of http.Server stop listening for connections but is allowed to finish servicing any requests it has already accepted. As soon as the previous http.Server stops listening, a new http.Server, with the new configuration, is started to accept connection.
3. Automatic hot restart. Basically the same as 2, but the trigger is some automatic detection of a changed configuration.

In 2 and 3 it's important to cleanly handle a bad configuration. The previous server should be left running until we've done as much work as possible to ensure a new server can be started to take its place.

[HuKeping]

by (3), maybe the latest viper can do that. There is a handler viper.OnConfigChange() to watch the events when configuration changed, it was supported by "github.com/fsnotify/fsnotify".

Since we use viper, we needn't to reinvent the wheel if certain conditions are met, I'll take a deeper look into it.

[endophage]

Yeah, I saw your PR updating viper. Kind of disappointed how many files that adds to support one simple use case but I agree, we shouldn't waste time building it ourselves.

[HuKeping]

yeah, too many additional files

[HuKeping]

Have talked to some colleague again, I think maybe we are a little bit over-designed about this feature.

Hot reload the debug level is more important than the other configuration items for them , because the log level will be set to default as info which may not recording enough information when something bad happens.

For other configuration, like the TLS, it's quite enough to shut down the relative node and reboot another one as long as it can be done gracefully.

So I think maybe the graceful shutdown signer and server are the one which should be in our roadmap?

Thoughts? @endophage @cyli @riyazdf

[endophage]

That definitely makes sense. I'd suggest using SIGUSR1 and SIGUSR2 to increment and decrement the logging level. That will allow it to be turned up or down as much as desired by sending the appropriate signal multiple times.

[cyli]

@HuKeping Would it be alright to close this? Your awesome logging fix has been merged, and as per the conversation above, restarting so long as we have graceful shutdown (#712 and #711) would be sufficient to reload other configuration.

[HuKeping]

sure, closing