Snapshotting pseudo state & log entry cache

[benbjohnson]

Overview

This pull request contains multiple changes/fixes for the raft package:

• Manage requests better while snapshotting. Currently a large snapshot will lock the log and queue up heartbeats & possibly cause an election timeout when it's done. There is now a Log.snapshotting field that is set using the sync/atomic package.
• Add a log entry cache so that temporarily disconnected streams can reconnect without requiring a snapshot.
• Wrap the Log.mu mutex with lock() and unlock() methods for lock contention debugging. There's also a printCaller() method that will log the file & line of the calling function to a lock.

[corylanou]

Did you still want this for debugging?

[corylanou]

nm, I see a bunch of them below, assuming this is needed for debugging in the wild.

[otoolep]

Interesting change, generally makes sense. I do have questions about how the system will respond if stuff that used to happen anyway, is now explicitly blocked by an active snapshot.

[otoolep]

+1, change generally makes sense.

[toddboom]

👍 from me too