shrinkable flux instance

[garlick]

Problem: Flux instances can mark nodes offline but cannot permanently remove (and free) them.

This issue is for the "shrink" operation, the easier half of "grow and shrink" as described in the original flux design document.

The initial solution could be constrained to

• operate at execution target (broker rank) granularity
• only support the removal of non-critical nodes

Here's a quick pass at a work breakdown

• provide a mechanism to request removal of set of execution targets from the flux instance
• post a (TBD) resource event to modify the resource set used in core
• make sure core tools reflect the current resource set
• send a (TBD) acquire shrink response to the scheduler
• modify schedulers to handle this response
• implement partial release of discarded nodes
• auto shrink on offline? flux assumes down nodes can come back, but this is only true for bootstrap from configuration #6641
• mechanism to shed nodes as they become idle (maybe shrink option to flux queue drain|idle
• rename the current fixed size attribute to tbon.universe-size or similar to nix the implication that it's a node count
• provide a new execution-targets or similar attribute that's an idset of valid ranks

See also:

• flux assumes down nodes can come back, but this is only true for bootstrap from configuration #6641

Edit: I removed the reference to "malleable jobs" and the TODO on a general way to shrink jobs as those items may get us off track of fulfilling this issue, which seems a bit narrower to me.