Optimize block processing - Part 3

[heifner]

Part 3 of 3 of #529. See #688 for part 2 and #619 for part 1.

• Add optional handler id to app().executor().post().
  • If handler id is provided and a existing function with the same handler id has already been posted with the same priority the function is not posted.
  • This allows for deduplication of redundant posts.
  • Specifically if an existing on_incoming_block() has been posted to the main thread, then another does not need to be added as the on_incoming_block will execute any blocks ready from the fork database.

Resolves #529

[ericpassmore]

Note:start
category: Other
component: Internal
summary: Optimize block processing - Part 3: deduplication of redundant posts.
Note:end

[greg7mdp]

This seems wrong to me. Shouldn't we want to skip handlers which have the same priority, but a different id?

[heifner]

The idea is the find the last posted function. If it same as you are attempting to post, don't bother.

[greg7mdp]

Could it be as simple as this?

Suggested change

	if (i != end) {
	// ordered iterator appears to only be a forward iterator
	auto p = i;
	for (; i != end; p = i, ++i) {
	if ((*i)->priority() != priority)
	break;
	}
	if ((*p)->priority() == priority && (*p)->id() == id)
	return;
	}
	if (i != end) {
	// ordered iterator appears to only be a forward iterator
	for (; i != end && (*i)->priority() == priority; ++i)
	if ((*i)->id() == id)
	return;
	}

[heifner]

No. You have to find the last posted handler with the same priority and see if that one has the same id.

[greg7mdp]

Why does it has to be the last posted one? Why does it matter if something else was posted later with the same priority?

[heifner]

This goes with the more conservative approach of only de-dup on matching id for a given priority when it was the last id posted.

Say you have items in the queue, u=unique, pb=process_incoming_block
q9(u) q8(pb) q7(u)

Say you want to post a q6(pb), it is possible this is not safe to de-dup because if q7(u) does something that needs q6(pb) to run then you end up with q6(pb) not running. So this goes with the more conservative route of only de-dup if q7(u) is a pb. Granted we could argue that q7(u) should post a pb if needs that to happen.

[greg7mdp]

The code I sent dedups only if it is the same id (pb) at the same priority, so in your example posting q6(pb) would not dedup as there is no other q6(pb) in the queue. Or am I missing something?

[heifner]

The 9,8,7 are just order numbers. They are all the same priority.
q9(u) q8(pb) q7(u)
So your code would find q8(pb) and return (de-dup). Want to only compare to q7(u) and then non-dedup.

[greg7mdp]

Is there a reason for this requirement, i.e would the code not work if we didn't check that it was the last posted? It seems to me that either we want to dedup, and then we should just do it, or not.

[heifner]

Been thinking about this. Maybe I am being too conservative. If the last item running needs to have a follow non-unique item it should post it.

[greg7mdp]

What was the rationale for not using unique_ptr anymore in the prio_queue?

[heifner]

What was the rationale for not using unique_ptr anymore in the prio_queue?

boost::heap doesn't support them. It uses copy in some places.

[greg7mdp]

boost::heap doesn't support them. It uses copy in some places.

annoying. Maybe it would be better to use shared_ptr, which would make it easier to be confident that there are no memory leaks (although there aren't now).