Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Do not merge] Changes to Rayon in fork for rustc #569

Closed
wants to merge 9 commits into from

Conversation

Zoxc
Copy link

@Zoxc Zoxc commented Apr 25, 2018

This just shows to changes intended for use in rustc via a rustc_rayon and rustc_rayon_core fork of Rayon.

@Zoxc Zoxc force-pushed the rustc branch 2 times, most recently from 1e863f4 to c3533d3 Compare April 27, 2018 13:31
@cuviper
Copy link
Member

cuviper commented May 3, 2018

Can you give a high level summary of what the changes are, and why they are needed?

@Zoxc
Copy link
Author

Zoxc commented May 4, 2018

I've made a commit for each feature added.

  • The thread local value is just a thread-local which gets read when jobs are created and set when they are executed. This means that jobs start out with the same thread-local value as their parent. It is added to allow Rayon jobs to access rustc's ImplicitCtxt which is a thread-local value which is updated during the lifetime of the thread-pool
  • main_handler is added to allow for an ergonomic way to set scoped thread-local state for worker threads, since APIs dealing with scoped things use closures and begin_handler and end_handler doesn't let you do this
  • The scoped thread pool is used so that we can point thread-locals in the work threads to the stack on the main thread. This allows us to have temporary "globals" for the entire thread pool.
  • ThreadLocal is added as an ergonomic and efficient way to create thread-locals with a lifetime shorter than the thread-pool

}

/// Sets the current thread-local value
pub fn set(value: usize) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok so this is some sort of "token" where:

  • it starts as 0
  • you can set it and get it
  • when you spawn a job, it inherits the value at the time that job was spawned?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

/// The type for a closure that gets invoked with a
/// function which runs rayon tasks.
/// The closure is passed the index of the thread on which it is invoked.
/// Note that this same closure may be invoked multiple times in parallel.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this distinct from the StartHandler?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see, it kind of "takes over" the thread

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It allows up to run code both before and after the Rayon worker, so it is more powerful than StartHandler and ExitHandler. You can see it used here.


impl<T> ThreadLocal<Vec<T>> {
/// Joins the elements of all the thread locals into one Vec
pub fn join(self) -> Vec<T> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from this method, I believe we could just use a "regular" thread-local value, no?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if @cuviper's "broadcast" primitive could be used to simulate this from a regular thread-local. I don't remember if it lets you return values — but it probably could. =)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If by "regular" thread-local value, you mean something like the thread_local crate, then yes. However that is less efficient since it doesn't exploit the fact that our thread pool has a fixed number of threads.

@nikomatsakis
Copy link
Member

My take on these changes:

  • I wouldn't want to land them in rayon proper yet (I know that's not the current plan). Mostly they seem quite reasonable but I'd want to spend some time thinking about whether this is the right approach and if we can polish the APIs.
    • In particular, I think that having some form of "task-local, inherited" state (and "worker-local") in Rayon makes a lot of sense.
    • (Reminder to myself that I need to carve out some dedicated Rayon time and in particular I think we need an RFC process, but I've been saying that for a while I know.)
  • However, the changes all look technically fine. I have to go review the rustc side to see how they all fit in. I would like to figure out how the scoped thread pools in particular fit in.

@nikomatsakis
Copy link
Member

I think we can close this – we are currently issuing rayon-rustc, which we should eventually try to sync, but this PR is probably done. Right @Zoxc? (Closing for now)

@cuviper cuviper mentioned this pull request Jul 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants