Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

std.Thread.Pool: process tree cooperation via a new jobserver protocol #20372

Closed
wants to merge 6 commits into from

Commits on Jun 30, 2024

  1. std.Thread.Pool: implement jobserverv2 protocol

    The host accepts N simultaneous connections and writes 1 byte to them
    each. Clients connect and read 1 byte in order to obtain a thread token.
    
    std.Thread.Pool now lazily spawns threads only when the work queue is
    non-empty. I think that was a bad idea and will revert it shortly.
    
    There is now a std.zig.initThreadPool wrapper that deals with:
    * Resolving a zig cache directory into a UNIX domain socket address.
    * Creating the "tmp" directory in .zig-cache but only if the listen
      failed due to ENOENT.
    * Deciding to connect to an existing jobserver, or become the host for
      child processes.
    andrewrk committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    c395df2 View commit details
    Browse the repository at this point in the history
  2. new thread pool jobserver integration

    std.Thread.Pool: back to spawning all threads in initialization because
    it's overall simpler. This scheme requires init to be passed a pointer
    to the struct.
    
    std.process.Child: implement integration with thread pool jobserver. The
    environment variable is called `JOBSERVERV2`. The API works based on
    assigning a thread pool to the child process.
    
    build runner: store the thread pool in std.Build.Graph so that it can be
    passed to child processes during the make phase.
    
    Fix not allocating +1 pollfds in previous commit.
    andrewrk committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    e51fd67 View commit details
    Browse the repository at this point in the history
  3. std.Thread.Pool: fix connection

    TCP cannot be used with UNIX domain sockets
    andrewrk committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    6d3a78b View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    dc207da View commit details
    Browse the repository at this point in the history
  5. make the main thread call waitAndWork instead of just wait

    The main thread has an implicit thread token which makes loitering
    illegal.
    andrewrk committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    86e68db View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. std.Thread.Pool: call shutdown on workers blocking on connect or read

    and increase kernel backlog to the maximum number. Without increasing
    the kernel backlog to the maximum number, I observed connect() to block
    indefinitely, even when another thread calls shutdown() on that socket
    file descriptor.
    andrewrk committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    a239e65 View commit details
    Browse the repository at this point in the history