How much concurrency can my system handle?

[jspooner]

I'd like to know the upper bound of concurrency for my system and I'm guessing this would be a popular request.

I've pushed the limits of my machine to see what is possible. I've seen the following errors and have experienced scripts that never exit.

1. Protocol::HTTP2::StreamError: Stream closed!
2. EOFError: Expected 15739 bytes, 15739 bytes short!

Async do |task|
  internet = Async::HTTP::Internet.instance
  APPS.each do |aid|
    task.async do |sub_task|
      aid_hash.each do |_k, v|
        sub_task.async do |sub_sub_task|
          response = internet.get(v, { 'accept-encoding' => 'gzip' })
          body = response.read
          # snip
          sub_sub_task.async do
            FileUtils.mkdir_p(short_path)
            File.open("#{short_path}/#{parts[3]}.html", 'w') { |f| f.write body }
          end
        end
      end
    end
  end
ensure
  internet&.close
end

This RubyConf video on Async Ruby mentioned a maximum number of ports but what are some other things to monitor?

[ioquatix]

My general advice is about 5,000 to 10,000 fibers per process before GC becomes a problem (hopefully we can fix this).