seems to be hanging on dns lookups

[jots]

I have a file of 1000 domain names, trying to get the front page of each with the following code. problem is that after about 13 lines, it hangs. if I ctrl-C it, I see a traceback indicating it's stuck in resolv.rb getaddresses.

Is there a way to replace the resolvers with an async one? also, I don't know why it stops after 13, I would think it would print out the first 200 lines because I have a semaphore for 200 at a time. am I doing that wrong?

#!/usr/bin/env ruby
#require "rubygems"
require "async"
require "async/barrier"
require "async/semaphore"
require "async/http/internet"

time = Time.now

OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE

system("mkdir test")
Async do |task|
  semaphore = Async::Semaphore.new(200)
  barrier = Async::Barrier.new(parent: semaphore)
  internet = Async::HTTP::Internet.new  
  File.open("domains.txt").each do |domain|
    domain.chomp!
    puts domain
    barrier.async do
      begin
        url = "https://#{domain}"
        body = internet.get(url).read
        File.open("test/#{domain}", "w") do |f|
          f.puts body
        end
        puts "did #{domain}"
      rescue => ex
        STDERR.puts "error: #{ex} #{domain}"
      end
    end
  end
  puts "done"
  barrier.wait
  STDERR.puts "Duration (barrier.wait): #{Time.now - time}s"
end

error when ctrl-C:

^C/root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:679:in `select': Interrupt
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:679:in `request'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:531:in `block in fetch_resource'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:1125:in `block (3 levels) in resolv'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:1123:in `each'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:1123:in `block (2 levels) in resolv'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:1122:in `each'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:1122:in `block in resolv'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:1120:in `each'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:1120:in `resolv'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:521:in `fetch_resource'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:507:in `each_resource'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:402:in `each_address'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:116:in `block in each_address'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:115:in `each'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:115:in `each_address'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:102:in `getaddresses'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/resolv.rb:51:in `getaddresses'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/async-2.5.0/lib/async/scheduler.rb:153:in `address_resolve'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/socket.rb:231:in `getaddrinfo'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/socket.rb:231:in `foreach'
	from /root/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/async-io-1.34.3/lib/async/io/host_endpoint.rb:57:in `connect'
	[...]

[ioquatix]

Does it always hang on the same domain?

[jots]

yes, looks like localstrong.store is the domain it hangs on.

[ioquatix]

Is there any chance you can share the repo with me? I'm happy if it remains private, you can just add me as a collaborator so I can check what's going on.

[jots]

thanks, I added you as a collaborator. one thing: i can't duplicate the problem locally which is annoying.