Timeout when trying to bootstrap or ssh spot request instances

[jeremywadsack]

I'm running into an issue where I can launch an instance and configure it without a problem. If I try to do the same with a spot request instance it hangs and eventually times out.

      def service
        @service ||= Fog::Compute.new(
            provider:              "AWS",
            aws_access_key_id:     Settings.amazon.access_key_id,
            aws_secret_access_key: Settings.amazon.secret_access_key,
            region:                Settings.parser.region
        )
      end

      def launch(name)
        options = {
            :name                      => name,
            :tags                      => {"Name": name},
            :flavor_id                 => Settings.parser.instance_type,
            :image_id                  => Settings.parser.image,
            :private_key_path          => File.join(Rails.root, "secrets", "parser"),
            :public_key_path           => File.join(Rails.root, "secrets", "parser.pub"),
            :username                  => "ec2-user",
            :groups                    => ["quick-start-1"],
            :availability_zone         => Settings.parser.zone,
            :iam_instance_profile_name => "parser-role",
            :block_device_mapping      => [
                {
                    "DeviceName"              => "/dev/xvda",
                    "Ebs.VolumeSize"          => 40,
                    "Ebs.DeleteOnTermination" => true,
                    "Ebs.VolumeType"          => "gp2"
                }
            ]
        }

        server = service.spot_requests.bootstrap(options.merge(
            :price => "0.03",
            :type  => "one-time",
        ))
      end

Calling launch(...) raises Fog::Errors::TimeoutError from the bootstrap call.

~/.rvm/gems/ruby-2.3.1@log-processing/gems/fog-core-1.44.3/lib/fog/core/wait_for.rb:9:in `block in wait_for': The specified wait_for timeout (600 seconds) was exceeded (Fog::Errors::TimeoutError)
	from ~/.rvm/gems/ruby-2.3.1@log-processing/gems/fog-core-1.44.3/lib/fog/core/wait_for.rb:6:in `loop'
	from ~/.rvm/gems/ruby-2.3.1@log-processing/gems/fog-core-1.44.3/lib/fog/core/wait_for.rb:6:in `wait_for'
	from ~/.rvm/gems/ruby-2.3.1@log-processing/gems/fog-core-1.44.3/lib/fog/core/model.rb:74:in `wait_for'
	from ~/.rvm/gems/ruby-2.3.1@log-processing/gems/fog-aws-1.4.0/lib/fog/aws/models/compute/server.rb:219:in `setup'
	from ~/.rvm/gems/ruby-2.3.1@log-processing/gems/fog-aws-1.4.0/lib/fog/aws/models/compute/spot_requests.rb:71:in `bootstrap'

The problem appears to be that it's waiting to be sshable? which never returns true so it times out.

I dug into that method and logged the errors and it appears to timeout on the connection:

54.215.70.245 #<Errno::ECONNREFUSED: Connection refused - connect(2) for 54.215.70.245:22>
54.215.70.245 #<Errno::ECONNREFUSED: Connection refused - connect(2) for 54.215.70.245:22>
54.215.70.245 #<Errno::ECONNREFUSED: Connection refused - connect(2) for 54.215.70.245:22>
54.215.70.245 #<Errno::ECONNREFUSED: Connection refused - connect(2) for 54.215.70.245:22>
54.215.70.245 #<Timeout::Error: execution expired>
54.215.70.245 #<Timeout::Error: execution expired>
54.215.70.245 #<Timeout::Error: execution expired>
54.215.70.245 #<Timeout::Error: execution expired>
54.215.70.245 #<Timeout::Error: execution expired>
54.215.70.245 #<Timeout::Error: execution expired>
54.215.70.245 #<Timeout::Error: execution expired>
54.215.70.245 #<Timeout::Error: execution expired>
54.215.70.245 #<Timeout::Error: execution expired>
54.215.70.245 #<Timeout::Error: execution expired>
54.215.70.245 #<Timeout::Error: execution expired>

While it's retrying AWS shows the instance as ready and I'm able to ssh into the instance using the credentials.

If I change this to servers.bootstrap then is works just fine, even though the sshable? call is the same for both.

I'm running fog-aws 1.4.0.

I'm new to spot instances so it's possible that I've configured something wrong, but I'm not sure what I need to change. Looking for guidance or anything I can do to help resolve this or help provide a fix.

[geemus]

It may just be a timing issue in terms of how/when things come up on AWS itself. I'm afraid I don't personally have much experience with spot instances though. You might consider decomposing the bootstrap method and adding a larger delay before it tries to check sshable? and/or a longer timeout, that might be enough to get you what you want. I'm certainly happy to discuss more if you have questions. Thanks!

[jeremywadsack]

@geemus Thanks for the suggestion. I had already decomposed #bootstrap for other reasons but dug into #sshable? I found that if I change the Timeout parameter in that method in fog-core to 11, then it works fine. If it's 10 or less it hangs until the 600s enclosing Timeout fails.

I can open a PR with that change, but I suspect the few extra second are likely going to be different for different cloud providers, instance types, zones, and other factors. Would it make sense to use an exponential decay for that call?

[geemus]

@jeremywadsack thanks for taking the time to dig-in. I imagine I set it at 10 because it was easy and seemed good-enough at the time. I would tend to agree that it is brittle though (especially across providers). Exponential backoff sounds good, as long as we can keep the code for it from getting to complex. Would you be up for taking a pass at that? Thanks!

[jeremywadsack]

Happy to take a stab at a Pull Request. You had it set at 8 by the way, which makes me think some kind of analysis. I found 11 to be the limit it my testing but suspect it would cause the same problem later without exponential back off.

On Wed, Jul 12, 2017 at 7:09 AM Wesley Beary ***@***.***> wrote: @jeremywadsack <https://github.com/jeremywadsack> thanks for taking the time to dig-in. I imagine I set it at 10 because it was easy and seemed good-enough at the time. I would tend to agree that it is brittle though (especially across providers). Exponential backoff sounds good, as long as we can keep the code for it from getting to complex. Would you be up for taking a pass at that? Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#372 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOurCOTUYEQhlSGIeRn8lyMaCNwAHV1ks5sNNN7gaJpZM4OI08i> .

-- Jeremy Wadsack

[geemus]

Sure, sounds good.

[jeremywadsack]

@geemus Opened a PR with a solution for fixing that. Let me know if this makes sense, any concerns, etc.

[geemus]

@jeremywadsack thanks, it's looking pretty good (sorry for the delay, I was on vacation last week). I'm going to go ahead and close this before I forget/lose it. I'm pretty confident we'll get your PR (or something quite like it) in soon. Thanks!