Downloading dependencies: "checking for connectivity" hangs for certain URLs.

[adrienbernede]

Context:

I’m on MacOS Catalina, using virtualenv configured to use the homebrew installed python3.8.5.
I install maestrowf in this environment with pip.

Then I’m relying on lulesh example to create my own study:
https://github.com/LLNL/maestrowf/blob/develop/samples/lulesh/lulesh_sample1_macosx.yaml

Problem:

I’m defining a git dependency:

      dependencies:
          git:
              - name: MFEM
                path: $(OUTPUT_PATH)
                url: https://github.com/mfem/mfem.git
                tag: v4.1

but it hangs at "checking for connectivity":

 [2020-10-13 14:02:42: INFO] Acquiring -- MFEM
[2020-10-13 14:02:42: INFO] Checking for connectivity to 'https://github.com/mfem/mfem.git'

With the following stack trace when aborting:

Traceback (most recent call last):
  File "/Users/bernede1/Environments/Python/maestrowf/bin/maestro", line 8, in <module>
    sys.exit(main())
  File "/Users/bernede1/Environments/Python/maestrowf/lib/python3.8/site-packages/maestrowf/maestro.py", line 430, in main
    rc = args.func(args)
  File "/Users/bernede1/Environments/Python/maestrowf/lib/python3.8/site-packages/maestrowf/maestro.py", line 240, in run_study
    study.setup_environment()
  File "/Users/bernede1/Environments/Python/maestrowf/lib/python3.8/site-packages/maestrowf/datastructures/core/study.py", line 392, in setup_environment
    self.environment.acquire_environment()
  File "/Users/bernede1/Environments/Python/maestrowf/lib/python3.8/site-packages/maestrowf/datastructures/core/studyenvironment.py", line 194, in acquire_environment
    value.acquire(substitutions=self.substitutions.values())
  File "/Users/bernede1/Environments/Python/maestrowf/lib/python3.8/site-packages/maestrowf/datastructures/environment/gitdependency.py", line 173, in acquire
    retcode = clone.wait()
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1079, in wait
    return self._wait(timeout=timeout)
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1804, in _wait
    (pid, sts) = self._try_wait(0)
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1762, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt

More:

It seems to depend on the repo:

OK:
https://github.com/LLNL/axom.git
https://github.com/LLNL/Umpire.git
https://github.com/LLNL/lulesh.git
https://github.com/prunkdump/arduino-variometer.git
https://github.com/mfem/web.git
https://github.com/geosx/geosx.git

KO:
https://github.com/spack/spack.git
https://github.com/mfem/mfem.git

[adrienbernede]

https://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait

maestrowf/maestrowf/datastructures/environment/gitdependency.py

Line 161 in 4cadf37

retcode = p.wait()

The use of wait() for a supprocess with PIPE on stdout and/or stderr can lead to a deadlock if the output is too large.