Impossible install from sources with pip and localized Visual Studio

[StrikerRUS]

It's impossible to install LightGBM with pip on Windows machine with localized Visual Studio.

Log:

C:\>pip install --no-binary :all: lightgbm
Collecting lightgbm
  Downloading lightgbm-2.0.6.tar.gz (216kB)
    100% |████████████████████████████████| 225kB 1.3MB/s
Requirement already satisfied: numpy in c:\program files\anaconda3\lib\site-packages (from lightgbm)
Requirement already satisfied: scipy in c:\program files\anaconda3\lib\site-packages (from lightgbm)
Requirement already satisfied: scikit-learn in c:\program files\anaconda3\lib\site-packages (from lightgbm)
Skipping bdist_wheel for lightgbm, due to binaries being disabled for it.
Installing collected packages: lightgbm
  Running setup.py install for lightgbm ... error
Exception:
Traceback (most recent call last):
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 27: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 27: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\commands\install.py", line 385, in run
    requirement_set.cleanup_files()
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\req\req_set.py", line 729, in cleanup_files
    req.remove_temporary_source()
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\req\req_install.py", line 977, in remove_temporary_source
    rmtree(self.source_dir)
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\_vendor\six.py", line 686, in reraise
    raise value
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\utils\__init__.py", line 102, in rmtree
    onerror=rmtree_errorhandler)
  File "C:\Program Files\Anaconda3\lib\shutil.py", line 488, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Program Files\Anaconda3\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Program Files\Anaconda3\lib\shutil.py", line 387, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\utils\__init__.py", line 114, in rmtree_errorhandler
    func(path)
PermissionError: [WinError 32] Процесс не может получить доступ к файлу, так как этот файл занят другим процессом: 'C:\\Users\\nekit\\AppData\\Local\\Temp\\pip-build-9bkszlwh\\lightgbm\\build_cpp'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\commands\install.py", line 385, in run
    requirement_set.cleanup_files()
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\utils\build.py", line 38, in __exit__
    self.cleanup()
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\utils\build.py", line 42, in cleanup
    rmtree(self.name)
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\_vendor\six.py", line 686, in reraise
    raise value
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\utils\__init__.py", line 102, in rmtree
    onerror=rmtree_errorhandler)
  File "C:\Program Files\Anaconda3\lib\shutil.py", line 488, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Program Files\Anaconda3\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Program Files\Anaconda3\lib\shutil.py", line 387, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\Program Files\Anaconda3\lib\site-packages\pip\utils\__init__.py", line 114, in rmtree_errorhandler
    func(path)
PermissionError: [WinError 32] Процесс не может получить доступ к файлу, так как этот файл занят другим процессом: 'C:\\Users\\nekit\\AppData\\Local\\Temp\\pip-build-9bkszlwh\\lightgbm'

It's known pip bug. And it is caused by logs that are produced by Visual Studio with Russian (I suppose any non-English) localization (if MinGW was localized it would cause installation fail too).

I had the same problem in my project and successfully solved it by shutting up logs from compilers. To be more precise by changing os.system() calls to subprocess.check_output().

If you guys do not mind described solution I can create a PR.

[StrikerRUS]

After the close look at setup.py file another question has raised: why not to try at first build from existing in folder windows Visual Studio solution file? It'll make possible not to install CMake.

[guolinke]

@StrikerRUS Thanks.
using cmake is more flexible, and bringing the unify settings for osx/linux/windows.

[StrikerRUS]

@guolinke
So, can I start rewriting calls in setup.py?

I understand your position about CMake and I don't suggest to replace CMake+VS building by sln+VS. I suggest to add this method on Windows: if sln+VS fails go to CMake+VS. What do you think?

[guolinke]

@StrikerRUS yeah, you can rewriting.

you can try the "sln+VS" first.

[StrikerRUS]

@guolinke
In addition, to speak about pip, I noticed that version from PyPI is 2.0.6, but from GitHub is 2.0.5.

[guolinke]

@StrikerRUS
Yeah, it is caused by some complicate reasons.
The next GitHub version will be 2.0.7 .