Installer should not create win32com/gen_py folder if it can't be written to by non-elevated processes

[Talkyn]

I seem to be having a problem with my install, though I installed from the most recent (correct) binary.
I am running Windows 10 64bit, Python 3.6.4 32bit, and have installed PyWin32 221 32bit from the exe.

Running the testall.py script (as well as a simple outlook script I was trying) results in a file not found error for a dicts.dat file. Navigating to the gen_py folder, I find that it is entirely empty. I've included the full trace below.

Can anyone figure out what I've done wrong?

c:\Program Files (x86)\Python36-32\Lib\site-packages\win32com\test>testall.py
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python36-32\lib\site-packages\win32com\client\gencache.py", line 60, in __init__
    _LoadDicts()
  File "C:\Program Files (x86)\Python36-32\lib\site-packages\win32com\client\gencache.py", line 110, in _LoadDicts
    f = open(os.path.join(win32com.__gen_path__, "dicts.dat"), "rb")
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Program Files (x86)\\Python36-32\\lib\\site-packages\\win32com\\gen_py\\dicts.dat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files (x86)\Python36-32\lib\site-packages\win32com\client\gencache.py", line 143, in GetGeneratePath
    os.stat(fname)
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\\Program Files (x86)\\Python36-32\\lib\\site-packages\\win32com\\gen_py\\__init__.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files (x86)\Python36-32\Lib\site-packages\win32com\test\testall.py", line 3, in <module>
    import win32com.client
  File "C:\Program Files (x86)\Python36-32\lib\site-packages\win32com\client\__init__.py", line 11, in <module>
    from . import gencache
  File "C:\Program Files (x86)\Python36-32\lib\site-packages\win32com\client\gencache.py", line 666, in <module>
    __init__()
  File "C:\Program Files (x86)\Python36-32\lib\site-packages\win32com\client\gencache.py", line 62, in __init__
    Rebuild()
  File "C:\Program Files (x86)\Python36-32\lib\site-packages\win32com\client\gencache.py", line 653, in Rebuild
    _SaveDicts()
  File "C:\Program Files (x86)\Python36-32\lib\site-packages\win32com\client\gencache.py", line 69, in _SaveDicts
    f = open(os.path.join(GetGeneratePath(), "dicts.dat"), "wb")
  File "C:\Program Files (x86)\Python36-32\lib\site-packages\win32com\client\gencache.py", line 145, in GetGeneratePath
    f = open(fname,"w")
PermissionError: [Errno 13] Permission denied: 'C:\\Program Files (x86)\\Python36-32\\lib\\site-packages\\win32com\\gen_py\\__init__.py'

[zspitz]

It seems this folder -- lib\site-packages\win32com\gen_py -- (which I guess is created when installing Python Windows Extensions) needs admin write privileges. See also #1140 .

[mhammond]

It seems this folder -- lib\site-packages\win32com\gen_py -- (which I guess is created when installing Python Windows Extensions) needs admin write privileges. See also #1140 .

Alternatively, deleting that directory should also work - win32com should then try and use the temp dir. I'll rename this issue to reflect that lib\site-packages\win32com\gen_py shouldn't be created by the installer if it is read-only for non-elevated processes.

[RossBoylan]

I think giving your regular user account full control over gen_py should also work; I just tried it. I figured if I was going to delete the folder I might as well try something less drastic. Obviously not a general solution.

[xjcl]

I'm also affected by this and don't have admin privileges, so none of the fixes help me. What environment variable do I set to control where the gen_py folder is written?

I've also posted this a question to StackOverflow (https://stackoverflow.com/q/65234005/2111778)

[mhammond]

All the logic exists in https://github.com/mhammond/pywin32/blob/master/com/win32com/__init__.py - look for __gen_path__. There's also a registry entry that can be set, but sadly that's under HKLM, so if you can't get admin access even temporarily, you will be unable to create it (and if you can get it temporarily, you can just delete the directory)

[xjcl]

I figured out the solution which is to set %TEMP%. I also wrote so much on StackOverflow

[mhammond]

Thanks for getting back! You mean %TEMP% was set to a non-writable folder? Yeah, I guess I can imagine that would cause many problems and TBH I don't think it's something pywin32 should try and account for.

[sschukat]

Hi, we had the same problem in our company environment. During the installation of our product the gen_py folder was created by the installing admin and then a non admin user got an error using gen_py. I'll fixed this with a temporary write in my fork with commit e735c2d. Not beautiful but working.

[mhammond]

During the installation of our product the gen_py folder was created by the installing admin and then a non admin user got an error using gen_py

Thanks, but I understand that problem - just not clear how to solve it without something like your patch, which I'm not that keen on - I'd probably prefer something like having the installer not create it in the first place and have win32com.init check - that would at least only be a read in the general case. I'm not super keen on that though because it might still have the same issue in some cases (eg, if a first run outside the installer happened to be by an admin user, non-admin users would suffer the same fate). It might instead be better to deprecate having it anywhere other than %TEMP%.

The comments above made me think that @xjcl had a slightly different problem and I was trying to confirm that.

[sschukat]

It might instead be better to deprecate having it anywhere other than %TEMP%.
I'll agree to that, This would fix any access right problems since files would not be shared among users. I did not want to change the behavior in general, that is why I used the trigger file.
There might be some problems in the path handling since per default the username is in the temp path and hence all kinds of unicode signs could occur (e.g .مد , äüö, こ, ...). Hence the test should cover this. A quick test with a hard coded path showed no problem

>>> import win32com
>>> win32com.__gen_path__
'd:\\Temp\\.مد , äüö, こ,\\gen_py'
>>> from win32com.client import gencache
>>> app = gencache.EnsureDispatch("Word.Application", 0)

[kxrob]

I'd probably prefer something like having the installer not create it in the first place and have win32com.init check - that would at least only be a read in the general case.

Maybe the win32com.gen_py.__path__ can be set up for 2 (or 3) search locations.
win32com.gen_py.__init__.py deals with that anyway somehow.

1. A user location - somewhere in site.getusersitepackages() could be a natural place, which is less volatile than %TEMP%. Unless a custom location is set via registry key. On a python user installation this location may be equal to Right-click grep menu broken #2 - and be dropped.
2. The normal install / admin location.

Generation simply tries to write to the locations in win32com.gen_py.__path__ in reverse order.
Thus it would still be possible to have shared admin generation.
dicts.dat of all locations need to be searched.

[mhammond]

Maybe the win32com.gen_py.__path__ can be set up for 2 (or 3) search locations.

That sounds like a can of worms for little value, particularly when writing (eg, would we try and write to every location and take the first writable one? That non-determinism seems problematic. If not, then how would that admin one be setup? I just don't see the use-case that makes these complications worthwhile. That said though, I guess I'd review a PR which addressed those kinds of issues.

[kxrob]

That sounds like a can of worms for little value, particularly when writing (eg, would we try and write to every location and take the first writable one?

Yes, the generation simply would go (only) to the first location which is successfully writable - trying the admin / install location first win32com.gen_py.__path__[::-1].

That non-determinism seems problematic. If not, then how would that admin one be setup?

Its deterministic - the shared install / admin location write would be preferred on write (but can principally be overriden for loading from user location.)

Initial setup doesn't matter much: could be either delayed to the first generation; or there could be even a simple static win32com/gen_py/ folder installation ( with __init__.py setting the __path__ = [userlocation, .., thisinstalllocation] ).

The approach is similar to sys.path, with system + user site installations. %PATH% mechanism, win32comext joining win32com etc.

The extra complexity probably is: The for loop trying the generation location __path__ list and the for loop over dicts.dat . The import search is already auto by python.
Integration of further locations, if somehow needed (pre-generated app local/read-only/distributed/zip/frozen code, ..), is possible w/o further complexity.

I just don't see the use-case that makes these complications worthwhile. That said though, I guess I'd review a PR which addressed those kinds of issues.

So the alternative would be the single user writable location without sharing - must exist and be writable.

%TEMP% is subject to random cleanup - so the application code must always ensure automatic re-creation.
Possible alternative: something like %USERPROFILE%\<pyver>\gen_py or site.getusersitepackages() +r'\win32com\gen_py'

[mhammond]

Its deterministic - the shared install / admin location write would be preferred on write (but can principally be overriden for loading from user location.)

What I mean was that it's only deterministic while run with the same permissions - the initial problem here is that it's first run by a user with admin permissions then later run without them. IOW, which directory is writable can change from invocation to invocation, which doesn't seem ideal.

%TEMP% is subject to random cleanup - so the application code must always ensure automatic re-creation.
Possible alternative: something like %USERPROFILE%\<pyver>\gen_py or site.getusersitepackages() +r'\win32com\gen_py'

Yeah, good point - site.getusersitepackages() probably does make the most sense.

[sschukat]

+1 for site.getusersitepackages