Linking to model success but not visible under spacy info.

[jreeter]

I have downloaded the md model and running the link with:

PS C:\dev\rasa_nlu> python -m spacy link c:\Python27\Lib\site-packages\en_core_web_sm en --force
Invalid switch - "Python27".

    Linking successful

    c:/Python27/Lib/site-packages/en_core_web_sm -->
    C:/Python27/lib/site-packages/spacy/data/en

    You can now load the model via spacy.load('en').

However it doesn't seem to be linked under spacy :

    Info about spaCy

    Python version     2.7.13
    Platform           Windows-10-10.0.14393
    spaCy version      1.7.3
    Installed models
    Location           C:\Python27\lib\site-packages\spacy

Info about spaCy

• Python version: 2.7.13
• Platform: Windows-10-10.0.14393
• spaCy version: 1.7.3
• Installed models:

[ines]

Thanks for the report! This is interesting... I need to test this, might be related to the pathlib issues with symlinks on Windows / Python 2 (see #909). Maybe the symlinks are not recognised correctly as directories here (which, according to pathlib, should work fine).

When you have a second, could you run the following code and tell me if it's getting the paths correctly and if not, which one fails / returns an empty list?

from spacy.util import get_data_path

data_path = get_data_path()
print([p for p in data_path.iterdir()])
print([p for p in data_path.iterdir() if p.is_dir()])
print([p.parts[-1] for p in data_path.iterdir()])
print([p.parts[-1] for p in data_path.iterdir() if p.is_dir()])

[jreeter]

@ines per your request:

PS C:\dev\rasa_nlu> python .\ines.py
[WindowsPath('C:/Python27/lib/site-packages/spacy/data/__init__.py'), WindowsPath('C:/Python27/lib/site-packages/spacy/data/__init__.pyc')]
[]
['__init__.py', '__init__.pyc']
[]

[ines]

Thanks! So it looks like the is_dir isn't actually the problem here and the symlink simply wasn't created in the first place. (And the success message shouldn't have been displayed, because clearly, it wasn't successful. Hm.)

I think the problem might be this:

Invalid switch - "Python27".

It somehow seems to be confused by the Windows path and thinks it's supposed to be a switch. What happens if you put the local path in quotes?

[jreeter]

@ines Same result

PS C:\dev\rasa_nlu> python -m spacy link "c:\Python27\Lib\site-packages\en_core_web_sm" en
Invalid switch - "Python27".

    Linking successful

    c:/Python27/Lib/site-packages/en_core_web_sm -->
    C:/Python27/lib/site-packages/spacy/data/en

    You can now load the model via spacy.load('en').

PS C:\dev\rasa_nlu> python -m spacy info

    Info about spaCy

    Python version     2.7.13
    Platform           Windows-10-10.0.14393
    spaCy version      1.7.3
    Installed models
    Location           C:\Python27\lib\site-packages\spacy

[jreeter]

@ines digging around some more I noticed that in the util\link code it tests for python 2.x and if os is windows, it tries to use mlink.

if util.is_python2() and util.is_windows():
        import subprocess
        command = ['mklink', '/d', link_path.as_posix(), model_path.as_posix()]
        subprocess.call(command, shell=True)

1. mklink doesn't seem to be recognized by Powershell not sure if the executing link.py in Powershell will execute the call to mklink subprocess in Powershell as well. If it does then that's an issue.

2. mklink expects the link and targets paths to use the windows back slash convention. Other wise you get the invalid switch error as seen above. So correct me if I am wrong but calling .as_posix() under the windows env is not correct.

I was able to manually link my model by calling the follow in cmd prompt (not powershell):
mklink /d C:\Python27\lib\site-packages\spacy\data\en c:\dev\rasa_nlu\en_core_web_md

Which resulted in spaCy recognizing my installed model:

PS C:\dev\rasa_nlu> python -m spacy info

    Info about spaCy

    Python version     2.7.13
    Platform           Windows-10-10.0.14393
    spaCy version      1.7.3
    Installed models    en
    Location           C:\Python27\lib\site-packages\spacy

[jreeter]

I have submitted a pull request to fix the issue of bad paths sent to mklink: #942

[ines]

Thanks so much for your analysis and submitting the PR 👍

The mlink fix originates from the symlink issues described in #909 and it seems to have solved the issue for this particular case. So I can only guess that in some setups, is_posix() is resolved correctly to the backslash Windows path, and in others, like yours, it isn't?

I hope there's no other pathlib bug or weirdness going on here and the Path is really interpreted correctly across Windows versions etc. Otherwise we might have to introduce some additional checks.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.