Error when trying to download 'fr_dep_news_trf'

[tristancatteeuw]

How to reproduce the problem

I was upgrading spacy to 3.0 in a new environment and followed these commands
pip install -U pip setuptools wheel
pip install -U spacy
python -m spacy download en_core_web_trf
python -m spacy download de_dep_news_trf
python -m spacy download fr_dep_news_trf

English and german worked fine but for french I got an error related to sentencepiece from what I understand.

Building wheels for collected packages: sentencepiece
  Building wheel for sentencepiece (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'C:\Users\Tristan\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Tristan\\AppData\\Local\\Temp\\pip-install-zfv2xj07\\sentencepiece_6b32d9bf018241f9ac2adf4b5f2a8a52\\setup.py'"'"'; __file__='"'"'C:\\Users\\Tristan\\AppData\\Local\\Temp\\pip-install-zfv2xj07\\sentencepiece_6b32d9bf018241f9ac2adf4b5f2a8a52\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\Tristan\AppData\Local\Temp\pip-wheel-etrnpchd'
       cwd: C:\Users\Tristan\AppData\Local\Temp\pip-install-zfv2xj07\sentencepiece_6b32d9bf018241f9ac2adf4b5f2a8a52\
  Complete output (15 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-3.9
  copying sentencepiece.py -> build\lib.win-amd64-3.9
  running build_ext
  building '_sentencepiece' extension
  creating build\temp.win-amd64-3.9
  creating build\temp.win-amd64-3.9\Release
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\include -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /EHsc /Tpsentencepiece_wrap.cxx /Fobuild\temp.win-amd64-3.9\Release\sentencepiece_wrap.obj /MT /I..\build\root\include
  clÿ: Ligne de commande warning D9025ÿ: substitution de '/MD' par '/MT'
  sentencepiece_wrap.cxx
  sentencepiece_wrap.cxx(2777): fatal error C1083: Impossible d'ouvrir le fichier includeÿ: 'sentencepiece_processor.h'ÿ: No such file or directory
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.27.29110\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
  ----------------------------------------
  ERROR: Failed building wheel for sentencepiece
  Running setup.py clean for sentencepiece
Failed to build sentencepiece
Installing collected packages: sentencepiece, protobuf, fr-dep-news-trf
  Attempting uninstall: sentencepiece
    Found existing installation: sentencepiece 0.1.95
    Uninstalling sentencepiece-0.1.95:
      Successfully uninstalled sentencepiece-0.1.95
    Running setup.py install for sentencepiece ... error
    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\Tristan\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Tristan\\AppData\\Local\\Temp\\pip-install-zfv2xj07\\sentencepiece_6b32d9bf018241f9ac2adf4b5f2a8a52\\setup.py'"'"'; __file__='"'"'C:\\Users\\Tristan\\AppData\\Local\\Temp\\pip-install-zfv2xj07\\sentencepiece_6b32d9bf018241f9ac2adf4b5f2a8a52\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\Tristan\AppData\Local\Temp\pip-record-d6jde4x3\install-record.txt' --single-version-externally-managed --user --prefix= --compile --install-headers 'C:\Users\Tristan\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\Include\sentencepiece'
         cwd: C:\Users\Tristan\AppData\Local\Temp\pip-install-zfv2xj07\sentencepiece_6b32d9bf018241f9ac2adf4b5f2a8a52\
    Complete output (15 lines):
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-3.9
    copying sentencepiece.py -> build\lib.win-amd64-3.9
    running build_ext
    building '_sentencepiece' extension
    creating build\temp.win-amd64-3.9
    creating build\temp.win-amd64-3.9\Release
    C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\include -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\include -IC:\Program Files (x86)\Microsoft Visual 
Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /EHsc /Tpsentencepiece_wrap.cxx /Fobuild\temp.win-amd64-3.9\Release\sentencepiece_wrap.obj /MT /I..\build\root\include
    clÿ: Ligne de commande warning D9025ÿ: substitution de '/MD' par '/MT'
    sentencepiece_wrap.cxx
    sentencepiece_wrap.cxx(2777): fatal error C1083: Impossible d'ouvrir le fichier includeÿ: 'sentencepiece_processor.h'ÿ: No such file or directory
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.27.29110\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
    ----------------------------------------
  Rolling back uninstall of sentencepiece
  Moving to c:\users\tristan\appdata\local\packages\pythonsoftwarefoundation.python.3.9_qbz5n2kfra8p0\localcache\local-packages\python39\site-packages\sentencepiece-0.1.95.dist-info\
   from C:\Users\Tristan\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\~entencepiece-0.1.95.dist-info
  Moving to c:\users\tristan\appdata\local\packages\pythonsoftwarefoundation.python.3.9_qbz5n2kfra8p0\localcache\local-packages\python39\site-packages\sentencepiece\
   from C:\Users\Tristan\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\~entencepiece
ERROR: Command errored out with exit status 1: 'C:\Users\Tristan\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Tristan\\AppData\\Local\\Temp\\pip-install-zfv2xj07\\sentencepiece_6b32d9bf018241f9ac2adf4b5f2a8a52\\setup.py'"'"'; __file__='"'"'C:\\Users\\Tristan\\AppData\\Local\\Temp\\pip-install-zfv2xj07\\sentencepiece_6b32d9bf018241f9ac2adf4b5f2a8a52\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\Tristan\AppData\Local\Temp\pip-record-d6jde4x3\install-record.txt' --single-version-externally-managed --user --prefix= --compile --install-headers 'C:\Users\Tristan\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\Include\sentencepiece' Check the logs for full command output.

Your Environment

• spaCy version: 3.0.1
• Platform: Windows-10-10.0.18362-SP0
• Python version: 3.9.1
• Pipelines: de_dep_news_trf (3.0.0), en_core_web_trf (3.0.0), fr_core_news_sm (3.0.0)

[adrianeboyd]

Hi, this error is that it's failing to compile sentencepiece, because the sdist for the required version is broken (it doesn't include the necessary source code). The required version, v0.1.91, was released before python 3.9 came out, so it doesn't have precompiled binary wheels through pip. I think the easiest solution is use python 3.8 instead, where there should be a binary wheel available.

Alternatively, if you'd like to continue using python 3.9 and you're comfortable compiling things, you can clone the github repo (https://github.com/google/sentencepiece), check out v0.1.91 and follow the instructions in the README: https://github.com/google/sentencepiece/tree/v0.1.91#installation. (To be honest, I'm not really sure how hard this is going to be in windows?)

Whatever you do, be sure that you end up with v0.1.91, since newer versions are know to segfault under certain conditions in use with transformers.

[tristancatteeuw]

Hi, this error is that it's failing to compile sentencepiece, because the sdist for the required version is broken (it doesn't include the necessary source code). The required version, v0.1.91, was released before python 3.9 came out, so it doesn't have precompiled binary wheels through pip. I think the easiest solution is use python 3.8 instead, where there should be a binary wheel available.

Alternatively, if you'd like to continue using python 3.9 and you're comfortable compiling things, you can clone the github repo (https://github.com/google/sentencepiece), check out v0.1.91 and follow the instructions in the README: https://github.com/google/sentencepiece/tree/v0.1.91#installation. (To be honest, I'm not really sure how hard this is going to be in windows?)

Whatever you do, be sure that you end up with v0.1.91, since newer versions are know to segfault under certain conditions in use with transformers.

Indeed this seemed to be the issue, thank you! However I do realize that this new french pipeline doesn't support NER, is this a planned feature for the future?

[adrianeboyd]

No, not unless we have a corpus that contains all the required types of annotation on the same documents. It doesn't work well to train a transformer pipeline with documents with "partial" annotation, where you have some docs just with tags+parses and some docs just with NER.