fix loading pkg while setup #71

Borda · 2019-08-08T07:33:15Z

fix loading package while setup #56

simple testing

virtualenv ~/Desktop/vEnv_tl -p python3
source ~/Desktop/vEnv_tl/bin/activate
pip install git+https://github.com/Borda/pytorch-lightning.git@fix-setup

williamFalcon · 2019-08-08T08:07:18Z

@Borda i hadn’t noticed the requirements change.

test-tube is missing, needs to be added.
numpy is also required.
why is this reading from requirements.txt? requirements.txt should be for devs no? maybe i misunderstood how the setup.py works but i thought convention was: required installs for the pacakge in setup.py, required installs for devs in requirements.txt

williamFalcon · 2019-08-08T08:08:47Z

yeah... just googled and what i mentioned looks correct:

https://www.google.com/amp/s/amp.reddit.com/r/Python/comments/3uzl2a/setuppy_requirementstxt_or_a_combination/

so, i’d rather go back to this setup with setup.py having the actual deps needed and requirements.txt having dev requirements.

is there an actual reason not to do this other than personal preference?

williamFalcon · 2019-08-08T08:13:10Z

and confirmed again here.

https://packaging.python.org/discussions/install-requires-vs-requirements/

Borda · 2019-08-08T08:31:47Z

#71 (comment) the variable requirements contains all packages from requirements.txt
this new list is also not complete and will cause issues in future

install_requires=['torch>=1.1.0', 'tqdm', 'test-tube>=0.6.7.6'],

for example memory.py needs import pandas as pd but pandas is missing in your list

williamFalcon · 2019-08-08T08:33:34Z

@Borda rolled back the setup.py changes. not sure this fixes the issue, but the package installed fine before when it was setup this way.

I want to keep the project simple and easy to read for researchers, it wasn't obvious what was happening before.

I also want to get back to moving features forward not engineering the package more. Once the install is fixed, I don't want to mess around with CI, setup.py, etc... anymore, don't have the bandwidth to fix issues I already dealt with months ago. The package was already stable and ready for new features :)

Borda · 2019-08-08T08:42:00Z

I want to keep the project simple and easy to read for researchers, it wasn't obvious what was happening before.

Sure, I agree, I just tried to fix some hidden/potential issues, the install_requirements was incomplete even before and it would crash on blank env anyway (it would be fine on Conda, there are some preinstalled packages)

I also want to get back to moving features forward not engineering the package more. Once the install is fixed, I don't want to mess around with CI, setup.py, etc... anymore, don't have the bandwidth to fix issues I already dealt with months ago. The package was already stable and ready for new features :)

Sure, you may leave the CI issues to me... and focus on features :)

williamFalcon · 2019-08-08T08:43:18Z

#71 (comment) the variable requirements contains all packages from requirements.txt
this new list is also not complete and will cause issues in future
install_requires=['torch>=1.1.0', 'tqdm', 'test-tube>=0.6.7.6'],
for example memory.py needs import pandas as pd but pandas is missing in your list

I'm sorry to be a stickler about this, please keep requirements.txt and setup.py separate.

i don't remember if memory was needed for dev or actual install. (pandas in needed by test-tube so it would get installed automatically).

In fact that's even how PyTorch officially does it:
https://github.com/pytorch/pytorch/blob/master/setup.py#L769
requirements.txt is for dev deps
setup.py for package deps.

setup.py installs each dep listed AND all of their deps...

Borda · 2019-08-08T08:48:02Z

your package pytorch_lightning requires pandas, see https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/root_module/memory.py#L10
the only reason why it did not crash yet is that pandas is preinstalled on Conda...
Then you need to make a statement that your package supports Conda only or you need to install all requirements first...

williamFalcon · 2019-08-08T08:49:24Z

@Borda like i said, test-tube requires pandas... it gets auto-installed.
but i agree that adding explicit pandas install is a good idea.

Borda · 2019-08-08T08:55:58Z

btw, pls, squash this PR before merging... :) #60 (comment)

williamFalcon · 2019-08-08T08:59:06Z

I just noticed the fancy squash button option haha :)

PS. just installed this branch in a conda env from scratch and it works fine. looks fixed now i guess?
I don't have virtualenv though, do you want to try on that? @Borda

base) user-mbp:~ user$ conda create --name lttest python=3.6
Collecting package metadata: done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.6.14
  latest version: 4.7.10

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /Users/user/miniconda3/envs/lttest

  added / updated specs:
    - python=3.6


The following NEW packages will be INSTALLED:

  ca-certificates    pkgs/main/osx-64::ca-certificates-2019.5.15-0
  certifi            pkgs/main/osx-64::certifi-2019.6.16-py36_1
  libcxx             pkgs/main/osx-64::libcxx-4.0.1-hcfea43d_1
  libcxxabi          pkgs/main/osx-64::libcxxabi-4.0.1-hcfea43d_1
  libedit            pkgs/main/osx-64::libedit-3.1.20181209-hb402a30_0
  libffi             pkgs/main/osx-64::libffi-3.2.1-h475c297_4
  ncurses            pkgs/main/osx-64::ncurses-6.1-h0a44026_1
  openssl            pkgs/main/osx-64::openssl-1.1.1c-h1de35cc_1
  pip                pkgs/main/osx-64::pip-19.1.1-py36_0
  python             pkgs/main/osx-64::python-3.6.9-h359304d_0
  readline           pkgs/main/osx-64::readline-7.0-h1de35cc_5
  setuptools         pkgs/main/osx-64::setuptools-41.0.1-py36_0
  sqlite             pkgs/main/osx-64::sqlite-3.29.0-ha441bb4_0
  tk                 pkgs/main/osx-64::tk-8.6.8-ha441bb4_0
  wheel              pkgs/main/osx-64::wheel-0.33.4-py36_0
  xz                 pkgs/main/osx-64::xz-5.2.4-h1de35cc_4
  zlib               pkgs/main/osx-64::zlib-1.2.11-h1de35cc_3


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate lttest
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) user-mbp:~ user$ act lttest
(lttest) user-mbp:~ user$ conda install pytorch torchvision -c pytorch
Collecting package metadata: done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.6.14
  latest version: 4.7.10

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /Users/user/miniconda3/envs/lttest

  added / updated specs:
    - pytorch
    - torchvision


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cffi-1.12.3                |   py36hb5b8e2f_0         214 KB
    mkl_fft-1.0.12             |   py36h5e564d8_0         158 KB
    mkl_random-1.0.2           |   py36h27c97d8_0         382 KB
    ninja-1.9.0                |   py36h04f5b5a_0          96 KB
    numpy-1.16.4               |   py36hacdab7b_0          49 KB
    numpy-base-1.16.4          |   py36h6575580_0         4.2 MB
    olefile-0.46               |           py36_0          48 KB
    pillow-6.1.0               |   py36hb68e598_0         595 KB
    pycparser-2.19             |           py36_0         173 KB
    pytorch-1.1.0              |          py3.6_0        50.0 MB  pytorch
    six-1.12.0                 |           py36_0          22 KB
    torchvision-0.3.0          |    py36_cuNone_1         1.7 MB  pytorch
    ------------------------------------------------------------
                                           Total:        57.6 MB

The following NEW packages will be INSTALLED:

  blas               pkgs/main/osx-64::blas-1.0-mkl
  cffi               pkgs/main/osx-64::cffi-1.12.3-py36hb5b8e2f_0
  freetype           pkgs/main/osx-64::freetype-2.9.1-hb4e5f40_0
  intel-openmp       pkgs/main/osx-64::intel-openmp-2019.4-233
  jpeg               pkgs/main/osx-64::jpeg-9b-he5867d9_2
  libgfortran        pkgs/main/osx-64::libgfortran-3.0.1-h93005f0_2
  libpng             pkgs/main/osx-64::libpng-1.6.37-ha441bb4_0
  libtiff            pkgs/main/osx-64::libtiff-4.0.10-hcb84e12_2
  mkl                pkgs/main/osx-64::mkl-2019.4-233
  mkl_fft            pkgs/main/osx-64::mkl_fft-1.0.12-py36h5e564d8_0
  mkl_random         pkgs/main/osx-64::mkl_random-1.0.2-py36h27c97d8_0
  ninja              pkgs/main/osx-64::ninja-1.9.0-py36h04f5b5a_0
  numpy              pkgs/main/osx-64::numpy-1.16.4-py36hacdab7b_0
  numpy-base         pkgs/main/osx-64::numpy-base-1.16.4-py36h6575580_0
  olefile            pkgs/main/osx-64::olefile-0.46-py36_0
  pillow             pkgs/main/osx-64::pillow-6.1.0-py36hb68e598_0
  pycparser          pkgs/main/osx-64::pycparser-2.19-py36_0
  pytorch            pytorch/osx-64::pytorch-1.1.0-py3.6_0
  six                pkgs/main/osx-64::six-1.12.0-py36_0
  torchvision        pytorch/osx-64::torchvision-0.3.0-py36_cuNone_1
  zstd               pkgs/main/osx-64::zstd-1.3.7-h5bba6e5_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
numpy-1.16.4         | 49 KB     | ########################################################## | 100%
torchvision-0.3.0    | 1.7 MB    | ########################################################## | 100%
mkl_random-1.0.2     | 382 KB    | ########################################################## | 100%
pycparser-2.19       | 173 KB    | ########################################################## | 100%
mkl_fft-1.0.12       | 158 KB    | ########################################################## | 100%
ninja-1.9.0          | 96 KB     | ########################################################## | 100%
pillow-6.1.0         | 595 KB    | ########################################################## | 100%
cffi-1.12.3          | 214 KB    | ########################################################## | 100%
numpy-base-1.16.4    | 4.2 MB    | ########################################################## | 100%
six-1.12.0           | 22 KB     | ########################################################## | 100%
pytorch-1.1.0        | 50.0 MB   | ########################################################## | 100%
olefile-0.46         | 48 KB     | ########################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(lttest) user-mbp:~ user$ pip install git+https://github.com/Borda/pytorch-lightning.git@fix-setup
Collecting git+https://github.com/Borda/pytorch-lightning.git@fix-setup
  Cloning https://github.com/Borda/pytorch-lightning.git (to revision fix-setup) to /private/var/folders/b6/lpck7y7j48q16n21vcdk0h7w0000gn/T/pip-req-build-nbhqumi2
  Running command git clone -q https://github.com/Borda/pytorch-lightning.git /private/var/folders/b6/lpck7y7j48q16n21vcdk0h7w0000gn/T/pip-req-build-nbhqumi2
  Running command git checkout -b fix-setup --track origin/fix-setup
  Switched to a new branch 'fix-setup'
  Branch fix-setup set up to track remote branch fix-setup from origin.
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting pandas>=0.20.3 (from pytorch-lightning==0.3.6.9)
  Using cached https://files.pythonhosted.org/packages/94/f0/3099fdb1ae94663561cd695b820f05b6f6d240c919ba179c076015de5e37/pandas-0.25.0-cp36-cp36m-macosx_10_9_x86_64.macosx_10_10_x86_64.whl
Collecting test-tube>=0.6.7.6 (from pytorch-lightning==0.3.6.9)
  Using cached https://files.pythonhosted.org/packages/c4/0b/2c25df2f72160dddfbcb7429f42b3e0a49b80a7a62d9deb98a91a734583c/test_tube-0.6.7.6.tar.gz
Requirement already satisfied: torch>=1.1.0 in ./miniconda3/envs/lttest/lib/python3.6/site-packages (from pytorch-lightning==0.3.6.9) (1.1.0)
Collecting tqdm (from pytorch-lightning==0.3.6.9)
  Using cached https://files.pythonhosted.org/packages/9f/3d/7a6b68b631d2ab54975f3a4863f3c4e9b26445353264ef01f465dc9b0208/tqdm-4.32.2-py2.py3-none-any.whl
Collecting pytz>=2017.2 (from pandas>=0.20.3->pytorch-lightning==0.3.6.9)
  Using cached https://files.pythonhosted.org/packages/87/76/46d697698a143e05f77bec5a526bf4e56a0be61d63425b68f4ba553b51f2/pytz-2019.2-py2.py3-none-any.whl
Requirement already satisfied: numpy>=1.13.3 in ./miniconda3/envs/lttest/lib/python3.6/site-packages (from pandas>=0.20.3->pytorch-lightning==0.3.6.9) (1.16.4)
Collecting python-dateutil>=2.6.1 (from pandas>=0.20.3->pytorch-lightning==0.3.6.9)
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting imageio>=2.3.0 (from test-tube>=0.6.7.6->pytorch-lightning==0.3.6.9)
  Using cached https://files.pythonhosted.org/packages/af/0a/943c965d372dae0b1f1482677d29030ab834351a61a9a632fd62f27f1523/imageio-2.5.0-py3-none-any.whl
Collecting tb-nightly==1.15.0a20190708 (from test-tube>=0.6.7.6->pytorch-lightning==0.3.6.9)
  Using cached https://files.pythonhosted.org/packages/b1/98/fa38deb0d26db3c88fc3c3267c622508648df9d67242d8a4d15c2f6bab08/tb_nightly-1.15.0a20190708-py3-none-any.whl
Collecting future (from test-tube>=0.6.7.6->pytorch-lightning==0.3.6.9)
Requirement already satisfied: six>=1.5 in ./miniconda3/envs/lttest/lib/python3.6/site-packages (from python-dateutil>=2.6.1->pandas>=0.20.3->pytorch-lightning==0.3.6.9) (1.12.0)
Requirement already satisfied: pillow in ./miniconda3/envs/lttest/lib/python3.6/site-packages (from imageio>=2.3.0->test-tube>=0.6.7.6->pytorch-lightning==0.3.6.9) (6.1.0)
Collecting absl-py>=0.4 (from tb-nightly==1.15.0a20190708->test-tube>=0.6.7.6->pytorch-lightning==0.3.6.9)
Collecting grpcio>=1.6.3 (from tb-nightly==1.15.0a20190708->test-tube>=0.6.7.6->pytorch-lightning==0.3.6.9)
  Using cached https://files.pythonhosted.org/packages/44/41/98f0c012ee3eed1a449df504343c3f3c5f9b0c2ae4d6eb31cf6c790a560c/grpcio-1.22.0-cp36-cp36m-macosx_10_9_x86_64.whl
Collecting werkzeug>=0.11.15 (from tb-nightly==1.15.0a20190708->test-tube>=0.6.7.6->pytorch-lightning==0.3.6.9)
  Using cached https://files.pythonhosted.org/packages/d1/ab/d3bed6b92042622d24decc7aadc8877badf18aeca1571045840ad4956d3f/Werkzeug-0.15.5-py2.py3-none-any.whl
Requirement already satisfied: setuptools>=41.0.0 in ./miniconda3/envs/lttest/lib/python3.6/site-packages (from tb-nightly==1.15.0a20190708->test-tube>=0.6.7.6->pytorch-lightning==0.3.6.9) (41.0.1)
Requirement already satisfied: wheel>=0.26; python_version >= "3" in ./miniconda3/envs/lttest/lib/python3.6/site-packages (from tb-nightly==1.15.0a20190708->test-tube>=0.6.7.6->pytorch-lightning==0.3.6.9) (0.33.4)
Collecting protobuf>=3.6.0 (from tb-nightly==1.15.0a20190708->test-tube>=0.6.7.6->pytorch-lightning==0.3.6.9)
  Using cached https://files.pythonhosted.org/packages/b5/24/d910231174d60c16b3b45db520d21581464049b28ba3562ecd5705c5d5c0/protobuf-3.9.1-cp36-cp36m-macosx_10_9_intel.whl
Collecting markdown>=2.6.8 (from tb-nightly==1.15.0a20190708->test-tube>=0.6.7.6->pytorch-lightning==0.3.6.9)
  Using cached https://files.pythonhosted.org/packages/c0/4e/fd492e91abdc2d2fcb70ef453064d980688762079397f779758e055f6575/Markdown-3.1.1-py2.py3-none-any.whl
Building wheels for collected packages: pytorch-lightning
  Building wheel for pytorch-lightning (PEP 517) ... done
  Stored in directory: /private/var/folders/b6/lpck7y7j48q16n21vcdk0h7w0000gn/T/pip-ephem-wheel-cache-ck1ba47n/wheels/46/76/22/df226c37c7815918c4a3a016f2d9f13622065e4e8ac01f174e
Successfully built pytorch-lightning
Building wheels for collected packages: test-tube
  Building wheel for test-tube (setup.py) ... done
  Stored in directory: /Users/user/Library/Caches/pip/wheels/9c/05/bd/107fea0a1db42992edd8183164b9972df9f70621de64f1d35c
Successfully built test-tube
Installing collected packages: pytz, python-dateutil, pandas, imageio, absl-py, grpcio, werkzeug, protobuf, markdown, tb-nightly, future, test-tube, tqdm, pytorch-lightning
Successfully installed absl-py-0.7.1 future-0.17.1 grpcio-1.22.0 imageio-2.5.0 markdown-3.1.1 pandas-0.25.0 protobuf-3.9.1 python-dateutil-2.8.0 pytorch-lightning-0.3.6.9 pytz-2019.2 tb-nightly-1.15.0a20190708 test-tube-0.6.7.6 tqdm-4.32.2 werkzeug-0.15.5
(lttest) user-mbp:~ user$ python
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 13:42:17)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytorch_lightning
>>> import pytorch_lightning as pl
>>> from pytorch_lightning import Trainer
>>> from pytorch_lightning import LightningModule
>>>

fix loading package while setup #56

e5ff62e

Borda mentioned this pull request Aug 8, 2019

pip installation using github repository incomplete #56

Closed

williamFalcon added 3 commits August 8, 2019 04:28

Update setup.py

72bbf55

Update __init__.py

d00c058

Update setup.py

4200662

use complete req.

9a76067

Update setup.py

2fb7571

Update setup.py

f509aff

williamFalcon merged commit d0ccbb8 into Lightning-AI:master Aug 8, 2019

Borda deleted the fix-setup branch August 8, 2019 10:02

felixdittrich92 mentioned this pull request Aug 2, 2021

un-packing bug : TypeError: wrapper() got an unexpected keyword argument #8677

Closed

OlfwayAdbayIgbay mentioned this pull request Aug 12, 2021

Predicting with custom dataloader overwrites "predict_dataloader()" method of module #8868

Closed

OlfwayAdbayIgbay mentioned this pull request Aug 20, 2021

UNet with Variable Input Shapes throws RuntimeError: CUDA error: an illegal memory access was encountered #9008

Closed

elbamos mentioned this pull request Aug 25, 2024

"FileExistsError: [Errno 17] File exists: '/000000_epoch_shape'" using the ddp_notebook strategy with data stored in MDS (mosaic streaming) format #20226

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix loading pkg while setup #71

fix loading pkg while setup #71

Borda commented Aug 8, 2019 •

edited

Loading

williamFalcon commented Aug 8, 2019

williamFalcon commented Aug 8, 2019 •

edited

Loading

williamFalcon commented Aug 8, 2019

Borda commented Aug 8, 2019 •

edited

Loading

williamFalcon commented Aug 8, 2019

Borda commented Aug 8, 2019

williamFalcon commented Aug 8, 2019 •

edited

Loading

Borda commented Aug 8, 2019

williamFalcon commented Aug 8, 2019

Borda commented Aug 8, 2019

williamFalcon commented Aug 8, 2019 •

edited

Loading

fix loading pkg while setup #71

fix loading pkg while setup #71

Conversation

Borda commented Aug 8, 2019 • edited Loading

williamFalcon commented Aug 8, 2019

williamFalcon commented Aug 8, 2019 • edited Loading

williamFalcon commented Aug 8, 2019

Borda commented Aug 8, 2019 • edited Loading

williamFalcon commented Aug 8, 2019

Borda commented Aug 8, 2019

williamFalcon commented Aug 8, 2019 • edited Loading

Borda commented Aug 8, 2019

williamFalcon commented Aug 8, 2019

Borda commented Aug 8, 2019

williamFalcon commented Aug 8, 2019 • edited Loading

Borda commented Aug 8, 2019 •

edited

Loading

williamFalcon commented Aug 8, 2019 •

edited

Loading

Borda commented Aug 8, 2019 •

edited

Loading

williamFalcon commented Aug 8, 2019 •

edited

Loading

williamFalcon commented Aug 8, 2019 •

edited

Loading