Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not export unnecessary conda env vars, use conda run #232

Merged
merged 10 commits into from
May 17, 2024

Conversation

rly
Copy link
Contributor

@rly rly commented Oct 9, 2023

Fix #230

@rly
Copy link
Contributor Author

rly commented Oct 9, 2023

Tests pass when run locally except for the error TypeError: float() argument must be a string or a real number, not 'coo_matrix' reported in #231.

@kushalkolar
Copy link
Collaborator

I wonder if this coo_matrix is due to some recent change with caiman.

@kushalkolar
Copy link
Collaborator

Because we didn't have it before.

if "CONDA_PREFIX" in os.environ.keys():
# add command to run the python script in the conda environment loaded
# at the time that this shell script was generated
f.write(f'conda run -p {os.environ["CONDA_PREFIX"]} python {module_path} {args_str}')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would assume that conda is findable in the shell env that that is launched by the subprocess and I don't think that's going to be the case if the conda binary isn't in the PATH env variable.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is "conda run" the best way to do this? If so then the fully path to the conda executable should be used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An alternative is to activate the conda environment and then run python, but conda activate within a shell script can be tricky.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would go with conda run

if "CONDA_PREFIX" in os.environ.keys():
f.write(
f'export CONDA_PREFIX={os.environ["CONDA_PREFIX"]}\n'
f'export CONDA_PYTHON_EXE={os.environ["CONDA_PYTHON_EXE"]}\n'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure this is no longer necessary?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean the CONDA_PYTHON_EXE

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe these env variables will not do anything if the conda environment is not activated and you call the default python below.

@rly
Copy link
Contributor Author

rly commented Nov 10, 2023

This works locally on my Mac M1 system.
Steps:

  1. create conda environment
  2. run through one of the notebooks like mcorr_cnmf.ipynb up to the df.iloc[0].caiman.run() step
  3. the cell runs successfully
  4. deactivate conda environment
  5. call the runfile bash script without any conda environment activated (not even base). the shell script runs successfully!

relevant env vars without any conda env activated are:

CONDA_CHANGEPS1=no
CONDA_EXE=/Users/rly/mambaforge/bin/conda
_CE_M=
_CE_CONDA=
CONDA_PYTHON_EXE=/Users/rly/mambaforge/bin/python
CONDA_SHLVL=0

@ethanbb
Copy link
Collaborator

ethanbb commented May 17, 2024

This is working for me (on Linux).

@kushalkolar kushalkolar merged commit 6a6a184 into nel-lab:master May 17, 2024
0 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Missing CONDA_PREFIX_1 env var to run cnmfe notebook on macOS
3 participants