Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use .condarc file to configure conda-standalone #99

Merged
merged 20 commits into from
Oct 3, 2024
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions news/99-configure-conda-standalone
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
### Enhancements

* Configure conda-standalone binaries with .condarc files. (#97 via #99)
jaimergp marked this conversation as resolved.
Show resolved Hide resolved
* Add environment variable `CONDA_RESTRICT_RC_SEARCH_PATH` and CLI option `--no-rc` to only load `.condarc` file delivered by `conda-standalone` bundle or `CONDARC` environment variable. (#99)

### Bug fixes

* <news item>

### Deprecations

* <news item>

### Docs

* <news item>

### Other

* <news item>
2 changes: 2 additions & 0 deletions recipe/.condarc
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
channels:
- conda-forge
2 changes: 2 additions & 0 deletions recipe/bld.bat
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ RENAME "%SP_DIR%\conda\utils.py" utils.py.bak || goto :error
COPY conda_src\conda\utils.py "%SP_DIR%\conda\utils.py" || goto :error
RENAME "%SP_DIR%\conda\deprecations.py" deprecations.py.bak || goto :error
COPY conda_src\conda\deprecations.py "%SP_DIR%\conda\deprecations.py" || goto :error
RENAME "%SP_DIR%\conda\base\constants.py" constants.py.bak || goto :error
COPY conda_src\conda\base\constants.py "%SP_DIR%\conda\base\constants.py" || goto :error

:: we need these for noarch packages with entry points to work on windows
COPY "conda_src\conda\shell\cli-%ARCH%.exe" entry_point_base.exe || goto :error
Expand Down
2 changes: 1 addition & 1 deletion recipe/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ set -euxo pipefail

# patched conda files
# new files in patches need to be added here
for fname in "core/path_actions.py" "utils.py" "deprecations.py"; do
for fname in "core/path_actions.py" "utils.py" "deprecations.py" "base/constants.py"; do
mv "$SP_DIR/conda/${fname}" "$SP_DIR/conda/${fname}.bak"
cp "conda_src/conda/${fname}" "$SP_DIR/conda/${fname}"
done
Expand Down
4 changes: 4 additions & 0 deletions recipe/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ source:
patches:
- ../src/conda_patches/0001-Rename-and-replace-entrypoint-stub-exe.patch
- ../src/conda_patches/0002-Manipulate-PATH-directly-instead-of-_call_ing-conda.patch
- ../src/conda_patches/0003-Restrict-search-paths.patch

- url: https://github.com/conda/constructor/archive/{{ constructor_version }}.tar.gz # [win]
sha256: 0ea4f6d563a53ebb03475dc6d2d88d3ab01be4e9d291fd276c79315aa92e5114 # [win]
Expand All @@ -29,6 +30,8 @@ build:
string: "g{{ GIT_FULL_HASH[:7] }}_py{{ pyver }}_{{ PKG_BUILDNUM }}"
ignore_run_exports:
- '*'
script_env:
- PYINSTALLER_CONDARC_DIR={{ RECIPE_DIR }}

requirements:
build:
Expand All @@ -48,6 +51,7 @@ test:
requires:
- pytest
- menuinst >={{ menuinst_lower_bound }}
- ruamel.yaml
source_files:
- tests
commands:
Expand Down
7 changes: 7 additions & 0 deletions src/conda.exe.spec
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,13 @@ elif sys.platform == "darwin":
]
extra_exe_kwargs["entitlements_file"] = os.path.join(HERE, "entitlements.plist")

# Add .condarc file to bundle to configure channels
# during the package building stage
if "PYINSTALLER_CONDARC_DIR" in os.environ:
condarc = os.path.join(os.environ["RECIPE_DIR"], ".condarc")
if os.path.exists(condarc):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be more explicit to simply pass the path to the condarc and then copy it? Or is it to save the rename logic?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know, pyinstaller does not support renaming, or at least not all versions do.

datas.append((condarc, "."))

a = Analysis(['entry_point.py', 'imports.py'],
pathex=['.'],
binaries=binaries,
Expand Down
91 changes: 91 additions & 0 deletions src/conda_patches/0003-Restrict-search-paths.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
diff --git a/conda/base/constants.py b/conda/base/constants.py
index d38502a48..b56724933 100644
--- a/conda/base/constants.py
+++ b/conda/base/constants.py
@@ -10,6 +10,7 @@ Another important source of "static" configuration is conda/models/enums.py.

import struct
from enum import Enum, EnumMeta
+from os import environ
from os.path import join

from ..common.compat import on_win
@@ -25,42 +26,47 @@ machine_bits = 8 * struct.calcsize("P")

APP_NAME = "conda"

-if on_win: # pragma: no cover
+if "CONDA_RESTRICT_RC_SEARCH_PATH" in environ:
SEARCH_PATH = (
- "C:/ProgramData/conda/.condarc",
- "C:/ProgramData/conda/condarc",
- "C:/ProgramData/conda/condarc.d",
+ "$CONDARC",
)
else:
- SEARCH_PATH = (
- "/etc/conda/.condarc",
- "/etc/conda/condarc",
- "/etc/conda/condarc.d/",
- "/var/lib/conda/.condarc",
- "/var/lib/conda/condarc",
- "/var/lib/conda/condarc.d/",
+ if on_win: # pragma: no cover
+ SEARCH_PATH = (
+ "C:/ProgramData/conda/.condarc",
+ "C:/ProgramData/conda/condarc",
+ "C:/ProgramData/conda/condarc.d",
+ )
+ else:
+ SEARCH_PATH = (
+ "/etc/conda/.condarc",
+ "/etc/conda/condarc",
+ "/etc/conda/condarc.d/",
+ "/var/lib/conda/.condarc",
+ "/var/lib/conda/condarc",
+ "/var/lib/conda/condarc.d/",
+ )
+
+ SEARCH_PATH += (
+ "$CONDA_ROOT/.condarc",
+ "$CONDA_ROOT/condarc",
+ "$CONDA_ROOT/condarc.d/",
+ "$XDG_CONFIG_HOME/conda/.condarc",
+ "$XDG_CONFIG_HOME/conda/condarc",
+ "$XDG_CONFIG_HOME/conda/condarc.d/",
+ "~/.config/conda/.condarc",
+ "~/.config/conda/condarc",
+ "~/.config/conda/condarc.d/",
+ "~/.conda/.condarc",
+ "~/.conda/condarc",
+ "~/.conda/condarc.d/",
+ "~/.condarc",
+ "$CONDA_PREFIX/.condarc",
+ "$CONDA_PREFIX/condarc",
+ "$CONDA_PREFIX/condarc.d/",
+ "$CONDARC",
)

-SEARCH_PATH += (
- "$CONDA_ROOT/.condarc",
- "$CONDA_ROOT/condarc",
- "$CONDA_ROOT/condarc.d/",
- "$XDG_CONFIG_HOME/conda/.condarc",
- "$XDG_CONFIG_HOME/conda/condarc",
- "$XDG_CONFIG_HOME/conda/condarc.d/",
- "~/.config/conda/.condarc",
- "~/.config/conda/condarc",
- "~/.config/conda/condarc.d/",
- "~/.conda/.condarc",
- "~/.conda/condarc",
- "~/.conda/condarc.d/",
- "~/.condarc",
- "$CONDA_PREFIX/.condarc",
- "$CONDA_PREFIX/condarc",
- "$CONDA_PREFIX/condarc.d/",
- "$CONDARC",
-)
-
DEFAULT_CHANNEL_ALIAS = "https://conda.anaconda.org"
CONDA_HOMEPAGE_URL = "https://conda.io"
ERROR_UPLOAD_URL = "https://conda.io/conda-post/unexpected-error"
9 changes: 9 additions & 0 deletions src/entry_point.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@
# See https://github.com/conda/conda-standalone/issues/86
del os.environ["SSLKEYLOGFILE"]

if "CONDARC" not in os.environ:
os.environ["CONDARC"] = os.path.join(sys.prefix, ".condarc")


def _create_dummy_executor(*args, **kwargs):
"use this for debugging, because ProcessPoolExecutor isn't pdb/ipdb friendly"
Expand Down Expand Up @@ -295,6 +298,12 @@ def _conda_main():
from conda.cli import main

_fix_sys_path()
try:
no_rc = sys.argv.index("--no-rc")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the plan for using this flag in constructor? Older conda-standalone versions will fail if the flag is passed 🤔 Or is it this here just for convenience and we will just set CONDA_RESTRICT_RC_SEARCH_PATH?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just submitted a draft PR: conda/constructor#863

Essentially, I will do a version check:

    if exe_name == "conda-standalone" and Version(exe_version) >= Version("24.9.0"):
        info["_ignore_condarcs_arg"] = "--no-rc"
    elif exe_name == "micromamba":
        info["_ignore_condarcs_arg"] = "--no-rc"
    else:
        info["_ignore_condarcs_arg"] = ""

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be tricky with cross-installers because we might not be able to run the to-be-bundled conda_exe, but I'll add that in the constructor PR.

os.environ["CONDA_RESTRICT_RC_SEARCH_PATH"] = "1"
del sys.argv[no_rc]
except ValueError:
pass
return main()


Expand Down
1 change: 1 addition & 0 deletions tests/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
pytest
menuinst>=2
ruamel.yaml
71 changes: 71 additions & 0 deletions tests/test_main.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import io
import json
import os
import shutil
import stat
Expand All @@ -8,6 +9,7 @@
from pathlib import Path

import pytest
from ruamel.yaml import YAML

# TIP: You can debug the tests with this setup:
# CONDA_STANDALONE=src/entry_point.py pytest ...
Expand Down Expand Up @@ -67,6 +69,75 @@ def test_constructor():
run_conda("constructor", "--help", check=True)


@pytest.mark.parametrize("search_paths", ("all_rcs", "--no-rc", "env_var"))
def test_conda_standalone_config(search_paths, tmp_path, monkeypatch):
expected_configs = {}
if recipe_dir := os.environ.get("PYINSTALLER_CONDARC_DIR"):
recipe_condarc = Path(recipe_dir, ".condarc")
if recipe_condarc.exists():
yaml = YAML()
with open(recipe_condarc) as crc:
recipe_config = YAML().load(crc)
expected_configs["recipe"] = recipe_config

config_args = ["--show-sources", "--json"]
if search_paths == "env_var":
monkeypatch.setenv("CONDA_RESTRICT_RC_SEARCH_PATH", "1")
elif search_paths == "--no-rc":
config_args.append("--no-rc")
else:
config_path = str(tmp_path / ".condarc")
expected_configs[config_path] = {
"channels": [
"defaults",
]
}
with open(config_path, "w") as crc:
yaml.dump(expected_configs[config_path], crc)
monkeypatch.setenv("CONDA_ROOT", str(tmp_path))
env = os.environ.copy()

proc = run_conda(
"config",
*config_args,
check=True,
capture_output=True,
text=True,
env=env,
)
condarcs = json.loads(proc.stdout)

tmp_root = None
if recipe_dir:
# Quick way to get the location conda-standalone is extracted into
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, is this stable across runs? I thought the name had some random characters everytime.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to be because I am using the {{ RECIPE_DIR }} environment variable. However, I decided to generalize the variable names.

proc = run_conda(
"python",
"-c",
"import sys; print(sys.prefix)",
check=True,
capture_output=True,
text=True,
env=env,
)
tmp_root = str(Path(proc.stdout).parent)

conda_configs = {}
for filepath, config in condarcs.items():
if Path(filepath).exists():
conda_configs[filepath] = config
elif recipe_dir and filepath.startswith(tmp_root):
conda_configs["recipe"] = config
if search_paths == "all_rcs":
# If the search path is restricted, there may be other .condarc
# files in the final config, so be less strict with assertions
for filepath, config in expected_configs.items():
assert (
conda_configs.get(filepath) == config
), f"Incorrect config for {filepath}"
else:
assert expected_configs == conda_configs


def test_extract_conda_pkgs(tmp_path: Path):
shutil.copytree(HERE / "data", tmp_path / "pkgs")
run_conda("constructor", "--prefix", tmp_path, "--extract-conda-pkgs", check=True)
Expand Down
Loading