Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a guide/walkthrough #355

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Conversation

jesshart
Copy link

Description

This is the draft walkthrough I mentioned on issue #350.

@jesshart jesshart requested a review from a team as a code owner February 18, 2023 01:17
@netlify
Copy link

netlify bot commented Feb 18, 2023

Deploy Preview for conda-lock ready!

Name Link
🔨 Latest commit f4476c5
🔍 Latest deploy log https://app.netlify.com/sites/conda-lock/deploys/641ca27597ccc300077de3e4
😎 Deploy Preview https://deploy-preview-355--conda-lock.netlify.app/walkthrough
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@jesshart jesshart changed the title create file with draft pasted resolves #350 Feb 18, 2023
@maresb maresb changed the title resolves #350 Add a guide/walkthrough Feb 18, 2023
@maresb
Copy link
Contributor

maresb commented Feb 18, 2023

Thanks a lot for this!!! Quickly skimming I really like the concept. I could see this turning into some sort of quickstart guide. I think people will have a substantial amount of feedback and suggestions, but working through those may be very valuable for bridging the gap between new and experienced users.

I am also not sure if/how exactly we will publish this. That is dependent on how this evolves, and also a broader consensus among the maintainers, so I can't make any promises. Also, our review capacity is a bit strained at the moment, so I really hope that spectators step up and offer their feedback.

Thanks a lot for all the careful work and thought you have put into this!

@maresb
Copy link
Contributor

maresb commented Feb 18, 2023

You should probably get pre-commit running locally on your side, it is simpler than pytest.

@maresb
Copy link
Contributor

maresb commented Feb 18, 2023

pre-commit.ci autofix

@maresb
Copy link
Contributor

maresb commented Feb 18, 2023

Make sure to pull that pre-commit formatting fix before making additional commits in order to prevent merge conflicts.

Comment on lines 209 to 241
## Adding a New Package via `conda`
We have `pandas` installed, but of course we want to use `matplotlib`.
1. Install `matplotlib`
```bash
(base) $ conda install -c conda-forge matplotlib
...
```
2. Add `matplotlib` to the `environment.yml` file.
```yml
# environment.yml
channels:
- conda-forge
- defaults
dependencies:
- python
- pandas
- matplotlib # NEW
platforms:
- linux-64
- osx-64
- win-64
- osx-arm64
```
3. Update the file using `conda-lock`
```bash
(base) $ conda-lock --update matplotlib
Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']...
...
```
Note: If you update to a specific version of a package, you will need to provide the version in the command. For example, `matplotlib=3.5` should be included after the `--update` flag. This update flag is useful for the following:
* Updating to the latest version of a package (no version specified)
* Updating a package to a specific version
* Adding a new package

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this actually work? I am trying to do this same type of thing. In my case, I want to add fsspec to https://github.com/firesim/firesim/blob/main/conda-reqs.conda-lock.yml. To do that, I:

  1. add fsspec to https://github.com/firesim/firesim/blob/main/conda-reqs.yaml
  2. conda-lock -f conda-reqs.yaml -p linux-64 --lockfile conda-reqs.conda-lock.yml --update fsspec

NOTE: I don't necessarily install the package into the conda environment because conda-lock doesn't use the contents of the currently active environment in it's calculation of the lockfile, does it?

However, I find that unless package you're trying to add is in the lockfile, the intersection() at

to_update = set(update).intersection(conda_locked)
filters the added package out of to_update and conda-lock effectively re-solves the entire environment, it doesn't use the update code.

Also, I think it is mildly confusing to suggest step 1 is installing the package into the currently active environment because it implies that having the packages installed somehow influences how conda-lock creates the lockfile, which I don't think is correct. If you are using a lockfile to reproducibly create an environment, it would probably be better to create the updated lockfile and then use it to create the environment. I haven't tried installing on top of an existing environment to see what happens but I'd expect it to be the same as removing the environment completely and reinstalling everything in the lockfile (but perhaps with optimizations for leaving unchanged packages alone).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The INFO logger line after Locking dependencies for ... is indicative of whether update is happening or not.

When update is being used, it should say something like:

INFO:conda_lock.conda_solver:to_update: ['matplotlib']

I suspect the output looks like :

INFO:conda_lock.conda_solver:linux-64 using specs ['python', 'pandas', 'matplotlib']

and that indicates that the entire environment is being resolved and --update is effectively being ignored.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mariusvniekerk, reading #158 it seems like incremental updating should be possible. If you could take a look at this use-case and comment on what I've found, it would be really helpful.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can confirm that what is shown in the walkthrough is in fact ..using specs... and appears to be ignoring the --update flag. I can be sure to remove this section, or at least clarify that the environment is being solved from scratch.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not very familiar with the --update flag, but if there's a suspected bug (or a missing warning message), let's open an issue for follow-up.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mariusvniekerk. I guess I was asking whether it is supposed to be possible to add new specs to the solve and ask to only update those new specs or if one currently must re-solve the whole environment.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That should work i think, but yeah, probably worth testing if it behaves in the way that you'd expect

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The more standard approach is to just delete and get a new solve

I can modify the document to recommend this approach. However, to your point,

That should work i think, but yeah, probably worth testing if it behaves in the way that you'd expect

It appears the --update flag is not behaving as expected. For example, when I used it to add matplotlib it behaved like this:

(base) $ conda-lock --update matplotlib
INFO:conda_lock.conda_solver:linux-64 using specs ['python', 'pandas', 'matplotlib']

This is in contrast to what was expected according to @timsnyder-siv:

INFO:conda_lock.conda_solver:to_update: ['matplotlib']

So it appears the command is ignoring the flag.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jesshart we should probably open a new issue referencing this. If we're feeling strong in our python-fu, PR'ing a test would also be helpful and if we're feeling particularly masterful, possibly a patch that helps conda-lock do this solve. I looked at the code enough to think I might be able to come up with a patch but it will probably take me longer to get right than I have time for right now. I'll open a ticket from this thread.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened #386 for this.

Sometimes a package installed via `conda` (even using the `conda-forge` channel) breaks my environment. I have found using `pip` as the *exception to the rule* has resolved most of these issues.

### Why is this process slow?
Solving the dependencies of dependencies for one operating system is time consuming. Out of the box, `conda-lock` will try to solve for 3 different operating systems (i.e., linux-64, osx-64, and win-64). However, `conda` recently announced partnering with a `mamba` solver. You could configure your `conda` instance to use a faster solver like this one. This is beyond the scope of this walkthrough. Regardless, it will still take time.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

conda-lock defaults to using mamba if its installed so we should just recommend that.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have been using miniconda with the libmamba solver enabled (ref). Is that much different from using mamba directly?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they're similar but not the same

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not trying to be obtuse here, but wouldn't a strong dependency like this discourage a variety of users from engaging conda-lock? I am happy to adapt this documentation, but wanted to approach this philosophical point first.

For example, I could make the point in the documentation so readers are not misled into thinking it only works with the mamba setup.

@jesshart
Copy link
Author

jesshart commented Mar 13, 2023

Document Changes

@jesshart
Copy link
Author

@maresb I updated the document per the conversations here. However, I forgot to follow your instructions,

Make sure to pull that pre-commit formatting fix before making additional commits in order to prevent merge conflicts.

Which may have caused a problem. I resolved the merge conflicts on my end, but it looks like they still broke during the pre-commit.ci - pr step. Do I need to do anything on my end?

@maresb
Copy link
Contributor

maresb commented Mar 15, 2023

@jesshart, if you like your local version, then the easiest solution is to do a force-push git push --force and this will replace the conflicting walkthrough-doc branch here with your local copy.

@mariusvniekerk
Copy link
Collaborator

pre-commit.ci autofix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants