Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

system_requirements files not shared between host/docker #3812

Closed
yangcha opened this issue Oct 20, 2018 · 8 comments · Fixed by #4354
Closed

system_requirements files not shared between host/docker #3812

yangcha opened this issue Oct 20, 2018 · 8 comments · Fixed by #4354
Assignees
@uilianries
Labels
complex: low priority: medium type: feature
Milestone
1.12

Comments

@yangcha
Copy link
Contributor

yangcha commented Oct 20, 2018 •
edited
Loading

I have an application depends on a library libA. libA's recipe has a system_requirements() method which installs a system library. libA's artifact is stored remotely. When I built the applications, it failed to find the system library installed by the libA's recipe.

Is system_requirements() transitive? If so, how to specify in the conanfile.txt or conanfile.py of application?
By the way, I don't want to build libA from source.
@yangcha
Copy link
Contributor Author

yangcha commented Oct 20, 2018

In fact, this is caused by the same reason in #2262

@lasote
Copy link
Contributor

lasote commented Oct 20, 2018

Everytime libA is installed the system requirements method will be called if it hasn't installed before so I don't understand exactly what is the issue. What do you mean with When I built the application? The app is built in a different computer, right? Is it not called when doing the conan install?

@yangcha
Copy link
Contributor Author

yangcha commented Oct 20, 2018 •
edited
Loading

Actually I run the build inside Docker container and the local cache is stored on a persistent volume. So rerun the build of the application will not install system package anymore.
It is a same problem as issue #2262. When I created this issue, I misunderstood the way how system_requirements() works.

@yangcha
Copy link
Contributor Author

yangcha commented Oct 20, 2018 •
edited
Loading

I think it is better to store system_reqs.txt alongside with systems, so local cache can be freely shared among different nodes.

If system_reqs.txt are stored in the user’s home directory instead of CONAN_USER_HOME, then problem will be perfectly resolved.

@lasote
Copy link
Contributor

lasote commented Oct 22, 2018

I wouldn't recommend sharing the cache with the containers, to be honest. It can be problematic with the permissions of the storage and for example with the system requirements. I would say it more solid to communicate with a conan server (or better Artifactory community edition)

@yangcha
Copy link
Contributor Author

yangcha commented Oct 22, 2018 •
edited
Loading

I wouldn't recommend sharing the cache with the containers, to be honest. It can be problematic with the permissions of the storage and for example with the system requirements. I would say it more solid to communicate with a conan server (or better Artifactory community edition)

I don't think permission is a big issue, because Docker can use -u to set the user/group.
The problem with conan server (or Artifactory) is that they can be slow to access. If the local cache can be shared locally, that would be a big plus.
By the way, all these system_reqs.txt are important for deploy the applications to target system. It would be nice to export all the system_reqs.txt in deploy() method.

@yangcha yangcha mentioned this issue Oct 22, 2018
Closed
@lasote lasote changed the title How to make system_requirements() transitive? system_requirements files not shared between host/docker Nov 20, 2018
@lasote
Copy link
Contributor

lasote commented Nov 20, 2018

Could a conan remove --system_reqs help? #2262 @bilke @yangcha

@lasote lasote assigned memsharded and unassigned lasote Nov 20, 2018
@yangcha
Copy link
Contributor Author

yangcha commented Nov 20, 2018 •
edited
Loading

Could a conan remove --system_reqs help? #2262 @bilke @yangcha

That will be great.

@lasote lasote assigned lasote and unassigned memsharded Nov 21, 2018
@lasote lasote removed their assignment Nov 21, 2018
@lasote lasote added this to the 1.12 milestone Jan 4, 2019
uilianries added a commit to uilianries/conan that referenced this issue Jan 21, 2019
@uilianries
conan-io#3812 Remove cached system_reqs
578975e 
- Add option to remove system_reqs from package cache.

Signed-off-by: Uilian Ries <uilianries@gmail.com>
uilianries added a commit to uilianries/conan that referenced this issue Jan 21, 2019
@uilianries
conan-io#3812 Add tests for remove --system_reqs
f1a944e 
- conan remove --system_reqs should remove global requirement folder.
- Should we support by package id?

Signed-off-by: Uilian Ries <uilianries@gmail.com>
uilianries added a commit to uilianries/conan that referenced this issue Jan 21, 2019
@uilianries
conan-io#3812 remove logger from cache
a166637 
Signed-off-by: Uilian Ries <uilianries@gmail.com>
@uilianries uilianries mentioned this issue Jan 21, 2019
Merged
5 tasks
uilianries added a commit to uilianries/conan that referenced this issue Jan 21, 2019
@uilianries
conan-io#3812 Update system_reqs description
1b76cba 
Signed-off-by: Uilian Ries <uilianries@gmail.com>
uilianries added a commit to uilianries/conan that referenced this issue Jan 22, 2019
@uilianries
conan-io#3812 --system-reqs can raise error
3612dd7 
- Conan remove --system-reqs forwards the exception message
  when an error occurs.
- Re-order remove command arguments
- Add test to check system-reqs error message

Signed-off-by: Uilian Ries <uilianries@gmail.com>
uilianries added a commit to uilianries/conan that referenced this issue Jan 22, 2019
@uilianries
conan-io#3812 Remove system-reqs by package id
4d3e7fc 
- Add support to parse package reference when `remove --system-reqs -p
 <ref>` is executed and remove ONLY the cache for the specific
  package id.

Signed-off-by: Uilian Ries <uilianries@gmail.com>
uilianries added a commit to uilianries/conan that referenced this issue Jan 22, 2019
@uilianries
Revert "conan-io#3812 Remove system-reqs by package id"
e6395d8 
This reverts commit 4d3e7fc.
uilianries added a commit to uilianries/conan that referenced this issue Jan 22, 2019
@uilianries
conan-io#3812 Ignore errors when remove system-reqs
55fc37b 
- Package ID is NOT supported for system-reqs
- Ignore error, like permission or path not found

Signed-off-by: Uilian Ries <uilianries@gmail.com>
uilianries added a commit to uilianries/conan that referenced this issue Jan 22, 2019
@uilianries
conan-io#3812 Remove system-reqs can raise errors
838f2e0 
- Add exception for permission denied and wrong pkg ref
- Add new test to check possible errors in system-reqs

Signed-off-by: Uilian Ries <uilianries@gmail.com>
uilianries added a commit to uilianries/conan that referenced this issue Jan 22, 2019
@uilianries
conan-io#3812 Fix test for Windows platform
2f52f4d 
Signed-off-by: Uilian Ries <uilianries@gmail.com>
uilianries added a commit to uilianries/conan that referenced this issue Jan 24, 2019
@uilianries
conan-io#3812 Replace rmtree by rmdir
6c2ec84 
Signed-off-by: Uilian Ries <uilianries@gmail.com>
@ghost ghost removed the stage: review label Jan 25, 2019
lasote pushed a commit that referenced this issue Jan 25, 2019
@uilianries @lasote
#3812 Remove system requirements from cache (#4354)
2311d1b 
* #3812 Remove cached system_reqs

- Add option to remove system_reqs from package cache.

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* #3812 Add tests for remove --system_reqs

- conan remove --system_reqs should remove global requirement folder.
- Should we support by package id?

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* #3812 remove logger from cache

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* #3812 Update system_reqs description

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* #3812 --system-reqs can raise error

- Conan remove --system-reqs forwards the exception message
  when an error occurs.
- Re-order remove command arguments
- Add test to check system-reqs error message

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* #3812 Remove system-reqs by package id

- Add support to parse package reference when `remove --system-reqs -p
 <ref>` is executed and remove ONLY the cache for the specific
  package id.

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* Revert "#3812 Remove system-reqs by package id"

This reverts commit 4d3e7fc.

* #3812 Ignore errors when remove system-reqs

- Package ID is NOT supported for system-reqs
- Ignore error, like permission or path not found

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* #3812 Remove system-reqs can raise errors

- Add exception for permission denied and wrong pkg ref
- Add new test to check possible errors in system-reqs

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* #3812 Fix test for Windows platform

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* #3812 Replace rmtree by rmdir

Signed-off-by: Uilian Ries <uilianries@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@uilianries uilianries

Labels
complex: low priority: medium type: feature
Projects
None yet
Milestone
1.12
Development

Successfully merging a pull request may close this issue.

#3812 Remove system requirements from cache uilianries/conan
6 participants
@jgsogo @lasote @uilianries @yangcha @danimtb @memsharded