[bug] source_credentials.json causes remote to expect credentials

[ottar-azn]

Describe the bug

Environment Details

Red Hat Enterprise Linux 8.9 (Ootpa)
Linux 4.18.0-477.15.1.el8_8.x86_64
cmake version 3.27.7
Python 3.11.5
Conan version 2.3.0

Description

When credentials are defined in source_credentials.json, with a URL that matches a URL defined for a Conan remote, the remote will require authentication information to be propagated even if the Conan remote repository itself is public and does not require authentication. This applies to virtual and local repositories and not remote repositories that are public.

Example Setup
Conan remote URL https://example-url.com/conan-remote

sources could exist at
https://example-url.com/source
https://example-url.com/differentsource

How to reproduce it

Commands

Behaviour without source_credentials.json

1. Configure an artifactory virtual conan repository named example-remote-name and set it up with the client
2. Assert that a virtual repository named example-remote-name exists

$ conan remote list
example-remote-name: https://example-url.com [Verify SSL: True, Enabled: True]

3. Attempt to install a package from that repository

$ conan install --requires zlib/1.3 -r example-remote-name

4. Observe that no authentication prompt is provided to the user

Behaviour with source_credentials.json

1. Configure an artifactory virtual conan repository named example-remote-name and set it up with the client
2. Assert that a virtual repository named example-remote-name exists

$ conan remote list example-remote-name: https://example-url.com [Verify SSL: True, Enabled: True]

3. Install the following source_credentials.json to the conan cache, $CONAN_HOME/source_credentials.json:

{% set artifactory_identity_token = os.getenv('ARTIFACTORY_IDENTITY_TOKEN') %}
{
   "credentials": [
     {
       "url": "https://example-url.com",
       "token": "{{ artifactory_identity_token }}" 
     }
   ]
}

3. Attempt to install a package from example-remote-name

$ conan install --requires zlib/1.3 -r example-remote-name

4. Observe that an authentication prompt is provided to the user

Logs

When source_credentials.json is present

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu14
compiler.libcxx=libstdc++11
compiler.version=8
os=Linux
[conf]
Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu14
compiler.libcxx=libstdc++11
compiler.version=8
os=Linux
[conf]

======== Computing dependency graph ========
zlib/1.3: Not found in local cache, looking in remotes...
zlib/1.3: Checking remote: example-remote-name
Please log in to "example-remote-name" to perform this action. Execute "conan remote login" command.
Remote 'example-remote-name' username:

WAITS FOR USER INPUT HERE   

When source_credentials.json is not present

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu14
compiler.libcxx=libstdc++11
compiler.version=8
os=Linux
[conf]
Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu14
compiler.libcxx=libstdc++11
compiler.version=8
os=Linux
[conf]

======== Computing dependency graph ========
zlib/1.3: Not found in local cache, looking in remotes...
zlib/1.3: Checking remote: example-remote-name
zlib/1.3: Downloaded recipe revision 5c0f3a1a222eebb6bff34980bcd3e024
Graph root
    cli
Requirements
    zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024 - Downloaded (example-remote-name)

======== Computing necessary packages ========
Requirements
    zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024:897e6fde718e72637d92b0b8ed929893a1976238#f26d128bbc37f6dbd9947b00b0fe7aaa - Download (example-remote-name)

======== Installing packages ========-
------- Downloading 1 package --------
zlib/1.3: Retrieving package 897e6fde718e72637d92b0b8ed929893a1976238 from remote 'example-remote-name'
zlib/1.3: Package installed 897e6fde718e72637d92b0b8ed929893a1976238
zlib/1.3: Downloaded package revision f26d128bbc37f6dbd9947b00b0fe7aaa
WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN: deprecated:     'cpp_info.names' used in: zlib/1.3

======== Finalizing install (deploy, generators) ========
cli: Generating aggregated env files
cli: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished successfully

[memsharded]

Hi @ottar-ct

Thanks for your report, and thanks specially for making it so detailed.

I am checking this, and I am not sure this is a bug or by design.

The source URL and the Conan packages URL cannot be the same. This is by design, it is impossible that the same URL can serve both Conan packages and generic source tarballs. So the "source_credentials.json" design is pretty simple, if the URL for any http requests matches the provided URL it will inject those credentials.

Can you please try using distinct URLs for the Conan repo and the sources repo? Because they are different, aren't they?

[ottar-azn]

Hi @memsharded

Thanks for the response,

The use case is:

• We have a large artifactory instance with many repositories of different types
  In the example this is https://example-url.com
  There are a number of conan remotes which would exist as

  • https://example-url.com/remote1
  • https://example-url.com/remote2
  • ...

  Some of these require authentication
  There are a number of sources which would exist as

  • https://example-url.com/source1
  • https://example-url.com/source2
  • ...

  Some of these require authentication

• There is value in being able to specify a base URL with which to apply authentication credentials automatically where they apply

• We don't want remotes to prompt for authentication where they have no authentication requirements. The alternative is working out a way to merge source_credentials.json for use cases across logical boundaries, which is going to be fragile and prone to error

source_credentials.json affecting the behaviour of conan remote authentication doesn't match my expectations of what a source_credentials.json file would do

[memsharded]

source_credentials.json affecting the behaviour of conan remote authentication doesn't match my expectations of what a source_credentials.json file would do

I understand what you mean, but the issue is that this credentials injection is designed as an http interceptor. So it is basically impossible to distinguish the "type" of URL and knowing if it is a Conan remote or not.

This was designed as a per-repository credential, and with the constraint that there cannot be overlap in URLs.

In your case, if your URLs for sources are like https://example-url.com/source1 then you can do

{
   "credentials": [
     {
       "url": "https://example-url.com/source",
       "token": "{{ artifactory_identity_token }}" 
     }
   ]
}

And that will work for every source* repo out there, because it is a text pattern, just a startswith call.

As another alternative, maybe it would also be recommended to gather all source repos in the server side in a virtual repo, so all the sources are available via a single URL? This is often a recommended good practice with Artifactory, to avoid the explosion of repo definitions in the client side (for all package managers, not only for Conan)

[ottar-azn]

Hi @memsharded , thanks for the reply

To expand on our use case
We're trying to overlap any number of repositories, the result of which is very individual specific, it doesn't make sense to use a virtual repository
i.e.,

User A requires access to repo's 1, 2, 3
User B requires access to repo's 1, 4, 5
User C requires access to repo's 2, 3, 5

Repositories of 1, 2, 3, 4, 5 are in different security domains, and should only be accessible to those within those domains
This would require a virtual repository for each user to satisfy the requirement - not practical

We can use the text matching pattern you suggested or merging of source credential files to work around the issue

I think from a user perspective its not intuitive that these two parts of the interface interact in this way, so if it's not a bug in the current design, we would want to request that as a feature.

[memsharded]

Ok, understood.

I am submitting #16425, to try to improve over this.

I think you are right, even if the initial design didn't consider the overlap, it might be better to take this into account. It requires a bit of propagation of information down to the interceptor, but thanks to the Conan 2 architecture, it is cleaner than I expected (I still have the Conan 1 codebase challenges in mind), so I am going to propose this to the team.

Thanks for the feedback!

[memsharded]

#16425 was merged, it will be included in next 2.5 release (end of month). Thanks again for the feedback!

[ottar-azn]

No worries, Thanks for addressing the issue.