Add [Conan] version service

[jtbandes]

Adds a new Version service for Conan Center packages.

Example: /conan/v/zeromq (https://conan.io/center/zeromq)

[shields-ci]

	Messages
📖	✨ Thanks for your contribution to Shields, @jtbandes!

Generated by 🚫 dangerJS against ef21b59

[calebcartwright]

Thanks for the PR! Are you aware of any documentation for the API? That's something we typically strive for with new services given the breadth of integration points we have, and I think it's going to be especially important in this case given what looks like non-trivial processing of the response

[jtbandes]

Unfortunately I didn’t find much. However, the conan CLI tool is a client and is totally open source: https://github.com/conan-io/conan

I believe my code mirrors what happens when you run conan search at the command line.

[jtbandes]

There's also some documentation at https://docs.gitlab.com/ee/api/packages/conan.html#search

By the way, does shields.io do any sort of caching? I found that some of these responses are slow for certain packages (such as boost) so I was wondering if the server does anything to reduce load for slow requests.

[calebcartwright]

By the way, does shields.io do any sort of caching? I found that some of these responses are slow for certain packages (such as boost) so I was wondering if the server does anything to reduce load for slow requests.

We do cache pretty extensively, but that's done post-badge creation (local/browser cache, CDN, etc.). We don't cache raw responses from upstream providers though, so a request for the same badge doesn't result in the upstream API call that often, but if there happens to be two separate badges that make the same API call then that can indeed happen twice.

Do you have a ballpark for how slow they are? The vast majority of the badges we provided are rendered by GitHub, and due to the way GitHub handles (and times out) on static content requests, we typically need the entire request/response loop for a badge to complete in 3-4 seconds max

[jtbandes]

I don't have a lot of data, but I just tried a handful of packages, and...it seems to vary a lot. Maybe their server is doing some caching, and/or maybe it depends on the number of versions of the package (although I didn't see any responses returning particularly large numbers of versions).

I tried querying boost, and got >10sec one time, but subsequent requests were ~4-5 sec.
Querying for nlohmann_json seems to be <1sec.
Querying for poco was >10sec the first time, and subsequent requests are ~1sec.
Querying for small, unpopular libraries (e.g. absent) seem to be <1sec.

Here's how you can test it out:

docker run -it --rm python:3.10 /bin/bash
pip install conan
time conan search -r conancenter nlohmann_json

I'll reach out to some conan folks and see if they can comment on whether there's a better API to use.

[calebcartwright]

Those numbers are rather concerning, and certainly give me pause. I'd be curious to know why the API is so sluggish, any chance there's some additional info we could send in the request that would reduce the scope of work on the backend and improve performance?

[jtbandes]

So far on the Conan Slack I haven't learned of any better solutions. I posted a question at conan-io/conan#10298.

Is this PR unshippable with the current API response times?

[calebcartwright]

Thanks for opening that issue! In my opinion, yes, the current response times aren't going to be acceptable, so let's put this on hold for a bit and wait to see how things evolve upstream

[jtbandes]

Hi @calebcartwright – based on other discussions around the upstream issue, I've given up on the Conan API and have reworked this version service to use the GitHub API to access each package's version info config.yml directly from conan-io/conan-center-index. This was recommended as a workaround at renovatebot/renovate#12009 (comment).

Looks like my attempt at mocking the github response has failed in CI (but it passes locally, probably due to the use of ConditionalGithubAuthV3Service), would appreciate any guidance on how to fix that! It looks like none of the other services that use fetchJsonFromRepo/fetchRepoContent are using mocks in their tests? 😟

[calebcartwright]

Not sure off hand, but nock is understandably very pedantic so the slightest variation in the actual request vs. the configured request in the test will result in the request being fired off.

Something I like to do when I'm having trouble getting nock to intercept is to enable TRACE_SERVICES when running the specific tests so that I can see the full output, e.g. TRACE_SERVICES=true npm run test:services -- --only conan

[jtbandes]

Ok, just added some conditional logic, that seems to do it! Please take a look when you get a chance 😄

[calebcartwright]

Thanks for following up, this does seem like a more viable path! A few thoughts/questions for you:

• We have some existing utilities for handling conditional skips of tests, check out skipWhen and noToken (e.g. Wheelmap tester)
• However, I think some refactoring to apply our common separation of concerns can simplify the testing story. Many of our classes incorporate a separate transform function that's responsible for taking the raw response and extracting/transforming/etc. as needed to get the desired value(s) for rendering. That not only helps clean up the code, but also makes that logic much more easily tested with a unit test (and often negates the need for the e2e style tests with nock intercepts)
• Could you explain why the custom version comparison is needed? Admittedly I've not done a line by line review, but more curious what necessitates it in the first place (as opposed to one of our existing version-related functions, semver, etc.). Will also confess I know of conan, but don't really know anything about it, including whether it applies any versioning conventions
• Fetching a yaml file from a GitHub repo seems like something with greater reuse/generic potential. No need for you to worry about that necessarily in this PR, but making a note while I'm thinking of it so that we can follow up at some point down the road

[jtbandes]

…I think some refactoring to apply our common separation of concerns can simplify the testing story. Many of our classes incorporate a separate transform function that's responsible for taking the raw response and extracting/transforming/etc. as needed to get the desired value(s) for rendering.

Thanks for the suggestion, I've removed the response mocking and just refactored to a separate function that processes the yaml, which can be tested in mocha.

Could you explain why the custom version comparison is needed? Admittedly I've not done a line by line review, but more curious what necessitates it in the first place (as opposed to one of our existing version-related functions, semver, etc.). Will also confess I know of conan, but don't really know anything about it, including whether it applies any versioning conventions

I also wasn't very closely familiar with Conan's version conventions. Basically I think each package is allowed to put whatever it wants as version names, there's no explicitly enforced format. So I was initially trying to match the logic they use here for version comparison: https://github.com/conan-io/conan/blob/0e22f9e84bb2953ef48de8e32decc5a1a3613fb1/conans/model/version.py#L131

However, after taking a look at the existing version parsing functionality in this repo, I decided to just use latest(), which requires much less custom code and should probably work fine for most packages.

Thanks for your thorough review!

[calebcartwright]

I also wasn't very closely familiar with Conan's version conventions. Basically I think each package is allowed to put whatever it wants as version names, there's no explicitly enforced format. So I was initially trying to match the logic they use here for version comparison: https://github.com/conan-io/conan/blob/0e22f9e84bb2953ef48de8e32decc5a1a3613fb1/conans/model/version.py#L131

However, after taking a look at the existing version parsing functionality in this repo, I decided to just use latest(), which requires much less custom code and should probably work fine for most packages.

Gotcha. If the simple route looks "right" then I'm fine with starting there 👍 We can always revisit and revert back to the more customized route if need be

[calebcartwright]

Thanks again for sticking with this! Things generally LGTM and I think this is on the verge of being ready. Few minor comments left inline below

[calebcartwright]

lgtm, thank you!

[jgsogo]

Hi there! (Here the team lead of ConanCenter)

Thanks a lot for this implementation! It is wonderful to see people out there implementing things from our own backlog 😅 We were already starting an effort to provide badges for the libraries in ConanCenter, using Shields.io. Besides the typical badge for each library version, we were also going to provide the latest version (the one implemented here) and maybe some other badges with extra data like the number of downloads or platforms where the library is available.

For dynamic badges, our plan was to use the endpoint implementation with an endpoint provided by us that returns the latest version for the given library, or the number of downloads,... It is out of scope right now to create a stable API to get information from ConanCenter, but we can create these very few endpoints with specific information, especially if they are going to be used by some cached services like Shields.io.

As you can read here (conan-io/conan-center-index#9418) we also have the problem of knowing which one is the latest version. We do our best in that config.yml file but it's not a silver bullet and may contain errors for some libraries. Anyway, that's probably the best proxy right now.

I would like to know how we can contribute, or modify our backlog so we can better serve our community. I would like to know (from the point of view of shields resources) if it is better to implement the badges using that shields-endpoint approach, or better to modify the logic here after we have the endpoint ready, or... well, we are open to any feedback or suggestions.

Thanks!

[jtbandes]

Summarizing our offline discussion from Cpplang Slack:

• This PR originally used the Conan search API to get the list of versions, but that was slow (see [question] Why is /v1/conans/search so slow? Is there a faster alternative? conan-io/conan#10298) and also required some difficult parsing logic around the version “references”. If you are able to add an API on the Conan side to directly return the latest version, this code could be modified to fetch that instead of fetching the YAML from GitHub.

• This PR originally tried to match the version comparison logic from the Conan python code, but changed to use the shields.io latest() function already in this repo since that seemed simplest and would work for the vast majority of libraries. A direct API call would be great.

• Personally I think the https://img.shields.io/conan/v/boost url is nicer than using the /endpoint feature. But I think you should get the shields maintainers’ opinion on when it is appropriate to add badge services to this repo vs. using the /endpoint feature. I was thinking conan could mimic npm and other package managers that have multiple shields services for versions, downloads, etc: https://github.com/badges/shields/tree/master/services/npm

  • Overall I think it’s a better design for Conan to offer APIs for “package stats” (latest version, number of downloads, etc), and update this code to extract the required info from that API response, rather than structuring the API response around the shields.io /endpoint expectations.

• If you need to authenticate API calls, it seems like this repo is set up to use various tokens per service, and I imagine you could add one for conan: https://github.com/badges/shields/blob/master/config/local.template.yml

[calebcartwright]

I would like to know (from the point of view of shields resources) if it is better to implement the badges using that shields-endpoint approach

Thanks for reaching out @jgsogo! In short, there's probably some easier/simpler options that don't require a full blown Endpoint badge, as the Endpoint model and schema isn't how we typically like to provide native badges anyway.

At a high level I'd describe the Endpoint badge as being a good solution for more individual, and often power user, type use cases. Examples would include scenarios where users want to manipulate our native service badges beyond the customization options we're willing/able to support, or where a user wants to create a badge with data from a platform that requires an unscopable, user-specific authorization mechanism that we can't utilize in the SaaS Shields.io. In many ways it's a more powerful/extensible version of our Dynamic Json/Xml/Yaml badges.

For our more first-class badges for various platforms, we strongly prefer to consume a standard API surface that provides the raw platform specific data for a package as opposed to the type of contract we expect on an Endpoint which is focused on rendering concerns (label, color, text of message, etc.). This helps provide better ergonomics to our users and ensures that the badges we provide for upstream platforms and services are consistent with our design goals and specification by default.

That being said, if Conan's documented API service includes an endpoint that delivers those key data points (version, downloads, etc.) that's perfect and we can certainly use those; just want to stress there's no need to force your API and response schema to align with our Endpoint Badge model. It's easier for us to e.g. grab a numeric value from a downloads or downloadCount type of field than to parse a number out of a '700 downloads' string in a message key, plus we'd ignore the other attributes in an Endpoint-style response like color/label/etc. anyway.

or better to modify the logic here after we have the endpoint ready, or... well, we are open to any feedback or suggestions.

Whenever there's an API ready for consumption anyone should feel to drop a note either here or in a new issue, or even submit a PR to change the implementation to take advantage. It shouldn't be too big a lift from the sounds of it, though we're definitely open to having some more detailed discussions if there's other considerations or topics of interest, including any concerns around request volume (I gather from some of the other threads there may be)

[jgsogo]

Thanks a lot for your words, @calebcartwright . We are currently working on that API, so we will adapt to your suggestion, it is also better for us to provide actual and meaningful data than to prepare an endpoint serving the fields requested by your shields-endpoint implementation (as you said, more about rendering concerns).

Just one more question before moving to actual implementation. We would like to keep these endpoints under control, so the suggestion about using a TOKEN would be great for us, or we can add shields-IPs to some whitelist so they can get to the actual endpoints. Which approach do you think is better if the API cannot be public?

Thanks!

[chris48s]

the suggestion about using a TOKEN would be great for us, or we can add shields-IPs to some whitelist so they can get to the actual endpoints. Which approach do you think is better if the API cannot be public?

The short answer is a token is preferable. There is a longer thread about why that is here #7574

[ilya-lavrenov]

Hi,
Is there a way to have a badge with number of package downloads from Conan's CCI ?