Generic datasource

[rarkins]

What would you like Renovate to be able to do?

Support the concept of a "generic" datasource, so that users can create an API matching its schema to massage unsupported datasources into something Renovate understands.

Did you already have any implementation ideas?

We take our existing "internal" datasource schema - i.e. the type definition for how datasources return results - and make that external/public in the form of a JSON schema potentially. Then users can e.g. scrape sources and publish them to a static website for Renovate to read.

Example:

Let's say we want to know the version of python supported by pyenv, and those are kept internal to their repo. So we create our own repo, called pyenv-versions and add a GitHub Action to run hourly and "scrape" the pyenv repo and generate a simple JSON file representing the versions, and then publish that to GitHub pages.

I can see two structures being supported:

1. One JSON file per package, so then you fetch <registryUrl><lookupName>.json and use the result directly
2. A single JSON file for all packages, so you fetch <registryUrl> and then use the result which exists inside the lookupName key in the JSON.

[rarkins]

@JamieMagee this was inspired by the vcpkg discussion and could open up some new possibilities for custom datasources without requiring much work from us or users. Although I'm not sure that it's actually the right way for vcpkg itself.

[JamieMagee]

Yeah, I've been thinking that something like this might be the end goal of #8647. Once we abstract so much away, there'll be a lot less required to implement a new datasource.

[fgreinacher]

I like this a lot, see also #4674

[SirUli]

I've started to do something similar by faking a pypi registry for totally different software (some internal repositories). Didn't get far enough at the moment but i would definitely favor this approach. Looking forward to it!

[NargiT]

definitely something I will use, right now I forked the project and maintain a custom datasource in order to achieve this. it's very painful specially when upstream code is refactored.

[JamieMagee]

@NargiT do you mind if I ask what the datasource you need is and if you would be willing to contribute it back?

[NargiT]

hello @JamieMagee,

Sorry for my late reply. We have a home made system to manage application configuration. Basically it's a set of files that are package as tar.gz and versioned. This tool expose a REST API and we can download a specific version for a given application.

I cannot contribute to upstream because the tool is not open source. Instead I extends Datasource and build Releases classes. Thus I can use regexManagers with my custom datasource to update my custom yaml.

It's still work in progress.

[NargiT]

@rarkins I feel like your solution would still need implementation and I was wondering if a plugin type system is not better.

For example, in my case, I had to create a custom data-source in order to fulfill my need.
This has drawbacks if I want to use fully renovate the self-hosted version.

• build the package
• build the docker image
• build the github action

of course my implementation is highly coupled to the internal API vs a json file. But since this issue is opened since 1 year now maybe a more modular structure is easier to support generic data sources.

For example a --plugin-directory that could allow to extend the amount of supported datasources. This would allow me to use all the renovate infrastructure already put in place (docker image, github actions...) and reduce the pain to maintain the whole process on my side.

[rarkins]

Adding a plugin architecture would extend our "public API" to areas which are today considered internal. Doing so would make it harder to refactor when we need to and ultimately slow us down for the 99%+ who don't need plugins. We are still evolving our internal interfaces regularly.

I think we'll do it some day, but for now the tradeoff isn't worth it.

[viceice]

@NargiT you can do something like our internal-tools¹ action. It is using renovate from npm and adding versionings / datasource extensions. It's probably a lot easier to maintain for you instead to build the whole renovate 😉

Footnotes

1. https://github.com/renovatebot/internal-tools ↩

[rarkins]

Related: https://github.com/flathub/flatpak-external-data-checker#json-checker

[reitzig]

Use Case: As-code config of $softwareA depends on version of $softwareB that actually runs. Ops would have to store that version of B in a digestable, maintainable format for Renovate to pick up and inject into the config of A (via magic comment).

Also related: #4459