Pseudo repository support for any kind of files

[ThraaxSession]

Request details

It might be awesome to have extended webdav support, as some artifacts are only zip files.
Also it would be awesome if each directory with artifacts has something like a meta.xml. Which keeps track of the latest package + hashing.

So each time a new upload was successful, it's possible to keep the track of the latest zip release.
This came in my mind as I want to upload my web client to a repository. With the same advantages like a maven repo.

[dzikoysk]

I don't know much about webdav 🤔 It sounds a little bit complex, do you know about requirements and why do you think it could be helpful particularly in terms of Maven-based projects?

[ThraaxSession]

Hi @dzikoysk
Well, it needn't be webdav specially. Webdav might be really too complicated as you would save the artefacts on filesystem and expose them like remote filesystem. (davfs)
But something like a common repository for zip files would be helpful.
It's helpfully as developers have often various kinds of artefacts, not only jars. It would be helpful if I develop for example a client application for different kind of operating systems. I could bundle them individually but keep them together as one version. That's how I would imagine it:

Directory structure:
/release/my/project/client/xxx

myclient-linux-xxx.zip
myclient-windows-xxx.zip
myclient-web-xxx.zip
meta.xml

meta.xml:

// it keeps track of the lastest releases like for maven repositories.

[dzikoysk]

I think this kind of feature is quite specific and dependent on given scenario - in general Reposilite allows you to upload any file atm to the repository, so such behavior should be probably implemented on the client side, just like e.g. Maven client. So client deploys not only those zip files, but also generates meta.xml (with metadata about those files and probably even more data you can configure during build phase) and Reposilite just stores it, so this kind of logic is a part of client and server stays quite generic. Not sure how you're deploying those files, but e.g. Gradle is pretty convenient solution for customizing any part of your build process, so you could probably write a script/plugin that handles it and it should work way better than any logic on server.

[ThraaxSession]

The meta.xml I assume is generated by the server currently, isn't it? I would see to update this meta.xml to updated by server, not client. As the client doesn't know the current versions. Do you know what I mean? Sure, uploading isn't a problem. I could upload a bunch of files, no matter gradle or via REST. But keep track of the current files, I would assume that the server update those meta informations. Else I would need create a plugin or extension for each build system which tracks down this meta information.

[dzikoysk]

maven-metadata.xml files are generated by Maven client, not server, that's the procedure:

• User runs deployment
• Maven downloads current metadata files from repository
• Maven reads downloaded metadata files and adds new necessary stuff about new files
• Maven deploys updated metadata files as well as other files

Reposilite, and any other repository manager, is not modyfing metadata files as it's not its responsibility. That's healthy, because it reduces a need to "guess" what clients meant when they've uploaded specified files.

[ThraaxSession]

So actually it means I would need to create a plugin for different buildsystems. That's what my ticket is about. It's not about maven here. It's about a simple routine here to upload generic files. (It can be a Reposilite/Javaline plugin)

Procedure:
I upload my testicals via rest against something like PUT /api/upload/generic/{the repo}/{version}/file.
It gets the file, it updates the meta.xml and as it knows which repo and version.
And with GET /api/upload/generic/{the repo}/version or whatever I can retrieve the current files.
With another API like GET /api/download/generic/{the repo}/{file}/{version} or even GET /api/donwload/generic/{the repo}/{file}/latest I could download it. file = 'myclient-linux'

I hope it's more clear why it doesn't work on client side. It's too generic for maven/gradle. As Flutter/Bazel/Cmake or what ever doesn't work with it.

Plus it doesn't need to be a zip file. I could upload as *.tar.gz, *.tar and *.zp. even *.exe or *.appimage or something would work. That's why it would be awesome.

[dzikoysk]

I feel like you're more comfortable with this topic than I am, would you like to try to prototype this as plugin? In theory you should be able to listen for DeployEvent and generate some kind of metadata file on top of that.

[ThraaxSession]

I could try. :)

[dzikoysk]

What's the status of this issue? :)

[ThraaxSession]

@dzikoysk that's a good question! I hope I've got time next week in my vacations. Then I'll be home and got a quiet place for coding. :D

[xuanswe]

I am also searching for a generic solution like this issue ticket.

It would be nice to see everything as a generic artifact.

And then, if we need maven, we can enable maven registry plugin.
If we need docker images and helm charts, we can enable OCI registry plugin.
If we need npm packages, we can enable npm registry plugin.
If we need python packages, we can enable python registry plugin.
If we need debian OS packages, we can enable debian registry plugin.
If we need RPM OS packages, we can enable RPM registry plugin.
Etc.

Basically, it would be nice to have a "universal build artifact management" system like https://cloud.google.com/artifact-registry, but open-source, lightweight and self-host.

[dzikoysk]

By default, Reposilite does not generate any kind of files, so as long as there's no such requirement from given build system, it basically may work like that. Unfortunately, some ecosystems (like e.g. OCI - e.g. Docker) require dedicated API, and it'd require probably quite a lot of work to cover all of that (#1618). As far as I remember, this issue is about adding some extra metadata for non Maven builds, but I think it could be covered with 3rd party plugin these days 🤔.

[xuanswe]

I know, each ecosystem requires a lot of stuffs, so the artifacts will work with other tools of the same ecosystem. That's why I think Reposilite should be generic at least first.
And then, each ecosystem should be a separated plugin.
Given enough time, the community will develop all plugins for each ecosystem.
For undeveloped ecosystem, at least, we can have a generic artifact type as a workaround until there's a plugin.

[xuanswe]

By default, Reposilite does not generate any kind of files, so as long as there's no such requirement from given build system, it basically may work like that.

Maybe I didn't understand Reposilite. So, do you mean, actually generic artifact will work already with Reposilite? Do I need gradle or maven to work with generic artifact?

PS: I didn't try Reposilite. Only searching for a solution but didn't try anything yet.

[dzikoysk]

I know, each ecosystem requires a lot of stuffs, so the artifacts will work with other tools of the same ecosystem. That's why I think Reposilite should be generic at least first.
And then, each ecosystem should be a separated plugin.
Given enough time, the community will develop all plugins for each ecosystem.
For undeveloped ecosystem, at least, we can have a generic artifact type as a workaround until there's a plugin.

That's how it's implemented atm, our storage system is an independent implementation that can be used by other plugins:

• https://github.com/dzikoysk/reposilite/tree/main/reposilite-backend/src/main/kotlin/com/reposilite/storage

And Maven integration is implemented as a plugin on top of that:

• https://github.com/dzikoysk/reposilite/tree/main/reposilite-backend/src/main/kotlin/com/reposilite/maven
• https://github.com/dzikoysk/reposilite/blob/main/reposilite-backend/src/main/kotlin/com/reposilite/maven/application/MavenPlugin.kt

Ofc UI is written with an assumption that there's just Maven atm, because that's our primary target, but that's something that we could work on if needed.

Maybe I didn't understand Reposilite. So, do you mean, actually generic artifact will work already with Reposilite? Do I need gradle or maven to work with generic artifact?

All you need to upload content to Reposilite is to make PUT/POST (we support both even tho Maven uses only PUT) with credentials (token + secret) specified as basic auth and it'll accept anything you'd like to put in given repository. Example:

• https://maven.reposilite.com/files/assets/twitter.gif

[xuanswe]

Oh, nice! Then, this is exactly the solution I need.
Thanks for your very quick replies and detailed answers!!!
I am going to try Reposilite next week for my home lab.
And, thank for developing and open-source this awesome project.

[dzikoysk]

Sure, let me know how it works for you. Even if Reposilite won't fulfill your requirements, it could be nice to have some kind of feedback that we could reconsider during e.g. Reposilite 4.x etc. :)

[xuanswe]

Sure, let me know how it works for you. Even if Reposilite won't fulfill your requirements, it could be nice to have some kind of feedback that we could reconsider during e.g. Reposilite 4.x etc. :)

I did a quick try and I like Reposilite. However, I think it would be nicer if Reposilite could learn from https://docs.gitea.com/next/usage/packages/overview

So, basically for home lab, I think the best option would be just using gitea, which could be used for 3 purposes at once: git repository, artifact repository and CI.

For more complicated setup requirements, such as small business, we could

• replace gitea packages with Reposilite (when it's similar or better than gitea packages)
• replace gitea actions with ex. Drone

[dzikoysk]

Well, by default Reposilite targets jvm projects and that's the priority - I need to filter feature requests, because I'm simply unable to provide a good support for all of these tools. Build tools that don't require any extra handling should work ootb, but Integrations like e.g. containers (OCI) brings a whole new spec. Atm I can only say, that they're plans to support containers & npm, I just didn't have time for it yet.

[dzikoysk]

I feel like this issue is a bit outdated & doesn't request a particular feature. Reposilite is able to host non-Maven related files, so in general it can be used as a generic repository for any kind of files. If someone has some issues related to that area, feel free to open a new issue that is focused on a specific use-case - it'll be easier to figure out further steps for us in terms of that support :)