New S3-plugin features for S3 artifacts management

[masterzen]

Hi,

Here is a pull request to add to the S3 plugin several features to allow S3 to become a more integrated repository for artifacts built by jenkins. The idea is for S3 to become a substitute of the master artifact storage area.

This pull request notably adds the following features:

• allow to upload from the slave (this is essential if like me you are
  using EC2 slaves with a master outside EC2, to prevent uploading from slave to master then from master to S3).
• S3 uploaded artifacts fingerprinting (to allow finding what builds produced what artifacts)
• S3 Copy artifact build step, based on the "Copy Artifact" plugin, it allows to download an artifact from a build using the S3 publisher plugin back into the workspace. This also produces fingerprints to link jobs together.
• automatically delete artifacts in S3 from deleted jobs/builds.
• Direct S3 download links of the artifacts in the job page or build home page.

This code is in production for about one week on my work jenkins installation with great success.

Please review this changeset, I'd be happy to modify it so that it can be merged upstream.
Many thanks!

Brice

[d6y]

Thank you - I can now build and create the s3.hpi file. I'll test some more later in the week.

[d6y]

Disappointingly I've not been able to progress testing this week. I thought I'd be able to this weekend, but I'm going to be away and I'm not sure how much time I'm going to get to work on this. But I've not forgotten about this request! Just apologising for not moving faster on it.

[d6y]

I tried to drop the s3.hpi into my ~/.hudson/plugins directory under Jenkins 1.456 and the received the following errors during startup:

Mar 25, 2012 6:14:11 PM jenkins.InitReactorRunner$1 onTaskFailed 
SEVERE: Failed Loading plugin s3
java.io.IOException: Dependency copyartifact (1.21) doesn't exist
at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:446)
at hudson.PluginManager$2$1$1.run(PluginManager.java:305)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
at jenkins.model.Jenkins$6.runTask(Jenkins.java:840)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

and

Mar 25, 2012 6:14:14 PM jenkins.InitReactorRunner$1 onTaskFailed  
SEVERE: Failed Initializing plugin s3
java.lang.NullPointerException
at hudson.PluginManager$2$1$2.run(PluginManager.java:322)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
at jenkins.model.Jenkins$6.runTask(Jenkins.java:840)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

I did this in the hope of retaining my existing S3 plugin settings.

I then removed in ~/.hudson/plugin/s3* files and folders and restarted and uploaded the s3.hpi from the advanced tab of the plugin manager in Jenkins. Doing this produced the "Failed Initializing plugin s3" error above.

I removed my ~/.hudson directory and tried again, and I still get the above errors.

I rebuilt the s3 plugin against 1.456 and the same problem continued.

Can you suggest what I should do next?

[masterzen]

The patch adds the S3 Copy Artifact feature that unfortunately requires to have the Copy Artifact plugin installed. This is I guess the cause of the first stack trace.
Can you try to install this plugin and try again?

We could theoretically remove this dependency, but we'll have to copy or re-implement code from this plugin.

Normally your previous S3 plugin settings should be compatible.

[d6y]

Thank you - I'll try that.

A thought: is there a way to add the copyartifact code as a library dependency in the POM? That is, use the code from the plugin without having to make the user install a separate plugin? I don't know if that's even possible.

[masterzen]

It is already in the POM, as I've followed these instructions:
https://wiki.jenkins-ci.org/display/JENKINS/Dependencies+among+plugins

[d6y]

Thank you - I've installed the plugin, ran a project, and it produced files on S3.

One change is the output on S3 is in a structure based on the project name and build number. In my case it's called: jobs/oth/5/output-1.0-SNAPSHOT.jar

That's a big change from the current plugin which leaves files in the top-level of the bucket you specify. Is there a setting to allow the plugin to produce the older behaviour?

[masterzen]

No, there is no option to restore the old behavior (it looks like I completely forgot this).
I'll add an option in a subsequent update, but if enabled, the user won't be able to use the S3 Copy Artifact part of the plugin (because we need to separate artifacts produced by different jobs).

[d6y]

Ok! Sounds good. I'll look out for that update.

[d6y]

Is there any progress on this change set or should we close this one?

[ndeloof]

incompatible change to bucket layout is a blocker imho. User expect plugin to be backward compatible and this will break the build chain. Should be an option, that S3CopyArtifact enables when used, but not the "classic" publisher (by default)

[masterzen]

Grr, I missed your last two comments because of the new github notifications.
I do agree that the default should be compatible with previous versions.
I just need to find enough time and motivation to finally fix that :)

[diranged]

This would really be great to have merged in... Any chance we can motivate you to get it pulled in?

[masterzen]

That's still on my list of things to do :)

Meanwhile you can build my patched version if you really need the functionality (I'm running it on production without any issues since I proposed this PR).

[jrabbe]

If you need motivation, @masterzen then have another vote. The current behavior of the S3 plugin is a huge blocker for us, since we need the hierarchy in our S3 bucket for our releases.

[masterzen]

@jrabbe I'm motivated :)
I just need to find the time to bring back the backward compatibility.

Meanwhile you can clone the repository, build the plugin (mvn hpi:hpi) and install it manually in your jenkins instance.

We're using it in production to host our artifacts on s3 for more than 2 years without any adverse effect. It is the base of some very complex pipelines where artifacts are stored in s3 and automatically downloaded in the next stage of the pipeline, etc...

[dmbeer]

Hi Guys,

I currently use the s3-plugin as is for deploing finished artifacts and delivering them afterwords via cloudfront or s3. We need to maintain the old capability, if anything it might be better to make the feature a separate plugin that depends on the s3-plugin. So that you have a customised s3-plugin for default artifact storage that does not interfere with standard s3-plugin upload and settings.

David

[masterzen]

@dmbeer it's easy to restore the original behavior (it's still there in fact), it just lacks the UI.

[masterzen]

Guys, I finally found the motivation to cleanup and finish this work, mostly clean up and restoring the original behavior of the plugin.

This happened in Pull Request #19.

Therefore I'm closing this one, as it has no object anymore.