s3Upload() doesn't work if `includePathPattern` matches multiple files #83

swisspol · 2018-05-02T06:29:44Z

Version 1.26

Assuming we have the following content in the build directory:

Houseparty-arm64-v8a.apk
Houseparty-armeabi-v7a.apk
Houseparty-x86.apk
mapping.txt

And we want to upload it to S3:

This works:

s3Upload(bucket: 'hp-client-builds', path: "jobs/$JOB_NAME/$BUILD_NUMBER/", includePathPattern: '*-arm64-v8a.apk', workingDir: 'build')
s3Upload(bucket: 'hp-client-builds', path: "jobs/$JOB_NAME/$BUILD_NUMBER/", includePathPattern: '*-armeabi-v7a.apk', workingDir: 'build')
s3Upload(bucket: 'hp-client-builds', path: "jobs/$JOB_NAME/$BUILD_NUMBER/", includePathPattern: '*-x86.apk', workingDir: 'build')
s3Upload(bucket: 'hp-client-builds', path: "jobs/$JOB_NAME/$BUILD_NUMBER/", includePathPattern: '*.txt', workingDir: 'build')

This doesn't work and only uploads mapping.txt:

s3Upload(bucket: 'hp-client-builds', path: "jobs/$JOB_NAME/$BUILD_NUMBER/", includePathPattern: '*.apk', workingDir: 'build')
s3Upload(bucket: 'hp-client-builds', path: "jobs/$JOB_NAME/$BUILD_NUMBER/", includePathPattern: '*.txt', workingDir: 'build')

This doesn't work either and doesn't upload anything:

s3Upload(bucket: 'hp-client-builds', path: "jobs/$JOB_NAME/$BUILD_NUMBER/", includePathPattern: '*', workingDir: 'build')

The text was updated successfully, but these errors were encountered:

hoegertn · 2018-06-17T17:02:26Z

can you try again with 1.27

llater · 2018-07-12T23:27:41Z

Version 1.27

I have the build directory:

Houseparty-arm64-v8a.apk
Houseparty-armeabi-v7a.apk
Houseparty-x86.apk
mapping.txt

I want to upload it to S3:

s3Upload(bucket: 'hp-client-builds', path: "jobs/$JOB_NAME/$BUILD_NUMBER/", includePathPattern: '*.apk', workingDir: 'build')
s3Upload(bucket: 'hp-client-builds', path: "jobs/$JOB_NAME/$BUILD_NUMBER/", includePathPattern: '*.txt', workingDir: 'build')

It only uploads mapping.txt. Still unresolved in 1.27

ryanneufeld · 2018-09-13T17:43:30Z

This issue is still unresolved in 1.31

hoegertn · 2018-09-13T18:44:42Z

Can you please describe your setup? I cannot reproduce the problem.

ryanneufeld · 2018-09-13T19:56:14Z

My pipeline looks similar to this:

dir('build') {
    withEnv(['GIT_SSH=run_ssh.sh']) {
        sh """
            ./make-package
            // This produces ${BUILD_PARENT}.tar.gz
            // This produces ${BUILD_PARENT}.tar.gz.sha1
        """
    }
}
withAWS(credentials: 'dash-build-s3upload', region: 'us-west-1') {
    s3Upload bucket: 's3bucketname', includePathPattern: "*tar*", workingDir: 'build'
}

I can confirm that the tar files are in the build directory.

And the following is what i'm actually using:

s3Upload bucket: 's3bucketname', file: "${BUILD_PARENT}.tar.gz", workingDir: 'build'
s3Upload bucket: 's3bucketname', file: "${BUILD_PARENT}.tar.gz.sha1", workingDir: 'build'

CroModder · 2018-11-13T19:01:44Z

Seems it's known bug but still waiting for taking into consideration:
https://issues.jenkins-ci.org/browse/JENKINS-47046

One of the workaround would be usage of pipeline findFiles:
https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#findfiles-find-files-in-the-workspace

hoegertn · 2018-11-13T19:40:55Z

The problem is that I cannot reproduce it on my test setup to debug the root cause.

CroModder · 2018-11-14T10:27:27Z

The problem is that I cannot reproduce it on my test setup to debug the root cause.

Is there an online playground for jenkins pipeline, or some other ways how to share the whole build job? Because the setup that is failing for me is literally the official example:
s3Upload(bucket:"my-bucket", path:'path/to/targetFolder/', includePathPattern:'**/*', workingDir:'dist', excludePathPattern:'**/*.svg')

kraenhansen · 2018-11-16T12:57:09Z

I am also seeing this error :/

eplodn · 2019-01-03T12:24:42Z

Same here.

This works:

pipeline{
    agent { node { label 'jenkins-host' } }
    stages {
        stage('test') {
            steps {
                script {
                    sh "rm -rf txt_dir || true"
                    sh "echo test1 >> test1.txt"
//                    sh "echo test2 >> test2.txt"
                    sh "mkdir -p txt_dir"
                    sh "mv *txt txt_dir" 
                    archiveArtifacts allowEmptyArchive: true,
                        artifacts: "**/*txt",
                        caseSensitive: false,
                        defaultExcludes: false,
                        onlyIfSuccessful: false
                
                    withAWS(endpointUrl:'http://100.64.0.165:9000',  // local minio.io
                            credentials:'128e57fa-140a-4463-ad37-b3821371f735') {
                        s3Upload bucket:'jenkins', path:"build-${env.BUILD_NUMBER}/", 
                                includePathPattern:'**/*txt', workingDir: "${env.WORKSPACE}"
                    }
                }
            }
        }
    }
}

Take out the comment on sh "echo test2 >> test2.txt" and it doesn't work.
It also doesn't say it's failed. Just "Upload complete".

Is there something I can do on my end to allow debugging/log files/etc.?

brockers · 2019-01-16T14:19:46Z

Having the same problem. Trying to upload the contents of an entire folder to the root of the S3 bucket, where the files list is something like:

ls -l assets/marketing/
-rw-rw-r--. 1 jenkins jenkins  85598 Jan 11 16:52 ai_logo.png
-rw-rw-r--. 1 jenkins jenkins   1559 Jan 11 16:52 favicon-16x16.png
-rw-rw-r--. 1 jenkins jenkins   2366 Jan 11 16:52 favicon-32x32.png
-rw-rw-r--. 1 jenkins jenkins   1150 Jan 11 16:52 favicon.ico
-rw-rw-r--. 1 jenkins jenkins 180092 Jan 11 16:52 header.jpg
-rw-rw-r--. 1 jenkins jenkins   3635 Jan 15 13:19 index.html
-rw-rw-r--. 1 jenkins jenkins  15173 Jan 11 16:52 logo.png
-rw-rw-r--. 1 jenkins jenkins    268 Jan 15 10:48 README.md
-rw-rw-r--. 1 jenkins jenkins    487 Jan 11 17:35 ribbon.css
-rw-rw-r--. 1 jenkins jenkins   1825 Jan 11 16:52 style.css

The following will fail to upload anything:

stage('deploy') {
	when {
		branch 'master'
	}
	steps {
		withAWS(endpointUrl:'https://s3.amazonaws.com', credentials:'aws_cred_id'){
			s3Upload(bucket:'staticwebsite-bucket', path:'', includePathPattern: '*', workingDir: 'assets/website/', acl:'PublicRead')
		}
	}
}

And this version will upload only one of each file type:

stage('deploy') {
	when {
		branch 'master'
	}
	steps {
		withAWS(endpointUrl:'https://s3.amazonaws.com', credentials:'aws_cred_id'){
			s3Upload(bucket:'staticwebsite-bucket'', path:'', includePathPattern: '*.css', workingDir: 'assets/website/', acl:'PublicRead')
			s3Upload(bucket:'staticwebsite-bucket'', path:'', includePathPattern: '*.png', workingDir: 'assets/website/', acl:'PublicRead')
			s3Upload(bucket:'staticwebsite-bucket'', path:'', includePathPattern: '*.jpg', workingDir: 'assets/website/', acl:'PublicRead')
			s3Upload(bucket:'staticwebsite-bucket'', path:'', includePathPattern: '*.ico', workingDir: 'assets/website/', acl:'PublicRead')
		}
	}
}

brockers · 2019-01-18T03:19:01Z

The only useful work-around while still using withAWS and s3Upload is currently by using a findFiles glob and looping through the resulting list. It works fine, and will make it easy to convert a Jenkinsfile over when s3Upload gets fixed, but here is an example for anyone else:

stage('deploy') {
	when {
		branch 'master'
	}
	steps {
		script {
			FILES = findFiles(glob: 'assets/website/**')
			withAWS(endpointUrl:'https://s3.amazonaws.com', credentials:'aws_cred_id'){
				FILES.each{ item -> 
					s3Upload(bucket: 'staticwebsite-bucket', acl: 'PublicRead', path: '', file: "${item.path}")
				}
			}
		}
	}
}

weidonglian · 2019-01-20T00:40:13Z

The only useful work-around while still using withAWS and s3Upload is currently by using a findFiles glob and looping through the resulting list. It works fine, and will make it easy to convert a Jenkinsfile over when s3Upload gets fixed, but here is an example for anyone else:
stage('deploy') {
	when {
		branch 'master'
	}
	steps {
		script {
			FILES = findFiles(glob: 'assets/website/**')
			withAWS(endpointUrl:'https://s3.amazonaws.com', credentials:'aws_cred_id'){
				FILES.each{ item -> 
					s3Upload(bucket: 'staticwebsite-bucket', acl: 'PublicRead', path: '', file: "${item.path}")
				}
			}
		}
	}
}

It does not keep the relative path, it will upload files in every sub folders into the root of the bucket.
Is there an easy way to get the relative path.
Really hope this ticket will be fixed soon.

brockers · 2019-01-22T02:05:25Z

Hey @weidonglian I haven't tried it but findFiles glob includes

${item.directory}

As a value. I would check that out (you may have to remove $PWD) and then put the value in for path:

I know it is a awful work-around, but it is the only thing I've been able to make work so far.

ryan-summers · 2019-02-21T17:29:00Z

I believe I may be able to clarify the issue a bit here. The problem appears to arise on Windows agents, but not on *nix agents.

Example pipeline

pipeline {
    agent { label 'master' }

    stages {
        stage('Generic files')
        {
            steps {
                dir('test') {
                    writeFile file: 'test.csv', text: 'fake csv file for testing'
                    writeFile file: 'test.log', text: 'fake log file for testing'

                    dir ('results') {
                        writeFile file: 'test.csv', text: 'fake csv file within results directory'
                        writeFile file: 'test.log', text: 'fake log file within results directory'
                    }
                }
            }
        }
    }

    post {
        always {
            withAWS(credentials: 'MY_CREDENTIALS', region: 'MY_REGION') {
                s3Upload(bucket: "test-bucket", includePathPattern: "test/results/*", path: "test-dir/")
                s3Upload(bucket: "test-bucket", includePathPattern: "test/results/test*", path: "test-dir/")
                s3Upload(bucket: "test-bucket", includePathPattern: "test/test*", path: "test-dir/")
            }
        }
    }

When running on a *nix agent, the commands all properly upload files.

When running on a windows agent (e.g. change master agent to a windows-specific agent), none of the s3Upload commands will upload a file.

hoegertn · 2019-02-21T17:35:05Z

Can you try what happens if you use \ as path separator?

swisspol · 2019-02-21T17:40:42Z

This bug does happen on Linux.

…

On Thu, Feb 21, 2019 at 9:35 AM Thorsten Hoeger ***@***.***> wrote: Can you try what happens if you use \ as path separator? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#83 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAmUy8p6KaRr-zQGLseuyBDnWcSrg-buks5vPtjKgaJpZM4Tu66k> .

ryan-summers · 2019-02-21T17:53:52Z

Using \\ as the path separator in the pipeline does not make the problem go away on a Windows agent. I can reliably upload using the above pipeline on Linux, but it will fail every time from a windows agent.

A potential work-around for this issue (if both windows-like and unix-like nodes are available) is to stash files on the Windows node and unstash them to a *nix node before the s3Upload command is executed.

ryanneufeld · 2019-02-21T23:16:09Z

@ryan-summers it totally happens on *nix agents as well. In fact, I didn't even know you could use a windows agent.

ryan-summers · 2019-02-21T23:18:32Z

Perhaps the windows vs. *nix agents upload is a different issue then.

smastrorocco · 2019-02-28T21:33:16Z

Any progress on this? Running into same issue on *nix agents as well.

tommymccallig · 2019-03-27T17:31:20Z

Seems to be an issue only on slaves. If you run the exact same pipeline on the master, it appears to work. See: https://issues.jenkins-ci.org/browse/JENKINS-44000

everactivetim · 2019-04-03T14:53:09Z

I was having this exact same problem, building the plugin from source resolved my issue (docker slaves).

techarchconsulting · 2019-04-26T21:49:41Z

This works fine in slave too..

Example Pipeline stage

stage('Deployment')
{
steps
{
script
{
def files = findFiles(glob: 'build/.')
withAWS(region:'us-east-1',credentials:'AutoDeployer')
{
files.each {s3Upload(file:"${it}", bucket:'mymnr.dev', path:"",pathStyleAccessEnabled:true, payloadSigningEnabled:true, acl:'PublicRead')}
}

                files = findFiles(glob: 'build/static/css/*.*')                    
                withAWS(region:'us-east-1',credentials:'AutoDeployer')
                {
                        files.each {s3Upload(file:"${it}", bucket:'mymnr.dev', path:"static/css/",pathStyleAccessEnabled:true, payloadSigningEnabled:true, acl:'PublicRead')}
                }
                
                files = findFiles(glob: 'build/static/js/**')
                withAWS(region:'us-east-1',credentials:'AutoDeployer')
                {
                        files.each {s3Upload(file:"${it}", bucket:'mymnr.dev', path:"static/js/",pathStyleAccessEnabled:true, payloadSigningEnabled:true, acl:'PublicRead')}
              
                }
            }
        }
    }

rudionrails · 2019-06-03T13:24:19Z

Hi, is there any progress on this or has it been resolved? I am on v1.36, using it to separate regular vs gz files. perhaps I am using it the wrong way also. Help appreciated.

    stage('Deploy') {
      when {
        branch "master"
      }

      steps {
        s3Upload(
          bucket: "${S3_BUCKET}",
          path: "${S3_PATH}",
          workingDir: "dist",
          includePathPattern: "**/*",
          excludePathPattern: "**/*.gz",
          acl: "PublicRead"
        )

        s3Upload(
          bucket: "${S3_BUCKET}",
          path: "${S3_PATH}",
          workingDir: "dist",
          includePathPattern: "**/*.gz",
          contentEncoding: "gzip",
          acl: "PublicRead"
        )
      }
    }

kraenhansen mentioned this issue Nov 16, 2018

Release artifact upload fixed again realm/realm-studio#1013

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

s3Upload() doesn't work if `includePathPattern` matches multiple files #83

s3Upload() doesn't work if `includePathPattern` matches multiple files #83

swisspol commented May 2, 2018

hoegertn commented Jun 17, 2018

llater commented Jul 12, 2018

ryanneufeld commented Sep 13, 2018

hoegertn commented Sep 13, 2018

ryanneufeld commented Sep 13, 2018 •

edited

Loading

CroModder commented Nov 13, 2018

hoegertn commented Nov 13, 2018

CroModder commented Nov 14, 2018

kraenhansen commented Nov 16, 2018

eplodn commented Jan 3, 2019 •

edited

Loading

brockers commented Jan 16, 2019

brockers commented Jan 18, 2019 •

edited

Loading

weidonglian commented Jan 20, 2019

brockers commented Jan 22, 2019

ryan-summers commented Feb 21, 2019 •

edited

Loading

hoegertn commented Feb 21, 2019

swisspol commented Feb 21, 2019 via email

ryan-summers commented Feb 21, 2019 •

edited

Loading

ryanneufeld commented Feb 21, 2019

ryan-summers commented Feb 21, 2019

smastrorocco commented Feb 28, 2019

tommymccallig commented Mar 27, 2019

everactivetim commented Apr 3, 2019

techarchconsulting commented Apr 26, 2019

rudionrails commented Jun 3, 2019 •

edited

Loading

s3Upload() doesn't work if includePathPattern matches multiple files #83

s3Upload() doesn't work if includePathPattern matches multiple files #83

Comments

swisspol commented May 2, 2018

hoegertn commented Jun 17, 2018

llater commented Jul 12, 2018

ryanneufeld commented Sep 13, 2018

hoegertn commented Sep 13, 2018

ryanneufeld commented Sep 13, 2018 • edited Loading

CroModder commented Nov 13, 2018

hoegertn commented Nov 13, 2018

CroModder commented Nov 14, 2018

kraenhansen commented Nov 16, 2018

eplodn commented Jan 3, 2019 • edited Loading

brockers commented Jan 16, 2019

brockers commented Jan 18, 2019 • edited Loading

weidonglian commented Jan 20, 2019

brockers commented Jan 22, 2019

ryan-summers commented Feb 21, 2019 • edited Loading

hoegertn commented Feb 21, 2019

swisspol commented Feb 21, 2019 via email

ryan-summers commented Feb 21, 2019 • edited Loading

ryanneufeld commented Feb 21, 2019

ryan-summers commented Feb 21, 2019

smastrorocco commented Feb 28, 2019

tommymccallig commented Mar 27, 2019

everactivetim commented Apr 3, 2019

techarchconsulting commented Apr 26, 2019

rudionrails commented Jun 3, 2019 • edited Loading

s3Upload() doesn't work if `includePathPattern` matches multiple files #83

s3Upload() doesn't work if `includePathPattern` matches multiple files #83

ryanneufeld commented Sep 13, 2018 •

edited

Loading

eplodn commented Jan 3, 2019 •

edited

Loading

brockers commented Jan 18, 2019 •

edited

Loading

ryan-summers commented Feb 21, 2019 •

edited

Loading

ryan-summers commented Feb 21, 2019 •

edited

Loading

rudionrails commented Jun 3, 2019 •

edited

Loading