Skip to content
This repository has been archived by the owner on Oct 1, 2024. It is now read-only.

Files copied to buildroot do not preserve symlinks #42

Closed
Icedmang opened this issue Feb 15, 2016 · 21 comments
Closed

Files copied to buildroot do not preserve symlinks #42

Icedmang opened this issue Feb 15, 2016 · 21 comments
Milestone

Comments

@Icedmang
Copy link

We have a number of files checked into an SVN repository that contain symbolic links to x86_64 binaries. We are trying to package up these symbolic links as part of the package but the symbolic links are replaced with the actual contents of the files with the name of the symbolic link.

The CI biuld is done through Jenkins and I've confirmed the files are checked out of SVN as symbolic links, however, during the copy to tmp-buildroot, it appears the symbolic links are lost in this process.

[INFO] Copying files to /home/jenkins/jenkins/workspace/test/target/rpm/test/tmp-buildroot/opt/test

I can't find any information on this topic and whether or not there is an option to preserve symbolic links except for the parameter, but I don't want to recreate the symbolic links through configuration, I would like to package them as-is.

      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>rpm-maven-plugin</artifactId>
        <version>2.1.4</version>
        <extensions>true</extensions>
        <executions>
          <execution>
            <id>attach-rpm</id>
            <goals>
              <goal>attached-rpm</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <group>Applications/Engineering</group>
          <needarch>x86_64</needarch>
          <targeOS>Linux</targeOS>

          <defineStatements>
            <defineStatement>_unpackaged_files_terminate_build 0</defineStatement>
          </defineStatements>

          <mappings>
            <mapping>
              <directory>/opt/test/</directory>
              <configuration>false</configuration>
              <username>user</username>
              <groupname>user</groupname>

              <sources>
                <source>
                  <location>test/</location>
                  <noDefaultExcludes>true</noDefaultExcludes>
                  <targetArchitecture>x86_64</targetArchitecture>
                  <targetOSName>linux</targetOSName>
                </source>
              </sources>

            </mapping>
          </mappings>
        </configuration>
      </plugin>
@dantran
Copy link
Contributor

dantran commented Feb 15, 2016

java copy does not preserve symlinks until java 7 this plugin also does not use any java 7 specifics either.

So it is best to manage the sym link yourself via rpm-plugin configurations

@Icedmang
Copy link
Author

Thanks Dan, I will have a look and try it and report back. (I was just looking at source code of the version 2.1.4 and just ended up at the org.codehaus.plexus.util.IOUtil class and was looking for a quick and dirty way to pass the NOFOLLOW_LINKS CopyOption https://docs.oracle.com/javase/tutorial/essential/io/copy.html)

edit: i just noticed you may have deleted your comment to try 2.1.5-SNAPSHOT.

@dantran
Copy link
Contributor

dantran commented Feb 16, 2016

😎 we need to fix up this plugin to support symlink which is possible now with minimal effort

@dantran
Copy link
Contributor

dantran commented Feb 16, 2016

@Icedmang could you give 2.1.5-SNAPSHOT deployed at sonatype a try or build this https://github.com/dantran/rpm-maven-plugin

I upgrade plexus-archiver to get symlink support.

@Icedmang
Copy link
Author

@dantran 2.1.5-SNAPSHOT from Sonatype seems to work! (It took me a bit to get our Nexus configured properly and my pom.xml to properly access SNAPSHOTs from Sonatype). The symbolic links are preserved (and I have many)

I did notice an Exception in the output but I'm unsure of the impact (yet):
[DEBUG] Cannot find ArtifactResolver with hint: project-cache-aware org.codehaus.plexus.component.repository.exception.ComponentLookupException: java.util.NoSuchElementException role: org.apache.maven.artifact.resolver.ArtifactResolver roleHint: project-cache-aware at org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:257) at org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:233) at org.apache.maven.shared.repository.DefaultRepositoryAssembler.contextualize(DefaultRepositoryAssembler.java:721) at org.sonatype.guice.plexus.lifecycles.PlexusLifecycleManager.contextualize(PlexusLifecycleManager.java:317) at org.sonatype.guice.plexus.lifecycles.PlexusLifecycleManager.manageLifecycle(PlexusLifecycleManager.java:292) at org.sonatype.guice.plexus.lifecycles.PlexusLifecycleManager.onProvision(PlexusLifecycleManager.java:148) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:108) at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:55) at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:68) at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1018) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:59) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:965) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1011) at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:961) at org.sonatype.guice.bean.locators.LazyBeanEntry.getValue(LazyBeanEntry.java:83) at org.sonatype.guice.plexus.locators.LazyPlexusBean.getValue(LazyPlexusBean.java:49) at org.sonatype.guice.bean.locators.EntryListAdapter$ValueIterator.next(EntryListAdapter.java:112) at org.apache.maven.plugin.assembly.archive.DefaultAssemblyArchiver.createArchive(DefaultAssemblyArchiver.java:181) at org.apache.maven.plugin.assembly.mojos.AbstractAssemblyMojo.execute(AbstractAssemblyMojo.java:436) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.jvnet.hudson.maven3.launcher.Maven3Launcher.main(Maven3Launcher.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239) at org.jvnet.hudson.maven3.agent.Maven3Main.launch(Maven3Main.java:178) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at hudson.maven.Maven3Builder.call(Maven3Builder.java:136) at hudson.maven.Maven3Builder.call(Maven3Builder.java:71) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.NoSuchElementException at java.util.Collections$EmptyIterator.next(Collections.java:4189) at org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:253) ... 54 more

@dantran
Copy link
Contributor

dantran commented Feb 16, 2016

@Icedmang about the exception, I dont see it at my maven 3.3.3 java 8 production build

what do you have?

@dantran
Copy link
Contributor

dantran commented Feb 16, 2016

@Icedmang did you test your RPM?

@Icedmang
Copy link
Author

@dantran I am using the following:

  • Java version: 1.8.0_66, vendor: Oracle Corporation
  • Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 05:51:28-0800)

Then I tried using

  • Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T04:57:37-07:00)

Note that I am using the -X option and the above dump is a DEBUG log message so it's likely hidden?
-X deploy site-deploy -Prun-its,coverage -Djacoco.outputDir=${WORKSPACE}/target

I get the same result for either Maven version 3.0.5 or 3.3.3 - the DEBUG message with the stack trace shows up. It will not show up without the -X option.

I am currently testing my packaged RPM and will report back

@Icedmang
Copy link
Author

There does not appear to be anything functionally different in the .rpm vs. the .tar.gz that we currently create... There is one caveat: the binary files have a different size in the .rpm vs. the .tar.gz. However, if I strip all binaries, the file sizes between the .rpm and .tar.gz are the same and there is no diff. The binaries in the .rpm are the ones that changed from their original form during packaging.

e.g.

# The file sizes are different for some reason.  The original source filesize is 1486163
[root@centos opt]# ls -l test-rpm/bin/binfile.x64Linux test-tarball/bin/binfile.x64Linux
-rwxrwxr-x. 1 user group 1478254 Feb 16 16:39 test-rpm/bin/binfile.x64Linux
-rw-r--r--. 1 user group 1486163 Feb 16 09:56 test-tarball/bin/binfile.x64Linux

# Permissions not set correctly in the tarball, so I set it manually
[root@centos opt]# chmod 775 test-tarball/bin/binfile.x64Linux
[root@centos opt]# ls -l test-rpm/bin/binfile.x64Linux test-tarball/bin/binfile.x64Linux
-rwxrwxr-x. 1 user group 1478254 Feb 16 16:39 test-rpm/bin/binfile.x64Linux
-rwxrwxr-x. 1 user group 1486163 Feb 16 09:56 test-tarball/bin/binfile.x64Linux

# I stripped the remaining symbols from the binaries
[root@centos opt]# strip test-rpm/bin/binfile.x64Linux test-tarball/bin/binfile.x64Linux

# Now the filesizes are the same
[root@centos opt]# ls -l test-rpm/bin/binfile.x64Linux test-tarball/bin/binfile.x64Linux
-rwxrwxr-x. 1 user group 1388056 Feb 16 17:04 test-rpm/bin/binfile.x64Linux
-rwxrwxr-x. 1 user group 1388056 Feb 16 17:04 test-tarball/bin/binfile.x64Linux

# Now there is no difference :/
[root@centos opt]# diff test-rpm/bin/binfile.x64Linux test-tarball/bin/binfile.x64Linux
[root@centos opt]# 

While I don't think this is a significant problem, I'm curious as to why the files are different while packaging the RPM

@dantran
Copy link
Contributor

dantran commented Feb 17, 2016

Glad it works out ( somewhat). I can't explain what you see. Must have something do with the rpmbuild during packaging

@Icedmang
Copy link
Author

I'm going to close this issue as it addresses my original question. Thank you so much for your help on this. You were very quick and responsive!

I'm looking forward to an official 2.1.5 release :)

@dantran
Copy link
Contributor

dantran commented Feb 17, 2016

@Icedmang please help test 2.1.5 staged at https://oss.sonatype.org/content/groups/staging

@Icedmang
Copy link
Author

@dantran Absolutely. I will report back in a few hours.

@Icedmang Icedmang reopened this Feb 17, 2016
@Icedmang
Copy link
Author

@dantran I am building the RPM now with version 2.1.5 from Staging. In the meantime, regarding the file size differences I mentioned earlier, I think it's related to what's described on this page

Is there a way through the plugin to include the following in the spec file? I could not find it
%global __os_install_post %{nil}

@dantran
Copy link
Contributor

dantran commented Feb 17, 2016

try defineStatements. here is mine

<defineStatements>
          <defineStatement>_unpackaged_files_terminate_build 0</defineStatement>
          <defineStatement>_tmppath ${project.build.directory}/rpm/tmp</defineStatement>
</defineStatements>

@Icedmang
Copy link
Author

@dantran Version 2.1.5 of rpm-maven-plugin works as expected. It still has different file sizes which I believe is due to reason I mentioned above and requiring the %global __os_install_post %{nil}. I don't believe the %define's will work here. I already have the following in place before I logged this issue:

        <defineStatements>
          <defineStatement>_unpackaged_files_terminate_build 0</defineStatement>
        </defineStatements>

@dantran
Copy link
Contributor

dantran commented Feb 18, 2016

maybe you can set it at your ~/.rpmrc ?

@Icedmang
Copy link
Author

@dantran I apologize, I was wrong to state that it wouldn't work in the defineStatement as you mentioned. I decided I should actually test it and you're right, it works as you said. Mine looks like:

          <defineStatements>
            <defineStatement>_unpackaged_files_terminate_build 0</defineStatement>
            <defineStatement>__os_install_post %{nil}</defineStatement>
          </defineStatements> 

I didn't want to include it in the ~/.rpmrc because I would have to do that to all the Jenkins nodes we have configured. I like it contained to the .rpm and not the user.

What should we do next so 2.1.5 is promoted from Staging to a Release?

@dantran
Copy link
Contributor

dantran commented Feb 18, 2016

nice, glad it works out. I need to announce a vote by tonight at mojohaus dev, it takes 3 days for the vote to pass

@dantran
Copy link
Contributor

dantran commented Feb 18, 2016

@dantran dantran added this to the 2.1.5 milestone Feb 18, 2016
@dantran dantran closed this as completed Feb 18, 2016
@Icedmang
Copy link
Author

Thanks again, @dantran

dantran referenced this issue Feb 18, 2016
plexus-interpolation-1.22, plexus-utils-3.0.22
zrhoffman added a commit to zrhoffman/trafficcontrol that referenced this issue Apr 25, 2024
zrhoffman added a commit to apache/trafficcontrol that referenced this issue Apr 26, 2024
…7999)

* Symlink from /opt/<component>/var/log to /var/log/<component>.

These symlinks are deprecated with the intent of removing them in ATC 9.0.0.

* Update rpm-maven-plugin to 2.2.0 to get symlink support

Symlink support is added in mojohaus/rpm-maven-plugin#42.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants