[MXNET-710] Change POM files to be able to regularly publish to Apache Release & Maven Central Repo #11862

nswamy · 2018-07-23T22:03:56Z

Description

This brings the ability to publish Maven packages using Apache Maven Release plugin and is integrated with the existing Make file System. The steps to perform the Scala Release is documented here:
https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process.

This process was used to publish both 1.2.0 and 1.2.1 releases. This could be further automated by integrating to the CI system to publish SNAPSHOTS to the Apache SnapShot repo which can be picked up to perform the Release.

@lanking520 @andrewfayres

lanking520

You can also try to change the

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-deploy-plugin</artifactId>
       <configuration>
          <skip>true</skip>
        </configuration>
     </plugin>

to the parent pom file and them apply changes to each assembly files.

lanking520 · 2018-07-24T19:05:50Z

Makefile

@@ -589,6 +589,7 @@ scalaclean:
 scalapkg:
 	(cd $(ROOTDIR)/scala-package; \
 		mvn package -P$(SCALA_PKG_PROFILE),$(SCALA_VERSION_PROFILE) -Dcxx="$(CXX)" \
+		    -Dbuild.platform="$(SCALA_PKG_PROFILE)" \


Can't we used this param in the profile directly?

I think I have tried this, I couldn't use read this from the profile in release/release-perform that is why I had to add an additional property.

-Darguments=""-Dbuild\.platform=\""$(SCALA_PKG_PROFILE)\""

lanking520 · 2018-07-24T19:07:34Z

Makefile

 			-Dcflags="$(CFLAGS)" -Dldflags="$(LDFLAGS)" \
 			-Dlddeps="$(LIB_DEP) $(ROOTDIR)/lib/libmxnet.a")

+scalarelease-dryrun:


Could you please also include explanations on these make target in the CWiki and possibly provide the links to explains the reason to run this and the following two make target below?

https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process#MXNet-ScalaReleaseProcess-MavenPublish

#if you don't set this it will fail with the error "gpg: signing failed: Inappropriate ioctl for device" export GPG_TTY=$(tty) #build scalapkg make scalapkg #make sure everything is working as expected make scalarelease-dryrun # Releases to Apache Snapshot make scaladeploy. This will upload to the snapshot repo # Verify that the package has been uploaded in https://repository.apache.org/content/repositories/snapshots/ # Preparing the release will create the new tag, automatically checking in on your behalf.(We have disabled CheckIn) make scalarelease-prepare # this will upload to the staging repository and make it for testing make scalarelease-perform

lanking520 · 2018-07-24T19:09:11Z

scala-package/assembly/osx-x86_64-cpu/pom.xml

+      <plugin>
+        <groupId>org.codehaus.mojo</groupId>
+        <artifactId>flatten-maven-plugin</artifactId>
+        <!--<version>1.1.0-SNAPSHOT</version>-->


Remove this comments?

I left it there because I am not using a version and to point out and remember that it is 1.1.0-SNAPSHOT(there is no released version)

lanking520 · 2018-07-24T19:12:14Z

scala-package/dev/compile-mxnet-backend.sh

+# below routine shamelessly copied from
+# https://github.com/apache/incubator-mxnet/blob/master/setup-utils/install-mxnet-osx-python.sh
+
+chkret() {


What is chkret?

chkret executes a command and checks return code.

lanking520 · 2018-07-24T19:13:02Z

scala-package/dev/compile-mxnet-backend.sh

+elif [[ $PLATFORM == "linux-x86_64-gpu" ]]
+then
+    echo "Building MXNet Backend on Linux GPU"
+    echo "Building MXNet Backend on Linux CPU"


remove this line?

nswamy

Thanks for the comments, I have responded and see if that addresses your concern.

nswamy · 2018-07-24T20:48:58Z

Makefile

 			-Dcflags="$(CFLAGS)" -Dldflags="$(LDFLAGS)" \
 			-Dlddeps="$(LIB_DEP) $(ROOTDIR)/lib/libmxnet.a")

+scalarelease-dryrun:


https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process#MXNet-ScalaReleaseProcess-MavenPublish

#if you don't set this it will fail with the error "gpg: signing failed: Inappropriate ioctl for device" export GPG_TTY=$(tty) #build scalapkg make scalapkg #make sure everything is working as expected make scalarelease-dryrun # Releases to Apache Snapshot make scaladeploy. This will upload to the snapshot repo # Verify that the package has been uploaded in https://repository.apache.org/content/repositories/snapshots/ # Preparing the release will create the new tag, automatically checking in on your behalf.(We have disabled CheckIn) make scalarelease-prepare # this will upload to the staging repository and make it for testing make scalarelease-perform

nswamy · 2018-07-24T20:51:03Z

scala-package/assembly/osx-x86_64-cpu/pom.xml

+      <plugin>
+        <groupId>org.codehaus.mojo</groupId>
+        <artifactId>flatten-maven-plugin</artifactId>
+        <!--<version>1.1.0-SNAPSHOT</version>-->


I left it there because I am not using a version and to point out and remember that it is 1.1.0-SNAPSHOT(there is no released version)

nswamy · 2018-07-24T20:51:25Z

scala-package/dev/compile-mxnet-backend.sh

+# below routine shamelessly copied from
+# https://github.com/apache/incubator-mxnet/blob/master/setup-utils/install-mxnet-osx-python.sh
+
+chkret() {


chkret executes a command and checks return code.

nswamy · 2018-07-24T21:07:04Z

Makefile

@@ -589,6 +589,7 @@ scalaclean:
 scalapkg:
 	(cd $(ROOTDIR)/scala-package; \
 		mvn package -P$(SCALA_PKG_PROFILE),$(SCALA_VERSION_PROFILE) -Dcxx="$(CXX)" \
+		    -Dbuild.platform="$(SCALA_PKG_PROFILE)" \


I think I have tried this, I couldn't use read this from the profile in release/release-perform that is why I had to add an additional property.

-Darguments=""-Dbuild\.platform=\""$(SCALA_PKG_PROFILE)\""

nswamy · 2018-07-24T21:12:53Z

@lanking520 regarding

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-deploy-plugin</artifactId>
       <configuration>
          <skip>true</skip>
        </configuration>
     </plugin>

I am not sure if I got your suggestion right, but that is exactly how it is done right now.

lanking520 · 2018-07-24T21:16:40Z

@nswamy The normal practice is to add the configuration file in the parent pom and all child modules will extend that. So if you add the maven-deploy-plugin in the parent pom, you will see it applies to every where in all of the modules. In that case, you only need to specify the <skip>false</skip> to the place you want to be instead of adding <skip>true</skip> in every pom file

nswamy · 2018-07-24T21:27:23Z

@lanking520 its already there
https://github.com/nswamy/incubator-mxnet/blob/35e064a707e4f694d3506628ac66196743cea7f6/scala-package/pom.xml#L164 in the parent and so does in the assembly pom files.
I understood what you are saying, agree it might not need to touch all files. It would be time consuming to test all of this(to make sure that the package release works fine). I prefer to keep it this way and probably change it during the next release cycle?

lanking520 · 2018-07-24T21:30:47Z

Agreed. We can create another PR specifically on cleaning the pom files and build cycle.

lanking520 · 2018-07-24T21:32:32Z

scala-package/pom.xml

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
-        <version>2.19</version>


Why do we degrade the version here?

good catch. will fix it

…e Release & Maven Central Repo (apache#11862) * pom file changes for maven builds

pom file changes for maven builds

3c1dcc3

nswamy requested review from szha and yzhliu as code owners July 23, 2018 22:03

Update pom.xml

3041a11

nswamy changed the title ~~Change POM files to be able to regularly publish to Apache Release & Maven Central Repo~~ [MXNET-710] Change POM files to be able to regularly publish to Apache Release & Maven Central Repo Jul 24, 2018

lanking520 suggested changes Jul 24, 2018

View reviewed changes

nswamy commented Jul 24, 2018

View reviewed changes

Update compile-mxnet-backend.sh

35e064a

lanking520 reviewed Jul 24, 2018

View reviewed changes

Update pom.xml

61457e3

lanking520 approved these changes Jul 24, 2018

View reviewed changes

nswamy mentioned this pull request Jul 25, 2018

Fix OpenCV 3.4.1 issue for Scala package for OSX #11193

Closed

nswamy merged commit 424fafe into apache:master Jul 25, 2018

lanking520 mentioned this pull request Aug 2, 2018

[MXNET-748] linker fixed on Scala issues #11989

Merged

5 tasks

XinYao1994 pushed a commit to XinYao1994/incubator-mxnet that referenced this pull request Aug 29, 2018

[MXNET-710] Change POM files to be able to regularly publish to Apach…

a0c1179

…e Release & Maven Central Repo (apache#11862) * pom file changes for maven builds

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MXNET-710] Change POM files to be able to regularly publish to Apache Release & Maven Central Repo #11862

[MXNET-710] Change POM files to be able to regularly publish to Apache Release & Maven Central Repo #11862

nswamy commented Jul 23, 2018 •

edited

Loading

lanking520 left a comment

lanking520 Jul 24, 2018

nswamy Jul 24, 2018

lanking520 Jul 24, 2018

nswamy Jul 24, 2018

lanking520 Jul 24, 2018

nswamy Jul 24, 2018

lanking520 Jul 24, 2018

nswamy Jul 24, 2018

lanking520 Jul 24, 2018

nswamy left a comment

nswamy Jul 24, 2018

nswamy Jul 24, 2018

nswamy Jul 24, 2018

nswamy Jul 24, 2018

nswamy commented Jul 24, 2018

lanking520 commented Jul 24, 2018

nswamy commented Jul 24, 2018

lanking520 commented Jul 24, 2018

lanking520 Jul 24, 2018

nswamy Jul 24, 2018

[MXNET-710] Change POM files to be able to regularly publish to Apache Release & Maven Central Repo #11862

[MXNET-710] Change POM files to be able to regularly publish to Apache Release & Maven Central Repo #11862

Conversation

nswamy commented Jul 23, 2018 • edited Loading

Description

lanking520 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nswamy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nswamy commented Jul 24, 2018

lanking520 commented Jul 24, 2018

nswamy commented Jul 24, 2018

lanking520 commented Jul 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nswamy commented Jul 23, 2018 •

edited

Loading