Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

smartics generator includes redundant module definitions #184

Closed
tdiesler opened this issue Dec 12, 2014 · 5 comments
Closed

smartics generator includes redundant module definitions #184

tdiesler opened this issue Dec 12, 2014 · 5 comments
Assignees
Labels
Milestone

Comments

@tdiesler
Copy link
Contributor

Currently there is no mechanism that guarantees the correct wiring to existing wildfly modules. As a result, redundant/incorrect module definitions are being generated.

org/apache/logging for example

[tdiesler@macbook modules]$ tar tzf target/wildfly-camel-modules-2.1.0-SNAPSHOT.tar.gz | grep jar | sort
modules/ca/uhn/hapi/base/main/hapi-base-2.2.jar
modules/com/jcraft/jsch/main/jsch-0.1.50.jar
modules/io/github/rometools/rome/main/rome-1.0.jar
modules/io/netty/netty-buffer/main/netty-buffer-4.0.23.Final.jar
modules/io/netty/netty-codec/main/netty-codec-4.0.23.Final.jar
modules/io/netty/netty-common/main/netty-common-4.0.23.Final.jar
modules/io/netty/netty-handler/main/netty-handler-4.0.23.Final.jar
modules/io/netty/netty-transport/main/netty-transport-4.0.23.Final.jar
modules/net/sf/saxon/he/main/Saxon-HE-9.5.1-5.jar
modules/org/apache/abdera/core/main/abdera-core-1.1.3.jar
modules/org/apache/abdera/i18n/main/abdera-i18n-1.1.3.jar
modules/org/apache/abdera/parser/main/abdera-parser-1.1.3.jar
modules/org/apache/activemq/camel/main/activemq-camel-5.10.0.jar
modules/org/apache/activemq/main/activemq-broker-5.10.0.jar
modules/org/apache/activemq/main/activemq-client-5.10.0.jar
modules/org/apache/activemq/main/activemq-jms-pool-5.10.0.jar
modules/org/apache/activemq/main/activemq-kahadb-store-5.10.0.jar
modules/org/apache/activemq/main/activemq-openwire-legacy-5.10.0.jar
modules/org/apache/activemq/main/activemq-pool-5.10.0.jar
modules/org/apache/activemq/main/activemq-protobuf-1.1.jar
modules/org/apache/activemq/main/activemq-ra-5.10.0.jar
modules/org/apache/activemq/main/activemq-spring-5.10.0.jar
modules/org/apache/camel/component/atom/main/camel-atom-2.14.0.jar
modules/org/apache/camel/component/bindy/main/camel-bindy-2.14.0.jar
modules/org/apache/camel/component/cdi/main/camel-cdi-2.14.0.jar
modules/org/apache/camel/component/cxf/main/camel-cxf-2.14.0.jar
modules/org/apache/camel/component/cxf/main/camel-cxf-transport-2.14.0.jar
modules/org/apache/camel/component/ftp/main/camel-ftp-2.14.0.jar
modules/org/apache/camel/component/hl7/main/camel-hl7-2.14.0.jar
modules/org/apache/camel/component/jaxb/main/camel-jaxb-2.14.0.jar
modules/org/apache/camel/component/jms/main/camel-jms-2.14.0.jar
modules/org/apache/camel/component/jmx/main/camel-jmx-2.14.0.jar
modules/org/apache/camel/component/jpa/main/camel-jpa-2.14.0.jar
modules/org/apache/camel/component/lucene/main/camel-lucene-2.14.0.jar
modules/org/apache/camel/component/mail/main/camel-mail-2.14.0.jar
modules/org/apache/camel/component/mina2/main/camel-mina2-2.14.0.jar
modules/org/apache/camel/component/mqtt/main/camel-mqtt-2.14.0.jar
modules/org/apache/camel/component/mvel/main/camel-mvel-2.14.0.jar
modules/org/apache/camel/component/netty4/main/camel-netty4-2.14.0.jar
modules/org/apache/camel/component/ognl/main/camel-ognl-2.14.0.jar
modules/org/apache/camel/component/quartz/main/camel-quartz-2.14.0.jar
modules/org/apache/camel/component/rss/main/camel-rss-2.14.0.jar
modules/org/apache/camel/component/saxon/main/camel-saxon-2.14.0.jar
modules/org/apache/camel/component/script/main/camel-script-2.14.0.jar
modules/org/apache/camel/component/soap/main/camel-soap-2.14.0.jar
modules/org/apache/camel/component/sql/main/camel-sql-2.14.0.jar
modules/org/apache/camel/component/weather/main/camel-weather-2.14.0.jar
modules/org/apache/camel/core/main/camel-core-2.14.0.jar
modules/org/apache/camel/core/xml/main/camel-core-xml-2.14.0.jar
modules/org/apache/camel/spring/main/camel-spring-2.14.0.jar
modules/org/apache/commons/codec/main/commons-codec-1.9.jar
modules/org/apache/commons/logging/main/commons-logging-1.0.3.jar
modules/org/apache/commons/net/main/commons-net-3.3.jar
modules/org/apache/commons/pool/main/commons-pool-1.6.jar
modules/org/apache/deltaspike/cdictrl/api/main/deltaspike-cdictrl-api-1.0.2.jar
modules/org/apache/deltaspike/core/api/main/deltaspike-core-api-1.0.2.jar
modules/org/apache/deltaspike/core/impl/main/deltaspike-core-impl-1.0.2.jar
modules/org/apache/geronimo/specs/geronimo-javamail_1/4_spec/main/geronimo-javamail_1.4_spec-1.7.1.jar
modules/org/apache/geronimo/specs/geronimo-osgi-locator/main/geronimo-osgi-locator-1.0.jar
modules/org/apache/geronimo/specs/geronimo-osgi-registry/main/geronimo-osgi-registry-1.0.jar
modules/org/apache/jakarta/regexp/main/jakarta-regexp-1.4.jar
modules/org/apache/james/apache-mime4j-core/main/apache-mime4j-core-0.7.2.jar
modules/org/apache/lucene/analyzers-common/main/lucene-analyzers-common-4.6.1.jar
modules/org/apache/lucene/core/main/lucene-core-4.6.1.jar
modules/org/apache/lucene/queries/main/lucene-queries-4.6.1.jar
modules/org/apache/lucene/queryparser/main/lucene-queryparser-4.6.1.jar
modules/org/apache/lucene/sandbox/main/lucene-sandbox-4.6.1.jar
modules/org/apache/mina/core/main/mina-core-2.0.7.jar
modules/org/apache/servicemix/bundles/javassist/main/org.apache.servicemix.bundles.javassist-3.12.1.GA_3.jar
modules/org/apache/servicemix/bundles/ognl/main/org.apache.servicemix.bundles.ognl-3.0.8_1.jar
modules/org/apache/ws/commons/axiom/axiom-api/main/axiom-api-1.2.14.jar
modules/org/apache/ws/commons/axiom/axiom-impl/main/axiom-impl-1.2.14.jar
modules/org/apache/xbean/spring/main/xbean-spring-3.16.jar
modules/org/codehaus/jackson/jackson-core-asl/main/jackson-core-asl-1.9.12.jar
modules/org/codehaus/jackson/jackson-mapper-asl/main/jackson-mapper-asl-1.9.12.jar
modules/org/codehaus/woodstox/stax2-api/main/stax2-api-3.1.1.jar
modules/org/codehaus/woodstox/woodstox-core-asl/main/woodstox-core-asl-4.1.4.jar
modules/org/fusesource/hawtbuf/main/hawtbuf-1.10.jar
modules/org/fusesource/hawtdispatch/main/hawtdispatch-1.21.jar
modules/org/fusesource/hawtdispatch/transport/main/hawtdispatch-transport-1.21.jar
modules/org/fusesource/mqtt-client/main/mqtt-client-1.10.jar
modules/org/jaxen/main/jaxen-1.1.1.jar
modules/org/jboss/gravia/main/gravia-provision-1.1.0.Beta44.jar
modules/org/jboss/gravia/main/gravia-repository-1.1.0.Beta44.jar
modules/org/jboss/gravia/main/gravia-resolver-1.1.0.Beta44.jar
modules/org/jboss/gravia/main/gravia-resource-1.1.0.Beta44.jar
modules/org/jboss/gravia/main/gravia-runtime-api-1.1.0.Beta44.jar
modules/org/jboss/gravia/main/gravia-runtime-embedded-1.1.0.Beta44.jar
modules/org/jboss/gravia/main/org.apache.felix.configadmin-1.8.0.jar
modules/org/jboss/gravia/main/org.apache.felix.http.bridge-2.2.1.jar
modules/org/jboss/gravia/main/org.apache.felix.log-1.0.1.jar
modules/org/jboss/gravia/main/org.apache.felix.metatype-1.0.8.jar
modules/org/jboss/gravia/main/org.apache.felix.scr-1.6.2.jar
modules/org/mvel/mvel2/main/mvel2-2.2.1.Final.jar
modules/org/osgi/core/main/org.osgi.core-4.3.1.jar
modules/org/osgi/enterprise/main/org.osgi.enterprise-5.0.0.jar
modules/org/quartz-scheduler/quartz/main/quartz-1.8.6.jar
modules/org/springframework/aop/main/aopalliance-1.0.jar
modules/org/springframework/aop/main/spring-aop-3.2.11.RELEASE.jar
modules/org/springframework/beans/main/spring-beans-3.2.11.RELEASE.jar
modules/org/springframework/context/main/spring-context-3.2.11.RELEASE.jar
modules/org/springframework/core/main/spring-core-3.2.8.RELEASE.jar
modules/org/springframework/expression/main/spring-expression-3.2.11.RELEASE.jar
modules/org/springframework/jdbc/main/spring-jdbc-3.2.11.RELEASE.jar
modules/org/springframework/jms/main/spring-jms-3.2.11.RELEASE.jar
modules/org/springframework/orm/main/spring-orm-3.2.11.RELEASE.jar
modules/org/springframework/transaction/main/spring-tx-3.2.11.RELEASE.jar

We support every module we ship. Therefore, its important that we only ship what we can support ;-)

  • Duplicate module definitions (possibly with multiple versions) must be avoided
  • License constraints must be applied

If this cannot be automated, we may need to use smartics to generate the next module definition "suggestion" and have an additional review + copy step.

An alternative may be to maintain an list like above in scm and make sure that every change to the list must manually be endorsed.

@tdiesler tdiesler added the bug label Dec 12, 2014
@tdiesler tdiesler added this to the 2.1.0 milestone Dec 12, 2014
@jamesnetherton
Copy link
Collaborator

We can help to guard against this by adding unwanted libraries into 'dependencyExcludes' on the smartics plugin configuration. Seems to work fine for commons-logging.

I'll see what else can be excluded.

@tdiesler
Copy link
Contributor Author

With the manual process that we had before there was the danger of missing dependencies. Unknown code paths could run into class not found errors.

With an automated approach there is the danger of dependency duplication or otherwise invalid module definitions. Modules may get generated that already exist in the platform somewhere or cannot be supported.

By generally excluding dependencies via 'dependencyExcludes' we create problem one again for the automated approach.

Ultimately, I believe some human intelligence will always be required to make the ultimate decision. Smartics could be used to make a proposal, that after being reviewed can make it upstream.

Specifically I suggest, we review the existing module list carefully now and make sure we don't already have invalid module definitions (e.g. sun jdk deps, license issues, etc). Then we generate to a location that does not automatically make it into the patch and is under SCM control. Every delta in the smartics output would be visible in the git diff. We can see, review and comment on proposed changes before they make it into the patch.

@tdiesler
Copy link
Contributor Author

mvn clean install -Dts.all results in dirty workspace

[tdiesler@macbook wildfly-camel]$ git diff
diff --git a/modules/etc/baseline/module-list.txt b/modules/etc/baseline/module-list.txt
index 8527e80..e657808 100644
--- a/modules/etc/baseline/module-list.txt
+++ b/modules/etc/baseline/module-list.txt
@@ -1,147 +1,147 @@
+./ca/uhn/hapi/base/main/hapi-base-2.2.jar
+./ca/uhn/hapi/base/main/module.xml
+./com/jcraft/jsch/main/jsch-0.1.51.jar
+./com/jcraft/jsch/main/module.xml
 ./io/github/rometools/rome/main/module.xml
 ./io/github/rometools/rome/main/rome-1.0.jar
 ./net/sf/saxon/he/main/module.xml
 ./net/sf/saxon/he/main/Saxon-HE-9.5.1-5.jar
-./org/quartz-scheduler/quartz/main/module.xml
-./org/quartz-scheduler/quartz/main/quartz-1.8.6.jar
-./org/mvel/mvel2/main/module.xml
-./org/mvel/mvel2/main/mvel2-2.2.1.Final.jar
-./org/springframework/orm/main/module.xml
-./org/springframework/orm/main/spring-orm-3.2.11.RELEASE.jar
-./org/springframework/jms/main/module.xml
-./org/springframework/jms/main/spring-jms-3.2.11.RELEASE.jar
-./org/springframework/beans/main/module.xml
-./org/springframework/beans/main/spring-beans-3.2.11.RELEASE.jar
-./org/springframework/context/main/module.xml
-./org/springframework/context/main/spring-context-3.2.11.RELEASE.jar
-./org/springframework/tx/main/module.xml
-./org/springframework/tx/main/spring-tx-3.2.11.RELEASE.jar
-./org/springframework/expression/main/module.xml
-./org/springframework/expression/main/spring-expression-3.2.11.RELEASE.jar
-./org/springframework/jdbc/main/module.xml
-./org/springframework/jdbc/main/spring-jdbc-3.2.11.RELEASE.jar
-./org/springframework/core/main/module.xml
-./org/springframework/core/main/spring-core-3.2.8.RELEASE.jar
-./org/springframework/aop/main/module.xml
-./org/springframework/aop/main/spring-aop-3.2.11.RELEASE.jar
-./org/springframework/aop/main/aopalliance-1.0.jar
-./org/osgi/enterprise/main/module.xml
-./org/osgi/enterprise/main/org.osgi.enterprise-5.0.0.jar
-./org/wildfly/camel/wildfly-camel-modules/main/module.xml
-./org/fusesource/mqtt-client/main/module.xml
-./org/fusesource/mqtt-client/main/mqtt-client-1.10.jar
-./org/fusesource/hawtdispatch/main/module.xml
-./org/fusesource/hawtdispatch/main/hawtdispatch-1.21.jar
-./org/fusesource/hawtdispatch/transport/main/module.xml
-./org/fusesource/hawtdispatch/transport/main/hawtdispatch-transport-1.21.jar
-./org/fusesource/hawtbuf/main/module.xml
-./org/fusesource/hawtbuf/main/hawtbuf-1.10.jar
-./org/jboss/gravia/main/gravia-provision-1.1.0.Beta44.jar
-./org/jboss/gravia/main/gravia-resource-1.1.0.Beta44.jar
-./org/jboss/gravia/main/gravia-repository-1.1.0.Beta44.jar
-./org/jboss/gravia/main/org.apache.felix.http.bridge-2.2.1.jar
-./org/jboss/gravia/main/module.xml
-./org/jboss/gravia/main/gravia-runtime-embedded-1.1.0.Beta44.jar
-./org/jboss/gravia/main/org.apache.felix.configadmin-1.8.0.jar
-./org/jboss/gravia/main/org.apache.felix.metatype-1.0.8.jar
-./org/jboss/gravia/main/org.apache.felix.log-1.0.1.jar
-./org/jboss/gravia/main/gravia-runtime-api-1.1.0.Beta44.jar
-./org/jboss/gravia/main/org.apache.felix.scr-1.6.2.jar
-./org/jboss/gravia/main/gravia-resolver-1.1.0.Beta44.jar
+./org/apache/abdera/core/main/abdera-core-1.1.3.jar
+./org/apache/abdera/core/main/module.xml
+./org/apache/abdera/i18n/main/abdera-i18n-1.1.3.jar
+./org/apache/abdera/i18n/main/module.xml
+./org/apache/abdera/parser/main/abdera-parser-1.1.3.jar
+./org/apache/abdera/parser/main/module.xml
+./org/apache/activemq/camel/main/activemq-camel-5.10.0.jar
+./org/apache/activemq/camel/main/module.xml
 ./org/apache/activemq/main/activemq-broker-5.10.0.jar
-./org/apache/activemq/main/module.xml
+./org/apache/activemq/main/activemq-client-5.10.0.jar
+./org/apache/activemq/main/activemq-jms-pool-5.10.0.jar
 ./org/apache/activemq/main/activemq-kahadb-store-5.10.0.jar
 ./org/apache/activemq/main/activemq-openwire-legacy-5.10.0.jar
 ./org/apache/activemq/main/activemq-pool-5.10.0.jar
-./org/apache/activemq/main/activemq-client-5.10.0.jar
 ./org/apache/activemq/main/activemq-protobuf-1.1.jar
-./org/apache/activemq/main/activemq-jms-pool-5.10.0.jar
 ./org/apache/activemq/main/activemq-ra-5.10.0.jar
 ./org/apache/activemq/main/activemq-spring-5.10.0.jar
-./org/apache/activemq/camel/main/activemq-camel-5.10.0.jar
-./org/apache/activemq/camel/main/module.xml
-./org/apache/xbean/spring/main/module.xml
-./org/apache/xbean/spring/main/xbean-spring-3.16.jar
-./org/apache/servicemix/bundles/ognl/main/module.xml
-./org/apache/servicemix/bundles/ognl/main/org.apache.servicemix.bundles.ognl-3.0.8_1.jar
-./org/apache/deltaspike/cdictrl/api/main/module.xml
-./org/apache/deltaspike/cdictrl/api/main/deltaspike-cdictrl-api-1.0.2.jar
-./org/apache/deltaspike/core/impl/main/deltaspike-core-impl-1.0.2.jar
-./org/apache/deltaspike/core/impl/main/module.xml
-./org/apache/deltaspike/core/api/main/module.xml
-./org/apache/deltaspike/core/api/main/deltaspike-core-api-1.0.2.jar
-./org/apache/ws/commons/axiom/axiom-api/main/module.xml
-./org/apache/ws/commons/axiom/axiom-api/main/axiom-api-1.2.14.jar
-./org/apache/ws/commons/axiom/axiom-impl/main/axiom-impl-1.2.14.jar
-./org/apache/ws/commons/axiom/axiom-impl/main/module.xml
-./org/apache/mina/core/main/module.xml
-./org/apache/mina/core/main/mina-core-2.0.7.jar
-./org/apache/abdera/i18n/main/module.xml
-./org/apache/abdera/i18n/main/abdera-i18n-1.1.3.jar
-./org/apache/abdera/parser/main/module.xml
-./org/apache/abdera/parser/main/abdera-parser-1.1.3.jar
-./org/apache/abdera/core/main/module.xml
-./org/apache/abdera/core/main/abdera-core-1.1.3.jar
-./org/apache/commons/net/main/module.xml
-./org/apache/commons/net/main/commons-net-3.3.jar
-./org/apache/camel/main/module.xml
-./org/apache/camel/component/jmx/main/module.xml
-./org/apache/camel/component/jmx/main/camel-jmx-2.14.1.jar
-./org/apache/camel/component/mvel/main/module.xml
-./org/apache/camel/component/mvel/main/camel-mvel-2.14.1.jar
-./org/apache/camel/component/weather/main/camel-weather-2.14.1.jar
-./org/apache/camel/component/weather/main/module.xml
-./org/apache/camel/component/cdi/main/module.xml
+./org/apache/activemq/main/module.xml
+./org/apache/camel/component/atom/main/camel-atom-2.14.1.jar
+./org/apache/camel/component/atom/main/module.xml
+./org/apache/camel/component/bindy/main/camel-bindy-2.14.1.jar
+./org/apache/camel/component/bindy/main/module.xml
 ./org/apache/camel/component/cdi/main/camel-cdi-2.14.1.jar
-./org/apache/camel/component/mina2/main/module.xml
-./org/apache/camel/component/mina2/main/camel-mina2-2.14.1.jar
+./org/apache/camel/component/cdi/main/module.xml
+./org/apache/camel/component/cxf/main/camel-cxf-2.14.1.jar
+./org/apache/camel/component/cxf/main/camel-cxf-transport-2.14.1.jar
+./org/apache/camel/component/cxf/main/module.xml
+./org/apache/camel/component/ftp/main/camel-ftp-2.14.1.jar
+./org/apache/camel/component/ftp/main/module.xml
+./org/apache/camel/component/hl7/main/camel-hl7-2.14.1.jar
+./org/apache/camel/component/hl7/main/module.xml
 ./org/apache/camel/component/jaxb/main/camel-jaxb-2.14.1.jar
 ./org/apache/camel/component/jaxb/main/module.xml
-./org/apache/camel/component/jms/main/module.xml
 ./org/apache/camel/component/jms/main/camel-jms-2.14.1.jar
-./org/apache/camel/component/atom/main/module.xml
-./org/apache/camel/component/atom/main/camel-atom-2.14.1.jar
-./org/apache/camel/component/soap/main/module.xml
-./org/apache/camel/component/soap/main/camel-soap-2.14.1.jar
+./org/apache/camel/component/jms/main/module.xml
+./org/apache/camel/component/jmx/main/camel-jmx-2.14.1.jar
+./org/apache/camel/component/jmx/main/module.xml
+./org/apache/camel/component/jpa/main/camel-jpa-2.14.1.jar
+./org/apache/camel/component/jpa/main/module.xml

@tdiesler tdiesler reopened this Dec 18, 2014
@jamesnetherton
Copy link
Collaborator

I've done a bit more work on this and the results can be found at commit a1d124e. There's also some additional documentation here.

I went through several iterations of trying to get something working with standalone scripts and attempting to modify the smartics plugin.

I've settled on a simple Groovy script which does some comparisons between the wildfly camel module dependencies and those supplied with the wildfly app server. Using Groovy at least keeps the build cross platform.

It's not 100% foolproof. Since it's comparing Maven group / artifact ids, there are scenarios where duplicate dependencies can slip though. E.g with netty where WF uses the 'super' netty-all jar but WF camel uses a selection of individual netty jars. In this instance it'll be for code review to rectify.

@tdiesler
Copy link
Contributor Author

tdiesler commented Jan 8, 2015

Does this generate to a location that goes not automatically into the patch and is under SCM control?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants