Code processors

This started as an output of #65 "Service discovery by parsing implementation source code".

Overview

• CI : sonar (see below)
• doc : Enunciate, Swagger (see below)
• generic http://spoon.gforge.inria.fr/SpoonVSuite/Main , samples (visitor...) http://spoon.gforge.inria.fr/Spoon/Examples
• bugs http://findbugs.sourceforge.net/
• code review, flexible http://www.hammurapi.com/dokuwiki/doku.php/products:hammurapi:start

2012.06.29 @mkalam-alami update

• Enunciate: Generates documentation & clients for web services. Uses apt-jelly for annotation processing.
• Qdox: Javadoc parser, can parse annotations/comments/etc, quite simple, not updated since 2010 but still has active members (a v2.0, with support of Java 1.7 is on its way). Used by Maven to parse code during plugins generations. (@mkalam-alami's preference for "code discovery")
• JavaParser: Java 1.5 code parser, not very active anymore.
• Parboiled: Generic grammar-based parser, has a maintained, Java 1.6-compatible grammar, except it doesn't parse the comments (can be fixed by tweaking the grammar). Recent, with active community.
• ANTLR: Parser generator, grammars are available for Java 1.6, but seems quite complex to use (Parboiled claims to be a good balance between 'hacky Regexps' & full-blown parser generators such as ANTLR).

Sonar

sonar architecture

sonar is

• a plugin aggregator (findbugs...)
• but also a core code analysis engine : squid http://www.developpez.net/forums/d922355/general-developpement/conception/usine-logicielle/qualimetrie/sonar-squid/

sonar architecture - core engine : squid

• api http://www.sonarsource.org/docs/2.0/apidocs/org/sonar/squid/Squid.html
• replaces ncss because unmaintained & not java 5 compatible http://javancss.codehaus.org/ , improves ncss on areas such as comments http://jira.codehaus.org/browse/SONAR-765
• both source & bytecode
• requires sonar to work & tightly bound, impossible to use it separately http://www.developpez.net/forums/d922355/general-developpement/conception/usine-logicielle/qualimetrie/sonar-squid/

sonar architecture - metrics

sonar-core-plugin :

• CorePlugin : hooks all extensions in
• an extension implements (Server)Extension and various business interfaces (ex. Chart, Metrics)
• ex. UserManagedMetrics implements api.measures.Metrics and api.ServerExtension

sonar-plugin-api :

• measures.Measure has name, value, a few variations...
• measures.CoreMetrics defines NCLOC, nb of files...

sonar-squid-java-plugin :

• bridges.Bridge interface has visitor methods : onFile(), onClass(), onMethod()...
• all impls created in bridges.BridgeFactory
• bridges.CopyBasicMeasuresBridge copies squid metrics to sonar ones
• ast.JavaAstScanner : here all AST visitors are plugged in getVisitorClasses()
• ex. KLOC visitor ast.visitor.LinesOfCodeVisitor uses Source to get metric
• ex. ClassVisitor measures nb of class, itf, abstract in visitToken()

sonar-ws-client :

provides & query metrics, rules...

TODO how to add nb ws metric :

• create a new AstVisitor, look at annotations in onClass(), hack JavaAstScanner to hook it in
• then either call EasySOA directly
• or hack create another bridge to put it in sonar and hack BridgeFactory to put it in, ?? hack CoreMetrics / implement a new ServerExtension, hack UI (?) or do something like CustomMeasuresWidget

sonar architecture - coding rules

• sonar-check-api
• check.Rule
• sonar-findbugs-plugin :
• FindbugsSensor parses findbugs report, creates & saves Violations in sonar SensorContext

TODO How to extend them :

• 1. Checkstyle or PMD conf
• 2. Sonar Rule api http://docs.codehaus.org/display/SONAR/Extend+coding+rules

sonar plugins

sonar plugins - test coverage

• cobertura (default)
• jacoco http://docs.codehaus.org/display/SONAR/JaCoCo+Plugin http://www.eclemma.org/jacoco/index.html
• emma (also within a single line filtering) http://docs.codehaus.org/display/SONAR/Sonar+Emma+Plugin http://emma.sourceforge.net/

sonar plugins - architecture

• sonarj : design conformance http://docs.codehaus.org/display/SONAR/SonarJ+Plugin
• total quality : combines metrics http://docs.codehaus.org/display/SONAR/Total+Quality+Plugin
• quality index : same http://docs.codehaus.org/display/SONAR/Quality+Index+Plugin

Swagger

what : REST API framework

• defines api spec / doc (trying to go beyond wadl, limitations : no versioning) (in json format) see http://swagger.wordnik.com/spec
• allows for api declaration (in java / scala by annotations extending jaxrs ones, in node.js in json https://github.com/wordnik/swagger-node-express/blob/master/Apps/petstore/petResources.js),
• server (in java, scala, node.js, with bonus for play2 see http://swagger.wordnik.com/downloads ) dynamically serving api spec / doc in json based on api declaration, ex. in java https://github.com/wordnik/swagger-core/blob/master/modules/swagger-jaxrs/src/main/scala/com/wordnik/swagger/jaxrs/ApiListing.scala
• online UI (including test playground) ex. http://developer.wordnik.com/docs ; on spine (js mvc), underscore, jquery, html / css, ruby (only to the js / html / css) https://github.com/wordnik/swagger-ui
• generation of api client (including custom test fw)

more about fct :

• annotations beyond jaxrs for api doc : API root, errors, more params doc (possible values...), generic security fw ex. https://github.com/ariesprayoga/swagger-java-sample-app/blob/master/src/main/java/com/wordnik/swagger/sample/resource/PetResource.java
• type model : resources / beans are modeled by a list of properties, whose type can be primitive (string, Date...), a model or list types ex. Array[StringValue], List, Map, Set, Array
• also ping, health, profiler (durations) https://github.com/wordnik/swagger-core/blob/master/modules/swagger-jaxrs-utils/src/main/scala/com/wordnik/resource/util/ https://github.com/wordnik/wordnik-oss/blob/master/modules/common-utils/src/main/scala/com/wordnik/util/perf/Profile.scala

more about impl :

• in java & scala, jackson..., build in ant + ivy, deployed in jetty
• conf'd in servlet config (auth filter impl...) ex. https://github.com/ariesprayoga/swagger-java-sample-app/blob/master/conf/web/web.xml
• has its own jarxs spec parser https://github.com/wordnik/swagger-core/tree/master/modules/swagger-jaxrs/src/main/scala/com/wordnik/swagger/jaxrs https://github.com/wordnik/swagger-core/blob/master/modules/swagger-core/src/main/scala/com/wordnik/swagger/core/SpecReader.scala
• test : unit, bdd, supports models, made simple using scala, see https://github.com/wordnik/swagger-core/blob/master/samples/java-jaxrs/src/test/scala/com/wordnik/test/swagger/integration/ResourceListingIT.scala
• also mongodb use & admin tools https://github.com/wordnik/wordnik-oss

examples :

• sample java app (on jersey, jetty, jackson, also scala) https://github.com/wordnik/swagger-core/tree/master/samples/java-jaxrs https://github.com/ariesprayoga/swagger-java-sample-app
• also sample node (uses json api model), js, play, scala apps https://github.com/wordnik/ https://github.com/wordnik/swagger-core/tree/master/samples

more about client gen codegen https://github.com/wordnik/swagger-codegen :

• generation from CLI of swagger service client in java, js (including ApiInvoker i.e. browser binding), ruby, python, php, scala, as3, csharp
• uses antlr templates https://github.com/wordnik/swagger-codegen/tree/master/conf/java/templates http://www.stringtemplate.org/
• also generates client tests using its own fw, which has TestSuite & Assertion of actual vs expected https://github.com/wordnik/swagger-codegen/tree/master/src/main/java/com/wordnik/swagger/testframework

integrating it with easysoa - ideas :

• (replaces easysoa doc & test ui) TODO in web discovery : recognize the swagger spec and upload it in the model as primary interface (instead of wadl) ; then make it useful : doc & test playground (add a link in the different environments to the api doc in the swagger online doc UI, possibly embedded), test (generate shell commands for CLI & test generation, or even embed it). NB. this renders copying the bare HTML doc itself useless.
• LATER ? in code discovery : parse swagger annotations and enrich the easysoa doc model with them as well as the easysoa doc ui, compare it with the json spec got from web discovery, ?? in client code compare spec with server's, add versioning...