Skip to content

Tutorial

Jump to bottom Edit New page
Joshua C Elliott edited this page Oct 23, 2013 · 61 revisions

Table of Contents

Introduction

Unstructured Information Management applications are software systems that analyze large volumes of unstructured information in order to discover knowledge that is relevant to an end user. An example UIM application might ingest plain text and identify entities, such as persons, places, organizations; or relations, such as works-for or located-at.

UIMA enables applications to be decomposed into components, for example "language identification" => "language specific segmentation" => "sentence boundary detection" => "entity detection (person/place names etc.)". Each component implements interfaces defined by the framework and provides self-describing metadata via XML descriptor files. The framework manages these components and the data flow between them. Components are written in Java or C++; the data that flows between components is designed for efficient mapping between these languages.

Getting Started

Setting up

Prerequisites

JDK

  1. Visit the Java SE Downloads page, and download the most recent, platform-appropriate binary for JDK 6.
  2. Install the JDK from the downloaded executable file. Note: If you are working in Windows, you will need to set the PATH environment variable manually.

git

  1. Visit the git page and download the most recent installer.
  2. Run the installer.
  3. If you've never used git before, the Pro Git book is an excellent guide. However, for the impatient, we recommend this section on setup and any git quickstart guide (this one is OK).

Maven

  1. Download the most recent stable version of Maven from the Apache Maven project page.
  2. Run the installer. Additional instructions can be found at the bottom of the project page.

Note: We use a standalone install of Maven because the Eclipse plugin version often runs into annoying build bugs. Pointing the plugin to a standalone installation has proven to be much more reliable.

Eclipse + plugins

  1. Download Eclipse IDE for Java Developers (Indigo or newer) from the Eclipse Download page. (Really, any version that has the Maven plugin already installed)
  2. Install Eclipse by extracting the downloaded package into a directory of your choosing.
  3. See the tips page for instructions on installing plugins.

UIMA SDK

Follow the instructions under "Install UIMA SDK" at the Apache UIMA page. See the Eclipse tips page for instructions on installing the UIMA plugins.

Getting the Code for the Tutorial Project

Run git clone https://github.com/oaqa/oaqa-tutorial.git in your preferred project directory.

Open Eclipse and specify the same project directory as your workspace. Once Eclipse is open, go to the workbench and File > Import > General/Existing Projects into Workspace. Set the root directory as your workspace directory. The tutorial project should show up in the "Projects" box, make sure its checkbox is checked and click Finish.

(This may also need to contain a section on the basics of Git, project information in a location on GitHub, and possibly information on Maven as well, things like Group Id, Artifact ID, Version. Depends on what method above is used.)

Directory structure

Note: need to create generic archetype for CSE project separate from hw1. Sub-Note: what is hw1?

The directory structure should like this: Note: the directory structure of what? What is meant by "all your descriptors"?

myproject
 |- pom.xml
 '- src
    '- main
      |- java
      |  '- **/*.java 
      '- resources
         |- mypipeline.yaml /* the entry point for your pipeline */
         |- **/*.yaml /* all your descriptors go into the resources folder */
         '- META-INF
            '- org.uimafit
               '- types.txt
```

### (Optional) Persistence/Database
Definitely a good idea to at least define OAQA database, what it is, how it is used.

### (Optional) UI

## Writing Extended Configuration Descriptors (ECD)
[Extended configuration descriptor (ECD)](https://github.com/oaqa/uima-ecd) extends the UIMA and uimaFit to balance between ease-of-use of flexibility. To this end, ECD provides three major features: (1) YAML based descriptors, (2) driver that supports multiple options for a component specified in the descriptor, and (3) driver that supports declarative options for a component.

Similar to collection processing engine descriptor (CPE) in UIMA
SDK, the key element of ECD is a YAML associative array
of three elements:
* collection-reader
* the actual pipeline, and
* the optional post-process pipeline

Similar to the UIMA SDK component descriptor (collection reader,pipeline annotator, and cas consumer), the building blocks of ECD are component descriptors (a YAML associative array) defines:
* The Java class that implements the component,
* Parameters and values the component requires.

### Basics
#### YAML Format
[YAML Ain't Markup Language (YAML)](http://yaml.org) is a human-readable data serialization format that provides a simple but rich syntax to represent a data structure, like our component descriptors as well as engine descriptors.

One important aspect of YAML is that indentation matters. Each indentation after a newline forms a block literal, similar to other indentation-sensitive languages such as Python. Also, parameters can be passed to fields using a colon and space. So for example,
```xml
configuration: 
  name: oaqa-tutorial
  author: oaqa
```

Here the parameters `name`, and `author` are set to `oaqa-tutorial` and `oaqa` respectively, and are passed as a block to `configuration`.

#### Components
When writing an ECD for a UIMA annotator, the first line specifies whether it is a subclass of another annotator descriptor, or whether it refers to a UIMA annotator class directly. The first line in an ECD component can contain one of the following:

* inherit: will look for a resource file within the class-path on the path specified by the doted syntax a.b.c => a/b/c.yaml. Inherited parameters. can be overridden directly on the body of the resource file.

* class: will look for the specified class within the class-path, and is intended to be used as a shorthand for classes that do not have configurable parameters.

For example, if `bar.yaml` refers to a concrete class `Bar.java` that resides in the package bar with some fixed parameter `fixed-param` set to `a`, then the YAML descriptor will look like:

```xml
# bar.yaml
class: bar.Bar
fixed-param: a
```

If a descriptor `foo.yaml` is a subclass of the `bar.yaml`, the YAML descriptor will look like:

```xml
# foo.yaml
inherit: bar
var: [x, y]
```
![resources][resources]
#### Parameters
Resources on a descriptor are configured using named parameters; any dash-separated string is a valid parameter name except for the reserved keywords: inherit, options, class and pipeline. The actual value of the parameter is either a Java primitive wrapper: Integer, Float, Long, Double, Boolean, or a String. For nested resources compound parameters are passed as Strings that are further parsed within the resource. For example, passing a RegEx pattern as a String parameter to a RoomAnnotator will look like:
```xml
class: annotators.RoomAnnotator

pattern: \\b[0-4]\\d[0-2]\\d\\d\\b
```
#### Cross-opts:
Combinatorial parameters are specified using the cross-opts mapping and declaring the desired values as elements on a list. For example for the following annotator descriptor:

```xml
class: annotators.RoomAnnotator

foo: bar

cross-opts:
    parameter-a: [value100,value200]
    parameter-b: [value300,value400] 
```

The configuration on Listing 2 will result in the 2x2 cross-product of configurations of the component: 
```xml
[foo:bar, parameter-a: value100, parameter-b: value300] 
[foo:bar, parameter-a: value200, parameter-b: value300]
[foo:bar, parameter-a: value100, parameter-b: value400] 
[foo:bar, parameter-a: value200, parameter-b: value400]
```

### CSE pipeline descriptors

#### Configuration
#### Phases

#### Pipeline

#### Options
![In-phase pipeline][inphasepipeline]

![Execution path][executionpathinphasepipeline] 
#### (Optional) Post-processing

## Creating a type system
Type system defines the domain: what is the subject matter being processed, and what kind of annotations are being produced, in a pipeline.

## Pipeline code

### Writing collection reader code

### Writing annotator code

### Writing cas consumer code



## Examples
These examples are based on the [UIMA SDK tutorial](http://uima.apache.org/d/uimaj-2.4.0/tutorials_and_users_guides.pdf). To see how they were migrated see section (Migrating your existing UIMA pipeline to CSE (and here)). 


### Example 1 (simple RoomNumberAnnotator)
For this example we will only use one type: RoomNumber. You can use the existing `TutorialTypeSystem.xml` under `src/main/resources/types/`. 

```xml
<?xml version="1.0" encoding="UTF-8" ?>
  <typeSystemDescription xmlns="http://uima.apache.org/resourceSpecifier">
    <name>TutorialTypeSystem</name>
    <description>Type System Definition for the tutorial examples - 
        as of Exercise 1</description>
    <vendor>Apache Software Foundation</vendor>
    <version>1.0</version>
    <types>
      <typeDescription>
        <name>org.apache.uima.tutorial.RoomNumber</name>
        <description></description>
        <supertypeName>uima.tcas.Annotation</supertypeName>
        <features>
          <featureDescription>
            <name>building</name>
            <description>Building containing this room</description>
            <rangeTypeName>uima.cas.String</rangeTypeName>
          </featureDescription>
        </features>
      </typeDescription>
    </types>
  </typeSystemDescription>
```

Simply include the `TutorialTypeSystem.xml` file to you META-INF/org.uimafit/types.txt file. Your types.txt file should look like:

```xml
classpath*:types/TutorialTypeSystem.xml
classpath*:types/SourceDocumentInformation.xml
```

Next, we will write the main yaml descriptor for the example. Notice that we are using the same collection-reader and cas consumer from before only we added the phase RoomNumberAnnotator. (src/main/resources/META-INF/oaqa-tutorial-ex1.yaml)

```YAML
configuration: 
  name: oaqa-tutorial
  author: oaqa

collection-reader:
  inherit: collection_reader.filesystem-collection-reader
  InputDirectory: data/
pipeline:
  - inherit: ecd.phase  
    name: RoomNumberAnnotator
    options: |
      - inherit: tutorial.ex1.RoomNumberAnnotator 

  - inherit: cas_consumer.AnnotationPrinter
```

Now we define the descriptor for the RoomNumberAnnotator by just specifying where the class is located. (src/main/resources/tutorial.ex1/RoomNumberAnnotator.yaml)

```yaml
class: org.apache.uima.tutorial.ex1.RoomNumberAnnotator
```
Finally, we write the code for the annotator, taken directly from the UIMA SDK tutorial (link). (src/main/java/org.apache.uima.tutorial.ex1/RoomNumberAnnotator.java)

```java
package org.apache.uima.tutorial.ex1;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.uima.analysis_component.JCasAnnotator_ImplBase;
import org.apache.uima.jcas.JCas;
import org.apache.uima.tutorial.RoomNumber;

/**
 * Example annotator that detects room numbers using Java 1.4 regular expressions.
 */
public class RoomNumberAnnotator extends JCasAnnotator_ImplBase {
  private Pattern mYorktownPattern = Pattern.compile("\\b[0-4]\\d-[0-2]\\d\\d\\b");
  private Pattern mHawthornePattern = Pattern.compile("\\b[JG1-4][1-2NS]-[A-Z]\\d\\d\\b");
  /**
   * @see JCasAnnotator_ImplBase#process(JCas)
   */
  public void process(JCas aJCas) {
    // get document text
    String docText = aJCas.getDocumentText();
    // search for Yorktown room numbers
    Matcher matcher = mYorktownPattern.matcher(docText);
    while (matcher.find()) {
      // found one - create annotation
      RoomNumber annotation = new RoomNumber(aJCas);
      annotation.setBegin(matcher.start());
      annotation.setEnd(matcher.end());
      annotation.setBuilding("Yorktown");
      annotation.addToIndexes();
    }
    // search for Hawthorne room numbers
    matcher = mHawthornePattern.matcher(docText);
    while (matcher.find()) {
      // found one - create annotation
      RoomNumber annotation = new RoomNumber(aJCas);
      annotation.setBegin(matcher.start());
      annotation.setEnd(matcher.end());
      annotation.setBuilding("Hawthorne");
      annotation.addToIndexes();
    }
  }
}
```
To run simply launch ex1.launch by right-clicking it and selecting Run As...-> ex1.

You should now be getting this output:

### Example 2 (passing parameters to an annotator)

```yaml
#oaqa-tutorial-ex2.yaml
configuration: 
  name: oaqa-tutorial
  author: oaqa

collection-reader:
  inherit: collection_reader.fs-collection-reader
  file: /data/UIMA_Seminars.txt
pipeline:
  - inherit: ecd.phase  
    name: RoomNumberAnnotator
    options: |
      - inherit: tutorial.ex2.RoomNumberAnnotator
  - inherit: cas_consumer.XmiWriterCasConsumer
```

```java
package org.apache.uima.tutorial.ex2;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.uima.UimaContext;
import org.apache.uima.analysis_component.AnalysisComponent;
import org.apache.uima.analysis_component.JCasAnnotator_ImplBase;
import org.apache.uima.analysis_engine.AnalysisEngineProcessException;
import org.apache.uima.jcas.JCas;
import org.apache.uima.resource.ResourceInitializationException;
import org.apache.uima.tutorial.RoomNumber;
import org.apache.uima.util.Level;

public class RoomNumberAnnotator extends JCasAnnotator_ImplBase {
  private Pattern[] mPatterns;
  private String[] mLocations;
 
  public void initialize(UimaContext aContext) throws ResourceInitializationException {
    super.initialize(aContext);
    // Get config. parameter values from oaqa-tutorial-ex2.yaml
    String[] patternStrings = (String[]) aContext.getConfigParameterValue("Patterns"); 
    mLocations = (String[]) aContext.getConfigParameterValue("Locations");

    // compile regular expressions
    mPatterns = new Pattern[patternStrings.length];
    for (int i = 0; i < patternStrings.length; i++) {
      mPatterns[i] = Pattern.compile(patternStrings[i]);
    }
  }

  /**
   * @see JCasAnnotator_ImplBase#process(JCas)
   */
  public void process(JCas aJCas) throws AnalysisEngineProcessException {
    // get document text
    String docText = aJCas.getDocumentText();
    // loop over patterns
    for (int i = 0; i < mPatterns.length; i++) {
      Matcher matcher = mPatterns[i].matcher(docText);
      while (matcher.find()) {
        // found one - create annotation
        RoomNumber annotation = new RoomNumber(aJCas);
        annotation.setBegin(matcher.start());
        annotation.setEnd(matcher.end());
        annotation.setBuilding(mLocations[i]);    
        annotation.addToIndexes();
        getContext().getLogger().log(Level.FINEST, "Found: " + annotation);
      }
    }
  }
}
```
To run simply launch ex2.launch by right-clicking it and selecting Run As...-> ex2.

You should now be getting this output:

### Example (UIMA-ASified RemoteRoomNumberAnnotator)

This example will show you how to run UIMA-AS annotators from within your CSE pipeline. The example will consist of two parts: (1) server-side setup, and (2) client-side setup. Since the server-side configuration is not currently supported by CSE, we will use the standard UIMA-AS SDK. You can read more about the purpose and usage of UIMA-AS [here](http://uima.apache.org/doc-uimaas-what.html).  

#### Server-side
1. Download the UIMA AS Asynchronous Scaleout from [here](http://uima.apache.org/downloads.cgi)
2. Decompress into desired directory where you want the remote service to run on.
3. cd into apache-uima-as-{version} 
4. Run `export UIMA_HOME=/path/to/directory/apache-uima-as-{version}/`
5. Run `./bin/startBroker.sh`

 This will output:
```xml
.
.
.
INFO  BrokerService                  - ActiveMQ 5.4.1 JMS Message Broker (localhost) is starting
INFO  BrokerService                  - For help or more information please see: http://activemq.apache.org/
INFO  ManagementContext              - JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
* INFO  TransportServerThreadSupport   - Listening for connections at: tcp://your-hostname:61616 *
.
.
```
 Look for the fourth line with `TransportServerThreadSupport` and copy the tcp address, in our case: `tcp://your-hostname:61616`; you will use it in later steps.
6. In our example we will run the example RoomNumberAnnotator from the UIMA-AS tutorial, but these instructions will apply to other annotators as well. As with any annotator in the previous examples, the UIMA-AS annotator will be composed of an annotator descriptor and a Java class implementation. However, in addition to these the the UIMA-AS annotator will require an additional _deployment_ descriptor. 

 Since currently CSE does not support server-side UIMA-AS annotators, the annotator descriptor will not be a CSE-style yaml file, but a "vanilla" UIMA XML descriptor provided in the UIMA-AS SDK. It will look like:
```xml
<!-- examples/descriptors/tutorial/ex2/RoomNumberAnnotator.xml -->
<analysisEngineDescription xmlns="http://uima.apache.org/resourceSpecifier" xmlns:xi="http://www.w3.org/2001/XInclude">
	<frameworkImplementation>org.apache.uima.java</frameworkImplementation>
	<primitive>true</primitive>
	<annotatorImplementationName>org.apache.uima.tutorial.ex2.RoomNumberAnnotator</annotatorImplementationName>
	<analysisEngineMetaData>
		<name>Room Number Annotator</name>
		<description>An example annotator that searches for room numbers in the IBM Watson
			research buildings.</description>
		<version>1.0</version>
		<vendor>The Apache Software Foundation</vendor>
		<configurationParameters>
			<configurationParameter>
				<name>Patterns</name>
				<description>List of room number regular expression
					pattterns.</description>
				<type>String</type>
				<multiValued>true</multiValued>
				<mandatory>true</mandatory>
			</configurationParameter>
			<configurationParameter>
				<name>Locations</name>
				<description>List of locations corresponding to the room number
					expressions specified by the Patterns parameter.</description>
				<type>String</type>
				<multiValued>true</multiValued>
				<mandatory>true</mandatory>
			</configurationParameter>
		</configurationParameters>
		<configurationParameterSettings>
			<nameValuePair>
				<name>Patterns</name>
				<value>
					<array>
						<string>\b[0-4]\d-[0-2]\d\d\b</string>
						<string>\b[G1-4][NS]-[A-Z]\d\d\b</string>
						<string>\bJ[12]-[A-Z]\d\d\b</string>
					</array>
				</value>
			</nameValuePair>
			<nameValuePair>
				<name>Locations</name>
				<value>
					<array>
						<string>Watson - Yorktown</string>
						<string>Watson - Hawthorne I</string>
						<string>Watson - Hawthorne II</string>
					</array>
				</value>
			</nameValuePair>
		</configurationParameterSettings>
		<typeSystemDescription>
			<imports>
				<import location="../ex1/TutorialTypeSystem.xml"/>
			</imports>
		</typeSystemDescription>
		<capabilities>
			<capability>
				<inputs></inputs>
				<outputs>
					<type>org.apache.uima.tutorial.RoomNumber</type>
					<feature>org.apache.uima.tutorial.RoomNumber:building</feature>
				</outputs>
				<languagesSupported></languagesSupported>
			</capability>
		</capabilities>
		<operationalProperties>
			<modifiesCas>true</modifiesCas>
			<multipleDeploymentAllowed>true</multipleDeploymentAllowed>
			<outputsNewCASes>false</outputsNewCASes>
		</operationalProperties>
	</analysisEngineMetaData>
</analysisEngineDescription>
```


 The code for the annotator will be exactly the same as the one from Example 2 and can also be found under `./examples/src/org/apache/uima/tutorial/ex2/RoomNumberAnnotator.java`. 

 Finally, the _deployment_ descriptor tells the broker from step 5 how to run the annotator:
```xml
<!-- ./examples/deploy/as/Deploy_RoomNumberAnnotator.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<analysisEngineDeploymentDescription 
  xmlns="http://uima.apache.org/resourceSpecifier">
  
  <name>Room Number Annotator</name>
  <description>Deploys the Room Number Annotator Primitive AE</description>
  
  <deployment protocol="jms" provider="activemq">
    <service>
      <inputQueue endpoint="RoomNumberAnnotatorQueue" brokerURL="tcp://your-hostname:61616"/>
      <topDescriptor>
       <import location="../../descriptors/tutorial/ex2/RoomNumberAnnotator.xml"/> 
      </topDescriptor>
    </service>
  </deployment>
</analysisEngineDeploymentDescription>
```

Notes:
    * The `inputQueue endpoint` tag contains the endpoint that will be used by the client-side wrapper to connect with the annotator.
    * The `location` is the path to the annotator descriptor. 

7. Run `./bin/deployAsyncService.sh ./examples/deploy/as/Deploy_RoomNumberAnnotator.xml` to launch the RoomNumberAnnotator as a remote UIMA-AS service. It should now display this output:

```xml
Service:Room Number Annotator Initialized. Ready To Process Messages From Queue:RoomNumberAnnotatorQueue
Press 'q'+'Enter' to quiesce and stop the service or 's'+'Enter' to stop it now.
Note: selected option is not echoed on the console.
```
 The broker is now listening on the port for requests for the RoomNumberAnnotator from the client-side on the RoomNumberAnnotatorQueue endpoint defined in the deployment descriptor from step 6. 

#### Client-side:
For the client-side we can use the same pipeline descriptor from Example 2, only changing the annotator from RoomNumberAnnotator to RemoteRoomNumberAnnotator:

```xml
#./src/main/resources/oaqa-tutorial-ex2-remote.yaml
configuration: 
  name: oaqa-tutorial
  author: oaqa
collection-reader:
  inherit: collection_reader.filesystem-collection-reader
  InputDirectory: data/
  
pipeline:
  - inherit: ecd.phase  
    name: RemoteRoomNumberAnnotator
    options: |
      - inherit: tutorial.ex2.RemoteRoomNumberAnnotator 

  - inherit: cas_consumer.AnnotationPrinter
```

Now, we write the `RemoteRoomNumberAnnotator.yaml` as follows:
```xml
class: edu.cmu.lti.oaqa.ecd.phase.adapter.JMSAdapterWrapper
brokerUrl: tcp://your-hostname:61616
endpoint: RoomNumberAnnotatorQueue
timeout: 5000
getmetatimeout: 5000
cpctimeout: 5000
```

Notes:
* The class is the path to the wrapper included in uima-ecd. It will be used for all client-side UIMA-AS descriptors.
* The brokerUrl is the address you copied from step 5. 
* The endpoint is the endpoint defined in the server-side setup (step 6).
* Various timeouts in milliseconds (will add more...)

Now you should be able to run the complete pipeline by running `launching/ex2-remote.launch` in Eclipse. The output should be identical to that in Example 2.



# Migrating your existing UIMA pipeline to CSE


* Have to use UIMA-fit style of declaring type system in META-INF/org.uimafit/types.txt (see slides)

* Have to use UIMA-fit hierarchy of cas_consumerimpl in order for it to work in the pipeline
 * initialize() -> initialize(UimaContext context)
     * initialize takes argument UimaContext context
 * instead of processCas() -> process(CAS aCas)

* Collection reader has to extend AbstractCollectionReader
    * can not use out-of-the-box UIMA collection readers
    * implement method getNextElement
        * returns the next data element in the pipeline
        * This is instead of using getNext(jcas)
    * Override and call super of initialize


# FAQ


# Acknowledgements 
Thanks to Zi Yang (@ziy), Elmer Garduno (@elmerg), and OAQA team members! 

# TODO
1. Take UIMA GUI screenshots of the output of each example.
2. How to create your own launch configuration using a yaml descriptor (ECD-Driver).

# References

[1] (Session 12: Configuration Space Exploration (SE class)) - (SE)  
[2] (Elmer's technical report) - (E)  
[3] (Zi's Slides) - (Z)  
[4] Zi HW0 - (hw0)  
[5] Zi HW1 - (hw1)  
[6] Zi HW2 - (hw2)  
[7] [UIMA SDK Tutorial](http://uima.apache.org/d/uimaj-2.4.0/index.html) (USDK)  
[resources]: https://github.com/oaqa/oaqa-tutorial/raw/master/resources/imgs/resources.png "resources"
[inphasepipeline]: https://github.com/oaqa/oaqa-tutorial/raw/master/resources/imgs/inphasepipeline.png "inphasepipeline"
[executionpathinphasepipeline]: https://github.com/oaqa/oaqa-tutorial/raw/master/resources/imgs/executionpathinphasepipeline.png "executionpathinphasepipeline"
[threephasepipeline]: https://github.com/oaqa/oaqa-tutorial/raw/master/resources/imgs/threephasepipeline.png "threephasepipeline"
Add a custom footer
Add a custom sidebar
Clone this wiki locally