Skip to content
This repository has been archived by the owner on Aug 19, 2020. It is now read-only.

Commit

Permalink
Clean up some README sections, fix branch for develop codecov
Browse files Browse the repository at this point in the history
  • Loading branch information
@mlhenderson
mlhenderson committed Sep 23, 2016
1 parent 30c5e63 commit 7b0606e
Showing 1 changed file with 1 addition and 76 deletions.
77 changes: 1 addition & 76 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Master branch status

Develop branch status
[![Build Status](https://travis-ci.org/kbase/data_api.svg?branch=develop)](https://travis-ci.org/kbase/data_api)
[![Coverage Status](http://codecov.io/github/kbase/data_api/coverage.svg?branch=develop)](http://codecov.io/github/kbase/data_api?branch=master)
[![Coverage Status](http://codecov.io/github/kbase/data_api/coverage.svg?branch=develop)](http://codecov.io/github/kbase/data_api?branch=develop)
![Coverage Graph](http://codecov.io/github/kbase/data_api/branch.svg?branch=develop&time=1y)

##### Table of Contents
Expand Down Expand Up @@ -187,18 +187,6 @@ The sub-repository will be cloned in the directory `test_resources`.
Starting the Redis instance:

redis-server redis.conf

## Starting the Data API services

Services can be started using the data_api_start_service.py script, which is in your path from a virtualenv install.

data_api_start_service.py --config deployment.cfg --service taxon --port 9101
data_api_start_service.py --config deployment.cfg --service assembly --port 9102

You can add a --kbase_url argument to indicate which service targets and configs from deployment.cfg to use.
For instance, to set the services to use local files and assume a running Redis instance:

data_api_start_service.py --config deployment.cfg --service assembly --port 9102 --kbase_url=dir_cache

The available targets are:

Expand All @@ -209,30 +197,6 @@ The sub-repository will be cloned in the directory `test_resources`.
- **dir_cache** : Use local test files, assume local Redis caching
- **dir_nocache** : Use local test files, do not attempt to cache using Redis

### Service logging

The Data API service logging is controlled by a file named, by default, "logging.conf", in the same directory as the configuration file from the `--config` option. You can choose another file with the `--log-config` option. If no explicit configuration is given, and the default file is not present, some basic default logging configuration will be used. Failure to open or parse a file provided with the `--log-config` option will cause the program to stop and exit.

For example:

* `data_api_start_service.py --service genome_annotation --config deployment.cfg` will look for a file `logging.conf` in the current directory.
* `data_api_start_service.py --service genome_annotation --config deployment.cfg --log-config ./logging.conf` is equivalent to the previous command.
* `data_api_start_service.py --service genome_annotation --config deployment.cfg --log-config /tmp/logging-test/logging.yaml` will configure logging from the named file in "/tmp/logging-test" instead of looking for "logging.conf" in the current directory. If that file does not exist, the program will stop with an error.

The configuration file is formatted using either the (older) ConfigParser format used by `logging.config.fileConfig` or the (newer) YAML format that is parsed and fed to `logging.config.dictConfig`; the format is auto-detected via trial and error. See the [logging.config module](https://docs.python.org/2.7/library/logging.config.html) documentation for details on the formats of these files. The `logging.conf` example in the root of the repository demonstrates how to configure output to a file and console, and control the level of logging for each service.

The output log format is largely controlled by the configuration file. Some of the messages have additional standardized formatting applied to the part in the `%(message)s` format code, i.e. the free-form string part of the log message. This standardized formatting is encoded the `doekbase.util.log_{event,start,end}` functions. (Note: This formatting can be altered, too, by changing the global `_MESSAGE_FORMAT` dict of the module; but this is not recommended). The basic idea is to reduce the free-form part to a simple name for the event, and put everything else in key/value pairs with a standard format. Also, messages at the beginning and the end of something (in cases where that makes sense) should be named so that they are easily matched. For example, the pair of log messages for the service starting and stopping look like this:

2016-02-18 05:34:39,260 [INFO] doekbase.data_api.annotation.genome_annotation.service.driver: start_service.begin | host='',port=9103
# ...
2016-02-18 05:35:01,910 [INFO] doekbase.data_api.annotation.genome_annotation.service.driver: start_service.end (22.649431) | host='',port=9103

Note the '.begin' and '.end' event suffixes, as well as the "|" separating the message into name from the key/value pairs of the values. The number in parentheses on the '.end' event is the duration in seconds since the corresponding '.start'. For a message that is not part of a begin/end pair, the format is the same:

2016-02-18 05:34:39,137 [INFO] doekbase.data_api_start_service: activating REDIS | host=localhost,port=6379

Not all, in fact right now not even most, of the messages from the services have this standard format extension. But the goal is to slowly convert them. (We'll see how that goes!)

## Testing

To verify all Data API code with local tests.
Expand All @@ -243,54 +207,15 @@ Not all, in fact right now not even most, of the messages from the services have

Install the [Test data](README.md#test-data)

Start each of the API services:

data_api_start_service.py --config deployment.cfg --service taxon --port 9101 &
data_api_start_service.py --config deployment.cfg --service assembly --port 9102 &
data_api_start_service.py --config deployment.cfg --service genome_annotation --port 9103 &

Run nosetests from the data_api source directory, which will use the test data:

nosetests -c nose.cfg -c nose-local.cfg -s doekbase.data_api

### TR;DL testing

An easier way to run the tests is to run the script `run-tests-local.sh` at the top level of the directory.

In addition to performing steps listed above, this does some sanity checks and also regenerates the Thrift
server and client stubs (in case you changed these). It also runs the Thrift preprocessor (see below).

### JavaScript tests

For the JavaScript API, all the code and tests live under `jslib`. See the README in that directory for more details.


### Example narratives

Retrieving and counting genomic features with a local data API client for a [GenomeAnnotation object] (https://narrative-ci.kbase.us/narrative/ws.3413.obj.1)
Retrieving and counting genomic features with direct data API access for a [GenomeAnnotation object] (https://narrative-ci.kbase.us/narrative/ws.3292.obj.1)
A [table] of genome properties for all genomes belonging to a taxon (https://narrative-ci.kbase.us/narrative/ws.3524.obj.1)
Panel of data quality plots for [GenomeAnnotation and Assembly objects] (https://narrative-ci.kbase.us/narrative/ws.3413.obj.1)

## Thrift preprocessor

For convenience, we have added a simple Thrift preprocessor. There is a driver program in the `bin` directory called `data_api_preprocess_thrift`.
This calls functions in the `doekbase.data_api.thrift_include` module to do the work.
"Why? Dear God, why?" you ask, since Thrift already has an include mechanism. Yes, it does, but this ends up complicating the namespace and adding files. And we just didn't feel
like dealing with that today. We wanted a dead-simple literal include mechanism.
So, what you do is add this line to your spec:

#%include api_shared

in your API's Thrift spec, and this special comment will get replaced with:

#%include api_shared
<contents of api_shared.thrift>
#%endinclude api_shared
In the driver script, you can specify include directories in which to look for the .thrift files to include. This is invoked automatically as part of rebuilding the
clients and servers, with the include directories being the current directory and the `thrift/specs/common` directory.

Note that includes can include other things (this is in fact what `api_shared.thrift` does), etc.

# FIN

0 comments on commit 7b0606e

Please sign in to comment.