FAB-15330 Support Node 10 Async Iterators

Add support for Node 10 Async Iterators for
node.js chaincode State and History iterators

Change-Id: I81e5c9b63009af4a114b9cf0c3da7e2822280ea3
Signed-off-by: Dave Kelsey <d_kelsey@uk.ibm.com>

.eslintrc

@@ -1,12 +1,13 @@
 {
     "env": {
+
         "es6": true,
         "node": true,
         "mocha": true
     },
 	"extends": "eslint:recommended",
     "parserOptions": {
-        "ecmaVersion": 8,
+        "ecmaVersion": 9,
         "sourceType": "module"
     },
     "rules": {

docs/tutorials/tutorials.json

@@ -4,5 +4,8 @@
 	},
 	"using-contractinterface":{
 		"title": "Using the Contract Interface"
-	}
+    },
+    "using-iterators": {
+        "title": "Working with apis that return iterators"
+    }
 }

docs/tutorials/using-iterators.md

@@ -0,0 +1,166 @@
+# Working with iterators
+The fabric-shim api provides capability to retrieve blocks of information. Information such as the history of a key, a set of keys and their values from a range of keys and also a set of keys and values when performing a rich query if your network is using couchdb to manage the world state.
+
+An example of these apis (but may not be a complete list) is given here
+
+- History
+  -	getHistoryForKey
+- Private data
+  -	getPrivateDataByPartialCompositeKey
+  -	getPrivateDataByRange
+  -	getPrivateDataQueryResult
+- Rich query
+  - getQueryResult
+  - getQueryResultWithPagination 
+- Range queries 
+  - getStateByPartialCompositeKey
+  -	getStateByPartialCompositeKeyWithPagination
+  - getStateByRange
+  - getStateByRangeWithPagination
+
+These api's are a request to return a set of data for which you need to iterate over using the provided iterator. Some of these apis will return the iterator directly and others return an iterator as part of an object property. In previous versions of the fabric-shim api you would need to know which ones did that and handle it appropriately and you need to check the documentation, but the rules are
+- all private data range queries return an object with just an iterator property containing the iterator
+- all Pagination queries return an object with an iterator property and metadata property
+- all other rich/range/history queries return just the iterator itself.
+
+These iterators were essentially asynchronous iterators (the next and close methods returned promises) but you couldn't use standard iterator capabilities such as for/of constructs in node because node could not work with the concept of asynchronous iterators.
+
+From fabric v2.0 onwards, node chaincode will be using node 10 as the node version and this has added support for asynchronous iterators. Also in fabric v2.0 onwards fabric-shim has added support to enable it's asynchronous iterators so that `for/of` can now be used, but note that they don't have full support so should not be used in generator functions.
+
+As a comparison lets present first how you would use iterators in previous releases and then show the new way.
+
+## How to use, the old way
+In the past, you might have coded something like this to process an iterator
+
+```javascript
+async function getAllResults(iterator) {
+    const allResults = [];
+    while (true) {
+        const res = await iterator.next();
+        if (res.value) {
+            // if not a getHistoryForKey iterator then key is contained in res.value.key
+            allResults.push(res.value.value.toString('utf8'););
+        }
+
+        // check to see if we have reached then end
+        if (res.done) {
+            // explicitly close the iterator            
+            await iterator.close();
+            return allResults;
+        }
+    }
+}
+```
+as iterator.next() returned an object of the form
+```javascript
+{value: KV|KeyModification object
+ done: true|false}
+```
+
+and the structures of the value property are best described by the typescript definitions
+
+```javascript
+interface KV {
+    key: string;
+    value: Buffer;
+    getKey(): string;
+    getValue(): ProtobufBytes;
+}
+
+interface KeyModification {
+    is_delete: boolean;
+    value: ProtobufBytes;
+    timestamp: Timestamp;
+    tx_id: string;
+    getIsDelete(): boolean;
+    getValue(): ProtobufBytes;
+    getTimestamp(): Timestamp;
+    getTxId(): string;
+}
+```
+
+and you would obtain an iterator as follows (depending on the api you are calling)
+
+```javascript
+// use await to get the iterator and pass it to getAllResults
+const iterator = await ctx.stub.getStateByRange(startKey, endKey);
+let results await getAllResults(iterator);
+
+// use await to get the object containing the iterator and metadata and
+// pass it to getAllResults. All Pagination type queries return an object
+// with iterator and metadata properties
+let response = await ctx.stub.getQueryResultWithPagination(JSON.stringify(query), 2);
+const {iterator, metadata} = response;
+let results = await getAllResults(iterator);
+
+// use await to get the object containing the iterator and metadata and
+// pass it to getAllResults. All Private Data type queries return an object
+// with only an iterator property
+let response = await ctx.stub.getPrivateDataByRange(collection, startKey, endKey);
+let results = await getAllResults(response.iterator);
+```
+
+## How to use, the new way
+The new way of using `for/await/of` in node.js makes things much easier. You don't have to worry about each of the api's returning a different value (is it an iterator or is it an object with an iterator in the iterator property). You also don't have to explicitly close the iterator any more. Here is a re-implementation of the `getAllResults` function
+
+```javascript
+async function getAllResults(promiseOfIterator) {
+    const allResults = [];
+    for await (const res of promiseOfIterator) {
+        // no more res.value.value ...
+        // if not a getHistoryForKey iterator then key is contained in res.key
+        allResults.push(res.value.toString('utf8'));
+    }
+
+    // iterator will be automatically closed on exit from the loop
+    // either by reaching the end, or a break or throw terminated the loop
+    return allResults;
+}
+```
+It's more concise, the only difference between the 2 signatures of the old versus the new is what you pass to it. Previously you passed the actual iterator but in the new version you pass the promise that will resolve to either an iterator or an object containing the iterator. So if we take the 3 previous calls to the various apis, how do they look now.
+
+```javascript
+// use await to get the iterator and pass it to getAllResults
+const promiseOfIterator = ctx.stub.getStateByRange(startKey, endKey);
+let results await getAllResults(promiseOfIterator);
+
+// use await to get the object containing the iterator and metadata and
+// pass it to getAllResults. All Pagination type queries return an object
+// with iterator and metadata properties
+let promiseOfIterator = ctx.stub.getQueryResultWithPagination(JSON.stringify(query), 2);
+let results = await getAllResults(promiseOfIterator);
+const metadata = (await promiseOfIterator).metadata;
+
+// use await to get the object containing the iterator and metadata and
+// pass it to getAllResults. All Private Data type queries return an object
+// with only an iterator property
+let promiseOfIterator = ctx.stub.getPrivateDataByRange(collection, startKey, endKey);
+let results = await getAllResults(promiseOfIterator);
+```
+Lets note the differences
+1. You do not use `await` when invoking the stub function. This means it will return a promise
+2. You don't have to worry about whether the promise will resolve to an iterator or an object containing the iterator. The returned value is handled in the same way in all cases
+3. In the case of pagination apis it's still easy to get the required metadata response.
+
+## example of getHistoryForKey
+All the functions that return a set of data, except one, return data in the KV structure format. The exception is getHistoryForKey whose dataset is of the form KeyModification. Below is a simple example of using getHistoryForKey.
+
+```javascript
+const promiseOfIterator = ctx.stub.getHistoryForKey(key);
+
+const results = [];
+for await (const keyMod of promiseOfIterator) {
+    const resp = {
+        timestamp: keyMod.timestamp,
+        txid: keyMod.tx_id
+    }
+    if (keyMod.is_delete) {
+        resp.data = 'KEY DELETED';
+    } else {
+        resp.data = keyMod.value.toString('utf8');
+    }
+    results.push(resp);
+}
+// results array contains the key history
+```
+

fabric-shim/lib/iterators.js

@@ -1,7 +1,7 @@
 'use strict';
 const ProtoLoader = require('./protoloader');
 const path = require('path');
-const EventEmitter = require('events');
+const logger = require('./logger').getLogger('lib/iterators.js');
 
 const _queryresultProto = ProtoLoader.load({
     root: path.join(__dirname, './protos'),
@@ -15,7 +15,7 @@ const _queryresultProto = ProtoLoader.load({
  * @class
  * @memberof fabric-shim
  */
-class CommonIterator extends EventEmitter {
+class CommonIterator {
 
     /**
 	 * constructor
@@ -25,7 +25,6 @@ class CommonIterator extends EventEmitter {
 	 * @param {object} response decoded payload
 	 */
     constructor(handler, channel_id, txID, response, type) {
-        super();
         this.type = type;
         this.handler = handler;
         this.channel_id = channel_id;
@@ -41,6 +40,7 @@ class CommonIterator extends EventEmitter {
 	 * if there is a problem
 	 */
     async close() {
+        logger.debug('close called on %s iterator for txid: %s', this.type, this.txID);
         return await this.handler.handleQueryStateClose(this.response.id, this.channel_id, this.txID);
     }
 
@@ -59,16 +59,15 @@ class CommonIterator extends EventEmitter {
 
 
     /*
-	 * creates a return value and emits an event
+	 * creates a return value
 	 */
     _createAndEmitResult() {
         const queryResult = {};
         queryResult.value = this._getResultFromBytes(this.response.results[this.currentLoc]);
         this.currentLoc++;
-        queryResult.done = !(this.currentLoc < this.response.results.length || this.response.has_more);
-        if (this.listenerCount('data') > 0) {
-            this.emit('data', this, queryResult);
-        }
+        // TODO: potential breaking change if it's assumed that if done == true then it has a valid value
+        queryResult.done = false;
+        // queryResult.done = !(this.currentLoc < this.response.results.length || this.response.has_more);
         return queryResult;
     }
 
@@ -92,22 +91,10 @@ class CommonIterator extends EventEmitter {
                     this.response = response;
                     return this._createAndEmitResult();
                 } catch (err) {
-                    // if someone is utilising the event driven way to work with
-                    // iterators (by explicitly checking for data here, not error)
-                    // then emit an error event. This means it will emit an event
-                    // even if no-one is listening for the error event. Error events
-                    // are handled specially by Node.
-                    if (this.listenerCount('data') > 0) {
-                        this.emit('error', this, err);
-                        return;
-                    }
+                    logger.error('unexpected error received getting next value: %s', err.message);
                     throw err;
                 }
             }
-            // no more, just return EMCA spec defined response
-            if (this.listenerCount('end') > 0) {
-                this.emit('end', this);
-            }
             return {done: true};
         }