Merge pull request #7 from dmlc/master

merge dmlc/master

doc/python/index.md

@@ -18,8 +18,8 @@ Code Examples
 
 Python API Documents
 --------------------
+* [High Level Model Training Related API](model.md)
 * [NDArray API](ndarray.md)
 * [Symbolic API](symbol.md)
 * [KVStore API](kvstore.md)
-* [Data Loading API](io.md)
-* [Model API](model.md)
+* [Data Loading API](io.md)

doc/python/model.md

@@ -4,12 +4,12 @@ The model API in mxnet as not really an API.
 It is a thin wrapper build on top of [ndarray](ndarray.md) and [symbolic](symbol.md)
 modules to make neural network training easy.
 
-* [Train a Model](#overloaded-operators) introduces operator overloading of symbols
-* [Serialization](#serialization) introduces how to save and load symbols.
-* [Multiple Outputs](#multiple-outputs) introduces how to configure multiple outputs
-* [API Reference](#api-reference) gives reference to all functions.
-* [Symbol Object Document](#mxnet.symbol.Symbol) gives API reference to the Symbol Object.
-
+* [Train a Model](#train-a-model) introduces basic training.
+* [Save the Model](#save-the-model)
+* [Periodically Checkpoint](#periodically-checkpoint)
+* [Initializer API Reference](#initializer-api-reference)
+* [Evaluation Metric API Reference](#initializer-api-reference)
+* [Optimizer API Reference](#optimizer-api-reference)
 
 Train a Model
 -------------
@@ -18,13 +18,13 @@ then call ```model.Feedforward.create``` to create a model for you.
 The following example creates a two layer neural networks.
 
 ```python
-batch_size = 100
+# configure a two layer neuralnetwork
 data = mx.symbol.Variable('data')
 fc1 = mx.symbol.FullyConnected(data, name='fc1', num_hidden=128)
-act1 = mx.symbol.Activation(fc1, name='relu1', act_type="relu")
-fc2 = mx.symbol.FullyConnected(act1, name = 'fc2', num_hidden = 64)
-softmax = mx.symbol.Softmax(fc2, name = 'sm')
-
+act1 = mx.symbol.Activation(fc1, name='relu1', act_type='relu')
+fc2 = mx.symbol.FullyConnected(act1, name='fc2', num_hidden=64)
+softmax = mx.symbol.Softmax(fc2, name='sm')
+# create a model
 model = mx.model.FeedForward.create(
      softmax,
      X=data_set,
@@ -44,18 +44,19 @@ We also provide a save and load function.
 ```python
 # save a model to mymodel-symbol.json and mymodel-0100.params
 prefix = 'mymodel'
-model.save(prefix, 100)
+iteration = 100
+model.save(prefix, iteration)
 
 # load model back
-model_loaded = mx.model.FeedForward.load(prefix, 100)
+model_loaded = mx.model.FeedForward.load(prefix, iteration)
 ```
 The advantage of this save and load function is they are language agnostic,
 and you should be able to save and load directly into cloud storage such as S3 and HDFS.
 
 Periodically Checkpoint
 -----------------------
 It is also helpful to periodically checkpoint your model after each iteration.
-To do so, you can simply add a checkpoint callback to the function.
+To do so, you can simply add a checkpoint callback ```do_checkpoint(path)``` to the function.
 The training process will automatically checkpoint to the specified place after
 each iteration.
 
@@ -65,8 +66,7 @@ model = mx.model.FeedForward.create(
      softmax,
      X=data_set,
      iter_end_callback=mx.model.do_checkpoint(prefix),
-     num_round=num_round,
-     learning_rate=0.01)
+     ...)
 ```
 You can load the model checkpoint later using ```Feedforward.load```.
 
@@ -82,6 +82,7 @@ model = mx.model.FeedForward.create(
      ctx=devices,
      ...)
 ```
+The training will be done in a data parallel way on the GPUs you specified.
 
 Initializer API Reference
 -------------------------

doc/python/ndarray.md

@@ -1,7 +1,100 @@
 NDArray API
 ===========
 
+NDArray package (`mxnet.ndarray`) contains tensor operations similar to `numpy.ndarray`. The syntax is similar except for some additional calls to deal with I/O and multi-devices.
 
+Create NDArray
+--------------
+Like `numpy`, you could create `mxnet.ndarray` like followings:
+```python
+>>> import mxnet as mx
+>>> a = mx.nd.zeros((100, 50))              # all-zero array of dimension 100x50
+>>> b = mx.nd.ones((256, 32, 128, 1))       # all-one array of dimension 256x32x128x1
+>>> c = mx.nd.array([[1, 2, 3], [4, 5, 6]]) # initialize array with contents
+```
+
+NDArray operations
+-------------------
+We provide some basic ndarray operations like arithmetic and slice operations. More operations are coming in handy!
+
+### Arithmetic operations
+```python
+>>> import mxnet as mx
+>>> a = mx.nd.zeros((100, 50))
+>>> a.shape
+(100L, 50L)
+>>> b = mx.nd.ones((100, 50))
+>>> c = a + b
+>>> d = a - b     # c and d will be calculated in parallel here!
+>>> b += d        # inplace operation, b's contents will be modified, but c and d won't be affected.
+```
+
+### Slice operations
+```python
+>>> import mxnet as mx
+>>> a = mx.nd.zeros((100, 50))
+>>> a[0:10] = 1   # first 10 rows will become 1
+```
+
+Conversion from/to `numpy.ndarray` and I/O
+--------------------------------
+MXNet NDArray supports pretty nature way to convert from/to `mxnet.ndarray` to/from `numpy.ndarray`:
+```python
+>>> import mxnet as mx
+>>> import numpy as np
+>>> a = np.array([1,2,3])
+>>> b = mx.nd.array(a)                  # convert from numpy array
+>>> b
+<mxnet.ndarray.NDArray object at ...>
+>>> b.asnumpy()                         # convert to numpy array
+array([ 1., 2., 3.], dtype=float32)
+```
+
+We also provide two convenient functions to help save and load file from I/O:
+```python
+>>> import mxnet as mx
+>>> a = mx.nd.zeros((100, 200))
+>>> mx.nd.save("/path/to/array/file", a)
+>>> mx.nd.save("s3://path/to/s3/array", a)
+>>> mx.nd.save("hdfs://path/to/hdfs/array", a)
+>>> from_file = mx.nd.load("/path/to/array/file")
+>>> from_s3 = mx.nd.load("s3://path/to/s3/array")
+>>> from_hdfs = mx.nd.load("hdfs://path/to/hdfs/array")
+```
+The good thing about using the above `save` and `load` interface is that:
+- You could use the format across all `mxnet` language bindings.
+- Already support S3 and HDFS.
+
+Multi-device support
+-------------------
+The device information is stored in `mxnet.Context` structure. When creating ndarray in mxnet, user could either use the context argument (default is CPU context) to create arrays on specific device or use the `with` statement as follows:
+```python
+>>> import mxnet as mx
+>>> cpu_a = mx.nd.zeros((100, 200))
+>>> cpu_a.context
+Context(device_type=cpu, device_id=0)
+>>> with mx.Context(mx.gpu(0)):
+>>>   gpu_a = mx.nd.ones((100, 200))
+>>> gpu_a.context
+Context(device_type=gpu, device_id=0)
+>>> ctx = mx.Context(mx.gpu(0))
+>>> gpu_b = mx.nd.zeros((100, 200), ctx)
+>>> gpu_b.context
+Context(device_type=gpu, device_id=0)
+```
+
+Currently, we *DO NOT* allow operations among arrays from different contexts. To allow this, use `copyto` member function to copy the content to different devices and continue computation:
+```python
+>>> import mxnet as mx
+>>> x = mx.nd.zeros((100, 200))
+>>> with mx.Context(mx.gpu(0)):
+>>>   y = mx.nd.zeros((100, 200))
+>>> z = x + y
+mxnet.base.MXNetError: [13:29:12] src/ndarray/ndarray.cc:33: Check failed: lhs.ctx() == rhs.ctx() operands context mismatch
+>>> cpu_y = mx.nd.zeros((100, 200))
+>>> y.copyto(cpu_y)
+>>> z = x + cpu_y
+```
 
 NDArray API Reference
 ---------------------

doc/python/symbol.md

@@ -3,8 +3,9 @@ MXNet Python Symbolic API
 * [How to Commpose Symbols](#overloaded-operators) introduces operator overloading of symbols
 * [Serialization](#serialization) introduces how to save and load symbols.
 * [Multiple Outputs](#multiple-outputs) introduces how to configure multiple outputs
-* [API Reference](#api-reference) gives reference to all functions.
-* [Symbol Object Document](#mxnet.symbol.Symbol) gives API reference to the Symbol Object.
+* [Symbol Creation API Reference](#symbol-creationapi-reference) gives reference to all functions.
+* [Symbol Object Document](#mxnet.symbol.Symbol) gives API reference to the Symbol Object
+* [Execution API Reference](#execution-api-reference) tell us on what executor can do.
 
 How to Compose Symbols
 ----------------------

include/mxnet/c_api.h

@@ -435,6 +435,25 @@ MXNET_DLL int MXSymbolListArguments(SymbolHandle symbol,
 MXNET_DLL int MXSymbolListOutputs(SymbolHandle symbol,
                                   mx_uint *out_size,
                                   const char ***out_str_array);
+/*!
+ * \brief Get a symbol that contains all the internals.
+ * \param symbol The symbol
+ * \param out The output symbol whose outputs are all the internals.
+ * \return 0 when success, -1 when failure happens
+ */
+MXNET_DLL int MXSymbolGetInternals(SymbolHandle symbol,
+                                   SymbolHandle *out);
+/*!
+ * \brief Get index-th outputs of the symbol.
+ * \param symbol The symbol
+ * \param index the Index of the output.
+ * \param out The output symbol whose outputs are all the internals.
+ * \return 0 when success, -1 when failure happens
+ */
+MXNET_DLL int MXSymbolGetOutput(SymbolHandle symbol,
+                                mx_uint index,
+                                SymbolHandle *out);
+
 /*!
  * \brief List auxiliary states in the symbol.
  * \param symbol the symbol

include/mxnet/symbolic.h

@@ -101,6 +101,12 @@ class Symbol {
    */
   Symbol operator () (const std::unordered_map<std::string, Symbol>& kwargs,
                       const std::string& name) const;
+  /*
+   * \brief Get all the internal nodes of the symbol.
+   * \return symbol A new symbol whose output contains all the outputs of the symbols
+   *  Including input variables and intermediate outputs.
+   */
+  Symbol GetInternals() const;
   /*!
    * \brief get the gradient graph
    * \param wrt with respect to the input

python/mxnet/metric.py

@@ -1,5 +1,6 @@
 # pylint: disable=invalid-name
 """Online evaluation metric module."""
+from .base import string_types
 import numpy as np
 
 class EvalMetric(object):
@@ -52,15 +53,34 @@ def update(self, pred, label):
         self.num_inst += label.size
 
 
-def create(name):
+class CustomMetric(EvalMetric):
+    """Calculate accuracy"""
+    def __init__(self, feval):
+        name = feval.__name__
+        if name.find('<') != -1:
+            name = 'custom(%s)' % name
+        super(CustomMetric, self).__init__(name)
+        self._feval = feval
+
+    def update(self, pred, label):
+        self.sum_metric += self._feval(pred, label)
+        self.num_inst += 1
+
+
+def create(metric):
     """Create an evaluation metric.
 
     Parameters
     ----------
-    name : str
-        The name of the metric
+    metric : str or callable
+        The name of the metric, or a function
+        providing statistics given pred, label NDArray.
     """
-    if name == 'acc' or name == 'accuracy':
+    if callable(metric):
+        return CustomMetric(metric)
+    if not isinstance(metric, string_types):
+        raise TypeError('metric should either be callable or str')
+    if metric == 'acc' or metric == 'accuracy':
         return Accuracy()
     else:
-        raise ValueError('Cannot find metric %s' % name)
+        raise ValueError('Cannot find metric %s' % metric)

python/mxnet/model.py

@@ -542,8 +542,10 @@ def fit(self, X, y=None, eval_data=None, eval_metric='acc',
         eval_data : DataIter or numpy.ndarray pair
             If eval_set is numpy.ndarray pair, it should be (valid_data, valid_label)
 
-        eval_metric : function
-            Evaluation metric function.
+        eval_metric : metric.EvalMetric or str or callable
+            The evaluation metric, name of evaluation metric.
+            Or a customize evaluation function that returns the statistics
+            based on minibatch.
 
         iter_end_callback : callable(iteration, symbol, arg_params, aux_states)
             A callback that is invoked at end of each iteration.
@@ -556,7 +558,7 @@ def fit(self, X, y=None, eval_data=None, eval_metric='acc',
         if self.arg_params is None:
             self._init_params(input_shape)
         # setup metric
-        if isinstance(eval_metric, str):
+        if not isinstance(eval_metric, metric.EvalMetric):
             eval_metric = metric.create(eval_metric)
         # setup optimizer
         optimizer = self.optimizer
@@ -666,8 +668,10 @@ def create(symbol, X, y=None, ctx=None,
         eval_data : DataIter or numpy.ndarray pair
             If eval_set is numpy.ndarray pair, it should be (valid_data, valid_label)
 
-        eval_metric : function
-            Evaluation metric function.
+        eval_metric : metric.EvalMetric or str or callable
+            The evaluation metric, name of evaluation metric.
+            Or a customize evaluation function that returns the statistics
+            based on minibatch.
 
         iter_end_callback : callable(iteration, symbol, arg_params, aux_states)
             A callback that is invoked at end of each iteration.

python/mxnet/symbol.py

@@ -153,6 +153,27 @@ def _compose(self, *args, **kwargs):
         check_call(_LIB.MXSymbolCompose(
             self.handle, name, num_args, keys, args))
 
+    def __getitem__(self, index):
+        if not isinstance(index, int):
+            raise TypeError('Symbol only support integer index to fetch i-th output')
+        handle = SymbolHandle()
+        check_call(_LIB.MXSymbolGetOutput(
+            self.handle, mx_uint(index), ctypes.byref(handle)))
+        return Symbol(handle=handle)
+
+    def get_internals(self):
+        """Get a new grouped symbol whose output contains all the internal outputs of this symbol.
+
+        Returns
+        -------
+        sgroup : Symbol
+            The internal of the symbol.
+        """
+        handle = SymbolHandle()
+        check_call(_LIB.MXSymbolGetInternals(
+            self.handle, ctypes.byref(handle)))
+        return Symbol(handle=handle)
+
     def list_arguments(self):
         """List all the arguments in the symbol.
 

src/c_api.cc

@@ -509,6 +509,25 @@ int MXSymbolCreateGroup(mx_uint num_symbols,
   API_END_HANDLE_ERROR(delete s);
 }
 
+int MXSymbolGetOutput(SymbolHandle symbol,
+                      mx_uint index,
+                      SymbolHandle *out) {
+  Symbol *s = new Symbol();
+  API_BEGIN();
+  *s = (*static_cast<Symbol*>(symbol))[index];
+  *out = s;
+  API_END_HANDLE_ERROR(delete s);
+}
+
+int MXSymbolGetInternals(SymbolHandle symbol,
+                         SymbolHandle *out) {
+  Symbol *s = new Symbol();
+  API_BEGIN();
+  *s = static_cast<Symbol*>(symbol)->GetInternals();
+  *out = s;
+  API_END_HANDLE_ERROR(delete s);
+}
+
 int MXSymbolCreateFromFile(const char *fname, SymbolHandle *out) {
   Symbol *s = new Symbol();
   API_BEGIN();