Default SEM to np.nan in SimpleExperiment and update docs

Summary: When a user runs a `SimpleExperiment` without specifying a `SEM`, Ax should assume `SEM = np.nan` rather than the current status `SEM = 0.0`. This diff fixes this and references.

Reviewed By: Balandat

Differential Revision: D25931956

fbshipit-source-id: 65301dfbd8e666ff93a177afbdf4290746945b1d

ax/core/simple_experiment.py

@@ -196,12 +196,14 @@ def evaluation_function_outer(
             # pyre-fixme[16]: `Optional` has no attribute `objective`.
             return {self.optimization_config.objective.metric.name: evaluation}
         elif isinstance(evaluation, (float, int)):
-            return {self.optimization_config.objective.metric.name: (evaluation, 0.0)}
+            return {
+                self.optimization_config.objective.metric.name: (evaluation, np.nan)
+            }
         elif isinstance(evaluation, (np.float32, np.float64, np.int32, np.int64)):
             return {
                 self.optimization_config.objective.metric.name: (
                     numpy_type_to_python_type(evaluation),
-                    0.0,
+                    np.nan,
                 )
             }
         raise Exception(  # pragma: no cover

docs/api.md

@@ -22,13 +22,13 @@ best_parameters, values, experiment, model = optimize(
             "type": "range",
             "bounds": [-5.0, 10.0],
         },
-        {  
+        {
             "name": "x2",
             "type": "range",
             "bounds": [0.0, 10.0],
         },
     ],
-    evaluation_function=lambda p: branin(p["x1"], p["x2"]),
+    evaluation_function=lambda p: (branin(p["x1"], p["x2"]), 0.0),
     minimize=True,
 )
 ```
@@ -82,7 +82,7 @@ branin_search_space = SearchSpace(
 exp = SimpleExperiment(
     name="test_branin",
     search_space=branin_search_space,
-    evaluation_function=lambda p: branin(p["x1"], p["x2"]),
+    evaluation_function=lambda p: (branin(p["x1"], p["x2"]), 0.0),
     objective_name="branin",
     minimize=True,
 )

docs/glossary.md

@@ -40,7 +40,7 @@ Dispatch abstraction that defines how a given [trial](glossary.md#trial) is to b
 ### Search space
 Continuous, discrete or mixed design space that defines the set of [parameters](glossary.md#parameter) to be tuned in the optimization, and optionally [parameter constraints](glossary.md#parameter-constraint) on these parameters. The parameters of the [arms](glossary.md#arm) to be evaluated in the optimization are drawn from a search space. [```[SearchSpace]```](/api/core.html#module-ax.core.search_space)
 ### SEM
-[Standard error](https://en.wikipedia.org/wiki/Standard_error) of the [metric](glossary.md#metric)'s mean, 0.0 for noiseless measurements.
+[Standard error](https://en.wikipedia.org/wiki/Standard_error) of the [metric](glossary.md#metric)'s mean, 0.0 for noiseless measurements. If no value is provided, defaults to `np.nan`, in which case Ax infers its value using the measurements collected during experimentation.
 ### Simple experiment
 Subclass of [experiment](glossary.md#experiment) that assumes synchronous evaluation (uses an evaluation function to get data for [trials](glossary.md#trial) right after they are suggested). Abstracts away certain details, and allows for faster instantiation. [```[SimpleExperiment]```](/api/core.html#module-ax.core.simple_experiment)
 ### Status quo