API/DOC/MAINT/ENH: update for experimental use (#100)

* ENH/MAINT: better choose max_epochs for adaptive runners * MAINT: preliminary work around resetting * DOC: show response rate for adaptive algs * API, DOC: offline embeddings
stsievert · Mar 31, 2021 · 0b9509b · 0b9509b
1 parent 98c1ae0
commit 0b9509b
Show file tree

Hide file tree

Showing 30 changed files with 909 additions and 451 deletions.
diff --git a/docs/source/algorithms.rst b/docs/source/algorithms.rst
@@ -1,5 +1,7 @@
 .. _adaptive-config:
 
+.. _alg-config:
+
 Algorithm configuration
 =======================
 
@@ -31,7 +33,8 @@ By default, ``samplers`` defaults to ``RandomSampling: {}``. We have to customiz
 
    targets: ["obj1", "obj2", "foo", "bar", "foobar!"]
    samplers:
-     RR: {"module": "TSTE"}
+     RR:
+       module: TSTE
 
 This will use :class:`~salmon.triplets.algs.TSTE`. If we want to customize to
 include different keyword arguments, we need to look close at the arguments for
@@ -44,7 +47,8 @@ configuration:
    samplers:
      RandomSampling: {}
      RR:
-       alpha: 1.1
+       module: TSTE
+       module__alpha: 1.1
 
 ``alpha`` is a keyword argument to
 :class:`~salmon.triplets.algs.TSTE`.
@@ -57,10 +61,8 @@ If we want to use two alternate configs for TSTE:
      RandomSampling: {}
      tste1:
        class: TSTE
-       optimizer__lr: 0.1
      tste2:
        class: RR
-       optimizer__lr: 0.1
    sampling:
      probs: {"RandomSampling": 20, "tste1": 40, "tste2": 40}
 
@@ -71,4 +73,3 @@ This would test out different optimization methods underlying the embedding.
        :class:`~salmon.triplets.algs.CKL` and
        :class:`~salmon.triplets.algs.TSTE` have identical input parameters,
        except ``CKL``'s input ``mu`` and ``TSTE``'s ``alpha``.
-
diff --git a/docs/source/api.rst b/docs/source/api.rst
@@ -1,13 +1,28 @@
 .. _alg-api:
 
-Algorithm API
-=============
+.. _api:
 
-.. warning::
+API
+===
 
-   These APIs are unstable.
+Offline embeddings
+------------------
 
-All triplet embedding algorithms must conform to this API:
+This class can be used to create embeddings from a set of downloaded responses
+from Salmon:
+
+.. autosummary::
+   :toctree: generated/
+   :template: class.rst
+
+   salmon.triplets.offline.OfflineEmbedding
+
+Samplers
+--------
+
+These classes are used to collect responses with Salmon. They can be configured
+using ``init.yaml`` as mentioned in :ref:`alg-config`. They all conform to the
+following API:
 
 .. autosummary::
    :toctree: generated/
@@ -24,7 +39,7 @@ Every class below inherits from :class:`~salmon.backend.alg.Runner`.
 
 
 Passive Algorithms
-------------------
+^^^^^^^^^^^^^^^^^^
 
 .. currentmodule:: salmon
 
@@ -36,24 +51,45 @@ Passive Algorithms
    salmon.triplets.algs.RoundRobin
 
 Active Algorithms
------------------
+^^^^^^^^^^^^^^^^^
 
-.. currentmodule:: salmon
+There are two base classes for every adaptive algorithm:
+
+.. autosummary::
+   :toctree: generated/
+   :template: only-init.rst
+
+   salmon.triplets.algs.Adaptive
+   salmon.triplets.algs.adaptive.Embedding
+
+The class :class:`~salmon.triplets.algs.Adaptive` runs the adaptive algorithm
+and depends on :class:`~salmon.triplets.algs.adaptive.Embedding` for
+optimization. To customize the optimization, all extra keyword arguments are
+passed to the optimizer.
+
+Then, all of these classes inherit from
+:class:`~salmon.triplets.algs.Adaptive`:
 
 .. autosummary::
    :toctree: generated/
    :template: only-init.rst
 
    salmon.triplets.algs.RR
    salmon.triplets.algs.TSTE
+   salmon.triplets.algs.SOE
    salmon.triplets.algs.STE
    salmon.triplets.algs.CKL
    salmon.triplets.algs.GNMDS
 
+We have tested out the top three algorithms---RR, TSTE and SOE---in our
+experiments. We use :class:`~salmon.triplets.algs.RR` for our adaptive sampling
+(which defaults to the noise model in :class:`~salmon.triplets.algs.TSTE`) and
+use :class:`~salmon.triplets.algs.SOE` for the offline embeddings.
+
 These adaptive algorithms are all the same except for the underlying noise
 model, with the exception of :class:`~salmon.triplets.algs.RR`.
 :class:`~salmon.triplets.algs.RR` introduces some randomness by fixing the head
-and adding the top ``5 * n`` triplets to the database. This is useful because
+and adding the top ``1 * n`` triplets to the database. This is useful because
 the information gain measure used by all of these algorithms (by default) is a
 rule-of-thumb.
 

diff --git a/docs/source/benchmarks/adaptive.rst b/docs/source/benchmarks/adaptive.rst
@@ -8,32 +8,70 @@ about a random question like random sampling. This can mean that higher
 accuracies are reached sooner, or that less human responses are required to
 reach a particular accuracy.
 
+This page will be concerned with the data scientist workflow, and every
+experiment below will use the same workflow a data scientists would:
+
+1. Launch Salmon.
+2. Simulate human users. [#noise]_
+3. Download the human responses from Salmon
+4. Generate the embedding offline.
+
+Unless explicitly mentioned, every experiment on this page will use the "alien
+eggs" dataset that's characterized by one parameter, the "smoothness" of each
+egg. For :math:`n=30` objects, that dataset looks like the following:
+
+.. image:: imgs/alien-eggs.png
+   :width: 100%
+   :align: center
+
+These are characterized by one parameter, and have a 1D embedding. However,
+let's embed into :math:`d=2` dimensions to simulate a mistake and to mirror
+prior work. [2]_ Unless explicitly mentioned, let's compare random and active
+sampling with this ``init.yaml`` configuration:
+
+.. code-block:: yaml
+
+   samplers:
+     RR: {random_state: 42}  # active or adaptive sampling
+     RandomSampling: {}  # random sampling
+
 .. note::
 
    This page shows results of experiments run with Salmon.
    For complete details, see https://github.com/stsievert/salmon-experiments
 
-Synthetic simulation
---------------------
+Response rate
+-------------
 
-Let's compare adaptive sampling and random sampling. Specifically, let's use
-Salmon like an experimentalist would:
+This section will provide evidence for the following points:
 
-1. Launch Salmon with the "alien eggs" dataset, with :math:`n=30` objects
-   embedded into :math:`d=2` dimensions.
-2. Simulate human users (10 users with mean response time of 1s).
-3. Download the human responses from Salmon
-4. Generate the embedding offline.
+1. **Using adaptive sampling will not degrade embedding quality or accuracy.**
+2. **Adaptive sampling finds higher quality embeddings with fewer responses.**
+   In one case, adaptive sampling required about 1,250 responses instead of
+   3,500 responses like random sampling to reach ground truth accuracy of 97%.
+3. **However, response rate determines how much adaptive gains are possible.**
+   Having 10 concurrent users will likely still present adaptive gains, but
+   they'll be smaller than if there were only 5 concurrent users.
 
-Let's run this process for adaptive and random sampling. When we do that, this
-is the graph that's produced:
+The number of users will be varied from 1 concurrent user to 10 concurrent
+users with a mean response time of 1 second. This matters for adaptive sampling
+because there will be less time for computation/searching. It doesn't matter
+for random sampling because no computation/searching is performed. How does
+the response rate affect the embedding accuracy?
 
-.. image:: imgs/synth-eg-acc.png
-   :width: 600px
+.. image:: imgs/accuracy.png
+   :width: 100%
    :align: center
 
-These are synthetic results, though they use a human noise model. These
-experiments provide evidence that Salmon works well with adaptive sampling.
+This graph shows two measures: accuracy on a set of test human responses (left)
+and responses that are 100% accurate on the ground truth dataset (right). The
+graph on the right is a measure of quality on the underlying embedding. The
+graph on the left shows that that this quality is reflected in hold-out
+performance on human responses.
+
+These experiments provide evidence that the adaptive sampling above works well
+in crowdsourcing settings. Additionally, they provide evidence that Salmon's
+adaptive sampling does not perform worse than random sampling.
 
 This measure provides evidence that Salmon's active sampling approach
 outperforms random sampling. If true, this is an improvement over existing
@@ -45,6 +83,8 @@ sampling" for (nearly) the same problem. [#same]_
 Simulation with human responses
 -------------------------------
 
+*This section uses the Zappos shoe dataset, not the alien eggs dataset*
+
 The Zappos shoe dataset has :math:`n=85` shoes, and asks every possible triplet
 4 times to crowdsourcing users. Let's run a simulation with Salmon on that that
 dataset. We'll embed into :math:`d = 3` dimensions, and have a response rate of
@@ -83,3 +123,6 @@ al. in their Figure 3. [1]_
            actually runs crowdsourcing experiments; Salmon's noise model is
            generated from those responses).
 
+
+.. [#noise] Specifically, with a noise model developed the human responses collected
+            for Fig. 3 of the NEXT paper. [2]_
diff --git a/docs/source/benchmarks/imgs/accuracy.png b/docs/source/benchmarks/imgs/accuracy.png
diff --git a/docs/source/benchmarks/imgs/alien-eggs.png b/docs/source/benchmarks/imgs/alien-eggs.png
diff --git a/docs/source/getting-started.rst b/docs/source/getting-started.rst
@@ -3,17 +3,17 @@
 Starting an experiment
 ======================
 
-Experiments can be initialized by vising ``http://[url]:8421/init``.
-
 .. note::
 
    By default, Salmon does not support HTTPS. Make sure the URL begins with
    ``http://``, not ``https://``.
 
-Initialization requires two actions:
+Initialization an experiment requires the following:
 
-1. Creating a username/password
-2. Launching an experiment.
+1. Visiting ``http://[url]:8421/init`` with the ``[url]`` from
+   :ref:`installation`,
+2. Creating a username/password
+3. Launching an experiment.
 
 First, type a username/password and hit "create user." After a user has been
 successfully created, hit the back button and launch an experiment. You have
@@ -68,7 +68,7 @@ Here's an example ``init.yaml`` YAML file for initialization:
    max_queries: 25
    samplers:
      RandomSampling: {}
-     RoundRobin: {}
+     RR: {}
 
 The top-level elements like ``max_queries`` and ``targets`` are called "keys"
 in YAML jargon. Here's documentation for each key:

diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -37,6 +37,7 @@ the same confidence.
    monitoring
    offline
    algorithms
+   api
 
 .. toctree::
    :maxdepth: 2
@@ -51,7 +52,6 @@ the same confidence.
 
    adaptive
    developers
-   api
 
 
 Indices and tables

diff --git a/docs/source/installation.rst b/docs/source/installation.rst
@@ -1,3 +1,5 @@
+.. _installation:
+
 Installation
 ============
 

diff --git a/docs/source/offline.rst b/docs/source/offline.rst
@@ -54,9 +54,12 @@ This code will generate an embedding:
    model.history_  # to view information on how well train/test performed
 
 Some customization can be done with ``model.history_``; it may not be necessary
-to train for 1,000,000 epochs. ``model.history_`` will include validation and
+to train for 400,000 epochs. ``model.history_`` will include validation and
 training scores, which might help limit the number of epochs.
 
+Documentation for :class:`~salmon.triplets.offline.OfflineEmbedding` is
+available on :ref:`api`.
+
 Embedding visualization
 -----------------------