Add notebook for DRIV

README.md

@@ -1,4 +1,4 @@
-[![Build Status](https://dev.azure.com/ms/EconML/_apis/build/status/Microsoft.EconML?branchName=master)](https://dev.azure.com/ms/EconML/_build/latest?definitionId=49&branchName=master)
+[![Build Status](https://dev.azure.com/ms/EconML/_apis/build/status/Microsoft.EconML?branchName=main)](https://dev.azure.com/ms/EconML/_build/latest?definitionId=49&branchName=main)
 [![PyPI version](https://img.shields.io/pypi/v/econml.svg)](https://pypi.org/project/econml/)
 [![PyPI wheel](https://img.shields.io/pypi/wheel/econml.svg)](https://pypi.org/project/econml/)
 [![Supported Python versions](https://img.shields.io/pypi/pyversions/econml.svg)](https://pypi.org/project/econml/)
@@ -46,10 +46,12 @@ For information on use cases and background material on causal inference and het
 
 # News
 
-**August 13, 2021:** Release v0.12.0, see release notes [here](https://github.com/Microsoft/EconML/releases/tag/v0.12.0)
+**January 31, 2022:** Release v0.13.0, see release notes [here](https://github.com/Microsoft/EconML/releases/tag/v0.13.0)
 
 <details><summary>Previous releases</summary>
 
+**August 13, 2021:** Release v0.12.0, see release notes [here](https://github.com/Microsoft/EconML/releases/tag/v0.12.0)
+
 **August 5, 2021:** Release v0.12.0b6, see release notes [here](https://github.com/Microsoft/EconML/releases/tag/v0.12.0b6)
 
 **August 3, 2021:** Release v0.12.0b5, see release notes [here](https://github.com/Microsoft/EconML/releases/tag/v0.12.0b5)
@@ -319,23 +321,81 @@ lb, ub = est.effect_interval(X_test, alpha=0.05)
 </details>
 
 <details>
-<summary>Orthogonal Instrumental Variables (click to expand)</summary>
+<summary>Double Machine Learning with Instrumental Variables (click to expand)</summary>
+
+* Orthogonal instrumental variable learner
+
+```Python
+from econml.iv.dml import OrthoIV
+
+est = OrthoIV(projection=False, 
+              discrete_treatment=True, 
+              discrete_instrument=True)
+est.fit(Y, T, Z=Z, X=X, W=W)
+treatment_effects = est.effect(X_test)
+lb, ub = est.effect_interval(X_test, alpha=0.05) # OLS confidence intervals
+```
+* Nonparametric double machine learning with instrumental variable
 
-* Intent to Treat Doubly Robust Learner (discrete instrument, discrete treatment)
+```Python
+from econml.iv.dml import NonParamDMLIV
+
+est = NonParamDMLIV(projection=False, 
+                    discrete_treatment=True, 
+                    discrete_instrument=True)
+est.fit(Y, T, Z=Z, X=X, W=W) # no analytical confidence interval available
+treatment_effects = est.effect(X_test)
+```
+</details>
+
+<details>
+<summary>Doubly Robust Machine Learning with Instrumental Variables (click to expand)</summary>
+
+* Linear final stage
+```Python
+from econml.iv.dr import LinearDRIV
+
+est = LinearDRIV(discrete_instrument=True, discrete_treatment=True)
+est.fit(Y, T, Z=Z, X=X, W=W)
+treatment_effects = est.effect(X_test)
+lb, ub = est.effect_interval(X_test, alpha=0.05) # OLS confidence intervals
+```
+
+* Sparse linear final stage
+
+```Python
+from econml.iv.dr import SparseLinearDRIV
+
+est = SparseLinearDRIV(discrete_instrument=True, discrete_treatment=True)
+est.fit(Y, T, Z=Z, X=X, W=W)
+treatment_effects = est.effect(X_test)
+lb, ub = est.effect_interval(X_test, alpha=0.05) # Debiased lasso confidence intervals
+```
+
+* Nonparametric final stage
+```Python
+from econml.iv.dr import ForestDRIV
+
+est = ForestDRIV(discrete_instrument=True, discrete_treatment=True)
+est.fit(Y, T, Z=Z, X=X, W=W)
+treatment_effects = est.effect(X_test)
+# Confidence intervals via Bootstrap-of-Little-Bags for forests
+lb, ub = est.effect_interval(X_test, alpha=0.05) 
+```
+
+* Linear intent-to-treat (discrete instrument, discrete treatment)
 
 ```Python
 from econml.iv.dr import LinearIntentToTreatDRIV
 from sklearn.ensemble import GradientBoostingRegressor, GradientBoostingClassifier
-from sklearn.linear_model import LinearRegression
 
 est = LinearIntentToTreatDRIV(model_y_xw=GradientBoostingRegressor(),
                               model_t_xwz=GradientBoostingClassifier(),
                               flexible_model_effect=GradientBoostingRegressor())
-est.fit(Y, T, Z=Z, X=X) # OLS inference by default
+est.fit(Y, T, Z=Z, X=X, W=W)
 treatment_effects = est.effect(X_test)
 lb, ub = est.effect_interval(X_test, alpha=0.05) # OLS confidence intervals
 ```
-
 </details>
 
 <details>
@@ -581,7 +641,7 @@ importances = policy.feature_importances_
   ![image](images/policy_tree.png)
 </details>
 
-To see more complex examples, go to the [notebooks](https://github.com/Microsoft/EconML/tree/master/notebooks) section of the repository. For a more detailed description of the treatment effect estimation algorithms, see the EconML [documentation](https://econml.azurewebsites.net/).
+To see more complex examples, go to the [notebooks](https://github.com/Microsoft/EconML/tree/main/notebooks) section of the repository. For a more detailed description of the treatment effect estimation algorithms, see the EconML [documentation](https://econml.azurewebsites.net/).
 
 # For Developers
 
@@ -598,12 +658,12 @@ We have added pytest marks to some tests to make it easier to run a subset, and
 
 ## Generating the documentation
 
-This project's documentation is generated via [Sphinx](https://www.sphinx-doc.org/en/master/index.html).  Note that we use [graphviz](https://graphviz.org/)'s 
+This project's documentation is generated via [Sphinx](https://www.sphinx-doc.org/en/main/index.html).  Note that we use [graphviz](https://graphviz.org/)'s 
 `dot` application to produce some of the images in our documentation, so you should make sure that `dot` is installed and in your path.
 
 To generate a local copy of the documentation from a clone of this repository, just run `python setup.py build_sphinx -W -E -a`, which will build the documentation and place it under the `build/sphinx/html` path. 
 
-The reStructuredText files that make up the documentation are stored in the [docs directory](https://github.com/Microsoft/EconML/tree/master/doc); module documentation is automatically generated by the Sphinx build process.
+The reStructuredText files that make up the documentation are stored in the [docs directory](https://github.com/Microsoft/EconML/tree/main/doc); module documentation is automatically generated by the Sphinx build process.
 
 # Blogs and Publications
 

azure-pipelines-steps.yml

@@ -6,7 +6,7 @@
 parameters:
   package: '-e .'
   images: ['ubuntu-18.04', 'macOS-10.15', 'windows-2019']
-  versions: ['3.6', '3.7', '3.8']
+  versions: ['3.6', '3.7', '3.8', '3.9']
   job: 
     job: Job
 

azure-pipelines.yml

@@ -4,7 +4,7 @@
 # https://docs.microsoft.com/azure/devops/pipelines/languages/python
 
 trigger:
-- master
+- main
 
 jobs:
 - job: 'EvalChanges'
@@ -70,7 +70,7 @@ jobs:
       - script: 'pip install git+https://github.com/slundberg/shap.git@d1d2700acc0259f211934373826d5ff71ad514de'
         displayName: 'Install specific version of shap'  
 
-      - script: 'pip install "sphinx < 3.2" sphinx_rtd_theme'
+      - script: 'pip install sphinx sphinx_rtd_theme'
         displayName: 'Install sphinx'
 
       - script: 'python setup.py build_sphinx -W'
@@ -83,24 +83,27 @@ jobs:
       - script: 'python setup.py build_sphinx -b doctest'
         displayName: 'Run doctests'
 
+
 - template: azure-pipelines-steps.yml
   parameters:
     versions: ['3.8']
     images: ['ubuntu-18.04']
     package: '-e .[tf,plt]'
     job:
-      job: 'Notebooks'
+      job: 'Notebooks_cust'
       dependsOn: 'EvalChanges'
       condition: eq(dependencies.EvalChanges.outputs['output.buildNbs'], 'True')
+      displayName: 'Notebooks (Customer Solutions)'
       steps:
       # Work around https://github.com/pypa/pip/issues/9542
-      - script: 'pip install -U numpy'
+      - script: 'pip install -U numpy~=1.21.0'
         displayName: 'Upgrade numpy'
 
-      - script: 'pip install pytest-runner && python setup.py pytest'
+      - script: 'pip install pytest pytest-runner jupyter jupyter-client nbconvert nbformat seaborn xgboost tqdm && python setup.py pytest'
         displayName: 'Unit tests'
         env:
           PYTEST_ADDOPTS: '-m "notebook"'
+          NOTEBOOK_DIR_PATTERN: 'CustomerScenarios'
 
       - task: PublishTestResults@2
         displayName: 'Publish Test Results **/test-results.xml'
@@ -109,6 +112,41 @@ jobs:
           testRunTitle: 'Notebooks'
         condition: succeededOrFailed()
 
+- template: azure-pipelines-steps.yml
+  parameters:
+    versions: ['3.8']
+    images: ['ubuntu-18.04']
+    package: '-e .[tf,plt]'
+    job:
+      job: 'Notebooks_noncust'
+      dependsOn: 'EvalChanges'
+      condition: eq(dependencies.EvalChanges.outputs['output.buildNbs'], 'True')
+      displayName: 'Notebooks (except Customer Solutions)'
+      steps:
+      # Work around https://github.com/pypa/pip/issues/9542
+      - script: 'pip install -U numpy~=1.21.0'
+        displayName: 'Upgrade numpy'
+
+      # shap 0.39 and sklearn 1.0 interact badly in these notebooks
+      # shap 0.40 has a bug in waterfall (https://github.com/slundberg/shap/issues/2283) that breaks our main tests
+      # but fixes the interaction here...
+      - script: 'pip install -U shap~=0.40.0'
+        displayName: 'Upgrade shap'
+
+      - script: 'pip install pytest pytest-runner jupyter jupyter-client nbconvert nbformat seaborn xgboost tqdm && python setup.py pytest'
+        displayName: 'Unit tests'
+        env:
+          PYTEST_ADDOPTS: '-m "notebook"'
+          NOTEBOOK_DIR_PATTERN: '(?!CustomerScenarios)'
+
+      - task: PublishTestResults@2
+        displayName: 'Publish Test Results **/test-results.xml'
+        inputs:
+          testResultsFiles: '**/test-results.xml'
+          testRunTitle: 'Notebooks'
+        condition: succeededOrFailed()
+
+
 # - job: 'AutoML'
 #   dependsOn: 'EvalChanges'
 #   condition: eq(dependencies.EvalChanges.outputs['output.testCode'], 'True')
@@ -166,7 +204,7 @@ jobs:
       condition: eq(dependencies.EvalChanges.outputs['output.testCode'], 'True')
       displayName: 'Run tests (main)'
       steps:
-      - script: 'pip install pytest-runner && python setup.py pytest'
+      - script: 'pip install pytest pytest-runner && python setup.py pytest'
         displayName: 'Unit tests'
         env:
           PYTEST_ADDOPTS: '-m "not (notebook or automl or dml or causal)" -n 2'
@@ -193,7 +231,7 @@ jobs:
       condition: eq(dependencies.EvalChanges.outputs['output.testCode'], 'True')
       displayName: 'Run tests (DML)'
       steps:
-      - script: 'pip install pytest-runner && python setup.py pytest'
+      - script: 'pip install pytest pytest-runner && python setup.py pytest'
         displayName: 'Unit tests'
         env:
           PYTEST_ADDOPTS: '-m "dml"'
@@ -220,7 +258,7 @@ jobs:
       condition: eq(dependencies.EvalChanges.outputs['output.testCode'], 'True')
       displayName: 'Run tests (Causal)'
       steps:
-      - script: 'pip install pytest-runner && python setup.py pytest'
+      - script: 'pip install pytest pytest-runner && python setup.py pytest'
         displayName: 'Unit tests'
         env:
           PYTEST_ADDOPTS: '-m "causal" -n 1'

doc/conf.py

@@ -4,7 +4,7 @@
 #
 # This file does only contain a selection of the most common options. For a
 # full list see the documentation:
-# http://www.sphinx-doc.org/en/master/config
+# http://www.sphinx-doc.org/en/main/config
 
 # -- Path setup --------------------------------------------------------------
 
@@ -59,9 +59,9 @@
                            'inherited-members': None,
                            'member-order': 'groupwise'}
 
-mathjax_config = {
-    'TeX': {
-        'Macros': {
+mathjax3_config = {
+    'tex': {
+        'macros': {
             'vec': [r'{\bf #1}', 1],
             'ldot': [r'\left\langle #1, #2 \right\rangle', 2],
             'E': r'\mathbb{E}',
@@ -81,8 +81,8 @@
 # source_suffix = ['.rst', '.md']
 source_suffix = '.rst'
 
-# The master toctree document.
-master_doc = 'index'
+# The root toctree document.
+root_doc = 'index'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -164,7 +164,7 @@
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-    (master_doc, 'econml.tex', 'econml Documentation',
+    (root_doc, 'econml.tex', 'econml Documentation',
      'Microsoft Research', 'manual'),
 ]
 
@@ -174,7 +174,7 @@
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
 man_pages = [
-    (master_doc, 'econml', 'econml Documentation',
+    (root_doc, 'econml', 'econml Documentation',
      [author], 1)
 ]
 
@@ -185,7 +185,7 @@
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-    (master_doc, 'econml', 'econml Documentation',
+    (root_doc, 'econml', 'econml Documentation',
      author, 'econml', 'One line description of project.',
      'Miscellaneous'),
 ]

doc/index.rst

@@ -1,4 +1,4 @@
-.. econml documentation master file, created by
+.. econml documentation root file, created by
    sphinx-quickstart on Fri Feb 15 12:08:35 2019.
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.

doc/spec/comparison.rst

@@ -35,7 +35,24 @@ Detailed estimator comparison
 +---------------------------------------------+--------------+--------------+------------------+-------------+-----------------+------------+--------------+--------------------+
 | :class:`.NonParamDML`                       | 1-d/Binary   |              |                  | Yes         |                 | Yes        |              | Yes                |
 +---------------------------------------------+--------------+--------------+------------------+-------------+-----------------+------------+--------------+--------------------+
-
+| :class:`.OrthoIV`                           | Any          | Yes          | Yes              | Yes         | Assumed         | Yes        | Yes          |                    |
++---------------------------------------------+--------------+--------------+------------------+-------------+-----------------+------------+--------------+--------------------+
+| :class:`.DMLIV`                             | Any          | Yes          |                  | Yes         | Assumed         | Yes        | Yes          | Yes                |
++---------------------------------------------+--------------+--------------+------------------+-------------+-----------------+------------+--------------+--------------------+
+| :class:`.NonParamDMLIV`                     | 1-d/Binary   | Yes          |                  | Yes         |                 | Yes        |              | Yes                |
++---------------------------------------------+--------------+--------------+------------------+-------------+-----------------+------------+--------------+--------------------+
+| :class:`.DRIV`                              | 1-d/Binary   | Yes          | Yes              | Yes         |                 |            |              | Yes                |
++---------------------------------------------+--------------+--------------+------------------+-------------+-----------------+------------+--------------+--------------------+
+| :class:`.LinearDRIV`                        | 1-d/Binary   | Yes          | Yes              | Yes         | Projected       |            |              |                    |
++---------------------------------------------+--------------+--------------+------------------+-------------+-----------------+------------+--------------+--------------------+
+| :class:`.SparseLinearDRIV`                  | 1-d/Binary   | Yes          | Yes              | Yes         | Projected       |            |              | Yes                |
++---------------------------------------------+--------------+--------------+------------------+-------------+-----------------+------------+--------------+--------------------+
+| :class:`.ForestDRIV`                        | 1-d/Binary   | Yes          | Yes              | Yes         |                 |            |              | Yes                |
++---------------------------------------------+--------------+--------------+------------------+-------------+-----------------+------------+--------------+--------------------+
+| :class:`.IntentToTreatDRIV`                 | Binary       | Yes          | Ye               | Yes         |                 |            |              | Yes                |
++---------------------------------------------+--------------+--------------+------------------+-------------+-----------------+------------+--------------+--------------------+
+| :class:`.LinearIntentToTreatDRIV`           | Binary       | Yes          | Yes              | Yes         | Projected       |            |              |                    |
++---------------------------------------------+--------------+--------------+------------------+-------------+-----------------+------------+--------------+--------------------+
 
 Treatment Type
     Some estimators can only estimate effects of particular kinds of treatments. 

doc/spec/estimation/dml.rst

@@ -589,8 +589,8 @@ Usage Examples
 ==================================
 
 For more extensive examples check out the following notebooks:
-`DML Examples Jupyter Notebook <https://github.com/microsoft/EconML/blob/master/notebooks/Double%20Machine%20Learning%20Examples.ipynb>`_,
-`Forest Learners Jupyter Notebook <https://github.com/microsoft/EconML/blob/master/notebooks/ForestLearners%20Basic%20Example.ipynb>`_.
+`DML Examples Jupyter Notebook <https://github.com/microsoft/EconML/blob/main/notebooks/Double%20Machine%20Learning%20Examples.ipynb>`_,
+`Forest Learners Jupyter Notebook <https://github.com/microsoft/EconML/blob/main/notebooks/ForestLearners%20Basic%20Example.ipynb>`_.
 
 .. rubric:: Single Outcome, Single Treatment
 

doc/spec/estimation/dr.rst

@@ -366,7 +366,7 @@ Usage FAQs
 
     If you care more about mean squared error than confidence intervals and hypothesis testing, then use the
     :class:`.DRLearner` class and choose a cross-validated final model (checkout the 
-    `Forest Learners Jupyter notebook <https://github.com/microsoft/EconML/blob/master/notebooks/ForestLearners%20Basic%20Example.ipynb>`_ 
+    `Forest Learners Jupyter notebook <https://github.com/microsoft/EconML/blob/main/notebooks/ForestLearners%20Basic%20Example.ipynb>`_ 
     for such an example).
     Also the check out the :ref:`Orthogonal Random Forest User Guide <orthoforestuserguide>` or the
     :ref:`Meta Learners User Guide <metalearnersuserguide>`.
@@ -516,7 +516,7 @@ Usage FAQs
 
 Check out the following Jupyter notebooks:
 
-* `Meta Learners Jupyter Notebook <https://github.com/microsoft/EconML/blob/master/notebooks/Metalearners%20Examples.ipynb>`_ 
-* `Forest Learners Jupyter Notebook <https://github.com/microsoft/EconML/blob/master/notebooks/ForestLearners%20Basic%20Example.ipynb>`_
+* `Meta Learners Jupyter Notebook <https://github.com/microsoft/EconML/blob/main/notebooks/Metalearners%20Examples.ipynb>`_ 
+* `Forest Learners Jupyter Notebook <https://github.com/microsoft/EconML/blob/main/notebooks/ForestLearners%20Basic%20Example.ipynb>`_
 
 

doc/spec/estimation/forest.rst

@@ -315,8 +315,8 @@ Usage Examples
 Here is a simple example of how to call :class:`.DMLOrthoForest`
 and what the returned values correspond to in a simple data generating process.
 For more examples check out our 
-`OrthoForest Jupyter notebook <https://github.com/Microsoft/EconML/blob/master/notebooks/Orthogonal%20Random%20Forest%20Examples.ipynb>`_ 
-and the `ForestLearners Jupyter notebook <https://github.com/microsoft/EconML/blob/master/notebooks/ForestLearners%20Basic%20Example.ipynb>`_ .
+`OrthoForest Jupyter notebook <https://github.com/Microsoft/EconML/blob/main/notebooks/Orthogonal%20Random%20Forest%20Examples.ipynb>`_ 
+and the `ForestLearners Jupyter notebook <https://github.com/microsoft/EconML/blob/main/notebooks/ForestLearners%20Basic%20Example.ipynb>`_ .
 
 
     .. testcode::