Separate core and dense state space.

CHANGES.rst

@@ -65,6 +65,10 @@ releases are available on `Anaconda.org
   the data preparation.
 - :gh:`300` aligns respy functions with the new data in :gh:`299`
   (:ghuser:`tobiasraabe`).
+- :gh:`310` introduces the separation between a core state space and dense dimensions of
+  the state space which reduces memory consumption by a lot and makes respy scalable.
+  :gh:`312` and :gh:`313` include changes to the simulation or maximum likelihood
+  estimation which pave the way for :gh:`310`. (:ghuser:`tobiasraabe`)
 - :gh:`314` fixes two parameters in KW97 and KW2000 (:ghuser:`tostenzel`,
   :ghuser:`tobiasraabe`).
 - :gh:`316` changes the invalid index value for the indexer to prevent silent errors
@@ -76,6 +80,10 @@ releases are available on `Anaconda.org
 - :gh:`323` adds an informative message if simulated individuals cannot be mapped to
   states in the state space (:ghuser:`mo2561057`, :ghuser:`tobiasraabe`).
 
+
+*Releases prior to the second version were published on PyPI, but deleted later. You can
+still checkout the following releases with the corresponding tags.*
+
 1.2.1 - 2019-05-19
 ------------------
 

LICENSE

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2015-2019
+Copyright (c) 2015-2020
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.rst

@@ -21,7 +21,7 @@ respy
 
 .. image:: https://img.shields.io/badge/code%20style-black-000000.svg
     :target: https://github.com/psf/black
-    
+
 
 ``respy``  is an open-source Python package for the simulation and estimation of a
 prototypical finite-horizon discrete choice dynamic programming model. We build on the

development/testing/regression.py

@@ -112,11 +112,13 @@ def _check_single(test, strict):
     """Check a single test."""
     params, options, exp_val = test
 
-    crit_val = calc_crit_val(params, options)
-
-    is_success = np.isclose(
-        crit_val, exp_val, rtol=TOL_REGRESSION_TESTS, atol=TOL_REGRESSION_TESTS
-    )
+    try:
+        crit_val = calc_crit_val(params, options)
+        is_success = np.isclose(
+            crit_val, exp_val, rtol=TOL_REGRESSION_TESTS, atol=TOL_REGRESSION_TESTS
+        )
+    except Exception:
+        is_success = False
 
     if strict is True:
         assert is_success, "Failed regression test."

docs/api.rst

@@ -1,7 +1,6 @@
 API
 ===
 
-
 User
 ----
 
@@ -24,6 +23,7 @@ solve
 .. autosummary::
     :toctree: _generated/
 
+    get_solve_func
     solve
 
 simulate
@@ -58,31 +58,3 @@ likelihood
 
     get_crit_func
     log_like
-
-
-Developer
----------
-
-config
-~~~~~~
-
-.. currentmodule:: respy.config
-
-.. autosummary::
-    :toctree: _generated/
-
-    INDEXER_DTYPE
-    INDEXER_INVALID_INDEX
-
-state space
-~~~~~~~~~~~
-
-.. currentmodule:: respy.state_space
-
-.. autosummary::
-    :toctree: _generated/
-
-    StateSpace
-    StateSpace.get_attribute_from_period
-    StateSpace.get_continuation_values
-    StateSpace.update_systematic_rewards

docs/conf.py

@@ -18,8 +18,8 @@ def setup(app):
 # -- Project information -----------------------------------------------------
 
 project = "respy"
-copyright = "2015-2019, Philipp Eisenhauer"  # noqa: A001
-author = "Philipp Eisenhauer"
+copyright = "2015-2020, The respy Development Team"  # noqa: A001
+author = "The respy Development Team"
 
 # The full version, including alpha/beta/rc tags.
 release = "2.0.0dev2"
@@ -49,6 +49,7 @@ def setup(app):
 autodoc_mock_imports = [
     "chaospy",
     "estimagic",
+    "joblib",
     "numba",
     "numpy",
     "pandas",
@@ -106,7 +107,7 @@ def setup(app):
 
     .. nbinfo::
 
-        Download the notebook :download:`here <https://github.com/OpenSourceEconomics/respy/blob/{{ env.config.release }}/{{ docname }}>`!
+        View and download the notebook `here <https://nbviewer.jupyter.org/github/OpenSourceEconomics/respy/tree/v{{ env.config.release }}/{{ docname }}>`_!
 
 """
 
@@ -117,7 +118,6 @@ def setup(app):
 # Configuration for autodoc
 autosummary_generate = True
 
-
 # -- Options for HTML output ----------------------------------------------
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for

docs/development/randomness-and-reproducibility.rst

@@ -59,16 +59,16 @@ parameters.
 respy.solve
 -----------
 
-.. currentmodule:: respy.solve
-
 Routines under ``respy.solve`` use a seed from the sequence initialized by
 ``options["solution_seed"]`` to control randomness. Apart from the draws,
 :func:`~respy.solve.solve` relies on the following function.
 
+.. currentmodule:: respy.interpolate
+
 .. autosummary::
     :toctree: ../_generated/
 
-    get_not_interpolated_indicator
+    _get_not_interpolated_indicator
 
 
 respy.simulate

docs/development/releases.rst

@@ -24,28 +24,21 @@ How to release a new version?
 2. Second, we need to create a final PR to prepare everything for the new version. The
    name of the PR and the commit message will be "Release vx.y.z". We need to
 
-   - update all references of the old version number (``setup.py``,
-     ``respy/__init__.py``, ``docs/conf.py``).
+   - use ``bumpversion part <dev|patch|minor|major>`` to increment the correct part of
+     the version number in all files.
    - update information in ``CHANGES.rst`` to have summary of the changes which
      can also be posted in the Github repository under the tag.
 
-   Merge the PR into master.
+3. Run ``python release.py`` and check whether you can actually release a new version.
+   If you experience errors, fix them here.
 
-3. After that, revisit the draft of the release. Make sure everything is fine. Now, you
-   click on "Publish release" which creates a version tag on the latest commit of the
-   specified branch. The tag will trigger a build on Travis-CI which will publish the
-   release on PypI.
-
-4. Make sure that the new release was indeed published by checking `PyPI
-   <https://pypi.org/project/respy/>`_.
-
-5. Spread the word!
+4. Merge the PR into master.
 
+5. After that, revisit the draft of the release. Make sure everything is fine. Now, you
+   click on "Publish release" which creates a version tag on the latest commit of the
+   specified branch. Make sure to target the master branch.
 
-Notes
------
+6. Visit `Anaconda.org <https://anaconda.org/OpenSourceEconomics/respy>`_ and check
+   whether the release is available.
 
-- Travis-CI only builds tags if "Build pushed branches" is active.
-- If you publish a release on PyPI, the same version number cannot be reused even if you
-  delete the release. This is a safety measure. If you are not sure whether the release
-  will work, create a release candidate instead and publish the real version later.
+7. Spread the word!

docs/getting_started/index.rst

@@ -11,6 +11,5 @@ Getting Started
    tutorial-robinson-crusoe.ipynb
    tutorial-simulation.ipynb
    tutorial-initial-conditions
-   tutorial-model-with-observables.ipynb
    tutorial-finite-mixture.ipynb
    tutorial-msm.ipynb

docs/getting_started/tutorial-finite-mixture.ipynb

@@ -218,7 +218,7 @@
        "    <tr>\n",
        "      <th>initial_exp_fishing_2</th>\n",
        "      <th>probability</th>\n",
-       "      <td>0.33</td>\n",
+       "      <td>0.34</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
@@ -229,7 +229,7 @@
        "category              name              \n",
        "initial_exp_fishing_0 probability   0.33\n",
        "initial_exp_fishing_1 probability   0.33\n",
-       "initial_exp_fishing_2 probability   0.33"
+       "initial_exp_fishing_2 probability   0.34"
       ]
      },
      "execution_count": 3,
@@ -242,7 +242,7 @@
     "category,name,value\n",
     "initial_exp_fishing_0,probability,0.33\n",
     "initial_exp_fishing_1,probability,0.33\n",
-    "initial_exp_fishing_2,probability,0.33\n",
+    "initial_exp_fishing_2,probability,0.34\n",
     "\"\"\"), index_col=[\"category\", \"name\"])\n",
     "initial_exp_fishing"
    ]
@@ -565,7 +565,7 @@
        "    <tr>\n",
        "      <th>initial_exp_fishing_2</th>\n",
        "      <th>probability</th>\n",
-       "      <td>0.3300</td>\n",
+       "      <td>0.3400</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th rowspan=\"2\" valign=\"top\">wage_fishing</th>\n",
@@ -620,7 +620,7 @@
        "lagged_choice_1_hammock constant                  1.0000\n",
        "initial_exp_fishing_0   probability               0.3300\n",
        "initial_exp_fishing_1   probability               0.3300\n",
-       "initial_exp_fishing_2   probability               0.3300\n",
+       "initial_exp_fishing_2   probability               0.3400\n",
        "wage_fishing            type_1                    0.2000\n",
        "                        type_2                    0.4000\n",
        "type_1                  initial_exp_fishing_0   -90.0000\n",
@@ -689,18 +689,7 @@
    "cell_type": "code",
    "execution_count": 9,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "C:\\Users\\tobia\\git\\respy\\respy\\pre_processing\\model_processing.py:599: UserWarning: The probabilities for parameter group \\binitial_exp_fishing_([0-9]+)\\b do not sum to one.\n",
-      "  category=UserWarning,\n",
-      "C:\\Users\\tobia\\git\\respy\\respy\\pre_processing\\model_processing.py:599: UserWarning: The probabilities for parameter group \\binitial_exp_fishing_([0-9]+)\\b do not sum to one.\n",
-      "  category=UserWarning,\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "simulate = rp.get_simulate_func(params, options)\n",
     "df = simulate(params)"
@@ -753,14 +742,14 @@
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td>0.000000</td>\n",
-       "      <td>0.659223</td>\n",
-       "      <td>0.340777</td>\n",
+       "      <td>0.656887</td>\n",
+       "      <td>0.343113</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
-       "      <td>0.000302</td>\n",
-       "      <td>0.337564</td>\n",
-       "      <td>0.662134</td>\n",
+       "      <td>0.000296</td>\n",
+       "      <td>0.338958</td>\n",
+       "      <td>0.660746</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
@@ -770,8 +759,8 @@
        "Type                       0         1         2\n",
        "Experience_Fishing                              \n",
        "0                   1.000000  0.000000  0.000000\n",
-       "1                   0.000000  0.659223  0.340777\n",
-       "2                   0.000302  0.337564  0.662134"
+       "1                   0.000000  0.656887  0.343113\n",
+       "2                   0.000296  0.338958  0.660746"
       ]
      },
      "execution_count": 10,
@@ -828,18 +817,18 @@
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
-       "      <td>0.438767</td>\n",
-       "      <td>0.561233</td>\n",
+       "      <td>0.440916</td>\n",
+       "      <td>0.559084</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
-       "      <td>0.590465</td>\n",
-       "      <td>0.409535</td>\n",
+       "      <td>0.591656</td>\n",
+       "      <td>0.408344</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
-       "      <td>0.705991</td>\n",
-       "      <td>0.294009</td>\n",
+       "      <td>0.700927</td>\n",
+       "      <td>0.299073</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
@@ -848,9 +837,9 @@
       "text/plain": [
        "Choice   fishing   hammock\n",
        "Type                      \n",
-       "0       0.438767  0.561233\n",
-       "1       0.590465  0.409535\n",
-       "2       0.705991  0.294009"
+       "0       0.440916  0.559084\n",
+       "1       0.591656  0.408344\n",
+       "2       0.700927  0.299073"
       ]
      },
      "execution_count": 11,
@@ -879,7 +868,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.3"
+   "version": "3.7.6"
   }
  },
  "nbformat": 4,