Merge pull request #611 from camsys/feature-sharrow-tt

[Phase 7] Complete Sharrow

.github/workflows/core_tests.yml

@@ -81,6 +81,7 @@ jobs:
         run: |
           python -m pytest --pyargs activitysim.cli
 
+
   cross-platform:
     # also test foundation cross platforms, but do not require a successful
     # completion before starting regional model tests
@@ -162,6 +163,7 @@ jobs:
         run: |
           python -m pytest --pyargs activitysim.cli
 
+
   regional_models:
     needs: foundation
     env:

.gitignore

@@ -72,6 +72,8 @@ _test_est
 *_local/
 *_local.*
 
+**/__sharrowcache__
+**/skims.zarr
 **/output/
 **/_generated_version.py
 docs/**/_generated

.pre-commit-config.yaml

@@ -3,8 +3,6 @@ repos:
 - repo: https://github.com/pre-commit/pre-commit-hooks
   rev: v4.3.0
   hooks:
-  - id: check-yaml
-    exclude: logging.yaml  # TODO don't exclude, will require fixing logging
   - id: end-of-file-fixer
     exclude: .*\.ipynb
   - id: trailing-whitespace

HOW_TO_RELEASE.md

@@ -5,9 +5,9 @@
 01. Check that the branch you intend to release is passing tests on Travis.
     If it's not passing there you should not release it.
 
-00. Start from a completely clean conda environment 
-    and git repository.  Assuming you have `conda` installed, you can do so 
-    by starting where ActivitySim is not yet cloned (e.g. in an empty 
+00. Start from a completely clean conda environment
+    and git repository.  Assuming you have `conda` installed, you can do so
+    by starting where ActivitySim is not yet cloned (e.g. in an empty
     directory) and running:
     ```sh
     conda create -n TEMP-ASIM-DEV python=3.9 git gh -c conda-forge --override-channels
@@ -19,7 +19,7 @@
 
 00. Per project policy, code on the main branch should have been released,
     but if you are *preparing* a release then the code should be on the `develop`
-    branch.  Switch to that branch now, and make sure it is synced to the 
+    branch.  Switch to that branch now, and make sure it is synced to the
     version on GitHub:
     ```sh
     git switch develop
@@ -28,45 +28,45 @@
 
 00. Update your Conda environment for testing.  We do not want to use an
     existing environment on your machine, as it may be out-of-date
-    and we want to make sure everything passes muster using the 
+    and we want to make sure everything passes muster using the
     most up-to-date dependencies available.  The following command
-    will update the active environment (we made this to be `TEMP-ASIM-DEV` 
+    will update the active environment (we made this to be `TEMP-ASIM-DEV`
     if you followed the directions above).
     ```sh
     conda env update --file=conda-environments/activitysim-dev.yml
     ```
-    If you add to the ActivitySim dependencies, make sure to also update 
-    the environments in `conda-environments`, which are used for testing 
-    and development.  If they are not updated, these environments will end 
+    If you add to the ActivitySim dependencies, make sure to also update
+    the environments in `conda-environments`, which are used for testing
+    and development.  If they are not updated, these environments will end
     up with dependencies loaded from *pip* instead of *conda-forge*.
 
-00. Run pycodestyle to ensure that the codebase passes all style checks.
+00. Run black to ensure that the codebase passes all style checks.
     This check should only take a few seconds.  These checks are also done on
     Travis and are platform independent, so they should not be necessary to
     replicate locally, but are listed here for completeness.
     ```sh
-    pycodestyle .
+    black --check --diff .
     ```
 
 00. Run the regular test suite on Windows. Travis tests are done on Linux,
     but most users are on Windows, and the test suite should also be run
     on Windows to ensure that it works on that platform as well.  If you
     are not preparing this release on Windows, you should be sure to run
-    at least through this step on a Windows machine before finalizing a 
-    release.  
-    
-    A few of the tests require pre-created data that is not included in the 
-    repository directly, but rather recreated on the fly before testing. The 
-    regular test suite takes some time to run, between about half an hour and 
+    at least through this step on a Windows machine before finalizing a
+    release.
+
+    A few of the tests require pre-created data that is not included in the
+    repository directly, but rather recreated on the fly before testing. The
+    regular test suite takes some time to run, between about half an hour and
     two hours depending on the specs of your machine.
     ```sh
     python activitysim/examples/placeholder_multiple_zone/scripts/two_zone_example_data.py
     python activitysim/examples/placeholder_multiple_zone/scripts/three_zone_example_data.py
     pytest .
     ```
- 
+
 00. Test the full-scale regional examples. These examples are big, too
-    large to run on Travis, and will take a lot of time (many hours) to 
+    large to run on Travis, and will take a lot of time (many hours) to
     download and run.
     ```sh
     mkdir tmp-asim
@@ -76,39 +76,39 @@
     call run_all_examples.bat
     ```
     These tests will run through the gamut even if some of them crash, so
-    if you don't sit and watch them go (please don't do this) you'll need 
+    if you don't sit and watch them go (please don't do this) you'll need
     to scan through the results to make sure there are no errors after the
     fact.
     ```sh
     python ../activitysim/examples/scan_examples_for_errors.py .
     ```
 
 00. Test the notebooks in `activitysim/examples/prototype_mtc/notebooks`.
-    There are also demo notebooks for estimation, but their functionality  
+    There are also demo notebooks for estimation, but their functionality
     is completely tested in the unit tests run previously.
 
-00. Use bump2version to tag the release commit and update the 
+00. Use bump2version to tag the release commit and update the
     version number.  The following code will generate a "patch" release,
-    incrementing the third value in the version number (i.e. "1.2.3" 
-    becomes "1.2.4").  Alternatively, make a "minor" or "major" release. 
-    The `--list` command will generate output to your console to confirm 
-    that the old and new version numbers are what you expect, before you 
-    push the commit (with the changed version in the code) and tags to 
+    incrementing the third value in the version number (i.e. "1.2.3"
+    becomes "1.2.4").  Alternatively, make a "minor" or "major" release.
+    The `--list` command will generate output to your console to confirm
+    that the old and new version numbers are what you expect, before you
+    push the commit (with the changed version in the code) and tags to
     GitHub.
     ```sh
     bump2version patch --list
     ```
-    
-    It is also possible to make a development pre-release. To do so, 
-    explicitly set the version number to the next patch plus a ".devN" 
-    suffix:  
-    
+
+    It is also possible to make a development pre-release. To do so,
+    explicitly set the version number to the next patch plus a ".devN"
+    suffix:
+
     ```sh
     bump2version patch --new-version 1.2.3.dev0 --list
     ```
-    
-    Then, when ready to make a "final" release, set the version by 
-    explicitly removing the suffix:  
+
+    Then, when ready to make a "final" release, set the version by
+    explicitly removing the suffix:
     ```sh
     bump2version patch --new-version 1.2.3 --list
     ```
@@ -118,26 +118,26 @@
     git push --tags
     ```
 
-00. For non-development releases, open a pull request to merge the proposed 
-    release into main. The following command will open a web browser for 
+00. For non-development releases, open a pull request to merge the proposed
+    release into main. The following command will open a web browser for
     you to create the pull request.
     ```sh
     gh pr create --web
     ```
     After creating the PR, confirm with the ActivitySim PMC that the release
     is ready before actually merging it.
-    
+
     Once final approval is granted, merge the PR into main.  The presence
     of the git tags added earlier will trigger automated build steps to
     prepare and deploy the release to pypi and conda-forge.
-    
+
 00. Create a "release" on GitHub.
     ```sh
     gh release create v1.2.3
     ```
     For a development pre-release, include the `--prerelease` argument.
     As the project's policy is that only formally released code is merged
-    to the main branch, any pre-release should also be built against a 
+    to the main branch, any pre-release should also be built against a
     non-default branch.  For example, to pre-release from the `develop`
     branch:
     ```sh
@@ -147,11 +147,11 @@
         --notes "this pre-release is for a cool new feature" \
         --title "Development Pre-Release"
     ```
-    
-00. Clean up your workspace, including removing the Conda environment used for 
-    testing (which will prevent you from accidentally using an old 
+
+00. Clean up your workspace, including removing the Conda environment used for
+    testing (which will prevent you from accidentally using an old
     environment when you should have a fresh up-to-date one next time).
     ```sh
     conda deactivate
     conda env remove -n TEMP-ASIM-DEV
-    ```
+    ```

activitysim/abm/models/__init__.py

@@ -26,9 +26,12 @@
     parking_location_choice,
     stop_frequency,
     summarize,
+    telecommute_frequency,
     tour_mode_choice,
     tour_od_choice,
     tour_scheduling_probabilistic,
+    transit_pass_ownership,
+    transit_pass_subsidy,
     trip_departure_choice,
     trip_destination,
     trip_matrices,
@@ -39,4 +42,5 @@
     trip_scheduling_choice,
     vehicle_allocation,
     vehicle_type_choice,
+    work_from_home,
 )

activitysim/abm/models/atwork_subtour_scheduling.py

@@ -41,6 +41,7 @@ def atwork_subtour_scheduling(
     estimator = estimation.manager.begin_estimation("atwork_subtour_scheduling")
 
     model_spec = simulate.read_model_spec(file_name=model_settings["SPEC"])
+    sharrow_skip = model_settings.get("sharrow_skip")
     coefficients_df = simulate.read_model_coefficients(model_settings)
     model_spec = simulate.eval_coefficients(model_spec, coefficients_df, estimator)
 
@@ -79,6 +80,7 @@ def atwork_subtour_scheduling(
         estimator=estimator,
         chunk_size=chunk_size,
         trace_label=trace_label,
+        sharrow_skip=sharrow_skip,
     )
 
     if estimator:

activitysim/abm/models/initialize_los.py

@@ -154,8 +154,16 @@ def compute_utilities_for_attribute_tuple(
         choosers_df, chunk_size, trace_label, chunk_tag=chunk_tag
     ):
         # we should count choosers_df as chunk overhead since its pretty big and was custom made for compute_utilities
-        assert chooser_chunk._is_view  # otherwise copying it is wasteful
-        chooser_chunk = chooser_chunk.copy()
+        if chooser_chunk._is_view:
+            chooser_chunk = (
+                chooser_chunk.copy()
+            )  # copy is needed when we start with a view
+        else:
+            # copying this is wasteful, but the code below edits the dataframe,
+            # which could have undesirable side effects.
+            # TODO: convert to Dataset or otherwise stop this copying, without
+            #       harming anything else.
+            chooser_chunk = chooser_chunk.copy()
         chunk.log_df(trace_label, "attribute_chooser_chunk", chooser_chunk)
 
         # add any attribute columns specified as column attributes in settings (the rest will be scalars in locals_dict)

activitysim/abm/models/joint_tour_participation.py

@@ -133,6 +133,8 @@ def participants_chooser(probs, choosers, spec, trace_label):
     In principal, this shold always occur eventually, but we fail after MAX_ITERATIONS,
     just in case there is some failure in program logic (haven't seen this occur.)
 
+    The return values are the same as logit.make_choices
+
     Parameters
     ----------
     probs : pandas.DataFrame

activitysim/abm/models/joint_tour_scheduling.py

@@ -71,6 +71,7 @@ def joint_tour_scheduling(tours, persons_merged, tdd_alts, chunk_size, trace_hh_
     estimator = estimation.manager.begin_estimation("joint_tour_scheduling")
 
     model_spec = simulate.read_model_spec(file_name=model_settings["SPEC"])
+    sharrow_skip = model_settings.get("sharrow_skip", False)
     coefficients_df = simulate.read_model_coefficients(model_settings)
     model_spec = simulate.eval_coefficients(model_spec, coefficients_df, estimator)
 
@@ -91,6 +92,7 @@ def joint_tour_scheduling(tours, persons_merged, tdd_alts, chunk_size, trace_hh_
         estimator=estimator,
         chunk_size=chunk_size,
         trace_label=trace_label,
+        sharrow_skip=sharrow_skip,
     )
 
     if estimator:

activitysim/abm/models/location_choice.py

@@ -10,9 +10,7 @@
     config,
     expressions,
     inject,
-    logit,
     los,
-    mem,
     pipeline,
     simulate,
     tracing,
@@ -116,6 +114,7 @@ def _location_sample(
     chunk_size,
     chunk_tag,
     trace_label,
+    zone_layer=None,
 ):
     """
     select a sample of alternative locations.
@@ -149,7 +148,13 @@ def _location_sample(
         )
         sample_size = 0
 
-    locals_d = {"skims": skims, "segment_size": segment_name}
+    locals_d = {
+        "skims": skims,
+        "segment_size": segment_name,
+        "orig_col_name": skims.orig_key,  # added for sharrow flows
+        "dest_col_name": skims.dest_key,  # added for sharrow flows
+        "timeframe": "timeless",
+    }
     constants = config.get_model_constants(model_settings)
     locals_d.update(constants)
 
@@ -175,6 +180,7 @@ def _location_sample(
         chunk_size=chunk_size,
         chunk_tag=chunk_tag,
         trace_label=trace_label,
+        zone_layer=zone_layer,
     )
 
     return choices
@@ -235,13 +241,7 @@ def aggregate_size_terms(dest_size_terms, network_los, model_settings):
     MAZ_size_terms = dest_size_terms.copy()
 
     # add crosswalk DEST_TAZ column to MAZ_size_terms
-    maz_to_taz = (
-        network_los.maz_taz_df[["MAZ", "TAZ"]]
-        .set_index("MAZ")
-        .sort_values(by="TAZ")
-        .TAZ
-    )
-    MAZ_size_terms[DEST_TAZ] = MAZ_size_terms.index.map(maz_to_taz)
+    MAZ_size_terms[DEST_TAZ] = network_los.map_maz_to_taz(MAZ_size_terms.index)
 
     MAZ_size_terms["avail_MAZ"] = np.where(
         (MAZ_size_terms.size_term > 0)
@@ -368,6 +368,7 @@ def location_presample(
         chunk_size,
         chunk_tag,
         trace_label,
+        zone_layer="taz",
     )
 
     # print(f"taz_sample\n{taz_sample}")
@@ -590,7 +591,13 @@ def run_location_simulate(
     skim_dict = network_los.get_default_skim_dict()
     skims = skim_dict.wrap("home_zone_id", alt_dest_col_name)
 
-    locals_d = {"skims": skims, "segment_size": segment_name}
+    locals_d = {
+        "skims": skims,
+        "segment_size": segment_name,
+        "orig_col_name": skims.orig_key,  # added for sharrow flows
+        "dest_col_name": skims.dest_key,  # added for sharrow flows
+        "timeframe": "timeless",
+    }
     constants = config.get_model_constants(model_settings)
     if constants is not None:
         locals_d.update(constants)
@@ -895,6 +902,8 @@ def iterate_location_choice(
 
     logger.debug("%s max_iterations: %s" % (trace_label, max_iterations))
 
+    choices_df = None  # initialize to None, will be populated in first iteration
+
     for iteration in range(1, max_iterations + 1):
 
         persons_merged_df_ = persons_merged_df.copy()
@@ -934,9 +943,7 @@ def iterate_location_choice(
                 and iteration > 1
             ):
                 # if a process ends up with no sampled workers in it, hence an empty choice_df_, then choice_df wil be what it was previously
-                if len(choices_df_) == 0:
-                    choices_df = choices_df
-                else:
+                if len(choices_df_) != 0:
                     choices_df = pd.concat([choices_df, choices_df_], axis=0)
                     choices_df_index = choices_df_.index.name
                     choices_df = choices_df.reset_index()