Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hot-fix: 0.4.21 run column isn't showing unique results #189

Merged
merged 4 commits into from
Sep 5, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,10 @@ testing/multi_config_test.py
testing/udo.py
testing/udo_test.py
simulations/regression_tests/poc
simulations/tickets

Simulation.md

monkeytype.sqlite3

notes.py
result
26 changes: 14 additions & 12 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ are still appended globally despite `append_config` being a method of Experiment
### New Features:
##### [Local Execution Mode (Default)](documentation/Simulation_Execution.md)
* Local Execution Mode (Default): Implicit parallelization of Monte-Carlo / Stochastic simulations (Automatically
selects Multi-Process / Threaded Mode if simulations are configure for a single run)
selects Multi-Threaded Mode if simulations are configured for more than a single run)
* **Backwards Compatibility:** `cadCAD.engine.ExecutionMode` accepts legacy execution modes from ver. `0.3.1`

##### cadCAD Post-Processing Enhancements / Modifications
Expand All @@ -28,17 +28,6 @@ selects Multi-Process / Threaded Mode if simulations are configure for a single
* **Subset** is a unique identifier of Monte Carlo simulations produced by parameter sweeps
* Note: Returning a single dataset was originally specified during the project’s inception instead of multiple per
simulation

* The `configs` `list` has been temporarily flattened to contain single run System Model `Configuration` objects to
support elastic workloads. This functionality will be restored in a subsequent release by a class that returns
`configs`'s original representation in ver. `0.3.1`.
* The conversion utilities have been provided to restore its original representation of configurations with
runs >= 1
* [System Configuration Conversions](documentation/System_Configuration.md)
* Configuration as List of Configuration Objects (as in ver. `0.3.1`)
* New: System Configuration as a Pandas DataFrame
* New: System Configuration as List of Dictionaries


##### Expandable state and policy update parameter space:
* Enables the development of feature enhancements that involve the use of additional parameters without requiring users
Expand All @@ -47,6 +36,19 @@ examples in documentation include an additional `**kwargs` parameter.
* [State Updates:](https://github.com/cadCAD-org/cadCAD/blob/master/documentation/README.md#state-update-functions)
* [Policy Updates:](https://github.com/cadCAD-org/cadCAD/blob/master/documentation/README.md#state-update-functions)

### Modifications:
* **Flattened Configuration List:**
* The `configs` `list` has been temporarily flattened to contain single run System Model `Configuration` objects to
support both fault tolerant simulation and elastic workloads for scalable micro-service architecture and
distributed computing. This functionality will be restored in a subsequent release by a class that returns
`configs`'s original representation in ver. `0.3.1`.
* The conversion utilities have been provided to restore its original representation of configurations with
runs >= 1
* [System Configuration Conversions](documentation/System_Configuration.md)
* Configuration as List of Configuration Objects (as in ver. `0.3.1`)
* New: System Configuration as a Pandas DataFrame
* New: System Configuration as List of Dictionaries


### May 29, 2020
* Packaging: Add [Nix](https://nixos.org/) derivation and shell for local development and distribution of cadCAD package
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
/ ___/ __` / __ / / / /| | / / / /
/ /__/ /_/ / /_/ / /___/ ___ |/ /_/ /
\___/\__,_/\__,_/\____/_/ |_/_____/
by cadCAD ver. 0.4.21
by cadCAD ver. 0.4.22
======================================
Complex Adaptive Dynamics
o i e
Expand All @@ -21,7 +21,7 @@ through simulation, with support for Monte Carlo methods, A/B testing and parame
# Getting Started


#### Change Log: [ver. 0.4.21](CHANGELOG.md)
#### Change Log: [ver. 0.4.22](CHANGELOG.md)

[Previous Stable Release (Deprecated / No Longer Supported)](https://github.com/cadCAD-org/cadCAD/tree/b9cc6b2e4af15d6361d60d6ec059246ab8fbf6da)

Expand Down
18 changes: 10 additions & 8 deletions cadCAD/engine/execution.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from typing import Callable, Dict, List, Any, Tuple
from pathos.multiprocessing import ThreadPool as TPool
from pathos.multiprocessing import ProcessPool as PPool
# from pathos.multiprocessing import ProcessPool as PPool
from collections import Counter

from cadCAD.utils import flatten
Expand Down Expand Up @@ -99,11 +99,12 @@ def threaded_executor(params):
tp.close()
return results

pp = PPool()
results = flatten(list(pp.map(lambda params: threaded_executor(params), new_params)))
pp.close()
pp.join()
pp.clear()
# pp = PPool()
# results = flatten(list(pp.map(lambda params: threaded_executor(params), new_params)))
results = flatten(list(map(lambda params: threaded_executor(params), new_params)))
# pp.close()
# pp.join()
# pp.clear()
# pp.restart()

return results
Expand All @@ -130,17 +131,18 @@ def local_simulations(
config_amt = len(configs_structs)
try:
_params = None
if config_amt == 1:
if config_amt == 1: # and configured_n != 1
_params = var_dict_list[0]
return single_proc_exec(
simulation_execs, _params, states_lists, configs_structs, env_processes_list,
Ts, SimIDs, Ns, ExpIDs, SubsetIDs, SubsetWindows, configured_n
)
elif config_amt > 1: # and config_amt < remote_threshold:
elif config_amt > 1: # and configured_n != 1 # and config_amt < remote_threshold
_params = var_dict_list
return parallelize_simulations(
simulation_execs, _params, states_lists, configs_structs, env_processes_list,
Ts, SimIDs, Ns, ExpIDs, SubsetIDs, SubsetWindows, configured_n
)
# elif config_amt > 1 and configured_n == 1:
except ValueError:
raise ValueError("\'sim_configs\' N must > 0")
1 change: 0 additions & 1 deletion cadCAD/engine/simulation.py
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,6 @@ def run_pipeline(
pipe_run: List[Dict[str, Any]] = self.state_update_pipeline(
sweep_dict, simulation_list, configs, env_processes, time_step, run, additional_objs
)

_, *pipe_run = pipe_run
simulation_list.append(pipe_run)

Expand Down
Binary file removed dist/cadCAD-0.4.21.tar.gz
Binary file not shown.
Binary file not shown.
Binary file added dist/cadCAD-0.4.22.tar.gz
Binary file not shown.
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
"""

setup(name='cadCAD',
version='0.4.21',
version='0.4.22',
description=short_description,
long_description=long_description,
url='https://github.com/cadCAD-org/cadCAD',
Expand Down