Log summary of simulation statuses

[nweires]

At the end of post-processing, print a summary of successful/failed/invalid simulations, grouped by upgrade.

Example output:

INFO:2023-12-15 16:30:59:buildstockbatch.cloud.docker_base:Final status of all simulations:
Baseline     Success: 8        Fail: 0        Invalid: 0        
Upgrade 01   Success: 8        Fail: 0        Invalid: 0        
Upgrade 02   Success: 8        Fail: 0        Invalid: 0        
Upgrade 03   Success: 8        Fail: 0        Invalid: 0        
Upgrade 04   Success: 2        Fail: 0        Invalid: 6        
Upgrade 05   Success: 8        Fail: 0        Invalid: 0        
Upgrade 06   Success: 0        Fail: 0        Invalid: 8        
Upgrade 07   Success: 8        Fail: 0        Invalid: 0        
Upgrade 08   Success: 8        Fail: 0        Invalid: 0        
Upgrade 09   Success: 8        Fail: 0        Invalid: 0        
Upgrade 10   Success: 8        Fail: 0        Invalid: 0        
Upgrade 11   Success: 8        Fail: 0        Invalid: 0        
Upgrade 12   Success: 8        Fail: 0        Invalid: 0        

Total        Success: 90       Fail: 0        Invalid: 14       

No results found for Upgrade 13
No results found for Upgrade 14

Future work:

• If useful, we could also show a few IDs of specific buildings that failed (to find them more quickly for debugging)
• Look for specific error messages in the logs and find a way to summarize those.

[github-actions]

File	Coverage
All files	86%	✅
base.py	91%	✅
exc.py	57%	✅
hpc.py	78%	✅
local.py	70%	✅
postprocessing.py	84%	✅
utils.py	91%	✅
cloud/docker_base.py	78%	✅
sampler/base.py	79%	✅
sampler/downselect.py	33%	✅
sampler/precomputed.py	93%	✅
sampler/residential_quota.py	61%	✅
test/shared_testing_stuff.py	85%	✅
test/test_docker.py	33%	✅
test/test_local.py	97%	✅
test/test_validation.py	97%	✅
workflow_generator/base.py	90%	✅
workflow_generator/commercial.py	53%	✅
workflow_generator/residential_hpxml.py	86%	✅

Minimum allowed coverage is 33%

Generated by 🐒 cobertura-action against da029d7

[mfathollahzadeh]

Thanks Natalie! This looks great and would be really helpful! I just added some thought to get your feedback on those and see if implementing those would add something to our guardrails. Thanks!

[mfathollahzadeh]

Thanks, Natalie!

[lathanh]

What do you think of also including:

• The sum for each status (e.g., total successes and total failures)

• The total of all statuses counted, and a warning (of sorts) if it's less than the total number of sims?

[nweires]

Sums: Sure, I can add that.

Comparing to the total: This is possible, but kind of annoying to do at this point in the job. To find the total number of simulations that should exist, I think we have to pull buildstock.csv back out of assets.tar.gz on the cloud and re-count how many rows it has. (We already have this info locally if running the full pipeline, but may not if running postprocessing only.) Thoughts on how important this is? Or a simpler way to find the intended total?

[lathanh]

If the total isn't readily available, I think it's fine to leave it out. We can easily pull that from elsewhere if needed.