Write checkpoint history #3347
Merged
Write checkpoint history #3347
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 8 commits
July 1, 2020 10:46
|
@ahnitz Can you take a look? |
ahnitz
approved these changes
Jul 10, 2020
lenona
pushed a commit
to lenona/pycbc
that referenced
this pull request
Sep 14, 2020
* add ability to append data in base_hdf write_data * add write start time method to base sampler io * add update_checkpoint_history function to base sampler file * add update sampler history to base mcmc file * write run start time when starting up run * call update checkpoint history in base mcmc's checkpoint * fix various bugs * add run end time attribute * handle no act or burn in having been calculated
OliverEdy
pushed a commit
to OliverEdy/pycbc
that referenced
this pull request
Apr 3, 2023
* add ability to append data in base_hdf write_data * add write start time method to base sampler io * add update_checkpoint_history function to base sampler file * add update sampler history to base mcmc file * write run start time when starting up run * call update checkpoint history in base mcmc's checkpoint * fix various bugs * add run end time attribute * handle no act or burn in having been calculated
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This makes it so that some history is preserved in the inference checkpoint file on each checkpoint. Right now, only the MCMC samplers make use of this, but nested samplers could pick up the functionality later. This can be used for comparing performance between samplers, and to diagnose issues (like growing ACTs).
On each checkpoint,
update_checkpoint_historyis called. This is a method ofBaseSamplerFile. It writes the current time and the time since the last checkpoint or last startup (which ever happened last). It then calls_update_sampler_history. This doesn't do anything inBaseSamplerFile, but can be overridden to write-sampler specific things. This is overriden byCommonMCMCMetadataIOto write the number of iterations, effective number of samples, burn-in iteration, and autocorrelation time. In the future, nested samplers could do something similar (e.g., writing the current dlogz). All information is written tosamlper_info/checkpoint_history/.To make this work, the ability to append data has been added to
BaseInferenceFile.write_data. I also had to add the time that pycbc inference starts up, so that the time since the last checkpoint could be accurately computed. This is stored asrun_start_timein the file'sattrs. For completeness, I also added arun_end_time, which is written bypycbc_inferencejust before exiting.I tested using the single template example in the docs, on emcee pt and epsie.
In another PR, I'll add an executable to plot the checkpoint history.