-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New chp type to run some common sidereal processing tasks #161
base: master
Are you sure you want to change the base?
Conversation
59988df
to
a4f486c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me overall, although I won't approve it because Richard should probably have the final say.
A few comments:
-
Should we consider using the 82-day rev06 stack in BlendStack instead of the rev03 stack? Occasionally in rev06 validation, we've seen hints of artifacts (e.g. vertical lines in delay spectra) that may be coming from the rev03 stack instead of individual rev06 days themselves, and presumably the rev06 stack is cleaner overall.
-
Should we consider another level of abstraction for
DelaySpectrumProcessing
? Most of the code in that class isn't specific to the precise reprocessing we want to do. Thus, we could consider bundling that code into a generic template (calledReProcessing
or something), and then haveDelaySpectrumProcessing
as a subclass of that, only specifying the modifications todefault_params
,default_script
etc. In the future, if we need to some other partial reprocessing of a revision, we'd only need to subclassReProcessing
and write the corresponding script.
Agreed
This can definitely be the default, although that should perhaps be a change for the standard Daily config in rev_07 of the daily pipeline. For reprocessing daily:rev_06, I think we should continue to use the rev_03 stack so no additional changes are introduced other than the new static mask.
I like this idea, although I think we will have to put some thought into the implementation so that we don't end up with a bunch of different |
c9bba86
to
c68d460
Compare
I've also added a task to get the static rfi mask and broadcast to the shape of the incoming data, and updated the reprocessing config to reflect the thresholdvisweight refactor |
26029c0
to
d77d0c2
Compare
9b57519
to
baf0b7b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks really nice. Even if we don't end up using it, having a ready-to-go implementation of the functionality will be great for when we do need to do something similar.
For instance I can imagine running Minori's bandpass averaging code over the processed days using all this which would be great.
83cfab4
to
f5211c0
Compare
f5211c0
to
6ef74b6
Compare
2913069
to
fc75b0e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great! I had only one minor comment to add. All of my previous comments have been addressed, so I'll leave it to @jrs65 to provide a final approval if he's happy as well
fc75b0e
to
54db07b
Compare
54db07b
to
c14dd9b
Compare
fdc82e7
to
baaa3bc
Compare
This adds a `delay` processing type to the types available in chp. The basic implementation of the new type loads a sidereal stream file, runs the standard daily processing masking, and produces a delay spectrum and high-pass filtered delay spectrum. A given revision of the `delay` type is linked to an existing revision of the `daily` type, so available tags are pulled from the completed tags of the linked `daily` revision.
baaa3bc
to
d764461
Compare
This PR introduces a new chp processing type, which runs the barebones daily pipeline steps to produce a delay spectrum from a sidereal stream. The type is linked to an existing daily processing revision, so the available tags are pulled from the tags completed by the linked day. The linked daily revision is by default the most recent, but this can be changed in the revconfig file after creation.
The idea is that anyone using this type is going to want to be customizing the jobtemplate in order to do something different from the linked pipeline revision. This just provides the barebones template, which should reproduce the delay spectrum that was created by the daily revision, assuming the venv is using the same commits (this would have to be manually configured).
It also adds a chp type for generating the wavelet spectrum of a given day based on radiocosmology/draco#237