Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Format data for analysis #131

Merged
merged 17 commits into from
Jun 21, 2022
Merged

Format data for analysis #131

merged 17 commits into from
Jun 21, 2022

Conversation

ngreenwald
Copy link
Member

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?
This PR adds a final notebook to organize the processed data so it's ready for analysis in ark.

  • It first renames all of the FOVs to have their original, user-generated name in a new Cohorts folder.
  • It then combines together data from the same experiment that were split across runs
  • Finally, it merges all of the FOVs together into a single image_data folder, giving each fov a unique prefix based on the run it was part of.
  • It also updates the logic of the fov renaming function to only copy over the selected folders

Remaining issues
Check that the notebooks run using the default base paths. Update the toffy_setup notebook with the 'cohorts` folder

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

ackagel
ackagel reviewed Jun 15, 2022
View reviewed changes
Copy link
Contributor

@ackagel ackagel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, mostly just a few questions

toffy/reorg.py

# get list of matching subfolders
partial_folders = io_utils.list_folders(cohort_dir, substrs=run_string)
partial_folders = [partial for partial in partial_folders if partial != run_string]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a risk that partial == run_string and that case causes issues?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope, there's a test case for that. It becomes the combined directory and the other fovs are copied into it

toffy/reorg.py Outdated Show resolved Hide resolved
alex-l-kong
alex-l-kong requested changes Jun 15, 2022
View reviewed changes
templates/5_rename_and_reorganize.ipynb Outdated Show resolved Hide resolved
toffy/reorg.py Outdated Show resolved Hide resolved
toffy/reorg.py Show resolved Hide resolved
@ngreenwald ngreenwald merged commit b551796 into main Jun 21, 2022
@ngreenwald ngreenwald deleted the reformat_run_dirs branch June 21, 2022 20:25
@ngreenwald ngreenwald mentioned this pull request Jun 21, 2022
Merged
@camisowers camisowers added the enhancement New feature or request label Sep 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@srivarra srivarra Awaiting requested review from srivarra

@camisowers camisowers Awaiting requested review from camisowers

@ackagel ackagel Awaiting requested review from ackagel

@alex-l-kong alex-l-kong Awaiting requested review from alex-l-kong

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ngreenwald @alex-l-kong @ackagel @camisowers