Add files via upload #558

iechevarriaz · 2020-06-11T07:42:05Z

Hello, this is my first attempt to contribute to a github repository. The aim of this PR is to try to get help in order to add Bos taurus as new species to the catalog and then adding an inferred Demographic model from literature.
Thank you!

grahamgower

Hi @iechevarriaz. It would be great to have Bos taurus in the catalog! I've opened an QC issue over at #559, which is a checklist of things that we'll look at before merging the new species into stdpopsim. This is quite a new process, so please let us know if you have any comments or questions. I've also left some comments next to your code. If you have specific questions, please fire away, and we'll do our best to help out.

grahamgower · 2020-06-11T09:27:03Z

BosTau.py

+
+# Create a string of chromosome lengths for easy parsing
+_chromosome_data = """\
+chr1    158534110


Could you add a comment here about which reference assembly (and version) the chromosome lengths come from?

If you have a URL for where you got it from that would be nice to have in a comment as well

The reference assembly comes from Ensembl genome browser... Genome assembly: ARS-UCD1.2 (GCA_002263795.2) . https://www.ensembl.org/Bos_taurus/Info/Annotation
Worked out the chromosome lengths from each "chromosome summary" in Ensembl. Copied and pasted.
Eg. for Chr1 ....https://www.ensembl.org/Bos_taurus/Location/Chromosome?r=1:421347-158534110
Chr2... https://www.ensembl.org/Bos_taurus/Location/Chromosome?r=2:14572483-14672483

BosTau.py

grahamgower · 2020-06-11T09:33:35Z

BosTau.py

+_genome = stdpopsim.Genome(
+    chromosomes=_chromosomes,
+    assembly_citations=[_assembly_citation])
+


You'll also need citations for the mutation rate and recombination rate used above.

grahamgower · 2020-06-11T09:36:36Z

BosTau.py

+###########################################################
+
+def _iid():
+    id = "IonaInferredDemography"


To get the tests to pass, this id will need to follow a specific naming convention. See https://stdpopsim.readthedocs.io/en/latest/development.html#adding-a-new-demographic-model and https://stdpopsim.readthedocs.io/en/latest/development.html#naming-conventions

@iechevarriaz this still needs to be fixed. Please use this format ${SomethingDescriptive}_${number_of_populations}${first_author_initial}${two_digit_date}.

BosTau.py

jeromekelleher · 2020-06-11T10:19:19Z

This is great, thanks @iechevarriaz! Welcome to stdpopsim, and thanks for taking the time to make this great contribution! It'll be fantastic to have cattle in the catalog.

From a quick scan of the code, I think you're most of the way there. We'll need to get the QC process going for the species data, which we're still working out, so please bear with us there.

ndukler · 2020-06-11T16:02:59Z

BosTau.py

+
+# Create a string of chromosome lengths for easy parsing
+_chromosome_data = """\
+chr1    158534110


If you have a URL for where you got it from that would be nice to have in a comment as well

ndukler · 2020-06-11T16:12:20Z

BosTau.py

+    The Runs of Homozygosity-based infer of Demography from MacLeod et al. 2013.
+    """
+    populations = [
+        stdpopsim.Population(id="FILL ME", description="FILL ME"),


Please add population id and description. See other implemented models for examples.

ndukler · 2020-06-11T16:14:51Z

BosTau.py

+                initial_size=90, growth_rate=0.0166,
+                metadata=populations[0].asdict())
+        ],
+        #migration_matrix=[


This can just be migration_matrix = [0] if you want to be explicit. Otherwise just delete.

BosTau.py

ndukler · 2020-06-11T16:17:49Z

BosTau.py

+_gen_time_citation = stdpopsim.Citation(
+    doi="",
+    year="",
+    author="",
+    reasons={stdpopsim.CiteReason.GEN_TIME})


You put 5 years as the generation time below, please add a citation for that number.

ndukler · 2020-06-11T16:18:41Z

BosTau.py

+_pop_size_citation = stdpopsim.Citation(
+    doi="",
+    year="",
+    author="",
+    reasons={stdpopsim.CiteReason.POP_SIZE})


You put 62,000 below, please add a citation for that number here if you think it is a representative population size for cattle.

ndukler · 2020-06-11T16:23:19Z

This is great, thanks @iechevarriaz! Welcome to stdpopsim, and thanks for taking the time to make this great contribution! It'll be fantastic to have cattle in the catalog.

From a quick scan of the code, I think you're most of the way there. We'll need to get the QC process going for the species data, which we're still working out, so please bear with us there.

I'd like to add my greetings as well :) Thanks for contributing!

agladstein · 2020-06-11T21:32:29Z

Welcome @iechevarriaz! Since you are adding both a new species and a new demographic model we will doing the QC process for both the new species and new demographic model.
Could you open a new issue with the Model QC issue template to get that started? Thanks!

Your BosTau.py file should be moved to stdpopsim/catalog/.

iechevarriaz · 2020-06-13T04:46:40Z

Thank you all for your commentaries and help!!!! And such a warm welcome! In the next couple of days, I'll be trying to resolve these citations and change parameters if needed. For that, I'm contacting people to point me in the right direction.

Here I've tried to put a descriptive word for the model, then the number of populations, the initial for the first author and the year of publication.

Added citations for mutation rates and recombination rates. Wrote the genome object with those citations.

ndukler

Looking good! There are some changes that need to be made based on some additional integration with Ensembl that we've added.

ndukler · 2020-08-16T14:05:19Z

BosTau.py

+_chromosome_data = """\
+chr1    158534110
+chr2    136231102
+chr3    121005158
+chr4    120000601
+chr5    120089316
+chr6    117806340
+chr7    110682743
+chr8    113319770
+chr9    105454467
+chr10   103308737
+chr11   106982474
+chr12   87216183
+chr13   83472345
+chr14   82403003
+chr15   85007780
+chr16   81013979
+chr17   73167244
+chr18   65820629
+chr19   63449741
+chr20   71974595
+chr21   69862954
+chr22   60773035
+chr23   52498615
+chr24   62317253
+chr25   42350435
+chr26   51992305
+chr27   45612108
+chr28   45940150
+chr29   51098607
+chrX    139009144
+"""


Sorry for the sudden change but this info should now be in an auto-generated file in this directory named genome_data.py this file is autogenerated by querying the ensembl REST API. We have a script for updating chromosome info for all species here but you probably want to tweak it to just add cow. (We really need to add at least a simple CLI to that file to make this easier). Finally, you will need to add an from . import genome_data to the top of this file and rename it __init__.py. Look at some of the other species for an example if you need them.

Hello @ndukler , I'm trying to run the script for getting the chromosome info but it crashes, it cannot find module called "maintenance". Moreover, I don't know how to tweak it to get only cattle info. That should generate a file called genome_data.py , is that correct? I've already made some changes including rename of BosTau.py to init.py

That's correct. Also I suppose you can run it so it updates everything then discard all changes that are not to bosTau. I'll look into running it and get back to you. If I can just generate genome_data.py I'll just send it to you.

ndukler · 2020-08-16T14:26:18Z

BosTau.py

+    _chromosomes.append(stdpopsim.Chromosome(
+        id=name, length=int(length),
+        mutation_rate=1.2e-8,
+        recombination_rate=9.26e-9)) # 25.5 crossovers per meiosis in males and 23.2 crossovers per meiosis in females gives an average of 24.35 crossovers per meiosis


Mutation and recombination rates look good! Relatively easy to see where you got them in the papers

ndukler · 2020-08-16T14:31:14Z

BosTau.py

+###########################################################
+
+def _iid():
+    id = "IonaInferredDemography"


@iechevarriaz this still needs to be fixed. Please use this format ${SomethingDescriptive}_${number_of_populations}${first_author_initial}${two_digit_date}.

ndukler · 2020-08-16T14:38:36Z

Also, I think we're far enough along that you can send me the files for the recombination map. They should be placed in a flat tarball with one file per chromosome.

agladstein · 2020-08-24T18:11:44Z

Hey @iechevarriaz, I just wanted to remind you that to start the demographic model QC process we'll need a new issue with the Model QC issue template.

ndukler · 2020-09-02T15:30:00Z

@iechevarriaz I've updated the script on downloading a new species genome info and updated the devdocs. I've also been working on the species QC and most of it is good to go except some of the assembly stuff (most of which is just handled by the automated download). If you pull master you should have everything you need to move forward. If possible can we wrap this up this week?

iechevarriaz · 2020-09-02T18:32:11Z

Hello @ndukler ! Thank you for the update... I have run the update_ensembl_data.py script and it fetches assemby_accession and assemby_name for BosTau, but it doesn't fill in the chromosome lengths...

grahamgower · 2020-09-02T19:10:53Z

Hey @iechevarriaz. It looks like a bug in maintenance/enseml.py. I'll open a ticket.

iechevarriaz · 2020-09-03T14:30:14Z

Thank you @grahamgower and @jeromekelleher ... now it works!!!!

Also, I think we're far enough along that you can send me the files for the recombination map. They should be placed in a flat tarball with one file per chromosome.

@ndukler Sorry but we don't have a recombination map for Bos taurus...

Is there anything else I should be doing?

ndukler · 2020-09-03T15:02:04Z

I think you're good. Just push the updates and the chromosome info.

Thank you @grahamgower and @jeromekelleher ... now it works!!!!

Also, I think we're far enough along that you can send me the files for the recombination map. They should be placed in a flat tarball with one file per chromosome.

@ndukler Sorry but we don't have a recombination map for Bos taurus...

Is there anything else I should be doing?

grahamgower · 2020-10-02T08:45:57Z

This looks like its basically done. I'm going to make some minor changes (in a new PR) so we can merge this.

grahamgower · 2020-10-02T12:09:49Z

Closing in favour of #600.

Add files via upload

844e2ee

grahamgower mentioned this pull request Jun 11, 2020

QC for BosTau #559

Closed

11 tasks

grahamgower requested changes Jun 11, 2020

View reviewed changes

jeromekelleher requested review from agladstein and ndukler June 11, 2020 10:19

ndukler suggested changes Jun 11, 2020

View reviewed changes

iechevarriaz added 2 commits June 12, 2020 00:18

Update BosTau.py

5294d8b

Update BosTau.py

a11bfd2

iechevarriaz added 7 commits June 13, 2020 06:21

Update BosTau.py

d319262

Here I've tried to put a descriptive word for the model, then the number of populations, the initial for the first author and the year of publication.

Update BosTau.py

0083e64

Update BosTau.py

f3a96c4

Added citations for mutation rates and recombination rates. Wrote the genome object with those citations.

Update BosTau.py

111e2d3

Update BosTau.py

d1c92c1

Update BosTau.py

47deb96

Update BosTau.py

2e63039

ndukler suggested changes Aug 16, 2020

View reviewed changes

ndukler reviewed Aug 16, 2020

View reviewed changes

Update and rename BosTau.py to __init__.py

8d9df0f

iechevarriaz mentioned this pull request Aug 25, 2020

QC for BosTau/HolsteinFriesian_1M13 #579

Closed

grahamgower mentioned this pull request Sep 2, 2020

maintenance/ensembl.py doesn't work for all species with chromosome-level assemblies #580

Closed

grahamgower closed this Oct 2, 2020

gregorgorjanc mentioned this pull request Apr 24, 2022

create list of authors popsim-consortium/adding-species-manuscript#32

Closed

grahamgower mentioned this pull request May 18, 2022

QC for BosTau #1239

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add files via upload #558

Add files via upload #558

iechevarriaz commented Jun 11, 2020

grahamgower left a comment

grahamgower Jun 11, 2020

ndukler Jun 11, 2020

iechevarriaz Jun 11, 2020

grahamgower Jun 11, 2020

grahamgower Jun 11, 2020

ndukler Aug 16, 2020

jeromekelleher commented Jun 11, 2020

ndukler Jun 11, 2020

ndukler Jun 11, 2020

ndukler Jun 11, 2020

ndukler Jun 11, 2020

ndukler Jun 11, 2020

ndukler commented Jun 11, 2020

agladstein commented Jun 11, 2020

iechevarriaz commented Jun 13, 2020

ndukler left a comment •

edited

Loading

ndukler Aug 16, 2020

iechevarriaz Aug 17, 2020

ndukler Aug 17, 2020 •

edited

Loading

ndukler Aug 16, 2020

ndukler Aug 16, 2020

ndukler commented Aug 16, 2020

agladstein commented Aug 24, 2020

ndukler commented Sep 2, 2020 •

edited

Loading

iechevarriaz commented Sep 2, 2020

grahamgower commented Sep 2, 2020

iechevarriaz commented Sep 3, 2020

ndukler commented Sep 3, 2020

grahamgower commented Oct 2, 2020

grahamgower commented Oct 2, 2020

Add files via upload #558

Add files via upload #558

Conversation

iechevarriaz commented Jun 11, 2020

grahamgower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeromekelleher commented Jun 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndukler commented Jun 11, 2020

agladstein commented Jun 11, 2020

iechevarriaz commented Jun 13, 2020

ndukler left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndukler Aug 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndukler commented Aug 16, 2020

agladstein commented Aug 24, 2020

ndukler commented Sep 2, 2020 • edited Loading

iechevarriaz commented Sep 2, 2020

grahamgower commented Sep 2, 2020

iechevarriaz commented Sep 3, 2020

ndukler commented Sep 3, 2020

grahamgower commented Oct 2, 2020

grahamgower commented Oct 2, 2020

ndukler left a comment •

edited

Loading

ndukler Aug 17, 2020 •

edited

Loading

ndukler commented Sep 2, 2020 •

edited

Loading