Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release v2.0.0 #90

Merged
merged 127 commits into from
Jan 16, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
127 commits
Select commit Hold shift + click to select a range
8f2a562
update system design diagram
Oct 5, 2022
c38d07a
parse host from CVODI-19 Data Portal as it should
priesgo Nov 1, 2022
a9c72fa
avoid writing the configuration in logs multiple times
priesgo Nov 1, 2022
eb09e14
log error
priesgo Nov 1, 2022
50e3b98
change acc by id in the COVID19 data portal API
priesgo Nov 1, 2022
7013ab3
change acc by id in the COVID19 data portal API
priesgo Nov 1, 2022
beb38bb
makes sure that run_accession is properly read and add binary search …
priesgo Nov 1, 2022
900bf9a
add search based on python set
priesgo Nov 1, 2022
0255a46
set accessor to have infinite retries
priesgo Nov 1, 2022
c3ecd14
add logs
priesgo Nov 1, 2022
114c985
rollbacks the change to use exclusion by coverage based on external c…
priesgo Nov 1, 2022
4da5194
fix issue building URL
priesgo Nov 1, 2022
8bf01f4
correct logs
priesgo Nov 1, 2022
8dce0c1
skip pangolin in the pipeline for COVID-19 data portal
priesgo Nov 2, 2022
53c3808
avoid interlock between nodes when waiting for completion
priesgo Nov 2, 2022
a320e67
avoid interlock between nodes when queueing jobs
priesgo Nov 2, 2022
d3435c8
fix mistake with list...
priesgo Nov 2, 2022
342857f
avoid processing missing pangolin results in C19 processor
priesgo Nov 2, 2022
f2955cb
avoid accessor issue with repeated samples in the API
priesgo Nov 2, 2022
50a23d4
introduce retry on Json parsing from the API
priesgo Nov 3, 2022
e55dd44
add some logs to COVID-19 Data Portal
priesgo Nov 7, 2022
f5c809b
query API by months
Nov 11, 2022
14b448c
update COVID19 data portal accessor to query by months
Nov 12, 2022
7f80f67
fix covid19 data portal accessor
priesgo Nov 12, 2022
8716508
make sure that repeated samples do not stop the whole accessor
priesgo Nov 15, 2022
f5441c1
add an alternative pipeline to rephase and reannotate mutations
priesgo Nov 22, 2022
2224e63
bump pipeline version
priesgo Nov 22, 2022
c5cbf3c
add support for complex variants in the data loading
priesgo Nov 25, 2022
c8d3348
include COMPLEX variant type in a drop down
priesgo Nov 25, 2022
1763d01
include rephasing option in the COVID19 data portal
priesgo Nov 30, 2022
046442b
dirty fix to improve performance
priesgo Nov 30, 2022
6adb3ab
enable configuring the nextflow profile via env variable
priesgo Dec 4, 2022
a4ede65
rollback dirty performance fix for massive rephasing
priesgo Dec 9, 2022
df99dae
Merge pull request #53 from TRON-Bioinformatics/fix-covid19-host
priesgo Dec 9, 2022
9894f3f
Implemented smooting of lineages plot with a simple moving average
johausmann Dec 15, 2022
6866990
Merge branch '54-smooth_lineages_plot' of https://github.com/TRON-Bio…
johausmann Dec 15, 2022
a13bcf0
Rebased branch against dev branch
johausmann Dec 15, 2022
f1109bd
Removed unused line pd.set_index() line
johausmann Dec 17, 2022
54e9137
Merge pull request #58 from TRON-Bioinformatics/54-smooth_lineages_plot
priesgo Dec 19, 2022
8a50c1d
add code coverage workflow to CI
priesgo Dec 20, 2022
e8b63ec
change something in code coverage
priesgo Dec 20, 2022
5fcc9bf
change some actions versions
priesgo Dec 20, 2022
e13b259
install covigator dependencies to run code coverage
priesgo Dec 20, 2022
5cb2ce5
add postgres service to code coverage workflow
priesgo Dec 20, 2022
c7a7c6c
remove code coverage action on PR, only push
priesgo Dec 20, 2022
1b0c03c
add codecov.yml to integrate code coverage results into PRs
priesgo Dec 20, 2022
f5a6262
add code coverage badge
priesgo Dec 20, 2022
55f86f8
Bump ipython from 7.27.0 to 7.31.1 in /docs
dependabot[bot] Dec 20, 2022
18b88a1
Bump numpy from 1.21.0 to 1.22.0
dependabot[bot] Dec 20, 2022
11b514f
fix codacy alerts on unused imports and method that could be a function
priesgo Dec 20, 2022
29b4a06
fix a method reported as too complex
priesgo Dec 20, 2022
85e8b2d
remove duplicated unit tests action
priesgo Dec 20, 2022
9a939de
Merge pull request #69 from TRON-Bioinformatics/code-coverage
priesgo Dec 20, 2022
3dc2931
bump version
priesgo Jan 3, 2023
735d5d9
make sure variants are only counted when they belong to a finished sa…
priesgo Jan 3, 2023
001d8fa
make sure variants are only counted when they belong to a finished sa…
priesgo Jan 3, 2023
a6b5929
update classification of mutations based on VAF
priesgo Jan 3, 2023
14da9e0
update test data
priesgo Jan 3, 2023
af11118
improves coverage in the VCF loader + remove unnecessary float conver…
priesgo Jan 4, 2023
d39ef7d
push coverage reports to codacy
priesgo Jan 4, 2023
7647510
fix VCF tests
priesgo Jan 4, 2023
35f974d
Merge pull request #72 from TRON-Bioinformatics/dependabot/pip/numpy-…
priesgo Jan 4, 2023
8ad145a
add documentation build to CI
priesgo Jan 4, 2023
7fa3bb0
Merge pull request #71 from TRON-Bioinformatics/dependabot/pip/docs/i…
priesgo Jan 4, 2023
8560dc8
do not send codacy
priesgo Jan 4, 2023
9896295
cover the VCF loader for COVID19 Data Portal
priesgo Jan 4, 2023
11a4c71
add db patch to reclassify variants by VAF
priesgo Jan 4, 2023
53fcae2
Merge pull request #77 from TRON-Bioinformatics/56/lq-clonal
priesgo Jan 4, 2023
cd3ed3c
add db patch to remove variants from beginning and end of genome
priesgo Jan 5, 2023
5d285a1
add db patch to set the right collection date in the variant observat…
priesgo Jan 5, 2023
7faf972
makes sure that the collection date is stored in variant observations
priesgo Jan 5, 2023
f9aba78
Fixed problems with NaN in column frequency_per_month in precomputed …
johausmann Jan 5, 2023
3f553b2
Remove unwanted uploaded files
johausmann Jan 5, 2023
3f301d7
fix the patch on dates for COVID19 Data Portal dataset
priesgo Jan 5, 2023
c86b27c
Merge pull request #78 from TRON-Bioinformatics/patch-variants-start-end
priesgo Jan 5, 2023
e77b9f9
add comment to avoid this issue from happening again
priesgo Jan 5, 2023
59c7b09
Merge pull request #76 from TRON-Bioinformatics/75-too-high-freqs
priesgo Jan 5, 2023
e7c7329
DB patches to introduce intrahost filters
priesgo Jan 7, 2023
ac84a30
first go introducing intrahost filtering in the VCF loader
priesgo Jan 7, 2023
c1eb24f
add tests
priesgo Jan 7, 2023
582a1ed
remove twitter!
priesgo Jan 7, 2023
d82ad0e
update export DB script
priesgo Jan 8, 2023
4e66c0b
update import DB script
priesgo Jan 8, 2023
ade554a
change backticks by $() notation
priesgo Jan 8, 2023
b2a5c13
change backticks by $() notation
priesgo Jan 8, 2023
917b584
remove unnnecessary quoting from bash import/export scripts
priesgo Jan 9, 2023
d7f0cf6
adding some missing quotes
priesgo Jan 9, 2023
79de65f
Merge pull request #79 from TRON-Bioinformatics/57-filter-hq-intrahost
priesgo Jan 9, 2023
65b42ef
fix codacy style issues
priesgo Jan 9, 2023
d69f5ec
add codacy badge
priesgo Jan 9, 2023
7db4f82
fix codacy detected issues
priesgo Jan 9, 2023
a3bc8a3
fix unit tests
priesgo Jan 9, 2023
6f67f97
Merge pull request #81 from TRON-Bioinformatics/codacy-things
priesgo Jan 9, 2023
1f0d657
merge
Jan 9, 2023
30e8630
update some screenshots
Jan 9, 2023
e1aea52
add some more screenshots
Jan 9, 2023
600fa1b
update all single screenshots
priesgo Jan 9, 2023
435121a
update figure 01
priesgo Jan 9, 2023
b8f5643
update figure 02
priesgo Jan 9, 2023
9e08ff7
update figure 03
priesgo Jan 9, 2023
76edd61
update size letters in figure 03
priesgo Jan 9, 2023
7674cb7
update figure 04
priesgo Jan 9, 2023
b6428f8
rollbacks session in the dashboard after DB error, this avoids a dash…
priesgo Jan 11, 2023
5185585
Merge pull request #84 from TRON-Bioinformatics/rollback-session-afte…
priesgo Jan 11, 2023
ec0f3d1
include teaser information for every tab
priesgo Jan 11, 2023
3e6a333
#80 change values of VAF filter
priesgo Jan 11, 2023
3a1072b
fix issue with intrahost query and samples without collection_date
priesgo Jan 11, 2023
cb872f3
include supplementary screenshots
priesgo Jan 11, 2023
f1d9b46
Updated raw SQL queries in queries.py to use SQLAlchemy text clause o…
johausmann Jan 11, 2023
d8ec9f5
Removed quotation mark from query that caused unit test to fail
johausmann Jan 12, 2023
781765c
avoid trying to fetch co-occurrence matrix in C19DP
priesgo Jan 13, 2023
22c64cd
edit the download tab to adapt to new classification of mutations
priesgo Jan 13, 2023
66db948
try to fix style issue reported by codacy
priesgo Jan 13, 2023
ac46a7d
try to fix style issue reported by codacy... again
priesgo Jan 13, 2023
1fa3f50
add word cloud + summary to overview page
priesgo Jan 13, 2023
18a9986
bump version
priesgo Jan 13, 2023
cc4460c
Merge pull request #87 from TRON-Bioinformatics/update_sql_queries
priesgo Jan 13, 2023
2b5212b
remove commented code
priesgo Jan 13, 2023
c191255
Merge pull request #83 from TRON-Bioinformatics/update-screenshots
priesgo Jan 13, 2023
a09fd13
some updates to the documentation
priesgo Jan 13, 2023
a7871f0
add word cloud to readme
priesgo Jan 13, 2023
391ce83
Merge pull request #88 from TRON-Bioinformatics/update-docs
priesgo Jan 13, 2023
157d8d5
fix bash escaped variables
priesgo Jan 14, 2023
e7beee9
fix "could be a function" issues from codacy
priesgo Jan 14, 2023
d45deb0
avoid too complex method
priesgo Jan 14, 2023
b3c106e
fix vcf loader + add missing storage of low quality clonal variants...
priesgo Jan 16, 2023
9a55f64
Merge pull request #91 from TRON-Bioinformatics/fix-codacy-issues
priesgo Jan 16, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
2. Click on '...'
3. Scroll down to '...'
4. See error

**Expected behavior**
Expand Down
23 changes: 23 additions & 0 deletions .github/workflows/documentation.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: Documentation
on: [push]
jobs:
run:
runs-on: ubuntu-20.04
env:
OS: ubuntu-20.04
PYTHON: '3.8'

steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.8'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools wheel coverage
pip install -r docs/requirements.txt

- name: Build documentation
run: |
cd docs; make clean html
21 changes: 13 additions & 8 deletions .github/workflows/unit_tests.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
name: Unit tests

on: [push]

jobs:
test:
run:
runs-on: ubuntu-20.04
env:
OS: ubuntu-20.04
PYTHON: '3.8'

# Service containers to run with `container-job`
services:
Expand All @@ -21,15 +22,16 @@ jobs:
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5

steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.8'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools wheel
- name: Build, install and run unit tests
pip install setuptools wheel coverage

- name: Generate Report
env:
# The hostname used to communicate with the PostgreSQL service container
POSTGRES_HOST: localhost
Expand All @@ -44,4 +46,7 @@ jobs:
python setup.py bdist_wheel
pip install numpy==1.21.0
pip install dist/covigator*.whl
python -m unittest discover covigator.tests.unit_tests
coverage run -m unittest discover covigator.tests.unit_tests

- name: Upload Coverage to Codecov
uses: codecov/codecov-action@v3
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,14 @@

[![PyPI version](https://badge.fury.io/py/covigator.svg)](https://badge.fury.io/py/covigator)
[![Run unit tests](https://github.com/TRON-Bioinformatics/covigator/actions/workflows/unit_tests.yml/badge.svg?branch=main)](https://github.com/TRON-Bioinformatics/covigator/actions/workflows/unit_tests.yml)
[![codecov](https://codecov.io/gh/TRON-Bioinformatics/covigator/branch/main/graph/badge.svg?token=J5Q8UV65PD)](https://codecov.io/gh/TRON-Bioinformatics/covigator)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/d6735b902b7b42e0a7cd423cebff69d2)](https://www.codacy.com/gh/TRON-Bioinformatics/covigator/dashboard?utm_source=github.com&utm_medium=referral&utm_content=TRON-Bioinformatics/covigator&utm_campaign=Badge_Grade)
[![Powered by Dash](https://img.shields.io/badge/powered%20by-Dash-orange.svg?style=flat&colorA=E1523D&colorB=007D8A)](https://dash.plotly.com/)
[![License](https://img.shields.io/badge/license-MIT-green)](https://opensource.org/licenses/MIT)
[![Documentation Status](https://readthedocs.org/projects/covigator/badge/?version=latest)](https://covigator.readthedocs.io/en/latest/?badge=latest)

![CoVigator art](covigator/dashboard/assets/wordcloud.png)

**CoVigator dashboard**: [https://covigator.tron-mainz.de](https://covigator.tron-mainz.de)

**CoVigator documentation**: [https://covigator.readthedocs.io/](https://covigator.readthedocs.io)
Expand Down Expand Up @@ -69,5 +73,5 @@ The project was kindly supported by
## How to cite

* Schrörs, B., Riesgo-Ferreiro, P., Sorn, P., Gudimella, R., Bukur, T., Rösler, T., Löwer, M., & Sahin, U. (2021).
Large-scale analysis of SARS-CoV-2 spike-glycoprotein mutants demonstrates the need for continuous screening of virus
isolates. PLOS ONE, 16(9), e0249254. [10.1371/journal.pone.0249254](https://doi.org/10.1371/journal.pone.0249254)
Large-scale analysis of SARS-CoV-2 spike-glycoprotein mutants demonstrates the need for continuous screening of virus
isolates. PLOS ONE, 16(9), e0249254. [10.1371/journal.pone.0249254](https://doi.org/10.1371/journal.pone.0249254)
Empty file added codecov.yml
Empty file.
4 changes: 2 additions & 2 deletions covigator/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
VERSION = "v1.1.0"
ANALYSIS_PIPELINE_VERSION = "v0.14.0"
VERSION = "v2.0.0"
ANALYSIS_PIPELINE_VERSION = "v0.15.0"

MISSENSE_VARIANT = "missense_variant"
SYNONYMOUS_VARIANT = "synonymous_variant"
Expand Down
27 changes: 15 additions & 12 deletions covigator/accessor/abstract_accessor.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from datetime import date, datetime
from datetime import date
from typing import Union
import requests
from covigator.misc.country_parser import CountryParser
Expand All @@ -7,13 +7,21 @@
from covigator.exceptions import CovigatorExcludedSampleTooEarlyDateException
from covigator.misc import backoff_retrier

NUMBER_RETRIES = 5
NUMBER_RETRIES = -1


class SampleCovid19:
pass


def _parse_abstract(value, _type):
try:
value = _type(value)
except (ValueError, TypeError):
value = None
return value


class AbstractAccessor:

def __init__(self):
Expand All @@ -33,15 +41,10 @@ def _parse_country(self, sample: Union[SampleEna, SampleCovid19]):
sample.continent_alpha_2 = parsed_country.continent_alpha_2
sample.continent = parsed_country.continent

def _parse_dates(self, sample: Union[SampleEna, SampleCovid19]):
sample.collection_date = self._parse_abstract(sample.collection_date, date.fromisoformat)
sample.first_created = self._parse_abstract(sample.first_created, date.fromisoformat)

@staticmethod
def _parse_dates(sample: Union[SampleEna, SampleCovid19]):
sample.collection_date = _parse_abstract(sample.collection_date, date.fromisoformat)
sample.first_created = _parse_abstract(sample.first_created, date.fromisoformat)
if sample.collection_date is not None and sample.collection_date < MINIMUM_DATE:
raise CovigatorExcludedSampleTooEarlyDateException

def _parse_abstract(self, value, type):
try:
value = type(value)
except (ValueError, TypeError):
value = None
return value
Loading