Impact App is a work-in-progress tool developed by the Social and Media Impact Evaluation group of the Barcelona Supercomputing Center with the goal of assist organizations and scholars to measure the scientific and social impact of their research projects.
By scientific impact, we understand the intellectual contribution to a person’s field of study within academia [1]. Within the various methods proposed in the literature to measure the scientific impact of research (e.g., [2], [3], [4]), we decided to employ the method called normalized impact of the scientific production, which is the methodology employed by the government of Spain to understand the impact of publications authored by scholars working in the country's research institutions [5].
| Name | Description |
|---|---|
| Number of publications (NP) | Number of papers published by the organization (or researcher). Papers are considered normal articles published in journals and in conference proceedings |
| Number of citations (C) | Total citations received by NP |
| Mean of citations per paper (CPP) | Average number of citations per publications |
| Percentage of not-cited papers (%ncP) | Proportion of papers that did not receive citations |
| Percentage of self-citation (%sc) | Proportion of citations in which the list of authors of the citing paper match the list of authors of the cited paper |
To understand whether a certain number of citations or a specific value of citations per publication is low or high, we compare them against some reference value. The following indicators are the international reference values used to normalize the assessment of scientific impact.
| Name | Description |
|---|---|
| Mean of field citation score (FCS) | Average of citations of all papers published in all journals of the field in which the institute has been active |
| CPP/FCS | Whether or not the average citations per publications is above (or below) the international average in the field |
The mean of citations by field and year can be obtained from the annual report of Elsevier.
- Select the range of years of evaluation. There should be at least four years between the current year and the final year of the evaluation period. The average "peak" in the number of citations happens in the third to a fourth-year after publication. A five-year period is then appropriate for impact evaluation [2];
- Compute the indicators for each year in the selected range;
- Evaluate how the institution performs within its field of activity.
Evaluate the scientific impact of a research institute between 2012 and 2015.
| Year | NP | C | CCP | %Pnc | %sc | FCS | CPP/FCS |
|---|---|---|---|---|---|---|---|
| 2012 | 70 | 450 | 6.2 | 3 | 7 | 3.1 | 2 |
| 2013 | 78 | 370 | 4.7 | 5 | 6 | 3.6 | 1.3 |
| 2014 | 81 | 407 | 5.0 | 7 | 8 | 3.5 | 1.4 |
| 2015 | 84 | 463 | 5.5 | 4 | 6 | 3.8 | 1.5 |
Currently, the primary sources of data are PubMed and PubMedCentral (PMC). The API of these sources are hit through Biopython—a PubMed API client.
Social impact is the social improvements achieved as a consequence of implementing the results of a particular research project or program. The computation of social impact is conducted by applying the Impact methodology proposed to monitor the impact of projects funded by the European Union [6].
- Connection to UN Sustainable Development Goals, EU2020 targets or other similar official targets;
- Percentage of improvement achieved concerning the starting situation;
- Transferability of the impact. The actions developed based on the project's findings have been transferred to other contexts besides the original one;
- Social impact published on scientific journals (with recognized impact), governmental or non-governmental official bodies;
- Sustainability. The impact achieved by the action developed based on the project’s findings has shown to be sustainable throughout time
| Organization | Project | SDG (UN 2030) [1] | % of Improvement achieved [2] | Description of achievement [3] | Sustainability [4] | Replicability [5] | Social impact report | Score-Impact [6] |
|---|---|---|---|---|---|---|---|---|
| Institution A | Project AA | No Poverty (SDG #1) | 5% | Reduce number of families in poverty situation | No | No | Scientific acticle | 4 |
| Project BB | Health | 21% | Reduce dengue contagion | Yes | Yes | Official report | 9 |
[1] Respond to at least one of the 2030 United Nations Social Development Goal (or other similar official social goals like EU2020)
[2] Percentage of the improvement achieved as a result of the actions taken during the programme execution
[3] Textual description of the achievement
[4] Evidences of the impact after the end of the project life span (Yes/No)
[5] Whether the programme has been implemented in more than one place (city, region, country) (Yes/No)
[6] The score of each impact is assigned according to the criteria that it fulfils:
- 10: The impact meets all the criteria, and has more than 30% of improvement
- 9: The impact meets all the criteria, and has more than 20% of improvement
- 8: The impact meets all the criteria, and has more than 10% of improvement
- 7: The impact meets all the criteria, and has some % of improvement (not available the specific %)
- 6: The impact responds to UN2030 strategy objectives, has achieved some % of improvement and meets at least 2 of the other criteria
- 5: The impact responds to UN2030 strategy objectives, has achieved some % of improvement and meets at least 1 of the other criteria
- 4: The impact responds to UN2030 strategy objectives and has achieved some % of improvement
- 3: The impact responds to other societal objectives, has achieved some % of improvement and meets at least 2 of the other criteria
- 2: The impact responds to other societal objectives, has achieved some % of improvement and meets at least 1 of the other criteria
- 1: The impact responds to other societal objectives and has achieved some % of improvement
Starting from a group of researcher names, ImpactApp queries the API of PubMed and PubMedCentral to collect information about the articles authored by the given researchers. With this data, the tool computes the scientific impact following the methodology mentioned before. The results of this computation are graphically displayed in dashboards, as shown in the next figures.
Initial dashboard showing an overview of the dataset used to compute the scientific impact of an institution
Dashboard outlining an overview of the scientific impact of the institution understudy
Dashboard reporting the results of applying the scientific impact methodology to a researcher of the institution under
study
A live demo of the application is available here
-
Run
git clone https://github.com/social-link-analytics-group-bsc/impact-app.git -
Get into the directory impact-app
-
Create a virtual environment
virtualenv env(instructions on how to install virtualenv on Ubuntu) -
Activate the virtual environment
source env/bin/activate -
Get into the directory backend
-
Execute
pip install -r requirements.txtto install dependencies. If an error occurs during the installation, it might be because some of these reasons: a) package python-dev is missing; b) package libmysqlclient-dev is missing c) the environment variables LC_ALL and/or LC_CTYPE are not defined or do not have a valid value -
Create a PostgreSQL database. Make sure your database collation is set to UTF-8
-
Rename the file backend/backend/settings.py.example as backend/backend/settings.py
-
Set the configuration parameters of the database in backend/backend/settings.py
DATABASES = {
...
'NAME': '',
'USER': '',
'PASSWORD': '',
'HOST': '',
'PORT': '',
...
}
-
Run
python manage.py migrateto set up the database schema -
Run
python manage.py createsuperuserto create an admin user -
Run the Django server
python manage.py runserver locahost:8000 -
Get into the backend directory and run
celery worker -A backend --loglevel=info. Celery is used to execute long-running tasks in background
ImpactApp uses pdftotext to extract text from pdf files. Pdftotext depends on
poppler a pdf rendering library that should be installed as part of the installation
process. This guidelines explains how to install poppler
in Ubuntu. To complete the installation of poppler some utilitarian functions of the library should be installed.
In Ubuntu poppler utils can be installed by executing sudo apt-get install poppler-utils.
Initially, the application was planned to be structured in two main modules: the frontend and the backend. In this way, we could separate the graphic interface from the business logic. The frontend could not be implemented and provisionally the UI was included "within the backend." Next, the structure of the backend is described.
...
├── backend <- Source code of the application
│ ├── dashboard <- Module that implements the dashboard
│ ├── data_collector <- Module used to collect information about scientific articles from academic repositories (scopus, pubmed)
| |── sci_impact <- Module that implements the scientific impact features of the application
| |── social_impact <- Module that implements the social impact features of the application
| |── static <- Directory with static resources (images, javascript and css files)
| |── template <- Files of the template used in the dashboard
...
Please use Github's issue tracker to report issues and suggestions.
Jorge Saldivar, Nataly Buslón, María José Rementería, and Salvador Capella