From 68d2f2530cb2718335e6a3b38f8721b438f07551 Mon Sep 17 00:00:00 2001 From: mattiagiupponi <51856725+mattiagiupponi@users.noreply.github.com> Date: Wed, 17 Feb 2021 09:29:28 +0100 Subject: [PATCH] [Fixes #6925] Thesauri improvements (#6951) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Thesaurus: Add model ThesaurusLabel, improve load_thesaurus managment command and add test coverage * Thesaurus: Add new migration file, stab thesaurus rdf file and managment command test * Thesaurus: Add copyright, file reformatted with black * Thesaurus: Increase test coverage for thesaurus load and class refactor * Resolves #6925: New Thesaurus form generated dynamically based on thesaurus avaiable, fix test on thesaurus load, add migration file for updated thesaurus schema * Resolves #6925: refactor thesaurus form, fix validation issue * Resolves #6925: fix metadata bug that prevent to save multiple key for each thesaurus * Bump urllib3 from 1.26.2 to 1.26.3 (#6908) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.2 to 1.26.3. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/1.26.3/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/1.26.2...1.26.3) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni * [Fixes #6880] Circle CI upload tests fail irregulary (#6881) * [Fixes #6880] Circle CI upload tests fail irregulary * CircleCI test fix: sometimes expires due to upload timeout in the test environment * - Avoid infinite loop on upload testing * Revert "CircleCI test fix: sometimes expires due to upload timeout in the test environment" This reverts commit 66139fdbf0b7510a9829a3e01254f41782fb7e1d. Co-authored-by: Alessio Fabiani Co-authored-by: afabiani * [Fixes #6914] Remove "add to basket" tool for documents and maps (#6915) * Added malnajdi as contributor * [Fixes #6910] meaningful filename for document download (#6911) * get meaningful document filenames on download * - Strip extension from document title before slugify it (e.g.: image.jpg instead of imagejpg.jpg) Co-authored-by: afabiani Co-authored-by: Alessio Fabiani * - CircleCI Upload Tests: trying to reduce more the risk of infinite loop on "wait_for_progress" * [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION (#6913) * [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION * [Fixes #6916] unit test for GEOSERVER_LOCATION * Bump django-cors-headers from 3.6.0 to 3.7.0 (#6901) Bumps [django-cors-headers](https://github.com/adamchainz/django-cors-headers) from 3.6.0 to 3.7.0. - [Release notes](https://github.com/adamchainz/django-cors-headers/releases) - [Changelog](https://github.com/adamchainz/django-cors-headers/blob/master/HISTORY.rst) - [Commits](https://github.com/adamchainz/django-cors-headers/compare/3.6.0...3.7.0) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump amqp from 5.0.3 to 5.0.5 (#6905) Bumps [amqp](https://github.com/celery/py-amqp) from 5.0.3 to 5.0.5. - [Release notes](https://github.com/celery/py-amqp/releases) - [Changelog](https://github.com/celery/py-amqp/blob/master/Changelog) - [Commits](https://github.com/celery/py-amqp/compare/v5.0.3...v5.0.5) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump pip from 21.0 to 21.0.1 (#6900) Bumps [pip](https://github.com/pypa/pip) from 21.0 to 21.0.1. - [Release notes](https://github.com/pypa/pip/releases) - [Changelog](https://github.com/pypa/pip/blob/master/NEWS.rst) - [Commits](https://github.com/pypa/pip/compare/21.0...21.0.1) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump coverage from 5.3.1 to 5.4 (#6903) Bumps [coverage](https://github.com/nedbat/coveragepy) from 5.3.1 to 5.4. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](https://github.com/nedbat/coveragepy/compare/coverage-5.3.1...coverage-5.4) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump pytest from 6.2.1 to 6.2.2 (#6907) Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.1 to 6.2.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pytest/compare/6.2.1...6.2.2) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump djangorestframework-gis from 0.16 to 0.17 (#6902) Bumps [djangorestframework-gis](https://github.com/openwisp/django-rest-framework-gis) from 0.16 to 0.17. - [Release notes](https://github.com/openwisp/django-rest-framework-gis/releases) - [Changelog](https://github.com/openwisp/django-rest-framework-gis/blob/master/CHANGES.rst) - [Commits](https://github.com/openwisp/django-rest-framework-gis/compare/v0.16.0...v0.17.0) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * - Algin setup.cfg to requirements.txt * [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has… (#6923) * [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has been uploaded * - Add REST APIs test suite to CircleCI * [Fixes #6918] Removal of QGIS support (#6919) * [Cleanup and Refactor] Remove QGIS server backend dependencies * [Cleanup and Refactor] Remove QGIS server backend dependencies * - Fix LGTM issues * Resolves #6925: add deprecation warning for old thesaurus settings * allow Basic authenticated requests in LOCKDOWN mode * Resolves #6925: change metadata template to let it reads the new thesaurus information as template tags, add tests for new template tags * Resolves #6925: add thesaurus in admin page * fix to avoid circular import * flake8 check fix * Resolves #6925: reformatting code for match Flake8 requirements * added tests * Resolves #6925: fix default values for all required in Thesaurus Model * Resolves #6925: add test for ThesaurusAvailableModelForm * Resolves #6925: minor refactor in ThesaurusForm initialization * Resolves #6925: thesaurus initial data * Resolves #6925: fix flake8 style errors * Refers #6925: add missing migration file * Resolves #6925: fix thesaurus show method by adding facet as filter * Resolves #6925: fix flake8 errors * Resolves #6925: fix lgmt bot issue * Resolves #6925: add missing prefix for thesaurus * Fixes #6925: remote RNDT information from the tests * Fixes #6925: italian wordings updated to english Co-authored-by: Giovanni Allegri Co-authored-by: allyoucanmap Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni Co-authored-by: Alessio Fabiani Co-authored-by: afabiani Co-authored-by: Florian Hoedt Co-authored-by: Mohammed Y. Alnajdi Co-authored-by: biegan --- geonode/base/fixtures/initial_data.json | 266 +++++++++++++++++- geonode/base/forms.py | 82 ++++-- .../management/commands/load_thesaurus.py | 47 +++- .../migrations/0050_auto_20210201_1008.py | 33 +++ .../migrations/0051_auto_20210202_1656.py | 28 ++ .../migrations/0052_auto_20210205_1631.py | 23 ++ geonode/base/models.py | 26 ++ geonode/base/templatetags/thesaurus.py | 26 ++ geonode/base/tests.py | 80 +++++- geonode/base/urls.py | 7 +- geonode/base/views.py | 18 ++ .../templates/catalogue/full_metadata.xml | 35 +++ geonode/context_processors.py | 12 +- geonode/layers/admin.py | 7 +- .../templates/layers/layer_metadata.html | 5 +- geonode/layers/templates/layouts/panels.html | 18 +- geonode/layers/views.py | 76 +++-- .../migrations/0030_auto_20210201_0953.py | 18 ++ .../migrations/0031_merge_20210205_0824.py | 14 + geonode/tests/data/thesaurus.rdf | 22 ++ geonode/tests/test_load_thesaurus.py | 147 ++++++++++ 21 files changed, 920 insertions(+), 70 deletions(-) create mode 100644 geonode/base/migrations/0050_auto_20210201_1008.py create mode 100644 geonode/base/migrations/0051_auto_20210202_1656.py create mode 100644 geonode/base/migrations/0052_auto_20210205_1631.py create mode 100644 geonode/base/templatetags/thesaurus.py create mode 100644 geonode/people/migrations/0030_auto_20210201_0953.py create mode 100644 geonode/people/migrations/0031_merge_20210205_0824.py create mode 100644 geonode/tests/data/thesaurus.rdf create mode 100644 geonode/tests/test_load_thesaurus.py diff --git a/geonode/base/fixtures/initial_data.json b/geonode/base/fixtures/initial_data.json index 74c433f5de3..564bf484af7 100644 --- a/geonode/base/fixtures/initial_data.json +++ b/geonode/base/fixtures/initial_data.json @@ -4852,5 +4852,269 @@ "fields": { "name": "TOPBAR_MENU" } - } + }, + { + "model": "base.thesaurus", + "pk": 1, + "fields": { + "identifier": "inspire_themes", + "title": "GEMET - INSPIRE themes, version 1.0", + "date": "2018-05-23T10:25:56", + "description": "GEMET - INSPIRE themes, version 1.0", + "slug": "", + "about": "http://inspire.ec.europa.eu/theme", + "card_min": 1, + "card_max": 0, + "facet": true + } + }, + { + "model": "base.thesaurus", + "pk": 4, + "fields": { + "identifier": "abc-all1", + "title": "Register of data of general interest", + "date": "2018-06-25", + "description": "Register of data of general interest", + "slug": "", + "about": "https://registry.geodati.gov.it/abc-all1", + "card_min": 0, + "card_max": 1, + "facet": true + } + }, + { + "model": "base.thesaurus", + "pk": 6, + "fields": { + "identifier": "limitation_on_public_access", + "title": "Limitations on public access", + "date": "2020-10-22T13:21:33", + "description": "Limitations on public access", + "slug": "", + "about": "http://inspire.ec.europa.eu/metadata-codelist/LimitationsOnPublicAccess", + "card_min": 1, + "card_max": 1, + "facet": true + } + }, + { + "model": "base.thesaurus", + "pk": 7, + "fields": { + "identifier": "limitation_on_public_access-2", + "title": "Conditions Applying To Access and Use", + "date": "2020-10-30T16:58:34", + "description": "Conditions Applying To Access and Use", + "slug": "", + "about": "http://inspire.ec.europa.eu/metadata-codelist/ConditionsApplyingToAccessAndUse", + "card_min": 0, + "card_max": -1, + "facet": true + } + }, + { + "model": "base.thesaurus", + "pk": 8, + "fields": { + "identifier": "foo_name", + "title": "Mocked Title", + "date": "2018-05-23T10:25:56", + "description": "Mocked Title", + "slug": "abc", + "about": "http://inspire.ec.europa.eu/theme", + "card_min": 1, + "card_max": -1, + "facet": true + } + }, + { + "model": "base.thesauruskeyword", + "pk": 1, + "fields": { + "about": "http://inspire.ec.europa.eu/theme/ad", + "alt_label": "ad", + "thesaurus": 1 + } + }, + { + "model": "base.thesauruskeyword", + "pk": 2, + "fields": { + "about": "http://inspire.ec.europa.eu/theme/au", + "alt_label": "au", + "thesaurus": 1 + } + }, + { + "model": "base.thesauruskeyword", + "pk": 3, + "fields": { + "about": "http://inspire.ec.europa.eu/theme/rs", + "alt_label": "rs", + "thesaurus": 1 + } + }, + { + "model": "base.thesauruskeyword", + "pk": 35, + "fields": { + "about": "http://inspire.ec.europa.eu/metadata-codelist/LimitationsOnPublicAccess/INSPIRE_Directive_Article13_1a", + "alt_label": "public access limited according to Article 13(1)(a) of the INSPIRE Directive [ITA]", + "thesaurus": 6 + } + }, + { + "model": "base.thesauruskeyword", + "pk": 44, + "fields": { + "about": "http://inspire.ec.europa.eu/metadata-codelist/ConditionsApplyingToAccessAndUse/noConditionsApply", + "alt_label": "No condition applicable", + "thesaurus": 7 + } + }, + { + "model": "base.thesauruskeyword", + "pk": 45, + "fields": { + "about": "http://inspire.ec.europa.eu/metadata-codelist/ConditionsApplyingToAccessAndUse/conditionsUnknown", + "alt_label": "Unknown Conditions", + "thesaurus": 7 + } + }, + { + "model": "base.thesauruskeyword", + "pk": 100, + "fields": { + "about": "http://test.com", + "alt_label": "Protected sites", + "thesaurus": 8 + } + }, + { + "model": "base.thesauruskeywordlabel", + "pk": 1, + "fields": { + "lang": "en", + "label": "Addresses", + "keyword": 1 + } + }, + { + "model": "base.thesauruskeywordlabel", + "pk": 2, + "fields": { + "lang": "de", + "label": "Adressen", + "keyword": 1 + } + }, + { + "model": "base.thesauruskeywordlabel", + "pk": 3, + "fields": { + "lang": "en", + "label": "Administrative units", + "keyword": 2 + } + }, + { + "model": "base.thesauruskeywordlabel", + "pk": 4, + "fields": { + "lang": "en", + "label": "Coordinate reference systems", + "keyword": 3 + } + }, + { + "model": "base.thesauruskeywordlabel", + "pk": 36, + "fields": { + "lang": "en", + "label": "public access limited according to Article 13(1)(a) of the INSPIRE Directive", + "keyword": 35 + } + }, + { + "model": "base.thesauruskeywordlabel", + "pk": 54, + "fields": { + "lang": "en", + "label": "no conditions to access and use", + "keyword": 44 + } + }, + { + "model": "base.thesauruskeywordlabel", + "pk": 56, + "fields": { + "lang": "en", + "label": "conditions to access and use unknown", + "keyword": 45 + } + }, + { + "model": "base.thesauruskeywordlabel", + "pk": 102, + "fields": { + "lang": "en", + "label": "Protected sites", + "keyword": 100 + } + }, + { + "model": "base.thesauruslabel", + "pk": 1, + "fields": { + "lang": "en", + "label": "Addresses", + "thesaurus": 1 + } + }, + { + "model": "base.thesauruslabel", + "pk": 2, + "fields": { + "lang": "de", + "label": "Adressen", + "thesaurus": 1 + } + }, + { + "model": "base.thesauruslabel", + "pk": 4, + "fields": { + "lang": "en", + "label": "Register of the reference data sets", + "thesaurus": 4 + } + }, + { + "model": "base.thesauruslabel", + "pk": 5, + "fields": { + "lang": "en", + "label": "Limitations on public access", + "thesaurus": 6 + } + }, + { + "model": "base.thesauruslabel", + "pk": 7, + "fields": { + "lang": "en", + "label": "Conditions Applying To Access and Use", + "thesaurus": 7 + } + }, + { + "model": "base.thesauruslabel", + "pk": 9, + "fields": { + "lang": "en", + "label": "Register of the reference data sets", + "thesaurus": 8 + } + } ] diff --git a/geonode/base/forms.py b/geonode/base/forms.py index f2817ad3df7..f695e82604e 100644 --- a/geonode/base/forms.py +++ b/geonode/base/forms.py @@ -20,43 +20,37 @@ import re import html import logging - -from tinymce.widgets import TinyMCE - -from .fields import MultiThesauriField - +from django.db.models.query import QuerySet +from bootstrap3_datetime.widgets import DateTimePicker from dal import autocomplete -from taggit.forms import TagField - from django import forms from django.conf import settings from django.contrib.auth import get_user_model from django.contrib.auth.models import Group from django.core import validators from django.db.models import Prefetch, Q -from django.forms import models -from django.forms import ModelForm +from django.forms import ModelForm, models from django.forms.fields import ChoiceField from django.forms.utils import flatatt +from django.utils.encoding import force_text from django.utils.html import format_html from django.utils.safestring import mark_safe from django.utils.translation import ugettext as _ - -from django.utils.encoding import ( - force_text, -) - -from bootstrap3_datetime.widgets import DateTimePicker from modeltranslation.forms import TranslationModelForm +from taggit.forms import TagField +from tinymce.widgets import TinyMCE -from geonode.base.models import HierarchicalKeyword, TopicCategory, Region, License, CuratedThumbnail, \ - ResourceBase -from geonode.base.models import ThesaurusKeyword, ThesaurusKeywordLabel -from geonode.documents.models import Document from geonode.base.enumerations import ALL_LANGUAGES +from geonode.base.models import (CuratedThumbnail, HierarchicalKeyword, + License, Region, ResourceBase, Thesaurus, + ThesaurusKeyword, ThesaurusKeywordLabel, ThesaurusLabel, + TopicCategory) from geonode.base.widgets import TaggitSelect2Custom +from geonode.documents.models import Document from geonode.layers.models import Layer +from .fields import MultiThesauriField + logger = logging.getLogger(__name__) @@ -320,6 +314,56 @@ class Meta: ) +class ThesaurusAvailableForm(forms.Form): + def __init__(self, *args, **kwargs): + super(ThesaurusAvailableForm, self).__init__(*args, **kwargs) + lang = settings.THESAURUS_DEFAULT_LANG if hasattr(settings, "THESAURUS_DEFAULT_LANG") else "en" + for item in Thesaurus.objects.all(): + tname = ThesaurusLabel.objects.values_list("label", flat=True).filter(id=item.id).filter(lang=lang) + if item.card_max == 0: + continue + elif item.card_max == 1 and item.card_min == 0: + self.fields[f"{item.id}"] = self._define_choicefield(item, False, tname, lang) + elif item.card_max == 1 and item.card_min == 1: + self.fields[f"{item.id}"] = self._define_choicefield(item, True, tname, lang) + elif item.card_max == -1 and item.card_min == 0: + self.fields[f"{item.id}"] = self._define_multifield(item, False, tname, lang) + elif item.card_max == -1 and item.card_min == 1: + self.fields[f"{item.id}"] = self._define_multifield(item, True, tname, lang) + + def cleanx(self, x): + cleaned_values = [] + for key, value in x.items(): + if isinstance(value, QuerySet): + for y in value: + cleaned_values.append(y.id) + elif value: + cleaned_values.append(value.id) + return ThesaurusKeyword.objects.filter(id__in=cleaned_values) + + @staticmethod + def _define_multifield(item, required, tname, lang): + return MultiThesauriField( + ThesaurusKeyword.objects.prefetch_related( + Prefetch( + "keyword", + queryset=ThesaurusKeywordLabel.objects.filter(keyword__thesaurus_id=item.id).filter(lang=lang), + ) + ), + widget=autocomplete.ModelSelect2Multiple(url=f"/base/thesaurus_available/?sysid={item.id}&lang={lang}"), + label=_(f"{tname[0] if len(tname) > 0 else item.title}"), + required=required, + ) + + @staticmethod + def _define_choicefield(item, required, tname, lang): + return models.ModelChoiceField( + label=f"{tname[0] if len(tname) > 0 else item.title}", + required=required, + queryset=ThesaurusKeywordLabel.objects.filter(keyword__thesaurus_id=item.id).filter(lang=lang), + ) + + class ResourceBaseDateTimePicker(DateTimePicker): def build_attrs(self, base_attrs=None, extra_attrs=None, **kwargs): diff --git a/geonode/base/management/commands/load_thesaurus.py b/geonode/base/management/commands/load_thesaurus.py index b95f195cd04..01af25d3f5c 100644 --- a/geonode/base/management/commands/load_thesaurus.py +++ b/geonode/base/management/commands/load_thesaurus.py @@ -18,11 +18,13 @@ # ######################################################################### +from typing import List from defusedxml import lxml as dlxml +from django.conf import settings from django.core.management.base import BaseCommand, CommandError -from geonode.base.models import Thesaurus, ThesaurusKeyword, ThesaurusKeywordLabel +from geonode.base.models import Thesaurus, ThesaurusKeyword, ThesaurusKeywordLabel, ThesaurusLabel class Command(BaseCommand): @@ -90,25 +92,46 @@ def load_thesaurus(self, input_file, name, store): if not scheme: raise CommandError("ConceptScheme not found in file") - title = scheme.find('dc:title', ns).text - descr = scheme.find('dc:description', ns).text if scheme.find('dc:description', ns) else title + titles = scheme.findall('dc:title', ns) + + default_lang = getattr(settings, 'THESAURUS_DEFAULT_LANG', None) + available_lang = get_all_lang_available_with_title(titles, LANG_ATTRIB) + thesaurus_title = determinate_value(available_lang, default_lang) + + descr = scheme.find('dc:description', ns).text if scheme.find('dc:description', ns) else thesaurus_title date_issued = scheme.find('dcterms:issued', ns).text + about = scheme.attrib.get(ABOUT_ATTRIB) - print('Thesaurus "{}" issued on {}'.format(title, date_issued)) + print('Thesaurus "{}" issued on {}'.format(thesaurus_title, date_issued)) thesaurus = Thesaurus() thesaurus.identifier = name - thesaurus.title = title + thesaurus.title = thesaurus_title thesaurus.description = descr + thesaurus.about = about thesaurus.date = date_issued if store: thesaurus.save() + for lang in available_lang: + if lang[0] is not None: + thesaurus_label = ThesaurusLabel() + thesaurus_label.lang = lang[0] + thesaurus_label.label = lang[1] + thesaurus_label.thesaurus = thesaurus + thesaurus_label.save() + for concept in root.findall('skos:Concept', ns): about = concept.attrib.get(ABOUT_ATTRIB) - alt_label = concept.find('skos:altLabel', ns).text + alt_label = concept.find('skos:altLabel', ns) + if alt_label is not None: + alt_label = alt_label.text + else: + concepts = concept.findall('skos:prefLabel', ns) + available_lang = get_all_lang_available_with_title(concepts, LANG_ATTRIB) + alt_label = determinate_value(available_lang, default_lang) print('Concept {} ({})'.format(alt_label, about)) @@ -157,3 +180,15 @@ def create_fake_thesaurus(self, name): tkl.lang = l tkl.label = keyword + "_l_" + l + "_t_" + name tkl.save() + +def get_all_lang_available_with_title(items: List, LANG_ATTRIB: str): + return [(item.attrib.get(LANG_ATTRIB), item.text) for item in items] + +def determinate_value(available_lang: List, default_lang: str): + sorted_lang = sorted(available_lang, key= lambda lang: '' if lang[0] is None else lang[0]) + for item in sorted_lang: + if item[0] is None: + return item[1] + elif item[0] == default_lang: + return item[1] + return available_lang[0][1] diff --git a/geonode/base/migrations/0050_auto_20210201_1008.py b/geonode/base/migrations/0050_auto_20210201_1008.py new file mode 100644 index 00000000000..f51b59453a8 --- /dev/null +++ b/geonode/base/migrations/0050_auto_20210201_1008.py @@ -0,0 +1,33 @@ +# Generated by Django 2.2.16 on 2021-02-01 10:08 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('base', '0049_resourcebase_resource_type'), + ] + + operations = [ + migrations.AddField( + model_name='thesaurus', + name='about', + field=models.CharField(blank=True, max_length=255, null=True), + ), + migrations.CreateModel( + name='ThesaurusLabel', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('lang', models.CharField(max_length=3)), + ('label', models.CharField(max_length=255)), + ('thesaurus', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='rel_thesaurus', to='base.Thesaurus')), + ], + options={ + 'verbose_name_plural': 'Thesaurus Labels', + 'ordering': ('lang',), + 'unique_together': {('thesaurus', 'lang')}, + }, + ), + ] diff --git a/geonode/base/migrations/0051_auto_20210202_1656.py b/geonode/base/migrations/0051_auto_20210202_1656.py new file mode 100644 index 00000000000..711e15e0cbe --- /dev/null +++ b/geonode/base/migrations/0051_auto_20210202_1656.py @@ -0,0 +1,28 @@ +# Generated by Django 2.2.16 on 2021-02-02 16:56 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('base', '0050_auto_20210201_1008'), + ] + + operations = [ + migrations.AddField( + model_name='thesaurus', + name='card_max', + field=models.IntegerField(default=True), + ), + migrations.AddField( + model_name='thesaurus', + name='card_min', + field=models.IntegerField(default=True), + ), + migrations.AddField( + model_name='thesaurus', + name='facet', + field=models.BooleanField(default=True), + ), + ] diff --git a/geonode/base/migrations/0052_auto_20210205_1631.py b/geonode/base/migrations/0052_auto_20210205_1631.py new file mode 100644 index 00000000000..2b7c17ede41 --- /dev/null +++ b/geonode/base/migrations/0052_auto_20210205_1631.py @@ -0,0 +1,23 @@ +# Generated by Django 2.2.16 on 2021-02-05 16:31 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('base', '0051_auto_20210202_1656'), + ] + + operations = [ + migrations.AlterField( + model_name='thesaurus', + name='card_max', + field=models.IntegerField(default=-1), + ), + migrations.AlterField( + model_name='thesaurus', + name='card_min', + field=models.IntegerField(default=0), + ), + ] diff --git a/geonode/base/models.py b/geonode/base/models.py index d19d63e78a0..084a3380d3e 100644 --- a/geonode/base/models.py +++ b/geonode/base/models.py @@ -471,6 +471,12 @@ class Thesaurus(models.Model): slug = models.CharField(max_length=64, default='') + about = models.CharField(max_length=255, null=True, blank=True) + + card_min = models.IntegerField(default=0) + card_max = models.IntegerField(default=-1) + facet = models.BooleanField(default=True) + def __str__(self): return "{0}".format(self.identifier) @@ -529,6 +535,26 @@ class Meta: unique_together = (("thesaurus", "alt_label"),) +class ThesaurusLabel(models.Model): + """ + Contains localized version of the thesaurus title + """ + # read from the RDF file + lang = models.CharField(max_length=3) + # read from the RDF file + label = models.CharField(max_length=255) + + thesaurus = models.ForeignKey('Thesaurus', related_name='rel_thesaurus', on_delete=models.CASCADE) + + def __str__(self): + return "{0}".format(self.label) + + class Meta: + ordering = ("lang",) + verbose_name_plural = 'Thesaurus Labels' + unique_together = (("thesaurus", "lang"),) + + class ResourceBaseManager(PolymorphicManager): def admin_contact(self): # this assumes there is at least one superuser diff --git a/geonode/base/templatetags/thesaurus.py b/geonode/base/templatetags/thesaurus.py new file mode 100644 index 00000000000..d7d6cf6577f --- /dev/null +++ b/geonode/base/templatetags/thesaurus.py @@ -0,0 +1,26 @@ +from django import template +from geonode.base.models import Thesaurus, ThesaurusKeywordLabel +from django.conf import settings + +register = template.Library() + + +@register.filter +def get_unique_thesaurus_set(thesaurus_from_keyword): + return set(thesaurus_from_keyword.values_list("thesaurus", flat=True)) + + +@register.filter +def get_keyword_label(keyword): + lang = "en" if not hasattr(settings, "THESAURUS_DEFAULT_LANG") else settings.THESAURUS_DEFAULT_LANG + return ThesaurusKeywordLabel.objects.filter(keyword=keyword).filter(lang=lang).first().label + + +@register.filter +def get_thesaurus_title(thesaurus_id): + return Thesaurus.objects.get(id=thesaurus_id).title + + +@register.filter +def get_thesaurus_date(thesaurus_id): + return Thesaurus.objects.get(id=thesaurus_id).date diff --git a/geonode/base/tests.py b/geonode/base/tests.py index bdb307acfd4..2889e851928 100644 --- a/geonode/base/tests.py +++ b/geonode/base/tests.py @@ -40,7 +40,10 @@ Menu, MenuItem, Configuration, - TopicCategory) + TopicCategory, + Thesaurus, + ThesaurusKeyword +) from django.conf import settings from django.template import Template, Context from django.contrib.auth import get_user_model @@ -51,6 +54,12 @@ from geonode.base.middleware import ReadOnlyMiddleware, MaintenanceMiddleware from geonode.base.models import CuratedThumbnail from geonode.base.templatetags.base_tags import get_visibile_resources +from geonode.base.templatetags.thesaurus import ( + get_unique_thesaurus_set, + get_keyword_label, + get_thesaurus_title, + get_thesaurus_date, +) from geonode.base.templatetags.user_messages import show_notification from geonode import geoserver from geonode.decorators import on_ogc_backend @@ -58,6 +67,8 @@ from django.core.files import File from django.core.management import call_command from django.core.management.base import CommandError +from geonode.base.forms import ThesaurusAvailableForm + test_image = Image.new('RGBA', size=(50, 50), color=(155, 0, 0)) @@ -879,3 +890,70 @@ def test_complex_tags_in_attribute(self): r = ResourceBase() filtered_value = r._remove_html_tags(tagged_value) self.assertEqual(filtered_value, attribute_target_value) + + +class TestTagThesaurus(TestCase): + # loading test thesausurs + @classmethod + def setUpTestData(cls): + from django.core import management + from os.path import dirname, abspath + + management.call_command( + "load_thesaurus", + file=f"{dirname(dirname(abspath(__file__)))}/tests/data/thesaurus.rdf", + name="foo_name", + stdout="out", + ) + + def setUp(self): + self.sut = Thesaurus( + identifier="foo_name", + title="Mocked Title", + date="2018-05-23T10:25:56", + description="Mocked Title", + slug="", + about="http://inspire.ec.europa.eu/theme", + ) + self.tkeywords = ThesaurusKeyword.objects.all() + + def test_get_unique_thesaurus_list(self): + tid = self.__get_last_thesaurus().id + actual = get_unique_thesaurus_set(self.tkeywords) + self.assertSetEqual({tid}, actual) + + @patch.dict("os.environ", {"THESAURUS_DEFAULT_LANG": "en"}) + def test_get_keyword_label(self): + actual = get_keyword_label(self.tkeywords[0]) + self.assertEqual("Addresses", actual) + + def test_get_thesaurus_title(self): + tid = self.__get_last_thesaurus().id + actual = get_thesaurus_title(tid) + self.assertEqual(self.sut.title, actual) + + def test_get_thesaurus_date(self): + tid = self.__get_last_thesaurus().id + actual = get_thesaurus_date(tid) + self.assertEqual(self.sut.date, actual) + + @staticmethod + def __get_last_thesaurus(): + return Thesaurus.objects.all().order_by("-id")[0] + + +class TestThesaurusAvailableForm(GeoNodeBaseTestSupport): + def setUp(self): + self.sut = ThesaurusAvailableForm + + def test_form_is_invalid_if_required_fields_are_missing(self): + actual = self.sut(data={}) + self.assertFalse(actual.is_valid()) + + def test_form_is_invalid_if_fileds_send_unexpected_values(self): + actual = self.sut(data={"8": [1, 2], "6": 1234}) + self.assertFalse(actual.is_valid()) + + def test_form_is_valid_if_fileds_send_expected_values(self): + actual = self.sut(data={"8": [1, 2], "6": 36}) + self.assertTrue(actual.is_valid()) diff --git a/geonode/base/urls.py b/geonode/base/urls.py index f433e4549fc..3a7ade5251d 100644 --- a/geonode/base/urls.py +++ b/geonode/base/urls.py @@ -21,7 +21,7 @@ from .views import ( ResourceBaseAutocomplete, RegionAutocomplete, - HierarchicalKeywordAutocomplete, ThesaurusKeywordLabelAutocomplete, OwnerRightsRequestView) + HierarchicalKeywordAutocomplete, ThesaurusKeywordLabelAutocomplete, OwnerRightsRequestView, ThesaurusAvailable) urlpatterns = [ @@ -43,6 +43,11 @@ name='autocomplete_hierachical_keyword', ), + url( + r'^thesaurus_available', + ThesaurusAvailable.as_view(), + name='thesaurus_available', + ), url( r'^thesaurus_autocomplete/$', ThesaurusKeywordLabelAutocomplete.as_view(), diff --git a/geonode/base/views.py b/geonode/base/views.py index c54e464c4ad..c5f7f4b1630 100644 --- a/geonode/base/views.py +++ b/geonode/base/views.py @@ -319,6 +319,24 @@ def get_results(self, context): ] +class ThesaurusAvailable(autocomplete.Select2QuerySetView): + def get_queryset(self): + tid = self.request.GET.get("sysid") + lang = self.request.GET.get("lang") + + qs = ThesaurusKeywordLabel.objects.filter(keyword__thesaurus__id=tid).filter(lang=lang).order_by('id') + return qs + + def get_results(self, context): + return [ + { + 'id': self.get_result_value(result.keyword), + 'text': self.get_result_label(result), + 'selected_text': self.get_selected_result_label(result), + } for result in context['object_list'] + ] + + class OwnerRightsRequestView(LoginRequiredMixin, FormView): template_name = 'owner_rights_request.html' form_class = OwnerRightsRequestForm diff --git a/geonode/catalogue/templates/catalogue/full_metadata.xml b/geonode/catalogue/templates/catalogue/full_metadata.xml index 485a815f269..60083b591e6 100644 --- a/geonode/catalogue/templates/catalogue/full_metadata.xml +++ b/geonode/catalogue/templates/catalogue/full_metadata.xml @@ -1,3 +1,4 @@ +{% load thesaurus %} {{layer.uuid}} @@ -350,6 +351,40 @@ {% endif %} + {% if layer.tkeywords %} + {% for thesaurus_id in layer.tkeywords|get_unique_thesaurus_set %} + + + {% for keyword in layer.tkeywords.all %} + {% if keyword.thesaurus.id == thesaurus_id %} + + {{keyword|get_keyword_label}} + + {% endif %} + {% endfor %} + + + + {{thesaurus_id|get_thesaurus_title}} + + + + + {{thesaurus_id|get_thesaurus_date}} + + + pubblicazione + + + + + + + + {% endfor %} + {% endif %} {% for region in layer.regions.all %} diff --git a/geonode/context_processors.py b/geonode/context_processors.py index 0ea8255eb90..add7156795f 100644 --- a/geonode/context_processors.py +++ b/geonode/context_processors.py @@ -18,19 +18,25 @@ # ######################################################################### +import warnings from django.conf import settings from geonode import get_version from geonode.catalogue import default_catalogue_backend from django.contrib.sites.models import Site from geonode.notifications_helper import has_notifications -from geonode.base.models import Configuration +from geonode.base.models import Configuration, Thesaurus def resource_urls(request): """Global values to pass to templates""" site = Site.objects.get_current() - + thesaurus = Thesaurus.objects.filter(facet=True).all() + if hasattr(settings, 'THESAURUS'): + warnings.warn( + 'Thesaurus settings is going to be' + 'deprecated in the future versions, please move the settings to ' + 'the new configuration ', FutureWarning) defaults = dict( STATIC_URL=settings.STATIC_URL, CATALOGUE_BASE_URL=default_catalogue_backend()['URL'], @@ -170,7 +176,7 @@ def resource_urls(request): False ), THESAURI_FILTERS=[t['name'] for t in [settings.THESAURUS, ] if - t.get('filter')] if hasattr(settings, 'THESAURUS') else None, + t.get('filter')] if hasattr(settings, 'THESAURUS') else [t.identifier for t in thesaurus], MAP_CLIENT_USE_CROSS_ORIGIN_CREDENTIALS=getattr( settings, 'MAP_CLIENT_USE_CROSS_ORIGIN_CREDENTIALS', False ), diff --git a/geonode/layers/admin.py b/geonode/layers/admin.py index 4f7c4bc0e33..10a65bca294 100644 --- a/geonode/layers/admin.py +++ b/geonode/layers/admin.py @@ -29,7 +29,7 @@ from geonode.layers.models import LayerFile, UploadSession from geonode.base.fields import MultiThesauriField -from geonode.base.models import ThesaurusKeyword, ThesaurusKeywordLabel +from geonode.base.models import ThesaurusKeyword, ThesaurusKeywordLabel, Thesaurus from dal import autocomplete @@ -98,6 +98,10 @@ class AttributeAdmin(admin.ModelAdmin): search_fields = ('attribute', 'attribute_label',) +class ThesaurusAdmin(admin.ModelAdmin): + model = Thesaurus + + class StyleAdmin(admin.ModelAdmin): model = Style list_display_links = ('sld_title',) @@ -120,3 +124,4 @@ class UploadSessionAdmin(admin.ModelAdmin): admin.site.register(Attribute, AttributeAdmin) admin.site.register(Style, StyleAdmin) admin.site.register(UploadSession, UploadSessionAdmin) +admin.site.register(Thesaurus, ThesaurusAdmin) diff --git a/geonode/layers/templates/layers/layer_metadata.html b/geonode/layers/templates/layers/layer_metadata.html index ee9d784c07a..ff8237cb01c 100644 --- a/geonode/layers/templates/layers/layer_metadata.html +++ b/geonode/layers/templates/layers/layer_metadata.html @@ -42,7 +42,7 @@

{% trans "Metadata" %} {% blocktrans with layer.titl Some of your original metadata may have been lost.{% endblocktrans %}

{% endif %} - {% if layer_form.errors or attribute_form.errors or category_form.errors or author_form.errors or poc.errors %} + {% if layer_form.errors or attribute_form.errors or category_form.errors or author_form.errors or poc.errors or tkeywords_form.errors %}

{% blocktrans %}Error updating metadata. Please check the following fields: {% endblocktrans %}

    {% if author_form.errors %} @@ -69,6 +69,9 @@

    {% trans "Metadata" %} {% blocktrans with layer.titl {% if category_form.errors %}
  • {{ category_form.errors.as_ul }}
    • {% endif %} + {% if tkeywords_form.errors %} +
  • {{ tkeywords_form.errors.as_ul }}
    • + {% endif %}

{% endif %} diff --git a/geonode/layers/templates/layouts/panels.html b/geonode/layers/templates/layouts/panels.html index 52a2fbc650a..c871a1391de 100644 --- a/geonode/layers/templates/layouts/panels.html +++ b/geonode/layers/templates/layouts/panels.html @@ -302,15 +302,6 @@
-
- - {{ layer_form.keywords }} -
- {% if THESAURI_FILTERS %} -
- {{ tkeywords_form }} -
- {% endif %}
@@ -362,6 +353,15 @@ {% endfor %}
+
+ + {{ layer_form.keywords }} +
+ {% if THESAURI_FILTERS %} +
+ {{tkeywords_form.as_p}} +
+ {% endif %}
diff --git a/geonode/layers/views.py b/geonode/layers/views.py index e4153d6e66c..1f09e09b77e 100644 --- a/geonode/layers/views.py +++ b/geonode/layers/views.py @@ -25,6 +25,7 @@ import base64 import traceback from types import TracebackType +import warnings import decimal import pickle from django.db.models import Q @@ -61,7 +62,7 @@ from django.forms.utils import ErrorList from geonode.base.auth import get_or_create_token -from geonode.base.forms import CategoryForm, TKeywordForm, BatchPermissionsForm +from geonode.base.forms import CategoryForm, TKeywordForm, BatchPermissionsForm, ThesaurusAvailableForm from geonode.base.views import batch_modify from geonode.base.models import ( Thesaurus, @@ -904,6 +905,7 @@ def layer_metadata( current_keywords = [keyword.name for keyword in layer.keywords.all()] topic_category = layer.category + topic_thesaurus = layer.tkeywords.all() # Add metadata_author or poc if missing layer.add_missing_metadata_author_or_poc() @@ -1018,7 +1020,11 @@ def layer_metadata( json.dumps(out), content_type='application/json', status=400) - tkeywords_form = TKeywordForm(request.POST) + if hasattr(settings, 'THESAURUS'): + tkeywords_form = TKeywordForm(request.POST) + else: + tkeywords_form = ThesaurusAvailableForm(request.POST, prefix='tkeywords') + # set initial values for thesaurus form if not tkeywords_form.is_valid(): logger.error(f"Layer Thesauri Keywords form is not valid: {tkeywords_form.errors}") out = { @@ -1041,31 +1047,42 @@ def layer_metadata( prefix="category_choice_field", initial=topic_category.id if topic_category else None) - # Keywords from THESAURUS management - layer_tkeywords = layer.tkeywords.all() - tkeywords_list = '' - lang = 'en' # TODO: use user's language - if layer_tkeywords and len(layer_tkeywords) > 0: - tkeywords_ids = layer_tkeywords.values_list('id', flat=True) - if hasattr(settings, 'THESAURUS') and settings.THESAURUS: - el = settings.THESAURUS - thesaurus_name = el['name'] - try: - t = Thesaurus.objects.get(identifier=thesaurus_name) - for tk in t.thesaurus.filter(pk__in=tkeywords_ids): - tkl = tk.keyword.filter(lang=lang) - if len(tkl) > 0: - tkl_ids = ",".join( - map(str, tkl.values_list('id', flat=True))) - tkeywords_list += "," + \ - tkl_ids if len( - tkeywords_list) > 0 else tkl_ids - except Exception: - tb = traceback.format_exc() - logger.error(tb) - tkeywords_form = TKeywordForm(instance=layer) + # Create THESAURUS widgets + lang = settings.THESAURUS_DEFAULT_LANG if hasattr(settings, 'THESAURUS_DEFAULT_LANG') else 'en' + if hasattr(settings, 'THESAURUS') and settings.THESAURUS: + warnings.warn('The settings for Thesaurus has been moved to Model, \ + this feature will be removed in next releases', DeprecationWarning) + layer_tkeywords = layer.tkeywords.all() + tkeywords_list = '' + if layer_tkeywords and len(layer_tkeywords) > 0: + tkeywords_ids = layer_tkeywords.values_list('id', flat=True) + if hasattr(settings, 'THESAURUS') and settings.THESAURUS: + el = settings.THESAURUS + thesaurus_name = el['name'] + try: + t = Thesaurus.objects.get(identifier=thesaurus_name) + for tk in t.thesaurus.filter(pk__in=tkeywords_ids): + tkl = tk.keyword.filter(lang=lang) + if len(tkl) > 0: + tkl_ids = ",".join( + map(str, tkl.values_list('id', flat=True))) + tkeywords_list += "," + \ + tkl_ids if len( + tkeywords_list) > 0 else tkl_ids + except Exception: + tb = traceback.format_exc() + logger.error(tb) + tkeywords_form = TKeywordForm(instance=layer) + else: + tkeywords_form = ThesaurusAvailableForm(prefix='tkeywords') + # set initial values for thesaurus form + for tid in tkeywords_form.fields: + values = [] + values = [keyword.id for keyword in topic_thesaurus if int(tid) == keyword.thesaurus.id] + tkeywords_form.fields[tid].initial = values + if request.method == "POST" and layer_form.is_valid() and attribute_form.is_valid( - ) and category_form.is_valid(): + ) and category_form.is_valid() and tkeywords_form.is_valid(): new_poc = layer_form.cleaned_data['poc'] new_author = layer_form.cleaned_data['metadata_author'] @@ -1156,14 +1173,17 @@ def layer_metadata( if not tkeywords_form.is_valid(): return HttpResponse(json.dumps({'message': "Invalid thesaurus keywords"}, status_code=400)) - tkeywords_data = tkeywords_form.cleaned_data['tkeywords'] - thesaurus_setting = getattr(settings, 'THESAURUS', None) if thesaurus_setting: + tkeywords_data = tkeywords_form.cleaned_data['tkeywords'] tkeywords_data = tkeywords_data.filter( thesaurus__identifier=thesaurus_setting['name'] ) layer.tkeywords.set(tkeywords_data) + elif Thesaurus.objects.all().exists(): + fields = tkeywords_form.cleaned_data + layer.tkeywords.set(tkeywords_form.cleanx(fields)) + except Exception: tb = traceback.format_exc() logger.error(tb) diff --git a/geonode/people/migrations/0030_auto_20210201_0953.py b/geonode/people/migrations/0030_auto_20210201_0953.py new file mode 100644 index 00000000000..2a9f31142fa --- /dev/null +++ b/geonode/people/migrations/0030_auto_20210201_0953.py @@ -0,0 +1,18 @@ +# Generated by Django 2.2.16 on 2021-02-01 09:53 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('people', '0029_auto_20201216_1335'), + ] + + operations = [ + migrations.AlterField( + model_name='profile', + name='language', + field=models.CharField(choices=[('af', 'Afrikaans'), ('ar', 'العربيّة'), ('ast', 'asturian'), ('az', 'Azərbaycanca'), ('bg', 'български'), ('be', 'беларуская'), ('bn', 'বাংলা'), ('br', 'brezhoneg'), ('bs', 'bosanski'), ('ca', 'català'), ('cs', 'česky'), ('cy', 'Cymraeg'), ('da', 'dansk'), ('de', 'Deutsch'), ('el', 'Ελληνικά'), ('en', 'English'), ('en-au', 'Australian English'), ('en-gb', 'British English'), ('eo', 'Esperanto'), ('es', 'español'), ('es-ar', 'español de Argentina'), ('es-mx', 'español de Mexico'), ('es-ni', 'español de Nicaragua'), ('es-ve', 'español de Venezuela'), ('et', 'eesti'), ('eu', 'Basque'), ('fa', 'فارسی'), ('fi', 'suomi'), ('fr', 'français'), ('fy', 'frysk'), ('ga', 'Gaeilge'), ('gl', 'galego'), ('he', 'עברית'), ('hi', 'Hindi'), ('hr', 'Hrvatski'), ('hu', 'Magyar'), ('ia', 'Interlingua'), ('id', 'Bahasa Indonesia'), ('io', 'ido'), ('is', 'Íslenska'), ('it', 'italiano'), ('ja', '日本語'), ('ka', 'ქართული'), ('kk', 'Қазақ'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', '한국어'), ('lb', 'Lëtzebuergesch'), ('lt', 'Lietuviškai'), ('lv', 'latvieš'), ('mk', 'Македонски'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'मराठी'), ('my', 'မြန်မာဘာသာ'), ('nb', 'norsk (bokmål)'), ('ne', 'नेपाली'), ('nl', 'Nederlands'), ('nn', 'norsk (nynorsk)'), ('os', 'Ирон'), ('pa', 'Punjabi'), ('pl', 'polski'), ('pt', 'Português'), ('pt-br', 'Português Brasileiro'), ('ro', 'Română'), ('ru', 'Русский'), ('sk', 'slovenský'), ('sl', 'Slovenščina'), ('sq', 'shqip'), ('sr', 'српски'), ('sr-latn', 'srpski (latinica)'), ('sv', 'svenska'), ('sw', 'Kiswahili'), ('ta', 'தமிழ்'), ('te', 'తెలుగు'), ('th', 'ภาษาไทย'), ('tr', 'Türkçe'), ('tt', 'Татарча'), ('udm', 'Удмурт'), ('uk', 'Українська'), ('ur', 'اردو'), ('vi', 'Tiếng Việt'), ('zh-cn', '简体中文'), ('zh-hans', '简体中文'), ('zh-hant', '繁體中文'), ('zh-tw', '繁體中文')], default='en', max_length=10, verbose_name='language'), + ), + ] diff --git a/geonode/people/migrations/0031_merge_20210205_0824.py b/geonode/people/migrations/0031_merge_20210205_0824.py new file mode 100644 index 00000000000..0ddcc278d06 --- /dev/null +++ b/geonode/people/migrations/0031_merge_20210205_0824.py @@ -0,0 +1,14 @@ +# Generated by Django 2.2.16 on 2021-02-05 08:24 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('people', '0030_auto_20210118_1112'), + ('people', '0030_auto_20210201_0953'), + ] + + operations = [ + ] diff --git a/geonode/tests/data/thesaurus.rdf b/geonode/tests/data/thesaurus.rdf new file mode 100644 index 00000000000..1eade728ad6 --- /dev/null +++ b/geonode/tests/data/thesaurus.rdf @@ -0,0 +1,22 @@ + + + + Italian register of the reference data sets + Register of the reference data sets + Mocked Title + 2018-05-23T10:25:56 + 2018-05-23T10:25:56 + + + Addresses + Adressen + Location of properties based on address identifiers, usually by road name, house number, postal code. + + + + Administrative units + Units of administration, dividing areas where Member States have and/or exercise jurisdictional rights, for local, regional and national governance, separated by administrative boundaries. + au + + + \ No newline at end of file diff --git a/geonode/tests/test_load_thesaurus.py b/geonode/tests/test_load_thesaurus.py new file mode 100644 index 00000000000..3fe33dfc866 --- /dev/null +++ b/geonode/tests/test_load_thesaurus.py @@ -0,0 +1,147 @@ +######################################################################### +# +# Copyright (C) 2020 OSGeo +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# +######################################################################### + +from django.test.testcases import SimpleTestCase +from geonode.base.models import Thesaurus, ThesaurusKeyword, ThesaurusKeywordLabel, ThesaurusLabel +from django.test import TestCase +from django.core import management +from defusedxml import lxml as dlxml +from geonode.base.management.commands.load_thesaurus import get_all_lang_available_with_title, determinate_value +import os + + +class TestLoadThesaurus(TestCase): + @classmethod + def setUpTestData(cls): + management.call_command( + "load_thesaurus", + file=f"{os.path.dirname(os.path.abspath(__file__))}/data/thesaurus.rdf", + name="foo_name", + stdout="out", + ) + + def setUp(self): + self.rdf_path = f"{os.path.dirname(os.path.abspath(__file__))}/data/thesaurus.rdf" + self.Thesaurus = Thesaurus( + identifier="foo_name", + title="Mocked Title", + date="2018-05-23T10:25:56", + description="Mocked Title", + slug="", + about="http://inspire.ec.europa.eu/theme", + ) + + def test_given_invalid_filename_will_raise_an_oserror(self): + with self.assertRaises(OSError): + management.call_command("load_thesaurus", file="abc", name="foo_name", stdout="out") + + def test_give_a_valid_name_should_save_the_expected_Thesaurus(self): + actual = self.__get_last_thesaurus() + self.assertEqual(self.Thesaurus.identifier, actual.identifier) + self.assertEqual(self.Thesaurus.title, actual.title) + self.assertEqual(self.Thesaurus.date, actual.date) + self.assertEqual(self.Thesaurus.description, actual.description) + self.assertEqual(self.Thesaurus.slug, actual.slug) + self.assertEqual(self.Thesaurus.about, actual.about) + + def test_give_a_valid_name_should_save_the_expected_ThesaurusKeyword(self): + tid = self.__get_last_thesaurus() + actual = ThesaurusKeyword.objects.filter(thesaurus=tid) + self.assertEqual(2, len(actual)) + + def test_give_a_valid_name_should_save_the_expected_ThesaurusKeywordLabel(self): + tkey = ThesaurusKeyword.objects.filter(thesaurus=self.__get_last_thesaurus())[0] + actual = ThesaurusKeywordLabel.objects.filter(keyword=tkey) + self.assertEqual(2, len(actual)) + + def test_give_a_valid_name_should_save_the_expected_ThesaurusLabel(self): + tid = self.__get_last_thesaurus() + actual = ThesaurusLabel.objects.all().filter(thesaurus=tid) + self.assertEqual(2, len(actual)) + + @staticmethod + def __get_last_thesaurus(): + return Thesaurus.objects.all().order_by("-id")[0] + + +class TestExtractLanguages(SimpleTestCase): + def setUp(self): + self.rdf_path = f"{os.path.dirname(os.path.abspath(__file__))}/data/thesaurus.rdf" + + def test_get_all_lang_available_should_return_all_the_lang_available_int_the_file(self): + titles = self.__load_titles() + XML_URI = "http://www.w3.org/XML/1998/namespace" + LANG_ATTRIB = "{" + XML_URI + "}lang" + actual = get_all_lang_available_with_title(titles, LANG_ATTRIB) + expected = [ + ("it", "Italian register of the reference data sets"), + ("en", "Register of the reference data sets"), + (None, "Mocked Title"), + ] + self.assertListEqual(expected, actual) + + def test_determinate_title_should_return_the_title_without_lang_if_available(self): + titles = [ + ("it", "Italian register of the reference data sets"), + ("en", "Register of the reference data sets"), + (None, "Mocked Title"), + ] + actual = determinate_value(titles, None) + self.assertEqual("Mocked Title", actual) + + def test_determinate_title_should_return_the_italian_lang_if_none_is_not_available(self): + titles = [ + ("it", "Italian register of the reference data sets"), + ("en", "Register of the reference data sets"), + ] + actual = determinate_value(titles, "it") + self.assertEqual("Italian register of the reference data sets", actual) + + def test_determinate_title_should_return_the_title_whithout_lang_even_if_localized_is_available(self): + titles = [ + ("it", "Italian register of the reference data sets"), + ("en", "Register of the reference data sets"), + (None, "Mocked Title"), + ] + actual = determinate_value(titles, "it") + self.assertEqual("Mocked Title", actual) + + def test_determinate_title_should_take_the_first_localized_title_when_default_one_is_not_available(self): + titles = [ + ("it", "Italian register of the reference data sets"), + ("en", "Register of the reference data sets"), + ] + actual = determinate_value(titles, "not-existing") + self.assertEqual("Italian register of the reference data sets", actual) + + def __load_titles(self): + RDF_URI = "http://www.w3.org/1999/02/22-rdf-syntax-ns#" + ns = { + "rdf": RDF_URI, + "foaf": "http://xmlns.com/foaf/0.1/", + "dc": "http://purl.org/dc/elements/1.1/", + "dcterms": "http://purl.org/dc/terms/", + "skos": "http://www.w3.org/2004/02/skos/core#", + } + + tfile = dlxml.parse(self.rdf_path) + root = tfile.getroot() + + scheme = root.find("skos:ConceptScheme", ns) + return scheme.findall("dc:title", ns)