[Fixes #6925] Thesauri improvements (#6951)

* Thesaurus: Add model ThesaurusLabel, improve load_thesaurus managment command and add test coverage

* Thesaurus: Add new migration file, stab thesaurus rdf file and managment command test

* Thesaurus: Add copyright, file reformatted with black

* Thesaurus: Increase test coverage for thesaurus load and class refactor

* Resolves #6925: New Thesaurus form generated dynamically based on thesaurus avaiable, fix test on thesaurus load, add migration file for updated thesaurus schema

* Resolves #6925: refactor thesaurus form, fix validation issue

* Resolves #6925: fix metadata bug that prevent to save multiple key for each thesaurus

* Bump urllib3 from 1.26.2 to 1.26.3 (#6908)

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.2 to 1.26.3.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.3/CHANGES.rst)
- [Commits](urllib3/urllib3@1.26.2...1.26.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Toni <toni.schoenbuchner@csgis.de>

* [Fixes #6880] Circle CI upload tests fail irregulary (#6881)

* [Fixes #6880] Circle CI upload tests fail irregulary

* CircleCI test fix: sometimes expires due to upload timeout in the test environment

* - Avoid infinite loop on upload testing

* Revert "CircleCI test fix: sometimes expires due to upload timeout in the test environment"

This reverts commit 66139fd.

Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it>
Co-authored-by: afabiani <alessio.fabiani@gmail.com>

* [Fixes #6914] Remove "add to basket" tool for documents and maps (#6915)

* Added malnajdi as contributor

* [Fixes #6910] meaningful filename for document download (#6911)

* get meaningful document filenames on download

* - Strip extension from document title before slugify it (e.g.: image.jpg instead of imagejpg.jpg)

Co-authored-by: afabiani <alessio.fabiani@gmail.com>
Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it>

* - CircleCI Upload Tests: trying to reduce more the risk of infinite loop on "wait_for_progress"

* [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION (#6913)

* [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION

* [Fixes #6916] unit test for GEOSERVER_LOCATION

* Bump django-cors-headers from 3.6.0 to 3.7.0 (#6901)

Bumps [django-cors-headers](https://github.com/adamchainz/django-cors-headers) from 3.6.0 to 3.7.0.
- [Release notes](https://github.com/adamchainz/django-cors-headers/releases)
- [Changelog](https://github.com/adamchainz/django-cors-headers/blob/master/HISTORY.rst)
- [Commits](adamchainz/django-cors-headers@3.6.0...3.7.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump amqp from 5.0.3 to 5.0.5 (#6905)

Bumps [amqp](https://github.com/celery/py-amqp) from 5.0.3 to 5.0.5.
- [Release notes](https://github.com/celery/py-amqp/releases)
- [Changelog](https://github.com/celery/py-amqp/blob/master/Changelog)
- [Commits](celery/py-amqp@v5.0.3...v5.0.5)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pip from 21.0 to 21.0.1 (#6900)

Bumps [pip](https://github.com/pypa/pip) from 21.0 to 21.0.1.
- [Release notes](https://github.com/pypa/pip/releases)
- [Changelog](https://github.com/pypa/pip/blob/master/NEWS.rst)
- [Commits](pypa/pip@21.0...21.0.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump coverage from 5.3.1 to 5.4 (#6903)

Bumps [coverage](https://github.com/nedbat/coveragepy) from 5.3.1 to 5.4.
- [Release notes](https://github.com/nedbat/coveragepy/releases)
- [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst)
- [Commits](nedbat/coveragepy@coverage-5.3.1...coverage-5.4)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pytest from 6.2.1 to 6.2.2 (#6907)

Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.1 to 6.2.2.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@6.2.1...6.2.2)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump djangorestframework-gis from 0.16 to 0.17 (#6902)

Bumps [djangorestframework-gis](https://github.com/openwisp/django-rest-framework-gis) from 0.16 to 0.17.
- [Release notes](https://github.com/openwisp/django-rest-framework-gis/releases)
- [Changelog](https://github.com/openwisp/django-rest-framework-gis/blob/master/CHANGES.rst)
- [Commits](openwisp/django-rest-framework-gis@v0.16.0...v0.17.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* - Algin setup.cfg to requirements.txt

* [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has… (#6923)

* [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has been uploaded

* - Add REST APIs test suite to CircleCI

* [Fixes #6918] Removal of QGIS support (#6919)

* [Cleanup and Refactor] Remove QGIS server backend dependencies

* [Cleanup and Refactor] Remove QGIS server backend dependencies

* - Fix LGTM issues

* Resolves #6925: add deprecation warning for old thesaurus settings

* allow Basic authenticated requests in LOCKDOWN mode

* Resolves #6925: change metadata template to let it reads the new thesaurus information as template tags, add tests for new template tags

* Resolves #6925: add thesaurus in admin page

* fix to avoid circular import

* flake8 check fix

* Resolves #6925: reformatting code for match Flake8 requirements

* added tests

* Resolves #6925: fix default values for all required in Thesaurus Model

* Resolves #6925: add test for ThesaurusAvailableModelForm

* Resolves #6925: minor refactor in ThesaurusForm initialization

* Resolves #6925: thesaurus initial data

* Resolves #6925: fix flake8 style errors

* Refers #6925: add missing migration file

* Resolves #6925: fix thesaurus show method by adding facet as filter

* Resolves #6925: fix flake8 errors

* Resolves #6925: fix lgmt bot issue

* Resolves #6925: add missing prefix for thesaurus

* Fixes #6925: remote RNDT information from the tests

* Fixes #6925: italian wordings updated to english

Co-authored-by: Giovanni Allegri <giohappy@gmail.com>
Co-authored-by: allyoucanmap <bovio.stefano@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Toni <toni.schoenbuchner@csgis.de>
Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it>
Co-authored-by: afabiani <alessio.fabiani@gmail.com>
Co-authored-by: Florian Hoedt <gannebamm@gmail.com>
Co-authored-by: Mohammed Y. Alnajdi <mohdnagfy@gmail.com>
Co-authored-by: biegan <bieganowski.rev@gmail.com>

geonode/base/fixtures/initial_data.json

@@ -4852,5 +4852,269 @@
       "fields": {
         "name": "TOPBAR_MENU"
       }
-    }
+    },
+    {
+      "model": "base.thesaurus",
+      "pk": 1,
+      "fields": {
+          "identifier": "inspire_themes",
+          "title": "GEMET - INSPIRE themes, version 1.0",
+          "date": "2018-05-23T10:25:56",
+          "description": "GEMET - INSPIRE themes, version 1.0",
+          "slug": "",
+          "about": "http://inspire.ec.europa.eu/theme",
+          "card_min": 1,
+          "card_max": 0,
+          "facet": true
+      }
+    },
+    {
+        "model": "base.thesaurus",
+        "pk": 4,
+        "fields": {
+            "identifier": "abc-all1",
+            "title": "Register of data of general interest",
+            "date": "2018-06-25",
+            "description": "Register of data of general interest",
+            "slug": "",
+            "about": "https://registry.geodati.gov.it/abc-all1",
+            "card_min": 0,
+            "card_max": 1,
+            "facet": true
+        }
+    },
+    {
+        "model": "base.thesaurus",
+        "pk": 6,
+        "fields": {
+            "identifier": "limitation_on_public_access",
+            "title": "Limitations on public access",
+            "date": "2020-10-22T13:21:33",
+            "description": "Limitations on public access",
+            "slug": "",
+            "about": "http://inspire.ec.europa.eu/metadata-codelist/LimitationsOnPublicAccess",
+            "card_min": 1,
+            "card_max": 1,
+            "facet": true
+        }
+    },
+    {
+        "model": "base.thesaurus",
+        "pk": 7,
+        "fields": {
+            "identifier": "limitation_on_public_access-2",
+            "title": "Conditions Applying To Access and Use",
+            "date": "2020-10-30T16:58:34",
+            "description": "Conditions Applying To Access and Use",
+            "slug": "",
+            "about": "http://inspire.ec.europa.eu/metadata-codelist/ConditionsApplyingToAccessAndUse",
+            "card_min": 0,
+            "card_max": -1,
+            "facet": true
+        }
+    },
+    {
+        "model": "base.thesaurus",
+        "pk": 8,
+        "fields": {
+            "identifier": "foo_name",
+            "title": "Mocked Title",
+            "date": "2018-05-23T10:25:56",
+            "description": "Mocked Title",
+            "slug": "abc",
+            "about": "http://inspire.ec.europa.eu/theme",
+            "card_min": 1,
+            "card_max": -1,
+            "facet": true
+        }
+    },
+    {
+      "model": "base.thesauruskeyword",
+      "pk": 1,
+      "fields": {
+          "about": "http://inspire.ec.europa.eu/theme/ad",
+          "alt_label": "ad",
+          "thesaurus": 1
+      }
+    },
+    {
+        "model": "base.thesauruskeyword",
+        "pk": 2,
+        "fields": {
+            "about": "http://inspire.ec.europa.eu/theme/au",
+            "alt_label": "au",
+            "thesaurus": 1
+        }
+    },
+    {
+        "model": "base.thesauruskeyword",
+        "pk": 3,
+        "fields": {
+            "about": "http://inspire.ec.europa.eu/theme/rs",
+            "alt_label": "rs",
+            "thesaurus": 1
+        }
+    },
+    {
+      "model": "base.thesauruskeyword",
+      "pk": 35,
+      "fields": {
+          "about": "http://inspire.ec.europa.eu/metadata-codelist/LimitationsOnPublicAccess/INSPIRE_Directive_Article13_1a",
+          "alt_label": "public access limited according to Article 13(1)(a) of the INSPIRE Directive [ITA]",
+          "thesaurus": 6
+      }
+    },
+    {
+      "model": "base.thesauruskeyword",
+      "pk": 44,
+      "fields": {
+          "about": "http://inspire.ec.europa.eu/metadata-codelist/ConditionsApplyingToAccessAndUse/noConditionsApply",
+          "alt_label": "No condition applicable",
+          "thesaurus": 7
+      }
+    },
+    {
+        "model": "base.thesauruskeyword",
+        "pk": 45,
+        "fields": {
+            "about": "http://inspire.ec.europa.eu/metadata-codelist/ConditionsApplyingToAccessAndUse/conditionsUnknown",
+            "alt_label": "Unknown Conditions",
+            "thesaurus": 7
+        }
+    },
+    {
+      "model": "base.thesauruskeyword",
+      "pk": 100,
+      "fields": {
+          "about": "http://test.com",
+          "alt_label": "Protected sites",
+          "thesaurus": 8
+      }
+    },
+    {
+      "model": "base.thesauruskeywordlabel",
+      "pk": 1,
+      "fields": {
+          "lang": "en",
+          "label": "Addresses",
+          "keyword": 1
+      }
+    },
+    {
+        "model": "base.thesauruskeywordlabel",
+        "pk": 2,
+        "fields": {
+            "lang": "de",
+            "label": "Adressen",
+            "keyword": 1
+        }
+    },
+    {
+        "model": "base.thesauruskeywordlabel",
+        "pk": 3,
+        "fields": {
+            "lang": "en",
+            "label": "Administrative units",
+            "keyword": 2
+        }
+    },
+    {
+        "model": "base.thesauruskeywordlabel",
+        "pk": 4,
+        "fields": {
+            "lang": "en",
+            "label": "Coordinate reference systems",
+            "keyword": 3
+        }
+    },
+    {
+      "model": "base.thesauruskeywordlabel",
+      "pk": 36,
+      "fields": {
+          "lang": "en",
+          "label": "public access limited according to Article 13(1)(a) of the INSPIRE Directive",
+          "keyword": 35
+      }
+    },
+    {
+      "model": "base.thesauruskeywordlabel",
+      "pk": 54,
+      "fields": {
+          "lang": "en",
+          "label": "no conditions to access and use",
+          "keyword": 44
+      }
+    },
+    {
+      "model": "base.thesauruskeywordlabel",
+      "pk": 56,
+      "fields": {
+          "lang": "en",
+          "label": "conditions to access and use unknown",
+          "keyword": 45
+      }
+    },
+    {
+        "model": "base.thesauruskeywordlabel",
+        "pk": 102,
+        "fields": {
+            "lang": "en",
+            "label": "Protected sites",
+            "keyword": 100
+        }
+    },
+    {
+      "model": "base.thesauruslabel",
+      "pk": 1,
+      "fields": {
+          "lang": "en",
+          "label": "Addresses",
+          "thesaurus": 1
+      }
+    },
+    {
+      "model": "base.thesauruslabel",
+      "pk": 2,
+      "fields": {
+          "lang": "de",
+          "label": "Adressen",
+          "thesaurus": 1
+      }
+    },
+    {
+      "model": "base.thesauruslabel",
+      "pk": 4,
+      "fields": {
+          "lang": "en",
+          "label": "Register of the reference data sets",
+          "thesaurus": 4
+      }
+    },
+    {
+      "model": "base.thesauruslabel",
+      "pk": 5,
+      "fields": {
+          "lang": "en",
+          "label": "Limitations on public access",
+          "thesaurus": 6
+      }
+    },
+    {
+      "model": "base.thesauruslabel",
+      "pk": 7,
+      "fields": {
+          "lang": "en",
+          "label": "Conditions Applying To Access and Use",
+          "thesaurus": 7
+      }
+    },
+    {
+      "model": "base.thesauruslabel",
+      "pk": 9,
+      "fields": {
+          "lang": "en",
+          "label": "Register of the reference data sets",
+          "thesaurus": 8
+      }
+  }
 ]

geonode/base/forms.py

@@ -20,43 +20,37 @@
 import re
 import html
 import logging
-
-from tinymce.widgets import TinyMCE
-
-from .fields import MultiThesauriField
-
+from django.db.models.query import QuerySet
+from bootstrap3_datetime.widgets import DateTimePicker
 from dal import autocomplete
-from taggit.forms import TagField
-
 from django import forms
 from django.conf import settings
 from django.contrib.auth import get_user_model
 from django.contrib.auth.models import Group
 from django.core import validators
 from django.db.models import Prefetch, Q
-from django.forms import models
-from django.forms import ModelForm
+from django.forms import ModelForm, models
 from django.forms.fields import ChoiceField
 from django.forms.utils import flatatt
+from django.utils.encoding import force_text
 from django.utils.html import format_html
 from django.utils.safestring import mark_safe
 from django.utils.translation import ugettext as _
-
-from django.utils.encoding import (
-    force_text,
-)
-
-from bootstrap3_datetime.widgets import DateTimePicker
 from modeltranslation.forms import TranslationModelForm
+from taggit.forms import TagField
+from tinymce.widgets import TinyMCE
 
-from geonode.base.models import HierarchicalKeyword, TopicCategory, Region, License, CuratedThumbnail, \
-    ResourceBase
-from geonode.base.models import ThesaurusKeyword, ThesaurusKeywordLabel
-from geonode.documents.models import Document
 from geonode.base.enumerations import ALL_LANGUAGES
+from geonode.base.models import (CuratedThumbnail, HierarchicalKeyword,
+                                 License, Region, ResourceBase, Thesaurus,
+                                 ThesaurusKeyword, ThesaurusKeywordLabel, ThesaurusLabel,
+                                 TopicCategory)
 from geonode.base.widgets import TaggitSelect2Custom
+from geonode.documents.models import Document
 from geonode.layers.models import Layer
 
+from .fields import MultiThesauriField
+
 logger = logging.getLogger(__name__)
 
 
@@ -320,6 +314,56 @@ class Meta:
     )
 
 
+class ThesaurusAvailableForm(forms.Form):
+    def __init__(self, *args, **kwargs):
+        super(ThesaurusAvailableForm, self).__init__(*args, **kwargs)
+        lang = settings.THESAURUS_DEFAULT_LANG if hasattr(settings, "THESAURUS_DEFAULT_LANG") else "en"
+        for item in Thesaurus.objects.all():
+            tname = ThesaurusLabel.objects.values_list("label", flat=True).filter(id=item.id).filter(lang=lang)
+            if item.card_max == 0:
+                continue
+            elif item.card_max == 1 and item.card_min == 0:
+                self.fields[f"{item.id}"] = self._define_choicefield(item, False, tname, lang)
+            elif item.card_max == 1 and item.card_min == 1:
+                self.fields[f"{item.id}"] = self._define_choicefield(item, True, tname, lang)
+            elif item.card_max == -1 and item.card_min == 0:
+                self.fields[f"{item.id}"] = self._define_multifield(item, False, tname, lang)
+            elif item.card_max == -1 and item.card_min == 1:
+                self.fields[f"{item.id}"] = self._define_multifield(item, True, tname, lang)
+
+    def cleanx(self, x):
+        cleaned_values = []
+        for key, value in x.items():
+            if isinstance(value, QuerySet):
+                for y in value:
+                    cleaned_values.append(y.id)
+            elif value:
+                cleaned_values.append(value.id)
+        return ThesaurusKeyword.objects.filter(id__in=cleaned_values)
+
+    @staticmethod
+    def _define_multifield(item, required, tname, lang):
+        return MultiThesauriField(
+            ThesaurusKeyword.objects.prefetch_related(
+                Prefetch(
+                    "keyword",
+                    queryset=ThesaurusKeywordLabel.objects.filter(keyword__thesaurus_id=item.id).filter(lang=lang),
+                )
+            ),
+            widget=autocomplete.ModelSelect2Multiple(url=f"/base/thesaurus_available/?sysid={item.id}&lang={lang}"),
+            label=_(f"{tname[0] if len(tname) > 0 else item.title}"),
+            required=required,
+        )
+
+    @staticmethod
+    def _define_choicefield(item, required, tname, lang):
+        return models.ModelChoiceField(
+            label=f"{tname[0] if len(tname) > 0 else item.title}",
+            required=required,
+            queryset=ThesaurusKeywordLabel.objects.filter(keyword__thesaurus_id=item.id).filter(lang=lang),
+        )
+
+
 class ResourceBaseDateTimePicker(DateTimePicker):
 
     def build_attrs(self, base_attrs=None, extra_attrs=None, **kwargs):