Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GNIP 82 - Thesauri improvements #6925

Closed
3 of 5 tasks
etj opened this issue Feb 3, 2021 · 7 comments
Closed
3 of 5 tasks

GNIP 82 - Thesauri improvements #6925

etj opened this issue Feb 3, 2021 · 7 comments
Assignees
Labels
gnip A GeoNodeImprovementProcess Issue
Milestone

Comments

@etj
Copy link
Contributor

etj commented Feb 3, 2021

WIP

GNIP 82 - Thesauri improvements

Overview

Thesauri implementation needs some revision, update and improvements.
Currently it's only possibile to have a single thesaurus in the GUI, while INSPIRE and other profiles require keywords from different thesauri.
We want to:

  • handle multiple thesauri
  • improve i18n in thesauri titles
  • improve cardinality handling

Proposed By

@etj (Emanuele Tajariol)

Assigned to Release

This proposal is for GeoNode ???

State

  • Under Discussion
  • In Progress
  • Completed
  • Rejected
  • Deferred

Motivation

Thesaurus code is stale and needs some revisions.
Currently it's only possibile to have a single thesaurus in the GUI, while INSPIRE and other profiles require keywords from different thesauri.
Furthermore, when compiling a 19115 metadata, recent INSPIRE specs (and related profiles) require both href and title of the cited thesaurus.

Proposal

Model

Thesaurus table (to be updated)

We need to support an href referencing the thesaurus; it's stored into the rdf:about field.

  • Add field about = models.CharField(max_length=255, null=True, blank=True)

We need to move into the DB the config currently kept in SETTINGS (see https://docs.geonode.org/en/master/admin/thesaurus/#configure-a-thesaurus-in-geonode).
This is an excerpt from the current doc:

name: (mandatory string) the identifier you used in the load_thesaurus commands.
required: (optional boolean) if True, a keyword of this thesaurus is mandatory to complete the metadata. Currently not implemented.
filter: (optional boolean) if True, a faceted list of keywords of this thesaurus will be presented on the search page.

  • name will not be ported, since it's a FK to the Thesaurus identifier;
  • required will be enhanced, in order to better support cardinality info; we'll have card_min and card_max
  • filter is a too general concept, so we can transform it into a field named facet

We'll add:

  • card_min integer
  • card_max integer
  • facet boolean

Warning: the change about moving the settings from the SETTINGS file into the DB is breaking backcompatibility, so we'll have to change the logic accordingly:

  • when one of the moved settings is needed,
    • we'll first check in the SETTINGS; if the setting is there, we'll use it and notify a warning about a deprecated setting.
    • only as a fallback step we'll check for the value in the DB

ThesaurusLabel table (new)

Contains localized version of the thesaurus title

  • lang = models.CharField(max_length=3)
  • label = models.CharField(max_length=255)
  • FK to Thesaurus

We'll use the localized label in the GUI, and the general title (in the Thesaurus class) for the xml template.

Commands

The command to load the thesaurus will be improved in order to support the thesaurus label localization.
In particular, we may have rdf files which only have localized titles, and a missing "general" title.
We need to handle all condition:

e.g.: only general title:

<skos:ConceptScheme rdf:about="http://geonetwork-opensource.org/thesaurus/naturalearth-and-seavox">
    <dc:title>Continents, countries, sea regions of the world.</dc:title>
    <dc:description>Generated from NaturalEarth datasets and SeaVox.</dc:description>
  • Thesaurus.title --> "Continents, countries, sea regions of the world."
  • ThesaurusLabel --> no records

e.g.: only localized titles:

<skos:ConceptScheme xmlns:skos="http://www.w3.org/2004/02/skos/core#" rdf:about="https://registry.geodati.gov.it/rndt-all1">
  <dc:title xmlns:dc="http://purl.org/dc/elements/1.1/" xml:lang="it">Registro dei dati di interesse generale per il RNDT</dc:title>
  <dc:title xmlns:dc="http://purl.org/dc/elements/1.1/" xml:lang="en">Register of the reference data sets for RNDT</dc:title>
  • ThesaurusLabel --> entries for it and en
  • Thesaurus.title --> We have to chose one.
    • A new setting will be introduced DEFAULT_THESAURUS_LANG, so that the title with the selected language will be chosen
      1. If a title with the language set in DEFAULT_THESAURUS_LANG, chose that one to fill the Thesaurus.title field
      2. Else take the title with en language
      3. Else take the first localized title found

Metadata Editing GUI

Modify the editing GUI so that thesauri selection is rendered in the first tab.

We want to display all of the thesauri set as required in the GUI.
The requirement is set in the DB (card_min, card_max), but we also have a legacy setting (https://docs.geonode.org/en/2.10/admin/thesaurus/index.html#configure-a-thesaurus-in-geonode)

In the GUI we will have, according to the card_max and card_min values,

  • card_max=0 --> Disabled, won't be shown in the GUI (card_min is not even checked in this case)
  • card_max=1 --> Single choice
    • card_min = 0 --> Single choice, optional --> rendered as a dropdown, with an empty/unselected element
    • card_min = 1 --> Single choice, required --> rendered as a dropdown
  • card_max=-1 --> Multiple choice
    • card_min = 0 --> [0..N] Multiple choices, optional --> rendered as a typeahead badges(?)
    • card_min = 1 --> [1..N] Multiple choices, required --> rendered as a typeahead badges(?) + apply check for the mandatory warning

If the legacy required boolean setting is in the configuration, we'll use it as if we'd found:

  • required = true
    • card_min = 1
    • card_max = -1
  • required = false
    • card_min = 0
    • card_max = -1

As stated elewhere also, when a legacy settings is found, we'll use it and notify a warning about the deprecated setting.

The displayed thesaurus label should be the localized one, if any, otherwise the generic title will be used.


GUI should be restructed as follows:

  • free keywords moved to the bottom
  • list of Thesauri keywords selectors

GeoNode - thesauri

Also, the validation checks should act according to the mandatory requirements of the displayed thesauri.

Metadata template

The XML metadata template has to be modified in order to include all of the thesauri keywords added to the layer metadata.
The thesaurus title to be put into the template is the generic (non localized) Thesaurus.title.

Settings changes

Backwards Compatibility

Thesaurus configuration is currently made in the settings file (as THESAURUS entry), and this is going to be moved to DB.
As noted in the proposal, if file configuration is present, it will be used instead of the DB data for a given thesaurus, but a warning log will be notified about the deprecation of the file setting.

Future evolution

Feedback

Update this section with relevant feedbacks, if any.

Voting

Project Steering Committee:

  • Alessio Fabiani: +1
  • Francesco Bartoli:
  • Giovanni Allegri: +1
  • Simone Dalmasso:
  • Toni Schoenbuchner: +1
  • Florian Hoedt: +1

Links

Remove unused links below.

mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 3, 2021
… on thesaurus avaiable, fix test on thesaurus load, add migration file for updated thesaurus schema
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 3, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 4, 2021
@giohappy giohappy changed the title Thesauri improvements GNIP 82 - Thesauri improvements Feb 4, 2021
@t-book t-book added the gnip A GeoNodeImprovementProcess Issue label Feb 4, 2021
@gannebamm
Copy link
Contributor

Just to be clear: If you mention 'will be moved to DB' you mean this will get its own model and therefore admin panel page, right?

I like the concept of the GNIP backend and frontend-wise.

mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 5, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 5, 2021
…ew thesaurus information as template tags, add tests for new template tags
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 5, 2021
@t-book
Copy link
Contributor

t-book commented Feb 5, 2021

My +1
( Ping @giohappy )

mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 5, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 5, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 5, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 5, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 5, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 5, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 5, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 10, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 15, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 15, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 15, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 16, 2021
mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 16, 2021
@afabiani afabiani reopened this Feb 17, 2021
@etj
Copy link
Contributor Author

etj commented Feb 23, 2021

Some missing details about the GUI:

  • The titles of the thesaurus should be localized, using the language expressed in the language currently choosen for the GUI, i.e. the record in ThesaurusLabel which matches the thesaurus FK and the lang.
    If no such language exists for that the thesaurus title, the default title will be displayed (Thesaurus.title).

  • In the same way, the thesaurus keywords should be localized, using the language currently choosen for the GUI.
    If such lang is not present for a given keyword (i.e. no record in ThesaurusKeywordLabel pointing to a ThesaurusKeyword with a given lang), the default keyword will be used (i.e. ThesaurusKeyword.alt_label).

Please note that when retrieving the records for GUI display, the DEFAULT_THESAURUS_LANG setting will not be used at all: that setting is only used when parsing the rdf files and importing the thesauri.

mattiagiupponi added a commit to geosolutions-it/geonode that referenced this issue Feb 23, 2021
etj added a commit to etj/geonode that referenced this issue Feb 24, 2021
etj added a commit to etj/geonode that referenced this issue Feb 25, 2021
afabiani pushed a commit that referenced this issue Feb 26, 2021
* [Issue #6925] Thesaurus admin pages

* [Issue #6925] Thesaurus admin pages
@etj
Copy link
Contributor Author

etj commented Mar 8, 2021

Thesauri editing in maps and documents should be added.

@etj etj reopened this Mar 8, 2021
@gannebamm
Copy link
Contributor

Is the loading of an RDF thesaurus still valid? See https://docs.geonode.org/en/master/admin/thesaurus/#loading-a-thesaurus

@etj
Copy link
Contributor Author

etj commented Mar 8, 2021

@gannebamm sure, we also made some improvements related to importing the items in various languages.
Found any issues loading RDF files?

@afabiani afabiani added this to the 3.2 milestone Mar 10, 2021
afabiani pushed a commit that referenced this issue Mar 10, 2021
* [Fixes: #7033] Mandatory fields in editor

* [Fixes #6925] Add new thesaurus behaviour for documents and maps
@afabiani afabiani reopened this Mar 10, 2021
@gannebamm
Copy link
Contributor

@etj No, I will need those soonish and was just curious if the old way of loading thesauri still work. Nice job.

afabiani pushed a commit that referenced this issue Mar 10, 2021
* [Fixes: #7033] Mandatory fields in editor

* [Fixes #6925] Add new thesaurus behaviour for documents and maps

* [Fixes #6925] Facet endpoint changed to serve default label if translated one does not exists

* [Fixes #6925] Facet endpoint changed to serve default label if translated one does not exists
@afabiani afabiani reopened this Mar 10, 2021
@etj etj mentioned this issue Apr 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
gnip A GeoNodeImprovementProcess Issue
Projects
None yet
Development

No branches or pull requests

6 participants