Extending table-schema types/formats

[yohanboniface]

Hi,

We work in the French administration. As you may know, we already use frictionless and table-schema in our tools, see for example https://validata.fr/ or http://schema.data.gouv.fr/.
We are now in the process of trying to describe, through table-schema, columns that contain national identifiers (like administrative references, or national company unique numbers, etc.).
We want to take this path to push table-schema one step forward as a national official standard.
Our current reflection is about how to describe such a column through table-schema, we are not yet talking about tools implementing such property (but our plan at this time is to have a frictionless plugin like frictionless-france that would deal with those formats).
And, to make this clear, we are not talking about changing the table-schema standard, but only using its current extension possibilities.
At this point, before moving forward, we'd like to have your feedback on what would be the best solution.
Here are some of the options we have identified:

• use a custom format nomenclature, for example:

  fields:
    - name: myfield
      type: string
      format: "fr:insee:commune"  # Invalid as it is

(which is not currently possible)

• use a custom format key, for example:

  fields:
    - name: myfield
      type: string
      formatFR: "fr:insee:commune"

• use a rich type:

  fields:
    - name: myfield
      type: string
      rdfType: "https://path/to/rdf/schema"

• use an extension of table-schema, like it's done with the Fiscal Data Package: https://specs.frictionlessdata.io/fiscal-data-package/#what-is-the-fiscal-data-package

Do you have any feedback to provide ? Any other users that have made such types extensions ?

Thanks in advance,

Yohan Boniface with @johanricher and @geoffreyaldebert

[yohanboniface]

Another option that we may consider, even if not having the exact same perimeter would be to allow external foreign keys, i.e. pointing to a resource from another data-packages designed by an URI.

Eg: with a new data-package property:

    fields:
    - name: state-code
    foreignKeys:
        fields: state-code
        reference:
            resource: states-code
            fields: code
            data-package: uri/to/datapackage

or, extending the resource property itself:

    fields:
    - name: state-code
    foreignKeys:
        fields: state-code
        reference:
            resource: uri/to/datapackage/states-code
            fields: code

Thoughts ?

[roll]

I think it might be useful to have a pattern defining best practices for this case

[roll]

I think taking into account the current direction of the Data Package Standard evolution, I would suggest using:

• Custom Properties
• Extensions Mechanism

It will allows to provide basically any domain-specific properties e.g. fr:format and constraint them to any domain-specific requirements with $schema property

WDYT @peterdesmet @khusmann

[pierrecamilleri]

We've made some progress in our thinking, and we chose to add a custom property (like the second example in the original post), with a custom profile to validate this property.

We are updating our tooling (Validata) to take account of this new property, and the schema remains compatible with all tools that take a standard Table Schema as input.

This solution seems to cover our needs, thanks for the hints @roll