November 2024

Changelog

The format is loosely based on Keep a Changelog, and this project adheres to Semantic Versioning, except that the date of the release is used as the patch version.

Added

• Filter by modification date

  • Filters have been added to listing endpoints that allow filtering based on the date assets were last modified (#381).

• Configuration options may now be changed with separate override files

  • The config.override.toml may be used to override default configuration settings (#383). The config.toml is now config.default.toml. Renaming your current config.toml to config.override.toml should allow seemless migration, though we recommend you those remove configurations from that file for which you use the defaults.

• Elastic Search endpoints can now be set to sort by id instead of relevance (#388)

• Elastic Search endpoints can now be set to do an exact match instead of a fuzzy match (#388)

Updated

• Make host ports of docker-compose configurable through .env

  • Moved some configuration options from docker-compose to environment variables (in .env), so different deployments only have to modify the .env file and not also docker-compose file. (#345)

• Default docker compose no longer mounts local files

  • This is to reduce the risk of accidentally deploying changes which are local-only (#386). For local development, add also the developer compose file. For convenience, we added two scripts. See the README for more information.

• HuggingFace dataset platform_resource_identifiers have been changed

  • The previous identifier of name/dataset led to issues when HF users changed their name or the name of the dataset (#392). The platform resource identifier is now the "_id" of the dataset; a hex string.

Fixed

• Platform count endpoint no longer has a detailed parameter.
  • The inclusion was an oversight, and using it led to an error. Now, only total platform count is available (#381).
• Environment variables in .env are no longer shared to all containers.
  • Variables are set as needed instead (#383).
• Delete triggers are now correctly created in the database on startup
  • Previously, no triggers were created, which could lead to e.g., an AIResource existing without its Dataset. (#392)

v1.3.20241030

What's Changed

• Fix some HuggingFace Datasets connector issues by @andrejridzik in #335
• Bump huggingface-hub from 0.20.3 to 0.23.4 by @dependabot in #329
• Include sanity check in deploy instructions by @PGijsbers in #337
• Bump types-python-dateutil from 2.8.19.14 to 2.9.0.20240316 by @dependabot in #289
• Bump mysqlclient from 2.2.1 to 2.2.4 by @dependabot in #338
• Bump mysql-connector-python from 8.2.0 to 9.0.0 by @dependabot in #340
• Make host ports of docker compose configurable through .env by @PGijsbers in #345
• Add keycloak aiod theme initial commit by @mrorro in #339
• Add service for database schema migration. Extend length of note.value field by @PGijsbers in #353
• Release/1.3.20240909 (#354) by @PGijsbers in #355
• Fix/360 by @PGijsbers in #361
• Fix/rebuild db by @PGijsbers in #322
• Add security linting, request timeouts by @PGijsbers in #376
• fix openml-connector skips dataset issue by @Taniya-Das in #377
• Add migration to extend distribution content_url to LONG by @PGijsbers in #379
• Feature/search by date by @AdrianAlcolea in #368

Full Changelog: v1.3.20240619...v1.3.20241030

v1.3.20240909

Changelog

The format is loosely based on Keep a Changelog, and this project adheres to Semantic Versioning, except that the date of the release is used as the patch version.

Added

• Alembic service for automated database migrations

  • Added new service (containarized in docker) to update the database schema programmatically using Alembic service. (#353)

• Increased maximum length of the value attribute of the note class

  • Added a migration to extend the maximum length of the value attribute of the note class and its respective tables. (#353)

Updated

• Make host ports of docker-compose configurable through .env

  • Moved some configuration options from docker-compose to environment variables (in .env), so different deployments only have to modify the .env file and not also docker-compose file. (#345)

• Update deployment instructions in README.md

  • Added sanity check for instructions for server deployment in Readme.md file. (#337)

• Keycloak theme

  • Added AIoD style theme to keycloak. (#339)

Fixed

• Bugfix/ fix huggingface connector
  • Fixed few bugs in the HuggingFace connector that prevented it from indexing some datasets. (#335)

Release 1.3.20240619 - Bugfixes and improvements

Changelog

The format is loosely based on Keep a Changelog, and this project adheres to Semantic Versioning, except that the date of the release is used as the patch version.

Added

• Improvement/backup routine

  • Hotfix: Added --users flag to ensure user details are included in the exported realm file. (#286)

• Update docker workflow

  • Enhanced workflow to publish images to Docker Hub under aiod/metadata_catalogue.
  • Conditions for publishing:
    • On release: pushes with tags latest and version-specific tag (e.g., v1.3.20240308).
    • On push to develop branch: pushes with tag develop.
    • Via workflow dispatch through GitHub Actions: pushes with provided tag.
  • Added capability to update Docker repository description based on docker-description.md.
  • Implemented build cache to optimize workflow efficiency.
  • Note: Further validation on develop and release triggers and flow setup will be conducted on a fork. (#302)

• Enhancement/fixed version tags

  • Fixed Docker image version tags for better version control.
  • Upgraded Keycloak to version 24.0.4 to resolve deployment bug (Reference: keycloak/keycloak#25120). (#314)

Fixed

• Bugfix/remove pytest workflow

  • Removed redundant pytest workflow since pre-commit triggers pytest tests. (#284)

• Minor typos in README.md

  • Corrected minor typos in the README.md file. (#287)

• Improvement/privacy issues

  • Restricted access to sensitive metadata from the former Drupal platform (now "ai4europe_cms"):
    • Access to name, given_name, and surname fields in the persons entity and email field in contacts restricted to users with "full_view_drupal_resources" permission.
    • Email field access in contacts for other platforms now requires authentication. (#298)

• Bugfix Elastic Search

  • Bugfix ensuring the correct search for assets of specific platforms using the platforms field. (#323)

• Bugfix huggingface validator

  • Replaced the buggy validator implementation with huggingface_hub.utils.validate_repo_id as per documentation. (#330)

Updated

• Specify the references require identifiers of contact details

  • Clarified the intended usage of references, specifying the need for identifiers of contact details. (#297)

• Rename drupal to ai4europe_cms

  • Renamed all instances of "drupal" to "ai4europe_cms" for better platform representation. (#312)

Notes

• Verification of the develop and release triggers and flow setup is pending and will be tested on a fork before finalizing the merge.
• Consideration for merging build and test jobs back to potentially speed up the Docker workflow process.

Release 1.3.20240308 - Backup scripts and bugfixes

Changelog

The format is loosely based on Keep a Changelog,
and this project adheres to Semantic Versioning, except that the date of the release is used as the patch version.

Added

• Automated Backup and Restoration System: Implemented automated incremental backup and restoration of any data related to the metadata catalogue (e.g., MySQL DB, Keycloak, connector states/logs, etc) (PR #275).

Fixed

• OpenML Connector Issue #266: Fixed the issue on PR #270 to ensure that the connector correctly detects the last asset uploaded to OpenML, preventing the offset of its state from increasing on subsequent runs. This ensures synchronization without skipping any assets.
• OpenML Connector empty distribution: Fixed on PR #270 to make sure that the connector correctly returns a distribution list when all the three fields: dependencies, installation_notes, and binary_url are filled.
• Zenodo Rate Limits Issue #267: Resolved on PR #272 to prevent skipping assets due to improper error handling. This includes:

1. Preventing connector to exceed harvesting rate limits;
2. Preventing connector to exceed the resumption token expiration time.
3. Avoiding connector to incorrectly raise an error on missing fields.

• Zenodo Uploader Error Message: Fixed truncation of error messages due to line breaks on PR #268.
• Hugging Face Connector Logging Operator: Updated the operator in connectors/huggingface/datasets.sh to prevent deletion of logs when the Docker container is restarted, fixed on PR #268.

Changed

• Connector Log Version Tags: Added version tags to the connector.log files to facilitate debugging, as implemented in PR #268.

Release 1.3.20240129 - Keycloak persistent and new features

Changelog

The format is loosely based on Keep a Changelog,
and this project adheres to Semantic Versioning, except that the date of the release is used as patch version.

Added

• OpenML MLModels Connector: Allows fetching metadata of ML models from openml to AIoD platform (#202).
• Zenodo Uploader: This allows data files (content) to be uploaded to Zenodo with the AIoD API endpoint .../upload/datasets/{identifier}/zenodo (#214).

Fixed

• Bugfix: Make sure the API redirects the response to download the content of a dataset to the client. This way, the download will be handled on the client side instead of downloading on the server side and passing it to the client afterwards. (#237).

Changed

• Huggingface Uploader: Refactored without major functional changes (#214).
• Keycloak state persistent: Makes sure that the Keycloak container saves the state to the filesystem (#244).
• Documentation: Typos fixed on README.md (#258).

1.2.20231219 - Finegrained authorization

Added

• Instead of only a single role (edit_aiod_resources), now roles such as crud_datasets, create_datasets work as well (#232)
• Add releases url to Swagger description (#233)

Fixed

• Bugfix: make sure Contact works when it specifies an organisation (#234)
• Minor fix in documentation (#228)

Changed

• Refactoring of Pydantic Annotations, without functional changes (#230)

Hotfix 1.1.20231205

Fixed

• Bugfix: make sure HuggingFace repository identifier is created correctly for the HuggingFace Uploader (#216)
• Security fix: make the API use credentials to ask Keycloak the token information: (#215)

Changed

• You will now get an error if you try to upload a HuggingFace or OpenML item with an incorrect platform_resource_identifier (platform-specific validation) (#216)

AIoD Metadata Catalogue in production

Changelog

The format is loosely based on Keep a Changelog,
and this project adheres to Semantic Versioning, except that the date of the release is used as patch version.

Added

• ElasticSearch + API search endpoints (#182, #207)
• Soft deletion (delete it after some time, so that logstash can synchronize for elasticsearch) (#166, #180, #184)
• Example scripts: how to authenticate / upload data (#159)
• Endpoints to get content (the actual data) (#175)
• Endpoint to get detailed count for all resources, or for single resource (#197)

Fixed

• Bugfixes for dataset connectors (#186, #191, #204)
• Bugfix for DCAT-AP conversion of datasets (#209)
• Making docker-compose work out-of the box on every machine (longer timeouts: #193, making sure all directories exist with correct permissions: #195, #196)

Changed

• Metadata changes: made contentUrl optional for MLModel distributions (#211)
• Improved the README (#174)
• Improved Swagger documentation (#208, #205, #210)
• Code cleanup (better "injection" of DbSession: #199, new SqlModel version with quite some changes: #200)

Release 1.0.20230911

Changelog

The format is loosely based on Keep a Changelog,
and this project adheres to Semantic Versioning, except that the date of the release is used as patch version.

Added

• Connector framework for synchronization. Implemented for datasets of OpenML, HuggingFace and Zenodo (#100)
• Metadata Model v2.0: inheritance with AIoDConcept with AIoDEntry, AIResource and AIAsset. (#125)
• New resource: Experiments (#125)
• New resource: MLModels (#125)
• New resource: Services (#125)
• New resource: Teams (#136)
• New endpoints: GET enums (e.g. list all licenses or languages) (#144)
• New endpoints: GET resources by resource_identifier, agents by agent_identifier etc. (#144)
• Security improvements: Depend-a-bot. Automatic dependency updates (#86)
• Security improvements: using public client (safer) in Swagger, better token-management, and configurable (#83)
• Development improvements: docker compose, making it easier to run the code (#115, #135, #145, #154)
• Development improvements: automatic tests for Pull Requests (#92, #93, #98, #101, #102, #111)

Fixed

• Security fixes: dependency updates (e.g. #148 and many more)

Changed

• Changed metadata (metadata v2.0): Case Study (#137)
• Changed metadata (metadata v2.0): Computational Assets (previously Computational Resource) (#139)
• Changed metadata (metadata v2.0): Datasets (#125)
• Changed metadata (metadata v2.0): Educational Resources (#140)
• Changed metadata (metadata v2.0): Event (#141)
• Changed metadata (metadata v2.0): News (#142)
• Changed metadata (metadata v2.0): Organisations (#125)
• Changed metadata (metadata v2.0): Persons (#125, #136)
• Changed metadata (metadata v2.0): Projects (#142)
• Changed metadata (metadata v2.0): Publications (#125)
• Fix after metadata change: Dataset Schema converters (DCAT-AP, schema.org) (#132)
• Fix after metadata change: Huggingface Data Uploader (#133)
• Fix after metadata change: Platform and platform identifier placed on each resource (#134)
• Fix after metadata change: Examples that fill the database (#138)
• Bumping versions to v1 in the API (#130)