Solr 9 error while trying to push to Datastore. Missing required field: site_id

[dpuertolas]

CKAN version

CKAN version: 2.10.4
Solr version: I have tried with version 2.10-solr9 and by modifying the schema.xml file from a Solr 9.6.1 image.
Xloader version: 1.0.1

Describe the bug

I am trying to update the Solr version used in my CKAN installation (from Solr 6 to Solr 9) after upgrading CKAN from 2.9 to 2.10.
I reindex, and the datasets load correctly, but when attempting to upload resources to the Datastore, I receive the following error:

Error: Unable to add package to search index: Solr returned an error: Solr responded with an error (HTTP 400): [Reason: [doc=null] missing required field: site_id]

I have found this error in two unresolved issues that have been open for over a year:

• CKAN - Custom plugin/theme error datastore using fluent presets #7750
• Solr error: missing required field #7730

They describe a similar behavior to mine.

Here is my list of plugins:

ckan.plugins = stats resource_proxy text_view image_view webpage_view datatables_view geo_view geojson_view wmts_view shp_view pdf_view datastore xloader harvest ckan_harvester csw_harvester gobierno_de_navarra_xlsx_harvest gobierno_de_navarra_json_harvest gobierno_de_navarra_csw_harvest base_templates scheming_datasets scheming_groups scheming_organizations fluent spatial_metadata spatial_query tableau_view datos_gob_es_federation gobierno_de_navarra activity

My portal is multilingual, so we use Fluent presets, although we are not using 'repeating_subfields' (as mentioned in some of the issues I referenced above).

I have checked the JSON that is sent to Solr before the error occurs, and it looks correct, containing the 'site_id' and the other fields.
When I tried modifying the Solr schema to make the 'site_id' field not required, I received the same error, but this time for the 'id' field.

[dpuertolas]

Solved:

Due to the multi-language feature, there were fields of type dict in the data dictionary when uploading to the datastore. To solve this, in my plugin I developed a function that transforms these fields into strings:

def _before_index_dump_dicts(self, pkg_dict):
    for key, value in pkg_dict.items():
        if isinstance(value, dict):
            pkg_dict[key] = json.dumps(value)
    return pkg_dict

I execute this function in the before_dataset_index method from the ckan.plugins.interfaces.IPackageController interface.

[amercader]

Glad you find the issue.
For anyone finding this error in the future, make sure that in all plugins you have loaded the before_dataset_index() (before_index() in old CKAN versions) return a valid dataset dict, and that the values of the fields can be indexed by Solr