Support amda template params #192

brenard-irap · 2025-01-22T16:05:50Z

This PR fixes issue #163

Now if you get data for "amda/jade_l5e180_def" parameters, an exception is raised to indicate that an "additional arguments" is needed.

data = spz.get_data("amda/jade_l5e180_def", "2021-01-01", "2021-01-02")

speasy.core.impex.exceptions.MissingTemplateArgs: Parameter jade_l5e180_def requires additional arguments to be used:
{
  "lookdir": {
    "name": "Look Direction",
    "type": "list",
    "default": "0",
    "items_list": {
      "0": {
        "name": "Look Dir. 0"
      },
      "1": {
        "name": "Look Dir. 1"
      },
      "2": {
        "name": "Look Dir. 2"
      },
      "3": {
        "name": "Look Dir. 3"
      },
      "4": {
        "name": "Look Dir. 4"
      },
      "5": {
        "name": "Look Dir. 5"
      },
      "6": {
        "name": "Look Dir. 6"
      },
      "7": {
        "name": "Look Dir. 7"
      },
      "8": {
        "name": "Look Dir. 8"
      },
      "9": {
        "name": "Look Dir. 9"
      },
      "10": {
        "name": "Look Dir. 10"
      },
      "11": {
        "name": "Look Dir. 11"
      },
      "12": {
        "name": "Look Dir. 12"
      },
      "13": {
        "name": "Look Dir. 13"
      },
      "14": {
        "name": "Look Dir. 14"
      },
      "15": {
        "name": "Look Dir. 15"
      },
      "16": {
        "name": "Look Dir. 16"
      },
      "17": {
        "name": "Look Dir. 17"
      },
      "18": {
        "name": "Look Dir. 18"
      },
      "19": {
        "name": "Look Dir. 19"
      },
      "20": {
        "name": "Look Dir. 20"
      },
      "21": {
        "name": "Look Dir. 21"
      },
      "22": {
        "name": "Look Dir. 22"
      },
      "23": {
        "name": "Look Dir. 23"
      },
      "24": {
        "name": "Look Dir. 24"
      },
      "25": {
        "name": "Look Dir. 25"
      },
      "26": {
        "name": "Look Dir. 26"
      },
      "27": {
        "name": "Look Dir. 27"
      },
      "28": {
        "name": "Look Dir. 28"
      },
      "29": {
        "name": "Look Dir. 29"
      },
      "30": {
        "name": "Look Dir. 30"
      },
      "31": {
        "name": "Look Dir. 31"
      },
      "32": {
        "name": "Look Dir. 32"
      },
      "33": {
        "name": "Look Dir. 33"
      },
      "34": {
        "name": "Look Dir. 34"
      },
      "35": {
        "name": "Look Dir. 35"
      },
      "36": {
        "name": "Look Dir. 36"
      },
      "37": {
        "name": "Look Dir. 37"
      },
      "38": {
        "name": "Look Dir. 38"
      },
      "39": {
        "name": "Look Dir. 39"
      },
      "40": {
        "name": "Look Dir. 40"
      },
      "41": {
        "name": "Look Dir. 41"
      },
      "42": {
        "name": "Look Dir. 42"
      },
      "43": {
        "name": "Look Dir. 43"
      },
      "44": {
        "name": "Look Dir. 44"
      },
      "45": {
        "name": "Look Dir. 45"
      },
      "46": {
        "name": "Look Dir. 46"
      }
    }
  }
}
For example:
===========================================================================
    data = spz.get_data(PRODUCT, START, STOP, additional_arguments={"lookdir": "0"})
===========================================================================

This tells you that the additional ‘lookdir’ argument is required to get the data for this parameter. For exemple:

data_lookdir_0 = spz.amda.get_parameter("jade_l5e180_def", "2021-01-01", "2021-01-02", additional_arguments={"lookdir": "0"})

data_lookdir_1 = spz.amda.get_parameter("jade_l5e180_def", "2021-01-01", "2021-01-02", additional_arguments={"lookdir": "1"})

And the same thing is available for access to VEX/IMA/extraparams data (and for all AMDA templated parameters):

data = spz.get_data(spz.inventories.tree.amda.Parameters.VEX.IMA.vex_ima_extra.vex_h_spec, "2012-01-01", "2012-01-02")

speasy.core.impex.exceptions.MissingTemplateArgs: Parameter vex_h_spec requires additional arguments to be used:
{
  "anode": {
    "name": "Anode",
    "type": "list",
    "default": "0",
    "items_list": {
      "0": {
        "name": "Anode 0"
      },
      "1": {
        "name": "Anode 1"
      },
      "2": {
        "name": "Anode 2"
      },
      "3": {
        "name": "Anode 3"
      },
      "4": {
        "name": "Anode 4"
      },
      "5": {
        "name": "Anode 5"
      },
      "6": {
        "name": "Anode 6"
      },
      "7": {
        "name": "Anode 7"
      },
      "8": {
        "name": "Anode 8"
      },
      "9": {
        "name": "Anode 9"
      },
      "10": {
        "name": "Anode 10"
      },
      "11": {
        "name": "Anode 11"
      },
      "12": {
        "name": "Anode 12"
      },
      "13": {
        "name": "Anode 13"
      },
      "14": {
        "name": "Anode 14"
      },
      "15": {
        "name": "Anode 15"
      }
    }
  }
}
For example:
===========================================================================
    data = spz.get_data(PRODUCT, START, STOP, additional_arguments={"anode": "0"})
===========================================================================

You just need to select the anode you want:

data_anode_0 = spz.get_data(spz.inventories.tree.amda.Parameters.VEX.IMA.vex_ima_extra.vex_h_spec, "2012-01-01", "2012-01-02", additional_arguments={"anode": "0"})
data_anode_1 = spz.get_data(spz.inventories.tree.amda.Parameters.VEX.IMA.vex_ima_extra.vex_h_spec, "2012-01-01", "2012-01-02", additional_arguments={"anode": "1"})

…try to use a templated parameter

speasy/core/impex/__init__.py

speasy/core/inventory/indexes.py

brenard-irap

I'm doing some more tests, but I'm already posting this first part of my review.

brenard-irap · 2025-02-07T13:10:15Z

speasy/webservices/amda/ws.py

+            warnings.warn(f"""Argument {arg.key} is not provided, using default value {v}
+You can set Derived Parameters inputs using:
+spz.get_data("amda/{product.spz_uid()}, start_time, stop_time, product_inputs={{'{k}': '{arg.default}' }})
+""", category=RuntimeWarning, skip_file_prefixes=(os.path.dirname(spz.__file__),))


It seems that skip_file_prefixes is only available since Python 3.12

brenard-irap · 2025-02-14T09:23:58Z

tests/test_speasy.py

+            "start_time": '2023-01-04T07:51',
+            "stop_time": '2023-01-04T07:52',
+            "disable_proxy": True,
+            "disable_cache": True,


"disable_cache": True is already set in test_get_data.
I need to remove it here to prevent "TypeError: speasy.core.requests_scheduling.request_dispatch.get_data() got multiple values for keyword argument 'disable_cache'" exception.

brenard-irap · 2025-02-14T09:26:42Z

tests/test_speasy.py

+            "disable_proxy": True,
+            "disable_cache": True,
+            "product_inputs": {
+                'lookdir': "100"


Lookdir 100 does not exist.
I replace it by Lookdir 1.

To sum up, here's how I modified the test:

{ "product": 'amda/jedi_i90_flux', "start_time": '2023-01-04T07:51', "stop_time": '2023-01-04T07:52', "disable_proxy": True, "product_inputs": { 'lookdir': "1" } }

brenard-irap · 2025-02-14T09:35:24Z

speasy/core/inventory/indexes.py

+    def __iter__(self):
+        return self._arguments.__iter__()
+
+class DerivedParameterIndex(ParameterIndex):


Derived parameter is a concept already used by AMDA for something else.
I suggest renaming it ‘ExtraParameterIndex’ or something like that

brenard-irap · 2025-02-14T10:38:47Z

Apart from the few points mentioned above, everything seems to be working perfectly.

- Uses dedicated indexes for arguments - allow per provider min proxy version - warns when using default values instead of error Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

sonarqubecloud · 2025-02-21T14:13:45Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

brenard-irap · 2025-02-21T15:10:18Z

In my opinion, the latest push is perfect!

brenard31 added 6 commits January 22, 2025 15:16

WIP

de2fa57

Propagate additional_arguments to proxy + add info message when user …

2f57248

…try to use a templated parameter

Add tests for AMDA templated parameters

9f5b40a

Add tests + fix for predefined templated parameters

422ec86

Parse parameter arguments as SpeasyIndex + Fix tests

735e61c

Fix obsdatatree.xml used for tests

a8520f8

jeandet self-requested a review January 23, 2025 10:13

github-advanced-security bot found potential problems Feb 6, 2025

View reviewed changes

speasy/core/impex/__init__.py Fixed Show fixed Hide fixed

speasy/core/inventory/indexes.py Fixed Show fixed Hide fixed

jeandet force-pushed the support-amda-template-params branch from c62b112 to 6cea08f Compare February 6, 2025 10:28

brenard-irap commented Feb 14, 2025

View reviewed changes

Some derived parameter generalisation

463f475

- Uses dedicated indexes for arguments - allow per provider min proxy version - warns when using default values instead of error Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

jeandet force-pushed the support-amda-template-params branch from 6cea08f to 463f475 Compare February 21, 2025 14:13

jeandet removed their request for review February 21, 2025 16:12

jeandet merged commit 4d10378 into SciQLop:main Feb 21, 2025
18 of 21 checks passed

jeandet mentioned this pull request Feb 21, 2025

[AMDA] Support AMDA "templated parameters" #163

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support amda template params #192

Support amda template params #192

brenard-irap commented Jan 22, 2025

brenard-irap left a comment

brenard-irap Feb 7, 2025

brenard-irap Feb 14, 2025

brenard-irap Feb 14, 2025

brenard-irap Feb 14, 2025

brenard-irap commented Feb 14, 2025

sonarqubecloud bot commented Feb 21, 2025

brenard-irap commented Feb 21, 2025

Support amda template params #192

Support amda template params #192

Conversation

brenard-irap commented Jan 22, 2025

brenard-irap left a comment

Choose a reason for hiding this comment

brenard-irap Feb 7, 2025

Choose a reason for hiding this comment

brenard-irap Feb 14, 2025

Choose a reason for hiding this comment

brenard-irap Feb 14, 2025

Choose a reason for hiding this comment

brenard-irap Feb 14, 2025

Choose a reason for hiding this comment

brenard-irap commented Feb 14, 2025

sonarqubecloud bot commented Feb 21, 2025

Quality Gate passed

brenard-irap commented Feb 21, 2025