[bug] changes for characteristics in shunt/trafo/trafo3w are not considered in stored nets and convert_format()

[SteffenMeinecke]

Bug report checklis

• Searched the issues page for similar reports

• Read the relevant sections of the documentation

• Browse the tutorials and tests for usefull code snippets and examples of use

• Reproduced the issue after updating with pip install --upgrade pandapower (or git pull)

• Tried basic troubleshooting (if a bug/error) like restarting the interpreter and checking the pythonpath

Reproducible Example

• the reproducible example is provided by an added test in Fix/wrong element columns in json files #2533

Issue Description and Traceback

If networks stored as (json) files were already updated after the last release, the format_version was updated, too. If further updates are made but not integrated to the (json) files, the loaded networks will not include the latest updates, even if these updates would usually be catched by the convert_format() function. This is because convert_format() won't do corrections to loaded data of the latest format_version.

1. Bug; At the moment, the changes for characteristics in shunt, trafo, and trafo3w (provided by add trafo/shunt characteristics and phase shifter logics #2490) are such changes, not integrated into stored (json) files. To show up, I added test_stored_networks.py.
   To fix the added, failing tests, the networks must be stored again including the latest updates.

@heckstrahler @mrifraunhofer @pawellytaev who is going to do the fixes?

2. Bug: The changes for characteristics are not catched by the convert_format() function. The following copy of the relevant code snippet shows that nothing is fitting included:

if Version(str(net.format_version)) < Version("3.0.0"):
    _convert_geo_data(net, elements_to_deserialize)
    _convert_group_element_index(net)
    _convert_trafo_controller_parameter_names(net)
    convert_trafo_pst_logic(net)

@heckstrahler @mrifraunhofer @pawellytaev who is going to fix this?

Expected Behavior

• all mandatory element columns exist in all networks that can be loaded from pandapower.networks

Installed Versions

INSTALLED VERSIONS

commit : 0691c5cf90477d3503834d983f69350f250a6ff7
python : 3.10.11
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : AMD64 Family 25 Model 80 Stepping 0, AuthenticAMD
byteorder : little
LC_ALL : None
LANG : de_DE.UTF-8
LOCALE : de_DE.cp1252

pandas : 2.2.3
numpy : 1.26.0
pytz : 2024.1
dateutil : 2.9.0
pip : 25.0
Cython : 0.29.33
sphinx : 7.3.7
IPython : 8.13.2
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.4
lxml.etree : 5.2.1
matplotlib : 3.9.1
numba : 0.60.0
numexpr : 2.10.2
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
psycopg2 : 2.9.9
pymysql : None
pyarrow : 16.1.0
pyreadstat : None
pytest : 8.2.1
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.15.1
sqlalchemy : None
tables : 3.9.2
tabulate : 0.9.0
xarray : None
xlrd : 2.0.1
xlsxwriter : 3.2.0
zstandard : 0.23.0
tzdata : 2024.1
qtpy : 2.4.1
pyqt5 : None

Label

• Relevant labels are selected

[pawellytaev]

@hilbrich @panos-xenos @JakobKirschner could you have a look into this?

[panos-xenos]

@pawellytaev @SteffenMeinecke the missing columns are only optional, so the networks coming from the json files should work without problems. A user may create a net manually without specifying all these columns. Same with the standard trafo types for example. Is this actually a problem?

cc: @hilbrich

EDIT: Speaking to Thomas Fabian today, I now realize that the issue is that the json files were re-created with v3.0, hence the logic doesn't go into the convert_format functions. I suggest we set up a short call for tomorrow (Feb 21) to align on the problem and potential solutions.

[panos-xenos]

@vogt31337, as discussed, please see attached mv_oberrhein.json file after going through the convert_format function (which includes the convert_trafo_pst_logic function). Let me know if you need anything else.

mv_oberrhein.json

[SteffenMeinecke]

Dear @panos-xenos,

I now realize that the issue is that the json files were re-created with v3.0, hence the logic doesn't go into the convert_format functions.

right, this is the main issue. Despite the characteristic columns are optional, I'd say that it is best to convert (by convert_format) the information of 2.14.x version, such as net.trafo.vk_percent_characteristic, into the new characteristic parameter columns. (My suggestion - without knowing about what you implemented - is that the characteristic models can be somehow converted into each other).

[hilbrich]

Hi @SteffenMeinecke ,

Dear @panos-xenos,

I now realize that the issue is that the json files were re-created with v3.0, hence the logic doesn't go into the convert_format functions.

right, this is the main issue. Despite the characteristic columns are optional, I'd say that it is best to convert (by convert_format) the information of 2.14.x version, such as net.trafo.vk_percent_characteristic, into the new characteristic parameter columns. (My suggestion - without knowing about what you implemented - is that the characteristic models can be somehow converted into each other).

That is possible, but I see an issue. Are there really testfiles that contain vk_percent_characteristics? The problem that I see is, that this only reflects tap_dependent_impedances, and we would also need to create the information about ratio and shift for every tap position. This is, because in 2.14.x, only tapdependent impedance was considered, but no tables.

[SteffenMeinecke]

Are there really testfiles that contain vk_percent_characteristics?

I guess, there was nobody who added a testfile for that. But I didn't get why this might be an issue in adding a function to convert_format(). But I am not so deep into it as you probably.

[SteffenMeinecke]

I think, it could be a good solution to newly convert the IEEE cases, since the from_ppc() has been adjusted/fixed within the last year.
We nevertheless need to reconvert and store the other json files, e.g. oberrhein...

[SteffenMeinecke]

In reconvert_power_system_test_case_jsons, I started to reconvert the power system test cases the originates from matpower cases.
json files which must be restored with the new trafo/shunt/trafo3w parameters are:

"case11_iwamoto",
"mv_oberrhein",
"lv_schutterwald",
"ieee_european_lv_asymmetric"