pynetbox greater than 7.0.0 fails to insert complex custom_fields

[costasd]

pynetbox version

v7.2.0

NetBox version

v3.5.9

Python version

3.11

Steps to Reproduce

Hello,

in our setup we introduced a complex custom_field in each device that is of the following form, in python notation:

{
  "metadata_config" : [
    {"a" : "b", "c": "d"},
    {"a":  "e", "c": "k"}
  ]
}

I found out that any version over 7.0.0, essentially anything containing the #518 fix that rewrote flatten_custom, is breaking updating this custom_field via pynetbox. update() returns fine, but all dicts are set as None into netbox.

The issue seems to lie with a cornercase in flatten_custom.
Below you can find a way to test this with 7.0.0, 7.2.0 and a candidate fix that I'll open a PR with.

Thanks!

to run it, just execute ./run.sh, after writing adding these two files into some directory.

filename: run.sh

#!/bin/bash

pip install pytest

echo "v7.0.0 works"
pip install pynetbox==7.0.0
pytest

echo "v7.2.0 fails"
pip install pynetbox==7.2.0
pytest

echo "candidate fix works"
pip install git+https://github.com/costasd/pynetbox@6fe8494f74cbe7779c98cb949af2ee8d95e74d0e
pytest

filename: test_custom_fields.py

import pynetbox

class TestComplexCustomFields:
    def test_delete_custom_field(self):
        nb = pynetbox.api('http://localhost:8080', token='0123456789abcdef0123456789abcdef01234567')
        field = nb.extras.custom_fields.get(name="metadata_config")
        nb.extras.custom_fields.delete([field])

    def test_create_custom_field(self):
        nb = pynetbox.api('http://localhost:8080', token='0123456789abcdef0123456789abcdef01234567')
        field = nb.extras.custom_fields.create(
            name="metadata_config",
            content_types=["dcim.device"],
            type="json",
            description="Format: [{\"key1\": \"value1\", \"key2\": \"value2\"}]",
            data_type="object",
            default=[])
        assert field

    def test_complex_structures_to_custom_field(self):
        nb = pynetbox.api('http://localhost:8080', token='0123456789abcdef0123456789abcdef01234567')

        device = nb.dcim.devices.get(1)
        dev_id = device.id

        # A list containing a dict
        data = [
            [],
            [{"a": "b"}],
            [{"a": "b", "c": "d"}],
            [{"a": "b", "c": "d"}, {"a": "e", "c": "k"}],
        ]
        for d in data:
            assert device.update({"custom_fields": {"metadata_config": d}})

            device = nb.dcim.devices.get(id=dev_id)
            assert device.custom_fields['metadata_config'] == d

Expected Behavior

Netbox should contain the structure we push.
ie. for:

{
  "metadata_config" : [
    {"a" : "b", "c": "d"},
    {"a":  "e", "c": "k"}
  ]
}

we should get the same back, either by browsing netbox or querying the relevant custom_field through pynetbox

Observed Behavior

Instead, the following gets stored into netbox and gets retrieved:

{
  "metadata_config" : [
    None,
    None
  ]
}

[squintfox]

@abhi1693 @arthanson Any chance of a merge or a comment here? I just lost a few hours of debugging eventually figuring out that pynetbox simply wasn't doing what I was asking it to do. It's specifically impactful for custom fields of type JSON.

[arthanson]

Closed as this has been merged.