0.10.1 bug: incorrect "ValueError: Circular reference detected" with multiple empty dictionaries

[Edward-Knight]

This only affects the latest version (0.10.1) and is fine in previous versions.

Example file that will cause the error (taken from the toml-lang/toml readme):

# [x] you
# [x.y] don't
# [x.y.z] need these
[x.y.z.w] # for this to work

[x] # defining a super-table afterward is ok

The fourth line here ([x.y.z.w]) will cause the new ValueError: Circular reference detected exception in toml.dumps.

Example:

With version 0.10.0:

>>> import toml
>>> toml.dumps({'x': {'y': {'z': {'w': {}}}}})
'[x.y.z.w]\n'
>>> 

With version 0.10.1:

>>> import toml
>>> toml.dumps({'x': {'y': {'z': {'w': {}}}}})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../toml/encoder.py", line 67, in dumps
    raise ValueError("Circular reference detected")
ValueError: Circular reference detected
>>> 

[andreoliwa]

I had a similar problem.
I thought it was an issue with OrderedDict and dots (.) on TOML keys.
But the weird thing is: it works for some cases and fails on very specific cases.

Check this gist: toml raises "ValueError: Circular reference detected" on some specific root keys.

For some reason, the id() function is returning the same value for different dict/OrderedDict objects:

toml/toml/encoder.py

Lines 62 to 67 in 9c9d4be

	outer_objs = [id(o)]
	while sections:
	section_ids = [id(section) for section in sections]
	for outer_obj in outer_objs:
	if outer_obj in section_ids:
	raise ValueError("Circular reference detected")

[ottowayi]

I'm seeing the same issue on Python 3.7 and 3.8. I'm using regulars dicts and just started getting the circular reference error after updating to 0.10.1. I think it has to do with id returning the same value for different objects somehow? What's even weirder is I added some tracking by using a dict of {id(section) : section} then the issue goes away and I can't replicate it the error.

[uiri]

This appears to be a bug in the fix for #217 , which was introduced between 0.10.0 and 0.10.1

My current hypothesis is that the Python runtime creates an empty dictionary, the dictionary gets thrown away and the next time an empty dictionary is "created", the runtime decides to reuse the already created dictionary that was empty and is no longer referenced over allocating a new dictionary.

[misho88]

I'm running into the same issue. I've been working around it, but it would be nice to have a fix.

To that end, I would like to suggest using gc.get_referents() instead of trying to use id() which definitely behaves weirdly on {}, [] and others. Garbage collection with reference cycles is why the gc module exists, so it should be safe to assume the implementation of get_referents() is reliable.

Illustrative example:

from gc import get_referents

def has_cycle(obj, visited=()):
    if obj in visited:
        return True
    return any(
        has_cycle(ref, visited + (obj,))
        for ref in get_referents(obj)
    )

d = { '1': { '2': { '3': { '4': {} } } } }
print(d, has_cycle(d))  # {'1': {'2': {'3': {'4': {}}}}} False
try:
    import toml
    toml.dumps(d)
except ValueError as e:
    print('false cycle detected:', e)  # false cycle detected: Circular reference detected
d['1']['2']['3']['4'] = d
print(d, has_cycle(d))  # {'1': {'2': {'3': {'4': {...}}}}} True

[dirkphilip]

I didn't see this issue yet, but I was running into the same problem and created a pull request that
should fix this issue as lijunzh mentioned. For my project, it would also be great if it could be merged and deployed soon.

[kwint]

Hi @uiri, we encounter this issue quite a bit in our setup, and we are therefore pinned to 0.10.0. But now that other dependencies (like black) are requiring toml 0.10.1 this is starting to hold us back a bit.

Do you have time to make a new release of toml in the near future?