Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New decorator/context manager to catch exceptions when not in strict mode. #3531

Merged
merged 29 commits into from
Sep 30, 2023

Conversation

jleibs
Copy link
Member

@jleibs jleibs commented Sep 29, 2023

What

Resolves: #3513

The permutations and indirections in our new API style made this fairly involved.

In an effort to be as-forgiving-as-possible I wanted to be able to handle exceptions down multiple levels in the construction hierarchy. For example:

rr.log("points", rr.Points3D(positions, colors='RED'))

Should produce an error:

my_points3d.py:12: RerunWarning: ColorBatch: ValueError(expected sequence of length of 3 or 4, received 1)

But still successfully log positions; just not colors.

However, if something else goes wrong in that constructor, or the things using it, the next level up in the stack, for example log should catch the error instead. In this case no data will be sent, but we'll still get a warning and not crash the program.

To handle this I introduced a hybrid decorator / context manager: catch_and_log_exceptions which tries to behave as sanely as possible in the context of a situation where we might recursively call multiple levels of functions, any one of which might produce an exception we would like to handle.

The basic idea is that the deepest context manager to see the error will handle it and produce the message. This means we can add the catch_and_log_exceptions decorator or context manager anywhere we know that we can try to recover sanely. We'll get an error produced from that location, but the code higher up in the stack will still run.

To improve the user experience, errors are appended to a warning list as they are encountered, and finally we output them when exiting the top-most context manager. This means we get warning lines that point to the user call-site rather than some arbitrary location within the guts of rerun SDK.

One irritation I discovered is that decorating __init__ methods causes two major issues. For one, it leaves the object in a potentially uninitialized state, just waiting to cascade into more exceptions in future. Additionally, it wreaks havoc on the type-checker. To accommodate this,__init__ functions follow a pattern where they use the context-manager directly and then fallback on a hopefully safer code-path.

The Archetype constructors now generally call __attr_clear__ which is a new helper function designed to call __attrs_init__ with None for all the known params.

I also code-gen a _clear helper to use as a similar fallback path in builder functions.

Recommend reviewing with:
image

Since a number of big snippets got indented when adding the context managers.

Checklist

@jleibs jleibs added the 🐍 Python API Python logging API label Sep 29, 2023
@jleibs jleibs changed the title Jleibs/swallow exceptions New decorator/context manager to catch exceptions when not in strict mode. Sep 29, 2023
@jleibs jleibs marked this pull request as ready for review September 29, 2023 04:10
colors=None,
labels=None,
class_ids=None,
instance_keys=None,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does one really have to spell out all the components here? That seems error prone, especially when adding new components.

Cannot we call the auto-generate a Arrows3_clear() method here instead?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this was annoying me too -- I think I have an idea for a better pattern.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, this is now replaced by a generated __attrs_clear__()

return Asset3D(blob=blob, media_type=media_type)
with catch_and_log_exceptions(context="Asset3D.from_file"):
return Asset3D(blob=blob, media_type=media_type)
return Asset3D._clear()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you use _clear(), but not everywhere. I'm confused

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_clear() currently returns a new Asset3D(), whereas the other functions (inside of an existing __init__) need to modify the current object to put it into a cleared state.

)

@classmethod
def _clear(cls) -> TextLog:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The clear-method is basically what we discussed in #3381 - maybe we should just make it public (call it clear)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's close, but I was worried about situations where it might still fail. It's still sort of a "try_clear_as_fallback".

rerun_py/rerun_sdk/rerun/error_utils.py Outdated Show resolved Hide resolved
# TODO(jleibs): Context/stack should be its own component.
context_descriptor = _build_warning_context_string(skip_first=depth_to_user_code + 1)

log("rerun", TextLog(text=f"{message}\n{context_descriptor}", level="WARN"), recording=recording)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should document this special entity name rerun somewhere

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jleibs jleibs merged commit d4041b4 into main Sep 30, 2023
@jleibs jleibs deleted the jleibs/swallow_exceptions branch September 30, 2023 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

The Python SDK should never raise an exception
2 participants