Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accessing nested values in import is non-ideal #2474

Closed
icanhazpython opened this issue Feb 9, 2022 · 4 comments
Closed

Accessing nested values in import is non-ideal #2474

icanhazpython opened this issue Feb 9, 2022 · 4 comments
Assignees
Milestone

Comments

@icanhazpython
Copy link

So I am importing via a JSON source. For some record "columns", nested objects are returned and I can access them within the import source and sync rule with dot-notation. Great! However, with my import source, these items can occasionally come back without any object at all, or as null. When this occurs, the below-linked error is triggered. I can't seem to figure a way around this other than building a custom importer. I feel that it would be more beneficial to the community however if this functionality was "fixed" so that it handled this situation more gracefully. Here are two possible proposals for fixing this:

  • Add a "filter" expression option to ALL import source modifiers (like we have on the sync rules). That way I could simply name the field in the filter, and it would only attempt to access the nested property if the field is not null.
  • Add an option to ALL import source modifiers that allows the user to decide how any error that occurs on a specific field is handled - the "decode a JSON string" import source modifier has an option like this, but I would propose having this option available universally on all modifier types, and additionally adding an option to set a default value other than null.

I think either of the above options (or even both!) would really benefit other users. If there is some other workaround or option I missed, please let me know!

'Data is not nested, cannot access %s: %s',

@icanhazpython
Copy link
Author

Upon further review, I realize that neither of my proposed fixes above would work, as the value access is attempted before any import source modifiers are applied.

I've submitted a PR with the simplest fix first: simply return null if the value is not a nested object.
#2584

Another idea I have to fix this:
Create an import source modifier that can extract the nested value, with choices for how to handle an error. The extracted value can then be placed into another variable for further processing if need be.

@icanhazpython
Copy link
Author

I submitted a second PR for my second fix option: #2585
This fix adds an import source modifier to access nested object properties with configurable error condition

@Thomas-Gelf
Copy link
Contributor

@icanhazpython: I have no better solution right now, this would require a new config flag at sync property level. Please have a look at the Combine Property Modifier, it should provide what your second pull request does

@Thomas-Gelf Thomas-Gelf self-assigned this Nov 3, 2022
@Thomas-Gelf Thomas-Gelf added this to the v1.10.2 milestone Nov 3, 2022
@Thomas-Gelf
Copy link
Contributor

Didn't like the idea of having a flag for this. I changed the default behavior, so for your use-case it should behave as expected out of the box. The following variants are now possible when accessing app.name:

  • { "app": null }: should shipnull
  • { "no app": "something" }: should ship null
  • { "app": "a string" }: should fail
  • { "app": false }: should fail
  • { "app": ["something"] }: should fail
  • { "app": { "name": "app" }: should ship "app"

Thomas-Gelf added a commit that referenced this issue Nov 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants