[Feature] Pass Through Columns for all joined tables in Transaction Details

[samw430]

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

The transaction_details table is built by joining together a number of source and intermediate tables. Some of these tables allow pass through columns that can be configured from the dbt_project.yml. Others do not. For those that don't you need to join the table back onto the transaction_details output and grab additional columns. This is computationally intensive for something that has a well supported pattern.

Can we add table pass through columns for all tables joined into transaction_details?

Describe alternatives you've considered

As described above the workaround is to rejoin tables to the output transaction_details

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

[fivetran-joemarkiewicz]

Hi @samw430 thanks so much for opening this Feature Request!

Based on your description it seems you are wanting to allow more flexibility for additional passthrough columns for all joined tables in the transaction_details model. I can definitely see how to computation behind additional post model joins would be unnecessarily duplicative. If we were to incorporate these feature we would need to include additional macros to pass in the respective fields. Further, we would want to create new variables to enable these macros to function as intended.

I could see this being a good enhancement to the package. I also notice you would be interested in contributing to the package! If you are, please let me know if you have any questions! Otherwise, my team can scope this out in our next quarter to review the implementation of the feature.

[ghost]

Hi! I don't think the problem is a missing macro but just the use inside the stg_ model. I'm experiencing the problem with vendors and items. You have a get_vendors_columns macro but then no {{ fivetran_utils.fill_pass_through_columns('vendors_pass_through_columns') }} is foreseen in stg_netsuite__vendors. Seems more like a bug rather than a FR. Can submit PR on this.

[fivetran-joemarkiewicz]

Hi @ropalloni the get_vendors_columns macro is used to ensure the staging model will have all fields necessary for the downstream transformations. For example, if you don't have the companyname in the vendors source, that macro will pass a null value through in order for the downstream transformations to not fail.

We currently do not have passthrough variables for the vendors or items models. You can see all supported passthrough columns in our README. If you are wanting to add these passthrough columns in a PR I would be happy to review it and look to integrate into the greater package. If so, you may need to make downstream adjustments in the dbt_netsuite models. But if implemented, we would be happy to incorporate these passthrough variables 😃

[rpalloni]

I guess the right place is here https://github.com/fivetran/dbt_netsuite_source/tree/bugfix/passthrough.
Should I commit in this branch or set up a new one? Or fork?

[fivetran-joemarkiewicz]

Hey @rpalloni if you wanted to add these new passthrough variables in a feature PR you would want to fork the package, make the updates in the fork, and the open a PR against our main branch in the package.

You can follow these steps for more details on the PR process.

[rpalloni]

@fivetran-joemarkiewicz done for vendors and items. Did not have time to check other "core" tables. fivetran/dbt_netsuite_source#23

[fivetran-joemarkiewicz]

Thanks for opening that PR @rpalloni! I can have someone on my team pick up the review of this PR in our next sprint (starting tomorrow).

I did have one question around the use of these passthrough columns. Are you planning to leverage them in the dbt_netsuite end models? Or do you only wish to leverage them in the staging models. If you are thinking the former, we may need to make some additional adjustments to bring those fields through to the end models.

[rpalloni]

I'm not using "end" models. Only stg and intermediate. If you could merge tomorrow, would be awesome!

[fivetran-joemarkiewicz]

Thanks so much for the clarification @rpalloni!

Unfortunately we will need to properly test your changes and make sure they are good to go for all users using the package. I would say a realistic timeframe would be early next week for a new release. We will follow up with any clarifying questions in the PR. Thanks again!

[rpalloni]

Hi @fivetran-joemarkiewicz any news on PR fix? Thanks

[fivetran-joemarkiewicz]

Hi @rpalloni apologies for the delay. I have planned to review on Thursday!

[fivetran-joemarkiewicz]

@rpalloni just wanted to past back here and let you know that the PR looked great and I have merged it into our next release branch!

We are planning to cut the next release once the dbt-utils v1.0.0 version is live. That should be sometime in early November.

[ghost]

Any news on this?
fivetran/dbt_netsuite_source#23 (comment)

[fivetran-joemarkiewicz]

Hi @rpalloni we are close to the next release! You can see my more detailed response within Issue #54

#54 (comment)

[fivetran-joemarkiewicz]

Closing out this feature as the requested updates have been rolled out!