Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azurerm_data_factory_pipeline: Support for activities #6224

Merged
merged 9 commits into from
Jun 2, 2020

Conversation

lawrencegripper
Copy link
Contributor

@lawrencegripper lawrencegripper commented Mar 23, 2020

This allows you to define the activities in the data factory pipeline. Would be great to get some thoughts on this approach and feedback so I can polish up and move out of draft. Currently also has the changes from #6195 which I'll rip out once ready for merge.

There is some complexity to this due to the complex setup of objects used to define activities. To work around this I've kept the activity definition as a JSON block.

REST: https://docs.microsoft.com/en-us/rest/api/datafactory/pipelines/createorupdate
SDK Issue: Azure/azure-sdk-for-go#7934 (explains some of the weird serialization stuff I do)

Integration test:
image

@ghost ghost added the size/XL label Mar 23, 2020
@ghost ghost added size/L and removed size/XL labels Mar 30, 2020
@lawrencegripper
Copy link
Contributor Author

I've updated to remove the devcontainer commit from #6195

@lawrencegripper lawrencegripper marked this pull request as ready for review March 30, 2020 10:04
@lawrencegripper
Copy link
Contributor Author

lawrencegripper commented May 7, 2020

If we think this is useful to include let me know and I'll fixup the formatting and update the docs to match the change.

Copy link
Member

@mbfrahry mbfrahry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @lawrencegripper. Thanks for getting this feature written up. It's one I've been wrestling with for awhile but struggled with because of how nested some of these activities could be. I think you've nailed the implementation here though.

To get it over the finish line, let's increase the complexity with some more tests by adding/removing/updating activities and then also updating the docs like you mentioned. I think we'll want an example on the docs page of how to use this since writing out the json could be a tad tricky. It should look similar to this where we have our basic example and then another example underneath to show how activities could look.

@lawrencegripper
Copy link
Contributor Author

@mbfrahry I'm off on leave for a bit as we just had a baby. I can look to pick this up again when back in around 4 weeks or happy for you to finish up while I'm away.

@ghost ghost removed the waiting-response label May 25, 2020
@mbfrahry
Copy link
Member

Heyo! Congrats @lawrencegripper! I'm happy to take a look when things clear up off my plate. Hope you enjoy your time with your baby

@ghost ghost added size/XL and removed size/L labels Jun 1, 2020
@ghost ghost added the documentation label Jun 1, 2020
@mbfrahry mbfrahry changed the title azurerm_data_factory_pipeline: Add ability to define activities azurerm_data_factory_pipeline: Support for activities Jun 2, 2020
Copy link
Member

@mbfrahry mbfrahry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mbfrahry mbfrahry merged commit cfae3b2 into hashicorp:master Jun 2, 2020
mbfrahry added a commit that referenced this pull request Jun 2, 2020
@ghost
Copy link

ghost commented Jul 2, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

@ghost ghost locked and limited conversation to collaborators Jul 2, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants