HDInsightOnDemandLinkedService class errors from multiple runs.

[glstream]

I am testing the use of On Demand clusters with the Python SDK. I can get a Cluster pipeline up and running fine, but when I manually delete the cluster and then re-porvision to test a new pre-action on the cluster I get the error:

{
"errorCode": "2310",
"message": "The cluster not found, ClusterEndpoint=https://sdkrun54bf5f8b-eb52-4b13-ab75-b53a22398a32.azurehdinsight.net/",
"failureType": "UserError",
"target": "spark-sqldb"
}

At the time of this error I have manually deleted the cluster that came from the previous pipeline. Why would an On Demand Linked service be looking for the URI of a previous cluster? If I wait a few hours my .py will execute fine and no longer have this specific error. Please let me know if code samples would make this issue more clear.

[lmazuel]

I feel like your question is more a HDInsight RestAPI / service question then a Python one? You should probably reach out directly to the HDInsight team directly, since this repo doesn't have HDInsight Python SDK :(

[arun1024]

@glstream would it be possible to share your code samples. I am trying to create HDInsight cluster using python sdk and there is not much documentation like Boto.

[hvermis]

This is by design. ADF keeps track of created OnDemand HDIs, and deletes them if they have not been used for the duration of the "time to live" property. If you delete it from outside, ADF will still try to submit your job on the cluster since it's not marked as deleted yet internally. So don't delete HDI on demand clusters, use time to live setting, and ADF will clean them up if they are not used. This is done to save time on bringing up the cluster and let you run multiple jobs one after another without waiting.