Expose RequestOptions for HLC methods causing noisy logs/tracing

[tg-msft]

The upcoming Azure.RequestOptions gives us the chance to expose ErrorOptions from our convenience methods. We're not planning to do this everywhere, but there have been a handful of cases where the noise of an expected exception degraded the observability/diagnosability experience for customers. This was discussed in detail in the ArchBoard meetings at Azure/azure-sdk#3391. This work item tracks enabling the first set of experiences for HLC libraries.

Foundational

• Add RequestOptions - [LLC] Move RequestOptions into Core autorest.csharp#1666
• Support RequestOptions via the AutoRest's HLC generator - Enable select HLC methods to use RequestOptions autorest.csharp#1734

Use internally for both sync/async overloads:

• (Azure.Storage.Blobs) Blob Download - [Feature Request] Consider allowing the ability to opt-out of distributed tracing for expected HTTP error responses #9934
• (Azure.Storage.Blobs) Blob Exists - [BUG] BlobBaseClient.Exists always logs exceptions to App Insights when the blob does not exist. #18592

Expose publicly both sync/async overloads:

• (Microsoft.Azure.WebJobs.Extensions.Storage) Receipt Manager - [BUG] Lease already present after upgrading to Microsoft.Azure.WebJobs.Extensions.Storage 5.0.0-beta.3 #21511

Could either expose publicly or add an If(Not)Exists method to use internally for both sync/async overloads:

• (Azure.Search.Documents) Get Document - [FEATURE REQ] Consider a GetDocumentIfExists method for Search #17445
• (Azure.Security.KeyVault.*) Delete - Delete throws error if secret doesn't exist #25125
• (Azure.Data.Tables) CreateIfNotExists, CreateTableIfNotExists, GetEntitiy - [TABLES] GetEntity/Async should not throw exception on 404 #16251
• (Azure.Data.AppConfiguration) Create - [FEATURE REQ] AppConfiguration ConfigurationClient is missing CreateIfNotExists API #14802 (comment)

[longtimedeveloper]

@tg-msft can you please add both 'Azure.Data.Tables.GetEntity' and 'Azure.Data.Tables.GetEntityAsync'?

Thank you!!

[tg-msft]

All of our sync/async APIs are required to have parity as validated by our analyzers, but I added a comment about it applying to both sync/async methods of all the affected APIs for good measure.

[pakrym]

I think some of these don't need a RequestOptions overload.

For example, in #18592 the fix is fully internal. The method needs to tell the pipeline that 404 is not an error in all cases.

[tg-msft]

Yes @pakrym, this was meant to track everything blocked on RequestOptions, but I just split it into three lists to make it clearer which would expose RequestOptions, which would merely make use of it, and which we could do either way.

[annelo-msft]

@pakrym, we are exploring how to address this without exposing RequestOptions in HLC public APIs.

[annelo-msft]

The Azure.Core and Generator work for this have been completed.

[heaths]

@annelo-msft I talked with @ShivangiReja about generating internal DPG protocol methods to add to our existing KV HLC, but it doesn't sound like quite what I want (we don't generate the core libs currently). Can you tell me which of the "new DPG" SDKs is about to ship with this I could use as an example of the expected signature to add manually?

We want to add a couple methods to each of the core KV clients that allow customers to, say, fetch a secret without throwing an exception on a 404 without adding an explicit method to do just that. IIRC, it was @KrzysztofCwalina or @tg-msft that recommended the RequestContext (or RequestOptions) over that.

[annelo-msft]

@heaths could you share links to the REST endpoints of the methods you'd like to add? Protocol method signatures are a fairly straightforward conversion from REST endpoint to protocol method, happy to share info on that. Or, for HLC service methods, you can add a RequestContext parameter to an overload of an existing method.

In either case, you will need to hand-write the code to implement using it in the method implementation. The WebPubSub API has GA'ed already, so WebPubSubServiceClient is an example implementation you can follow.

One thing to note is that we are still developing the DPG generator. If your library isn't generated, then you won't get automatic updates when we regenerate the libraries in the repo.

[heaths]

Since we have a thread going on internally that seems to have progressed a bit further, let's continue this conversation there. 🙂

[Archomeda]

Is there any target date for this? I keep hitting a lot of warning logs for various methods like CreateIfNotExists, DeleteIfExists, UpsertEntity, etc. For now I'm resorting to just up the Azure.Core logging to error, but I would prefer that these warnings would just not show up at all for expected results.

[annelo-msft]

@Archomeda, the foundational pieces have shipped. Which library are you interested in addressing warning logs in?

[Archomeda]

@annelo-msft This is regarding the Azure.Data.Tables and Azure.Storage.Blobs libraries.

[diligent176]

OMG, will someone please fix this 409 error in Table Storage... 10 years!!

#109

[christothes]

related to #27852

[schafersm-fsa]

I have been following this issue for a few months now as I am having a similar problem that is outlined in the following issue: #28084

With the release of Azure.Data.Tables 12.7.0 I was expecting the failed dependency errors in app insights when calling TableClient.CreateIfNotExistsAsync to no longer show but this is not the case.

Am I missing something here?

[christothes]

@schafersm-fsa -

Could you clarify which dependency call is failing and with which status code?

[schafersm-fsa]

@christothes - It is Table.Create with a status code of 409.

[christothes]

Hi @schafersm-fsa It looks like there is a bug here related to how we log in general. Tracking that here: #33080

[heaths]

Unchecking #17445 because it wasn't actually completed. The bot closed it with no other action taken.

[heaths]

@schaabs @christothes @AlexanderSher should Core also do something to this effect for the shared challenge-based auth such that the initial 401s aren't adversely tracing? Or is that warranted (to trace)?

For more context, I often get questions from first and third parties about why they are seeing so many 401s in their logs e.g., App Insights, from the Key Vault SDK, which uses the shared challenge-based auth as a base. Is it worth logging/tracing then?

[github-actions]

Hi @tg-msft, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.

[jasarsoft]

when?

[github-actions]

Hi @tg-msft, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.