Outdated API version in the Metricbeat Azure module Billing set

[toby-sutor]

It seems like we are referencing an outdated API version in the (Beta) Azure module / Billing metricset for Metricbeat.

E.g. in

beats/x-pack/metricbeat/module/azure/billing/client.go

Line 13 in 496e3c9

"github.com/Azure/azure-sdk-for-go/services/consumption/mgmt/2019-01-01/consumption"

beats/x-pack/metricbeat/module/azure/billing/service.go

Line 10 in 496e3c9

"github.com/Azure/azure-sdk-for-go/services/consumption/mgmt/2019-01-01/consumption"

beats/x-pack/metricbeat/module/azure/billing/data.go

Line 12 in 496e3c9

"github.com/Azure/azure-sdk-for-go/services/consumption/mgmt/2019-01-01/consumption"

we’re referring to API version 2019-01-01 but in the Microsoft docs the API version is 2019-10-01.

This leads to error:

INFO module/wrapper.go:259 Error fetching data for metricset azure.billing: error retrieving usage information: 
Retrieving usage details failed in client: consumption.UsageDetailsClient#List: Failure responding to request:
StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="400" 
Message="Subscription scope usage is not supported for current api version. Please use api version after 2019-10-01

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[narph]

@toby-sutor ,
I am not able to reproduce this issue and I can retrieve the data fine.
Also we cannot update until the azure sdk will offer us the new version https://github.com/Azure/azure-sdk-for-go/tree/master/services/consumption/mgmt (we would find an workaround in case there are critical issues).
The error could also tell us that the subscription might not have the rights to access the data.
These APIs currently only support Enterprise Enrollments and Web Direct Subscriptions (with a few exceptions), can you tell us what is the type of subscription you have currently configured?

[narph]

I have created an issue for mapping the 2019-10-01 consumption API's inside the sdk Azure/azure-sdk-for-go#12452.
The latest version seems to support more subscription types than the ones mentioned above.

[toby-sutor]

As discussed offline, from what I found at https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/consumption-api-overview and more specifically here: https://docs.microsoft.com/en-us/partner-center/azure-plan-available it appears that there is no API access available for CSP subscriptions, which has been the case here. Hence the error message we receive is misleading.

[narph]

reopening this, the new version 2019-10-01 seems to not support filtering by the usage dates.

[willemdh]

Please fix this...

[willemdh]

When will this get some attention..? This module hasn't worked at all for many months now. You can as well just delete it...

[kasper112]

I still have the same problem any news here ?

using

Azure Billing Metrics
Version
1.0.1

[tommyers-elastic]

thanks all for your patience with this

we are now actively working on this - we will update progress here when we have a more concrete timeline for shipping the fix

[MohammadBangee]

thanks all for your patience with this

we are now actively working on this - we will update progress here when we have a more concrete timeline for shipping the fix

Dear @tommyers-elastic
Has this issue been resolved?
I am still facing this issue on MetricBeat and Elastic Agent.

[DougBellinger]

We are also experiencing this with elastic 7.17 in the cloud

[udaydaayam]

we are experiencing as well.
11:07:24.136
elastic_agent.metricbeat
[elastic_agent.metricbeat][error] Error fetching data for metricset azure.billing: error retrieving usage information: Retrieving usage details failed in client: consumption.UsageDetailsClient#List: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="400" Message="Subscription scope usage is not supported for current api version. Please use api version after 2019-10-01 (Request ID: 0277fbdc-23db-48ce-80e9-bccb982c0814)"

[zmoog]

Hey @MohammadBangee @DougBellinger

We fixed this issue by upgrading Usage Details API in Beats to version 2019-10-01.

Here's the availability of the fix across releases of Beats (Metricbeat) and Elastic Agent:

• 8.4: included in 8.4.0 due for a public release later in August.
• 8.3: included in 8.3.3 available today.
• 7.17: targeting 7.17.6 due in mid September merged in the 7.17 branch; it will be released in 7.17.8.

[zmoog]

@udaydaayam, which version of Beats are you currently using?

[willemdh]

Hello @tommyers-elastic ,

We just updated to 7.17.7 and it appears this issue is still not fixed.. When the billing metricset is configured, the following error is logged on metricbeat start:

Nov 02 13:59:29 myhostname metricbeat[8884]: ERROR module/wrapper.go:259 Error fetching data for metricset azure.billing: error retrieving usage information: Retrieving usage details failed in client: consumption.UsageDetailsClient#List: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="400" Message="Subscription scope usage is not supported for current api version. Please use api version after 2019-10-01 (Request ID: 0ae2ddfe-aeb6-4be4-949f-a3e2dae3de47)"

This has been broken for so long.....

[zmoog]

Hello @willemdh, I am very sorry for this delay; we had many issues during the fix's backport. It was finally merged (see these two PRs, #32297 and #33023), but unfortunately, it couldn't make it for 7.17.7.

Since we merged it in the 7.17 branch, It will be available in 7.17.8.

[willemdh]

Really sorry to hear that we spent 2 days upgrading our pr cluster because this thread mentioned it was going to be fixed in 7.16....
What a waste of time..

[zmoog]

I am so sorry to hear this. I feel your pain. My apologies for not providing you with more timely updates and precise wording ¹.

Let me try to help you with the next steps.

To discover what's released, you can reference the Beats release notes. For 7.17.x, you can check https://www.elastic.co/guide/en/beats/libbeat/7.17/release-notes.html and check if we included the expected change in a release. In the release notes, you can look for the PR number or the description CHANGELOG.next.asciidoc file.

Please contact me if you need any help or have questions. I'll do my best to assist you.

Footnotes

1. I used the word "targeting 7.17.6" because until we merge the fix in the branch, we are not sure it can hit that version. ↩