Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Azure Billing] Upgrade Usage Details API to version 2019-10-01 #31970

Merged

Conversation

zmoog
Copy link
Contributor

@zmoog zmoog commented Jun 17, 2022

What does this PR do?

This PR upgrades the Usage Details API client to version 2019-10-01 to add support for the existing "legacy" and the new "modern" 1 Azure account types.

MIgration

In the migration from the original (2019-01-01) version to the new (2019-10-01) version, we kept the same field naming tried to maintain the semantics as close as possible to the original.

In addition, we added a few new fields that were available on both "legacy" and "modern" account types:

  • usage_date
  • account_id
  • unit_price
  • quantity

There are lots of other fields available for both legacy and modern account types, but we plan to introduce them with different PRs and according to provided value and users' interests.

Patch Management

To accomplish this goal, we had to patch the existing Usage Details API client to make it work with modern customers with a Microsoft Customer Agreement.

Solution A (currently implemented)

To host the patch, we created a fork of the original Azure/azure-sdk-for-go repo.

We applied the patch on top of the azure/azure-sdk-for-go@v59.0.0; see https://github.com/elastic/azure-sdk-for-go/releases/tag/v59.0.0-elastic-1 for more details about branch and tag.

As a long-term solution, we are also getting in touch with Microsoft to update the OpenAPI spec used to generate the clients, so we don't have to maintain any patch anymore.

Solution B (pivoted)

Since the Usage Details API and other clients are generated from OpenAPI specs, we could generate and patch our own API clients using the AutoRest, the same toolchain used by Microsoft.

Pros:

  • Avoid forking the azure-sdk-for-go repo
  • Use the latest API specs that Microsoft stopped generating for the previous-gen client)

I tested this approach in the zmoog/azure-sdk-clients and it works.

Why is it important?

Right now, a few people using modern 1 Azure accounts cannot use the Azure Billing metrics module due to errors. For more information on this problem, check the following issues:

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Footnotes

  1. In 2021, Azure introduced new plans creating a distinction between "legacy" and "modern" account types. 2

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 17, 2022
@zmoog zmoog added the Team:Cloud-Monitoring Label for the Cloud Monitoring team label Jun 17, 2022
@mergify mergify bot assigned zmoog Jun 17, 2022
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 17, 2022
@zmoog zmoog added enhancement needs_team Indicates that the issue/PR needs a Team:* label labels Jun 17, 2022
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 17, 2022
@botelastic
Copy link

botelastic bot commented Jun 17, 2022

This pull request doesn't have a Team:<team> label.

@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 17, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-07-11T08:07:50.921+0000

  • Duration: 130 min 26 sec

Test stats 🧪

Test Results
Failed 0
Passed 22218
Skipped 1914
Total 24132

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@zmoog zmoog force-pushed the azure-usagedetailsapi-upgrade-2019-10-01 branch 4 times, most recently from 31c290b to 4d876f2 Compare June 22, 2022 14:59
@zmoog
Copy link
Contributor Author

zmoog commented Jun 23, 2022

/test

@zmoog zmoog marked this pull request as ready for review June 23, 2022 09:01
@zmoog zmoog requested review from a team as code owners June 23, 2022 09:01
@zmoog zmoog requested review from rdner and faec and removed request for a team June 23, 2022 09:01
metricbeat/docs/fields.asciidoc Outdated Show resolved Hide resolved
x-pack/metricbeat/module/azure/billing/_meta/fields.yml Outdated Show resolved Hide resolved
x-pack/metricbeat/module/azure/billing/_meta/fields.yml Outdated Show resolved Hide resolved
x-pack/metricbeat/module/azure/billing/client_test.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/azure/billing/mock_service.go Outdated Show resolved Hide resolved
Copy link
Contributor

@tommyers-elastic tommyers-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome work - no blockers from me!

NOTICE.txt Show resolved Hide resolved
go.mod Outdated Show resolved Hide resolved
metricbeat/docs/fields.asciidoc Outdated Show resolved Hide resolved
x-pack/metricbeat/module/azure/billing/billing.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/azure/billing/billing.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/azure/billing/client.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/azure/billing/service.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/azure/billing/data.go Outdated Show resolved Hide resolved
@zmoog zmoog force-pushed the azure-usagedetailsapi-upgrade-2019-10-01 branch from 471e9bc to d49342b Compare June 28, 2022 15:41
@zmoog
Copy link
Contributor Author

zmoog commented Jun 29, 2022

/package

@zmoog zmoog added backport-v8.3.0 Automated backport with mergify backport-v8.2.0 Automated backport with mergify labels Jun 29, 2022
@mergify
Copy link
Contributor

mergify bot commented Jul 6, 2022

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b azure-usagedetailsapi-upgrade-2019-10-01 upstream/azure-usagedetailsapi-upgrade-2019-10-01
git merge upstream/main
git push upstream azure-usagedetailsapi-upgrade-2019-10-01

@tommyers-elastic tommyers-elastic merged commit 1f232dc into elastic:main Jul 11, 2022
mergify bot pushed a commit that referenced this pull request Jul 11, 2022
* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 1f232dc)

# Conflicts:
#	go.sum
#	x-pack/metricbeat/module/azure/billing/billing.go
#	x-pack/metricbeat/module/azure/billing/client.go
#	x-pack/metricbeat/module/azure/billing/data.go
#	x-pack/metricbeat/module/azure/billing/mock_service.go
#	x-pack/metricbeat/module/azure/billing/service.go
mergify bot pushed a commit that referenced this pull request Jul 11, 2022
* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 1f232dc)
@willemdh
Copy link

In what version should this be fixed?

@zmoog zmoog deleted the azure-usagedetailsapi-upgrade-2019-10-01 branch July 19, 2022 15:54
@zmoog
Copy link
Contributor Author

zmoog commented Jul 19, 2022

@willemdh It will be available in 8.4. I'm currently checking to backport it to the next 8.3.x and 7.17.x releases.

zmoog added a commit that referenced this pull request Jul 20, 2022
…ersion 2019-10-01 (#32298)

* [Azure Billing] Upgrade Usage Details API to version 2019-10-01 (#31970)

* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

Co-authored-by: Maurizio Branca <maurizio.branca@gmail.com>
@zmoog
Copy link
Contributor Author

zmoog commented Aug 5, 2022

Quick update about the availability of the fix across currently planned releases of Beats (Metricbeat) and Elastic Agent:

  • 8.4: included in 8.4.0 due for a public release later in August.
  • 8.3: included in 8.3.3 available today.
  • 7.17: targeting 7.17.6 due in mid-September.

zmoog added a commit that referenced this pull request Aug 24, 2022
* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 1f232dc)

# Conflicts:
#	go.sum
#	x-pack/metricbeat/module/azure/billing/billing.go
#	x-pack/metricbeat/module/azure/billing/client.go
#	x-pack/metricbeat/module/azure/billing/data.go
#	x-pack/metricbeat/module/azure/billing/mock_service.go
#	x-pack/metricbeat/module/azure/billing/service.go
zmoog added a commit that referenced this pull request Aug 25, 2022
* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 1f232dc)

# Conflicts:
#	go.sum
#	x-pack/metricbeat/module/azure/billing/billing.go
#	x-pack/metricbeat/module/azure/billing/client.go
#	x-pack/metricbeat/module/azure/billing/data.go
#	x-pack/metricbeat/module/azure/billing/mock_service.go
#	x-pack/metricbeat/module/azure/billing/service.go
zmoog added a commit that referenced this pull request Aug 31, 2022
* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 1f232dc)

# Conflicts:
#	go.sum
#	x-pack/metricbeat/module/azure/billing/billing.go
#	x-pack/metricbeat/module/azure/billing/client.go
#	x-pack/metricbeat/module/azure/billing/data.go
#	x-pack/metricbeat/module/azure/billing/mock_service.go
#	x-pack/metricbeat/module/azure/billing/service.go
zmoog added a commit that referenced this pull request Sep 11, 2022
* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 1f232dc)

# Conflicts:
#	go.sum
#	x-pack/metricbeat/module/azure/billing/billing.go
#	x-pack/metricbeat/module/azure/billing/client.go
#	x-pack/metricbeat/module/azure/billing/data.go
#	x-pack/metricbeat/module/azure/billing/mock_service.go
#	x-pack/metricbeat/module/azure/billing/service.go
zmoog added a commit that referenced this pull request Oct 10, 2022
* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 1f232dc)

# Conflicts:
#	go.sum
#	x-pack/metricbeat/module/azure/billing/billing.go
#	x-pack/metricbeat/module/azure/billing/client.go
#	x-pack/metricbeat/module/azure/billing/data.go
#	x-pack/metricbeat/module/azure/billing/mock_service.go
#	x-pack/metricbeat/module/azure/billing/service.go
zmoog added a commit that referenced this pull request Oct 11, 2022
* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 1f232dc)

# Conflicts:
#	go.sum
#	x-pack/metricbeat/module/azure/billing/billing.go
#	x-pack/metricbeat/module/azure/billing/client.go
#	x-pack/metricbeat/module/azure/billing/data.go
#	x-pack/metricbeat/module/azure/billing/mock_service.go
#	x-pack/metricbeat/module/azure/billing/service.go
zmoog added a commit that referenced this pull request Oct 13, 2022
* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 1f232dc)

# Conflicts:
#	go.sum
#	x-pack/metricbeat/module/azure/billing/billing.go
#	x-pack/metricbeat/module/azure/billing/client.go
#	x-pack/metricbeat/module/azure/billing/data.go
#	x-pack/metricbeat/module/azure/billing/mock_service.go
#	x-pack/metricbeat/module/azure/billing/service.go
zmoog added a commit that referenced this pull request Oct 18, 2022
* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 1f232dc)

# Conflicts:
#	go.sum
#	x-pack/metricbeat/module/azure/billing/billing.go
#	x-pack/metricbeat/module/azure/billing/client.go
#	x-pack/metricbeat/module/azure/billing/data.go
#	x-pack/metricbeat/module/azure/billing/mock_service.go
#	x-pack/metricbeat/module/azure/billing/service.go
zmoog added a commit that referenced this pull request Oct 18, 2022
…version 2019-10-01 (#32297)

* [Azure Billing] Upgrade Usage Details API to version 2019-10-01 (#31970)

* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 1f232dc)

# Conflicts:
#	go.sum
#	x-pack/metricbeat/module/azure/billing/billing.go
#	x-pack/metricbeat/module/azure/billing/client.go
#	x-pack/metricbeat/module/azure/billing/data.go
#	x-pack/metricbeat/module/azure/billing/mock_service.go
#	x-pack/metricbeat/module/azure/billing/service.go

* Fix conflicts

* Clean up

* Remove unrelated changelog entries

* go mod tidy

Co-authored-by: Maurizio Branca <maurizio.branca@gmail.com>
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.3.0 Automated backport with mergify enhancement Team:Cloud-Monitoring Label for the Cloud Monitoring team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Outdated API version in the Metricbeat Azure module Billing set
6 participants