-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Azure Billing] Upgrade Usage Details API to version 2019-10-01 #31970
[Azure Billing] Upgrade Usage Details API to version 2019-10-01 #31970
Conversation
This pull request doesn't have a |
31c290b
to
4d876f2
Compare
/test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
awesome work - no blockers from me!
471e9bc
to
d49342b
Compare
/package |
This pull request is now in conflicts. Could you fix it? 🙏
|
…pi-upgrade-2019-10-01
…pi-upgrade-2019-10-01
…pi-upgrade-2019-10-01
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc)
In what version should this be fixed? |
@willemdh It will be available in 8.4. I'm currently checking to backport it to the next 8.3.x and 7.17.x releases. |
…ersion 2019-10-01 (#32298) * [Azure Billing] Upgrade Usage Details API to version 2019-10-01 (#31970) * Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range Co-authored-by: Maurizio Branca <maurizio.branca@gmail.com>
Quick update about the availability of the fix across currently planned releases of Beats (Metricbeat) and Elastic Agent:
|
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
…version 2019-10-01 (#32297) * [Azure Billing] Upgrade Usage Details API to version 2019-10-01 (#31970) * Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go * Fix conflicts * Clean up * Remove unrelated changelog entries * go mod tidy Co-authored-by: Maurizio Branca <maurizio.branca@gmail.com>
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
What does this PR do?
This PR upgrades the Usage Details API client to version 2019-10-01 to add support for the existing "legacy" and the new "modern" 1 Azure account types.
MIgration
In the migration from the original (2019-01-01) version to the new (2019-10-01) version, we kept the same field naming tried to maintain the semantics as close as possible to the original.
In addition, we added a few new fields that were available on both "legacy" and "modern" account types:
usage_date
account_id
unit_price
quantity
There are lots of other fields available for both legacy and modern account types, but we plan to introduce them with different PRs and according to provided value and users' interests.
Patch Management
To accomplish this goal, we had to patch the existing Usage Details API client to make it work with modern customers with a Microsoft Customer Agreement.
Solution A (currently implemented)
To host the patch, we created a fork of the original Azure/azure-sdk-for-go repo.
We applied the patch on top of the azure/azure-sdk-for-go@v59.0.0; see https://github.com/elastic/azure-sdk-for-go/releases/tag/v59.0.0-elastic-1 for more details about branch and tag.
As a long-term solution, we are also getting in touch with Microsoft to update the OpenAPI spec used to generate the clients, so we don't have to maintain any patch anymore.
Solution B (pivoted)
Since the Usage Details API and other clients are generated from OpenAPI specs, we could generate and patch our own API clients using the AutoRest, the same toolchain used by Microsoft.
Pros:
I tested this approach in the zmoog/azure-sdk-clients and it works.
Why is it important?
Right now, a few people using modern 1 Azure accounts cannot use the Azure Billing metrics module due to errors. For more information on this problem, check the following issues:
Checklist
CHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.Author's Checklist
How to test this PR locally
Related issues
Use cases
Screenshots
Logs
Footnotes
In 2021, Azure introduced new plans creating a distinction between "legacy" and "modern" account types. ↩ ↩2