-
Notifications
You must be signed in to change notification settings - Fork 4.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Azure Billing] Upgrade Usage Details API to version 2019-10-01 (#31970)
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
- Loading branch information
Showing
16 changed files
with
422 additions
and
123 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| // Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| // or more contributor license agreements. Licensed under the Elastic License; | ||
| // you may not use this file except in compliance with the Elastic License. | ||
|
|
||
| package billing | ||
|
|
||
| import ( | ||
| "testing" | ||
| "time" | ||
|
|
||
| "github.com/stretchr/testify/assert" | ||
| ) | ||
|
|
||
| func TestPreviousDayFrom(t *testing.T) { | ||
| t.Run("returns the previous day as time interval to collect metrics", func(t *testing.T) { | ||
| referenceTime, err := time.Parse("2006-01-02 15:04:05", "2007-01-09 09:41:00") | ||
| assert.NoError(t, err) | ||
| expectedStartTime, err := time.Parse("2006-01-02 15:04:05", "2007-01-08 00:00:00") | ||
| assert.NoError(t, err) | ||
| expectedEndTime, err := time.Parse("2006-01-02 15:04:05", "2007-01-08 23:59:59") | ||
| assert.NoError(t, err) | ||
|
|
||
| actualStartTime, actualEndTime := previousDayFrom(referenceTime) | ||
|
|
||
| assert.Equal(t, expectedStartTime, actualStartTime) | ||
| assert.Equal(t, expectedEndTime, actualEndTime) | ||
| }) | ||
| } |
Oops, something went wrong.