Allow multiple resources to be specified per metric (#683)

* Add a v2 serializer

- Renaming the v1 serializer to `V1Serializer` to avoid having multiple classes all called `ConfigurationSerializer`.
- Added the initial framework for the `V2Serializer`.

* Implement v2 AzureMetadataDeserializer

* Deserialize v2 metricDefaults

* Make v2 deserializers inherit from Deserializer

- Updated all the v2 deserializers.
- Moved IDeserializer next to Deserializer, and adjusted the access levels from internal to public to allow unit testing.
- Updated the V2Serializer to inherit from Deserializer.

* Implemented AggregationDeserializer

* Implement v2 Scraping deserializer

* Implement v2 MetricDefinition deserialization

* Pull utility methods to Deserializer object

I've pulled a few utility methods up to the Deserializer object, and updated the v2 subclasses to use them.

* Refactor common deserialization assertions

Most of the tests were checking that a property had been set, or that it was null. As a result I've pulled those two assertions out to a helper class, and refactored the tests to use that helper.

* Implement v2 container instance deserializer

- Added the deserializer.
- Added some extra assertions to `DeserializerTestHelpers` to help with the situation where a deserializer returns a base type, but we want to assert against another type.

* Implement v2 container registry deserializer

- Implemented the container registry deserializer.
- Created a base class for azure resource deserializers to make sure ResourceGroupName is always deserialized.
- Created a base class for the resource deserializer tests.

* Implement v2 cosmos db deserializer

* Implement v2 generic resource deserializer

* Implement v2 network interface deserializer

* Implement v2 postgre deserializer

* Implement v2 redis cache deserializer

* Implemented v2 service bus queue deserializer

* Implement v2 virtual machine deserializer

* Implement v2 secret deserializer

* Implement v2 storage queue deserializer

* Implement v2 azure metric config deserializer

* Implement v2 metric aggregation deserializer

I also added a `GetTimespan()` method to `Deserializer`, and updated the `AggregationDeserializer` to use that.

* Remove duplicate enum deserialization method

Removed `GetNullableEnum()` and altered `GetEnum()` to return a nullable enum. The rationale for this is that the configuration model should have null properties where nothing was supplied in the yaml.

* Add v2 serialization integration test

- Added a test that creates a v2 model, serializes it, deserializes it using the V2 deserializer, and then verifies it was deserialized correctly.
- Implemented the AzureResourceDeserializerFactory.
- Fixed a small typo where I had put "metrics" instead of "resources" as a yaml tag name in the MetricDefinitionDeserializer.
- Defaulted the version in MetricsDeclarationV2 so you don't need to explicitly specify it when creating a v2 model.

* Remove unnecessary type argument on null helpers

The `AssertPropertyNull()` methods didn't actually need the property type argument because we can assert null against an object.

* Rename v1 and v2 serializers

They both handle deserialization, so I've renamed them to `{Version}Deserializer`. I've also rename the `InterpretYamlStream()` method in the v1 deserializer to `Deserialize()` since that's what it does.

* Connect v2 format to application

I've hooked the v2 format into the application so that it can actually be used.

* Remove v1 config objects

* Rename v2 objects back to v1

* Update example config for new format

I've converted the example config so that it uses the new multi-resource format.

* Updates as a result of PR feedback

- Pulled Deserializer methods to extension methods.
- Fixed some places where I'd left names as `v2`.
- Removed the duplicated code for creating the deserializer in test classes to fix code factor tests.
- Added missing doc-comments to resources.
- Renamed DeserializerTestHelpers to YamlAssert.
- Replaced mock loggers with NullLogger.Instance.
- Various other fixes.

* Fixing build issues

- Removed TODO.
- Removed unnecessary using directives.

src/Promitor.Core.Scraping/Configuration/Providers/MetricsDeclarationProvider.cs

@@ -7,6 +7,7 @@
 using Promitor.Core.Scraping.Configuration.Model;
 using Promitor.Core.Scraping.Configuration.Providers.Interfaces;
 using Promitor.Core.Scraping.Configuration.Serialization;
+using Promitor.Core.Scraping.Configuration.Serialization.v1.Model;
 
 namespace Promitor.Core.Scraping.Configuration.Providers
 {
@@ -15,9 +16,9 @@ public class MetricsDeclarationProvider : IMetricsDeclarationProvider
         private readonly ConfigurationSerializer _configurationSerializer;
         private readonly IConfiguration _configuration;
 
-        public MetricsDeclarationProvider(IConfiguration configuration, ILogger logger, IMapper mapper)
+        public MetricsDeclarationProvider(IConfiguration configuration, ILogger logger, IMapper mapper, IDeserializer<MetricsDeclarationV1> v1Deserializer)
         {
-            _configurationSerializer = new ConfigurationSerializer(logger, mapper);
+            _configurationSerializer = new ConfigurationSerializer(logger, mapper, v1Deserializer);
             _configuration = configuration;
         }
 
@@ -42,7 +43,7 @@ public virtual MetricsDeclaration Get(bool applyDefaults = false)
                     }
                     if (metric.AzureMetricConfiguration?.Aggregation.Interval == null)
                     {
-                        metric.AzureMetricConfiguration.Aggregation.Interval = config.MetricDefaults.Aggregation.Interval;
+                        metric.AzureMetricConfiguration.Aggregation.Interval = config.MetricDefaults.Aggregation?.Interval;
                     }
 
                     // Apply the default scraping interval if none is specified

src/Promitor.Core.Scraping/Configuration/Serialization/ConfigurationSerializer.cs

@@ -17,11 +17,13 @@ public class ConfigurationSerializer
     {
         private readonly ILogger _logger;
         private readonly IMapper _mapper;
+        private readonly IDeserializer<MetricsDeclarationV1> _v1Deserializer;
 
-        public ConfigurationSerializer(ILogger logger, IMapper mapper)
+        public ConfigurationSerializer(ILogger logger, IMapper mapper, IDeserializer<MetricsDeclarationV1> v1Deserializer)
         {
             _logger = logger;
             _mapper = mapper;
+            _v1Deserializer = v1Deserializer;
         }
 
         public MetricsDeclaration Deserialize(string rawMetricsDeclaration)
@@ -55,8 +57,7 @@ private MetricsDeclaration InterpretYamlStream(YamlStream metricsDeclarationYaml
             switch (specVersion)
             {
                 case SpecVersion.v1:
-                    var v1Serializer = new v1.Core.ConfigurationSerializer(_logger);
-                    var v1Config = v1Serializer.InterpretYamlStream(rootNode);
+                    var v1Config = _v1Deserializer.Deserialize(rootNode);
 
                     return _mapper.Map<MetricsDeclaration>(v1Config);
                 default:
@@ -81,21 +82,7 @@ private SpecVersion DetermineDeclarationSpecVersion(YamlMappingNode mappingNode)
             return (SpecVersion)specVersion;
         }
 
-        public string Serialize(MetricsDeclaration metricsDeclaration)
-        {
-            Guard.NotNull(metricsDeclaration, nameof(metricsDeclaration));
-
-            var serializer = YamlSerialization.CreateSerializer();
-            var rawMetricsDeclaration = serializer.Serialize(metricsDeclaration);
-            return rawMetricsDeclaration;
-        }
-
-        /// <summary>
-        /// Allows a v1 version of the config to be serialized.
-        /// </summary>
-        /// <param name="metricsDeclaration">A v1 version of the config.</param>
-        /// <returns>The serialized yaml.</returns>
-        public string Serialize(MetricsDeclarationV1 metricsDeclaration)
+        public string Serialize(object metricsDeclaration)
         {
             Guard.NotNull(metricsDeclaration, nameof(metricsDeclaration));
 

src/Promitor.Core.Scraping/Configuration/Serialization/Deserializer.cs

@@ -6,20 +6,20 @@
 
 namespace Promitor.Core.Scraping.Configuration.Serialization
 {
-    internal abstract class Deserializer<TObject>
+    public abstract class Deserializer<TObject> : IDeserializer<TObject>
     {
         protected ILogger Logger { get; }
 
-        internal Deserializer(ILogger logger)
+        protected Deserializer(ILogger logger)
         {
             Guard.NotNull(logger, nameof(logger));
 
             Logger = logger;
         }
 
-        internal abstract TObject Deserialize(YamlMappingNode node);
+        public abstract TObject Deserialize(YamlMappingNode node);
 
-        internal List<TObject> Deserialize(YamlSequenceNode nodes)
+        public List<TObject> Deserialize(YamlSequenceNode nodes)
         {
             Guard.NotNull(nodes, nameof(nodes));
 

src/Promitor.Core.Scraping/Configuration/Serialization/IDeserializer.cs

@@ -0,0 +1,26 @@
+using System.Collections.Generic;
+using YamlDotNet.RepresentationModel;
+
+namespace Promitor.Core.Scraping.Configuration.Serialization
+{
+    /// <summary>
+    /// An object that can deserialize a yaml node into an object.
+    /// </summary>
+    /// <typeparam name="TObject">The type of object that can be deserialized.</typeparam>
+    public interface IDeserializer<TObject>
+    {
+        /// <summary>
+        /// Deserializes the specified node.
+        /// </summary>
+        /// <param name="node">The node to deserialize.</param>
+        /// <returns>The deserialized object.</returns>
+        TObject Deserialize(YamlMappingNode node);
+
+        /// <summary>
+        /// Deserializes an array of elements.
+        /// </summary>
+        /// <param name="node">The node to deserialize.</param>
+        /// <returns>The deserialized objects.</returns>
+        List<TObject> Deserialize(YamlSequenceNode node);
+    }
+}

src/Promitor.Core.Scraping/Configuration/Serialization/YamlMappingNodeExtensions.cs

@@ -0,0 +1,102 @@
+using System;
+using System.Collections.Generic;
+using YamlDotNet.RepresentationModel;
+
+namespace Promitor.Core.Scraping.Configuration.Serialization
+{
+    public static class YamlMappingNodeExtensions
+    {
+        /// <summary>
+        /// Gets the string value of the specified yaml property.
+        /// </summary>
+        /// <param name="node">The node containing the property.</param>
+        /// <param name="propertyName">The name of the property.</param>
+        /// <returns>The string value of the property.</returns>
+        public static string GetString(this YamlMappingNode node, string propertyName)
+        {
+            if (node.Children.TryGetValue(propertyName, out var propertyNode))
+            {
+                return propertyNode.ToString();
+            }
+
+            return null;
+        }
+
+        /// <summary>
+        /// Gets the value of the specified yaml property converted to an enum.
+        /// </summary>
+        /// <typeparam name="TEnum">The type of enum to return.</typeparam>
+        /// <param name="node">The node containing the property.</param>
+        /// <param name="propertyName">The property name.</param>
+        /// <returns>The enum value, or null if the property doesn't exist.</returns>
+        public static TEnum? GetEnum<TEnum>(this YamlMappingNode node, string propertyName)
+            where TEnum: struct
+        {
+            if (node.Children.TryGetValue(propertyName, out var propertyNode))
+            {
+                return System.Enum.Parse<TEnum>(propertyNode.ToString());
+            }
+
+            return null;
+        }
+
+        /// <summary>
+        /// Gets the contents of the specified property as a dictionary.
+        /// </summary>
+        /// <param name="node">The node containing the property.</param>
+        /// <param name="propertyName">The name of the property.</param>
+        /// <returns>The child items of the property as a dictionary.</returns>
+        public static Dictionary<string, string> GetDictionary(this YamlMappingNode node, string propertyName)
+        {
+            if (node.Children.TryGetValue(propertyName, out var propertyNode))
+            {
+                var result = new Dictionary<string, string>();
+
+                foreach (var (key, value) in ((YamlMappingNode) propertyNode).Children)
+                {
+                    result[key.ToString()] = value.ToString();
+                }
+
+                return result;
+            }
+
+            return null;
+        }
+
+        /// <summary>
+        /// Gets the value of the specified yaml property converted to a <see cref="TimeSpan"/>.
+        /// </summary>
+        /// <param name="node">The node containing the property.</param>
+        /// <param name="propertyName">The name of the property.</param>
+        /// <returns>The value converted to a timespan, or null if the property doesn't exist.</returns>
+        public static TimeSpan? GetTimeSpan(this YamlMappingNode node, string propertyName)
+        {
+            if (node.Children.TryGetValue(propertyName, out var propertyNode))
+            {
+                return TimeSpan.Parse(propertyNode.ToString());
+            }
+
+            return null;
+        }
+
+        /// <summary>
+        /// Deserializes a child object using the specified deserializer.
+        /// </summary>
+        /// <typeparam name="TObject">The type of object to deserialize.</typeparam>
+        /// <param name="node">The yaml node.</param>
+        /// <param name="propertyName">The name of the property to deserialize.</param>
+        /// <param name="deserializer">The deserializer to use.</param>
+        /// <returns>The deserialized property, or null if the property does not exist.</returns>
+        public static TObject DeserializeChild<TObject>(
+            this YamlMappingNode node, string propertyName, IDeserializer<TObject> deserializer)
+            where TObject: class
+        {
+            if (node.Children.TryGetValue(propertyName, out var propertyNode))
+            {
+                return deserializer.Deserialize((YamlMappingNode)propertyNode);
+            }
+
+            return null;
+        }
+    }
+}

src/Promitor.Core.Scraping/Configuration/Serialization/v1/Core/AggregationDeserializer.cs

@@ -5,30 +5,31 @@
 
 namespace Promitor.Core.Scraping.Configuration.Serialization.v1.Core
 {
-    internal class AggregationDeserializer : Deserializer<AggregationV1>
+    public class AggregationDeserializer : Deserializer<AggregationV1>
     {
-        internal AggregationDeserializer(ILogger logger) : base(logger)
+        private const string IntervalTag = "interval";
+
+        private readonly TimeSpan _defaultAggregationInterval = TimeSpan.FromMinutes(5);
+
+        public AggregationDeserializer(ILogger logger) : base(logger)
         {
         }
 
-        internal override AggregationV1 Deserialize(YamlMappingNode node)
+        public override AggregationV1 Deserialize(YamlMappingNode node)
         {
-            var aggregation = new AggregationV1();
+            var interval = node.GetTimeSpan(IntervalTag);
 
-            var interval = TimeSpan.FromMinutes(5);
-            if (node.Children.ContainsKey("interval"))
-            {
-                var rawIntervalNode = node.Children[new YamlScalarNode("interval")];
-                interval = TimeSpan.Parse(rawIntervalNode.ToString());                
-            }
-            else
+            var aggregation = new AggregationV1 {Interval = interval};
+
+            if (aggregation.Interval == null)
             {
-                Logger.LogWarning("No default aggregation was configured, falling back to {AggregationInterval}", interval.ToString("g"));
+                aggregation.Interval = _defaultAggregationInterval;
+                Logger.LogWarning(
+                    "No default aggregation was configured, falling back to {AggregationInterval}",
+                    aggregation.Interval?.ToString("g"));
             }
 
-            aggregation.Interval = interval;
-
             return aggregation;
         }
     }
-}
+}

src/Promitor.Core.Scraping/Configuration/Serialization/v1/Core/AzureMetadataDeserializer.cs

@@ -1,32 +1,28 @@
-using GuardNet;
-using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Logging;
 using Promitor.Core.Scraping.Configuration.Serialization.v1.Model;
 using YamlDotNet.RepresentationModel;
 
 namespace Promitor.Core.Scraping.Configuration.Serialization.v1.Core
 {
-    internal class AzureMetadataDeserializer : Deserializer<AzureMetadataV1>
+    public class AzureMetadataDeserializer : Deserializer<AzureMetadataV1>
     {
-        internal AzureMetadataDeserializer(ILogger logger) : base(logger)
+        private const string TenantIdTag = "tenantId";
+        private const string SubscriptionIdTag = "subscriptionId";
+        private const string ResourceGroupNameTag = "resourceGroupName";
+
+        public AzureMetadataDeserializer(ILogger logger) : base(logger)
         {
         }
 
-        internal override AzureMetadataV1 Deserialize(YamlMappingNode node)
+        public override AzureMetadataV1 Deserialize(YamlMappingNode node)
         {
-            Guard.NotNull(node, nameof(node));
-
-            var tenantId = node.Children[new YamlScalarNode("tenantId")];
-            var subscriptionId = node.Children[new YamlScalarNode("subscriptionId")];
-            var resourceGroupName = node.Children[new YamlScalarNode("resourceGroupName")];
-
-            var azureMetadata = new AzureMetadataV1
-            {
-                TenantId = tenantId?.ToString(),
-                SubscriptionId = subscriptionId?.ToString(),
-                ResourceGroupName = resourceGroupName?.ToString()
-            };
+            var metadata = new AzureMetadataV1();
 
-            return azureMetadata;
+            metadata.TenantId = node.GetString(TenantIdTag);
+            metadata.SubscriptionId = node.GetString(SubscriptionIdTag);
+            metadata.ResourceGroupName = node.GetString(ResourceGroupNameTag);
+
+            return metadata;
         }
     }
-}
+}

src/Promitor.Core.Scraping/Configuration/Serialization/v1/Core/AzureMetricConfigurationDeserializer.cs

@@ -1,41 +1,38 @@
-using GuardNet;
-using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Logging;
 using Promitor.Core.Scraping.Configuration.Serialization.v1.Model;
 using YamlDotNet.RepresentationModel;
 
 namespace Promitor.Core.Scraping.Configuration.Serialization.v1.Core
 {
-    internal class AzureMetricConfigurationDeserializer : Deserializer<AzureMetricConfigurationV1>
+    public class AzureMetricConfigurationDeserializer : Deserializer<AzureMetricConfigurationV1>
     {
-        private readonly MetricAggregationDeserializer _metricAggregationDeserializer;
-        private readonly YamlScalarNode _metricNode = new YamlScalarNode("metricName");
-        private readonly YamlScalarNode _aggregationNode = new YamlScalarNode("aggregation");
+        private const string MetricNameTag = "metricName";
+        private const string AggregationTag = "aggregation";
+        private readonly IDeserializer<MetricAggregationV1> _aggregationDeserializer;
 
-        internal AzureMetricConfigurationDeserializer(ILogger logger) : base(logger)
+        public AzureMetricConfigurationDeserializer(IDeserializer<MetricAggregationV1> aggregationDeserializer, ILogger logger)
+            : base(logger)
         {
-            _metricAggregationDeserializer = new MetricAggregationDeserializer(logger);
+            _aggregationDeserializer = aggregationDeserializer;
         }
 
-        internal override AzureMetricConfigurationV1 Deserialize(YamlMappingNode node)
+        public override AzureMetricConfigurationV1 Deserialize(YamlMappingNode node)
         {
-            Guard.NotNull(node, nameof(node));
-
-            var metricName = node.Children[_metricNode];
-
-            MetricAggregationV1 metricAggregation = null;
-            if (node.Children.ContainsKey(_aggregationNode))
+            return new AzureMetricConfigurationV1
             {
-                var aggregationNode = (YamlMappingNode) node.Children[_aggregationNode];
-                metricAggregation = _metricAggregationDeserializer.Deserialize(aggregationNode);
-            }
+                MetricName = node.GetString(MetricNameTag),
+                Aggregation = DeserializeAggregation(node)
+            };
+        }
 
-            var azureMetricConfiguration = new AzureMetricConfigurationV1
+        private MetricAggregationV1 DeserializeAggregation(YamlMappingNode node)
+        {
+            if (node.Children.TryGetValue(AggregationTag, out var aggregationNode))
             {
-                MetricName = metricName?.ToString(),
-                Aggregation = metricAggregation
-            };
+                return _aggregationDeserializer.Deserialize((YamlMappingNode) aggregationNode);
+            }
 
-            return azureMetricConfiguration;
+            return null;
         }
     }
-}
+}