Seperate internal metrics declaration model from the v1 (de)seri… (#652)

* Create version-specific serialization objects

We need to support both serializing and deserializing the metric config. Currently, Promitor relies on the model matching the yaml schema, and just using the standard serialization from YamlDotNet.

The problem with this is that if we introduce multiple config formats, we end up with a situation where the model objects used by the application don't match both (or either) of the config formats anymore, and so serialization won't work. Given that most of the tests currently rely on this serialization, it would end up with a pretty complicated change to sort this.

So that we can still generate the old `v1` format, what I've done is created separate serialization objects for both config versions. This means that if you need to write out a config in a certain format, you can use those objects to define the configuration.

NOTE: this commit doesn't compile but I want to add the files like this to make it easier to see what I'm doing as I make further changes. It can be rebased into future commits if it makes sense.

* Implement builder pattern for v1 models

Renamed all the models in the v1 namespace to `<ModelName>Builder` and added a `Build()` method to each of them. The reason for this is to distinguish between the model objects used by the application, and the models used for serialization.

I've also added additional constructors to the resource model objects. The reason I did this was to make it easier for me to be sure that the builders were populating all of the properties on each object. I've left the default constructors in place.

* Remove GetSecretValue from builder

I've removed the `GetSecretValue()` method from the v1 `SecretBuilder` object because it's only needed by the model object, not the serialization objects.

* Convert tests to use v1 config builders

- Updated the tests to use the v1 builders. This should exactly match the previous behaviour.
- Temporarily added an extra `Serialize()` method to `ConfigurationSerializer`. This is to allow the tests to serialize the v1 configuration objects. This will need to be removed / rethought before these changes are complete.

* Rename "Builder" objects

I've renamed the v1 config objects from `{Type}Builder` to `{Type}V1` since they aren't really builders. I'll remove the `Build()` methods in a further commit once there's an alternate way of mapping them.

* Use AutoMapper to map between config and runtime

- Added AutoMapper and created a profile for mapping between the V1 config model and the runtime model.
- Removed the IVersionedConfigurationSerializer interface because it was forcing the version deserializers to return the runtime model.
- Added a test to verify the AutoMapper config is valid. I haven't added any specific tests yet because there aren't any custom mappings.

* Fix tests after AutoMapper changes

A lot of the tests rely on fully serializing and then deserializing into the runtime model. Because of that, I've just updated them for now to use the v1 mapper profile and actually use AutoMapper in the tests.

* Remove `Build()` method from config objects

I've removed the `Build()` method from all the v1 config objects since it is no-longer required now that AutoMapper is handling the mapping.

* Reorder MetricDefinition constructor arguments

* Remove redundant type parameter

src/Promitor.Core.Scraping/Configuration/Model/Metrics/MetricDefinition.cs

@@ -4,6 +4,24 @@ namespace Promitor.Core.Scraping.Configuration.Model.Metrics
 {
     public abstract class MetricDefinition
     {
+        protected MetricDefinition()
+        {
+        }
+
+        protected MetricDefinition(string name,
+            string description,
+            string resourceGroupName,
+            Dictionary<string, string> labels,
+            Scraping scraping, AzureMetricConfiguration azureMetricConfiguration)
+        {
+            AzureMetricConfiguration = azureMetricConfiguration;
+            Description = description;
+            Name = name;
+            ResourceGroupName = resourceGroupName;
+            Labels = labels;
+            Scraping = scraping;
+        }
+
         /// <summary>
         ///     Configuration about the Azure Monitor metric to scrape
         /// </summary>

src/Promitor.Core.Scraping/Configuration/Model/Metrics/ResourceTypes/ContainerInstanceMetricDefinition.cs

@@ -1,7 +1,19 @@
-namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
+using System.Collections.Generic;
+
+namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
 {
     public class ContainerInstanceMetricDefinition : MetricDefinition
     {
+        public ContainerInstanceMetricDefinition()
+        {
+        }
+
+        public ContainerInstanceMetricDefinition(AzureMetricConfiguration azureMetricConfiguration, string description, string name, string resourceGroupName, string containerGroup, Dictionary<string, string> labels, Scraping scraping)
+            : base(name, description, resourceGroupName, labels, scraping, azureMetricConfiguration)
+        {
+            ContainerGroup = containerGroup;
+        }
+
         public string ContainerGroup { get; set; }
         public override ResourceType ResourceType { get; } = ResourceType.ContainerInstance;
     }

src/Promitor.Core.Scraping/Configuration/Model/Metrics/ResourceTypes/ContainerRegistryMetricDefinition.cs

@@ -1,7 +1,19 @@
-namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
+using System.Collections.Generic;
+
+namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
 {
     public class ContainerRegistryMetricDefinition : MetricDefinition
     {
+        public ContainerRegistryMetricDefinition()
+        {
+        }
+
+        public ContainerRegistryMetricDefinition(AzureMetricConfiguration azureMetricConfiguration, string description, string name, string resourceGroupName, string registryName, Dictionary<string, string> labels, Scraping scraping)
+            : base(name, description, resourceGroupName, labels, scraping, azureMetricConfiguration)
+        {
+            RegistryName = registryName;
+        }
+
         public string RegistryName { get; set; }
         public override ResourceType ResourceType { get; } = ResourceType.ContainerRegistry;
     }

src/Promitor.Core.Scraping/Configuration/Model/Metrics/ResourceTypes/CosmosDbMetricDefinition.cs

@@ -1,7 +1,19 @@
-namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
+using System.Collections.Generic;
+
+namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
 {
     public class CosmosDbMetricDefinition : MetricDefinition
     {
+        public CosmosDbMetricDefinition()
+        {
+        }
+
+        public CosmosDbMetricDefinition(AzureMetricConfiguration azureMetricConfiguration, string description, string name, string resourceGroupName, string dbName, Dictionary<string, string> labels, Scraping scraping)
+            : base(name, description, resourceGroupName, labels, scraping, azureMetricConfiguration)
+        {
+            DbName = dbName;
+        }
+
         public string DbName { get; set; }
 
         public override ResourceType ResourceType { get; } = ResourceType.CosmosDb;

src/Promitor.Core.Scraping/Configuration/Model/Metrics/ResourceTypes/GenericAzureMetricDefinition.cs

@@ -1,7 +1,20 @@
-namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
+using System.Collections.Generic;
+
+namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
 {
     public class GenericAzureMetricDefinition : MetricDefinition
     {
+        public GenericAzureMetricDefinition()
+        {
+        }
+
+        public GenericAzureMetricDefinition(AzureMetricConfiguration azureMetricConfiguration, string description, string name, string resourceGroupName, string filter, string resourceUri, Dictionary<string, string> labels, Scraping scraping)
+            : base(name, description, resourceGroupName, labels, scraping, azureMetricConfiguration)
+        {
+            Filter = filter;
+            ResourceUri = resourceUri;
+        }
+
         public string Filter { get; set; }
         public override ResourceType ResourceType { get; } = ResourceType.Generic;
         public string ResourceUri { get; set; }

src/Promitor.Core.Scraping/Configuration/Model/Metrics/ResourceTypes/NetworkInterfaceMetricDefinition.cs

@@ -1,7 +1,19 @@
+using System.Collections.Generic;
+
 namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
 {
     public class NetworkInterfaceMetricDefinition : MetricDefinition
     {
+        public NetworkInterfaceMetricDefinition()
+        {
+        }
+
+        public NetworkInterfaceMetricDefinition(AzureMetricConfiguration azureMetricConfiguration, string description, string name, string resourceGroupName, string networkInterfaceName, Dictionary<string, string> labels, Scraping scraping)
+            : base(name, description, resourceGroupName, labels, scraping, azureMetricConfiguration)
+        {
+            NetworkInterfaceName = networkInterfaceName;
+        }
+
         public string NetworkInterfaceName { get; set; }
         public override ResourceType ResourceType { get; } = ResourceType.NetworkInterface;
     }

src/Promitor.Core.Scraping/Configuration/Model/Metrics/ResourceTypes/PostgreSqlMetricDefinition.cs

@@ -1,7 +1,19 @@
-namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
+using System.Collections.Generic;
+
+namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
 {
     public class PostgreSqlMetricDefinition : MetricDefinition
     {
+        public PostgreSqlMetricDefinition()
+        {
+        }
+
+        public PostgreSqlMetricDefinition(AzureMetricConfiguration azureMetricConfiguration, string description, string name, string resourceGroupName, string serverName, Dictionary<string, string> labels, Scraping scraping)
+            : base(name, description, resourceGroupName, labels, scraping, azureMetricConfiguration)
+        {
+            ServerName = serverName;
+        }
+
         public string ServerName { get; set; }
         public override ResourceType ResourceType { get; } = ResourceType.PostgreSql;
     }

src/Promitor.Core.Scraping/Configuration/Model/Metrics/ResourceTypes/RedisCacheMetricDefinition.cs

@@ -1,7 +1,19 @@
-namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
+using System.Collections.Generic;
+
+namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
 {
     public class RedisCacheMetricDefinition : MetricDefinition
     {
+        public RedisCacheMetricDefinition()
+        {
+        }
+
+        public RedisCacheMetricDefinition(AzureMetricConfiguration azureMetricConfiguration, string description, string name, string resourceGroupName, string cacheName, Dictionary<string, string> labels, Scraping scraping)
+            : base(name, description, resourceGroupName, labels, scraping, azureMetricConfiguration)
+        {
+            CacheName = cacheName;
+        }
+
         public string CacheName { get; set; }
         public override ResourceType ResourceType { get; } = ResourceType.RedisCache;
     }

src/Promitor.Core.Scraping/Configuration/Model/Metrics/ResourceTypes/ServiceBusQueueMetricDefinition.cs

@@ -1,7 +1,20 @@
-namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
+using System.Collections.Generic;
+
+namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
 {
     public class ServiceBusQueueMetricDefinition : MetricDefinition
     {
+        public ServiceBusQueueMetricDefinition()
+        {
+        }
+
+        public ServiceBusQueueMetricDefinition(AzureMetricConfiguration azureMetricConfiguration, string description, string name, string resourceGroupName, string ns, string queueName, Dictionary<string, string> labels, Scraping scraping)
+            : base(name, description, resourceGroupName, labels, scraping, azureMetricConfiguration)
+        {
+            Namespace = ns;
+            QueueName = queueName;
+        }
+
         public string Namespace { get; set; }
         public string QueueName { get; set; }
         public override ResourceType ResourceType { get; } = ResourceType.ServiceBusQueue;

src/Promitor.Core.Scraping/Configuration/Model/Metrics/ResourceTypes/StorageQueueMetricDefinition.cs

@@ -1,7 +1,21 @@
+using System.Collections.Generic;
+
 namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
 {
     public class StorageQueueMetricDefinition : MetricDefinition
     {
+        public StorageQueueMetricDefinition()
+        {
+        }
+
+        public StorageQueueMetricDefinition(AzureMetricConfiguration azureMetricConfiguration, string description, string name, string resourceGroupName, string accountName, string queueName, Secret sasToken, Dictionary<string, string> labels, Scraping scraping)
+            : base(name, description, resourceGroupName, labels, scraping, azureMetricConfiguration)
+        {
+            AccountName = accountName;
+            QueueName = queueName;
+            SasToken = sasToken;
+        }
+
         public string AccountName { get; set; }
         public string QueueName { get; set; }
         public Secret SasToken { get; set; }

src/Promitor.Core.Scraping/Configuration/Model/Metrics/ResourceTypes/VirtualMachineMetricDefinition.cs

@@ -1,7 +1,19 @@
+using System.Collections.Generic;
+
 namespace Promitor.Core.Scraping.Configuration.Model.Metrics.ResourceTypes
 {
     public class VirtualMachineMetricDefinition : MetricDefinition
     {
+        public VirtualMachineMetricDefinition()
+        {
+        }
+
+        public VirtualMachineMetricDefinition(AzureMetricConfiguration azureMetricConfiguration, string description, string name, string resourceGroupName, string virtualMachineName, Dictionary<string, string> labels, Scraping scraping)
+            : base(name, description, resourceGroupName, labels, scraping, azureMetricConfiguration)
+        {
+            VirtualMachineName = virtualMachineName;
+        }
+
         public string VirtualMachineName { get; set; }
         public override ResourceType ResourceType { get; } = ResourceType.VirtualMachine;
     }

src/Promitor.Core.Scraping/Configuration/Providers/MetricsDeclarationProvider.cs

@@ -1,5 +1,6 @@
 using System;
 using System.IO;
+using AutoMapper;
 using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.Logging;
 using Promitor.Core.Configuration.Model.Metrics;
@@ -14,9 +15,9 @@ public class MetricsDeclarationProvider : IMetricsDeclarationProvider
         private readonly ConfigurationSerializer _configurationSerializer;
         private readonly IConfiguration _configuration;
 
-        public MetricsDeclarationProvider(IConfiguration configuration, ILogger logger)
+        public MetricsDeclarationProvider(IConfiguration configuration, ILogger logger, IMapper mapper)
         {
-            _configurationSerializer = new ConfigurationSerializer(logger);
+            _configurationSerializer = new ConfigurationSerializer(logger, mapper);
             _configuration = configuration;
         }
 

src/Promitor.Core.Scraping/Configuration/Serialization/ConfigurationSerializer.cs

@@ -2,10 +2,12 @@
 using System.IO;
 using System.Linq;
 using System.Runtime.Serialization;
+using AutoMapper;
 using GuardNet;
 using Microsoft.Extensions.Logging;
 using Promitor.Core.Scraping.Configuration.Model;
 using Promitor.Core.Scraping.Configuration.Serialization.Enum;
+using Promitor.Core.Scraping.Configuration.Serialization.v1.Model;
 using Promitor.Core.Serialization.Yaml;
 using YamlDotNet.RepresentationModel;
 
@@ -14,10 +16,12 @@ namespace Promitor.Core.Scraping.Configuration.Serialization
     public class ConfigurationSerializer
     {
         private readonly ILogger _logger;
+        private readonly IMapper _mapper;
 
-        public ConfigurationSerializer(ILogger logger)
+        public ConfigurationSerializer(ILogger logger, IMapper mapper)
         {
             _logger = logger;
+            _mapper = mapper;
         }
 
         public MetricsDeclaration Deserialize(string rawMetricsDeclaration)
@@ -52,7 +56,9 @@ private MetricsDeclaration InterpretYamlStream(YamlStream metricsDeclarationYaml
             {
                 case SpecVersion.v1:
                     var v1Serializer = new v1.Core.ConfigurationSerializer(_logger);
-                    return v1Serializer.InterpretYamlStream(rootNode);
+                    var v1Config = v1Serializer.InterpretYamlStream(rootNode);
+
+                    return _mapper.Map<MetricsDeclaration>(v1Config);
                 default:
                     throw new Exception($"Unable to interpret YAML stream for spec version '{specVersion}'");
             }
@@ -83,5 +89,19 @@ public string Serialize(MetricsDeclaration metricsDeclaration)
             var rawMetricsDeclaration = serializer.Serialize(metricsDeclaration);
             return rawMetricsDeclaration;
         }
+
+        /// <summary>
+        /// Allows a v1 version of the config to be serialized.
+        /// </summary>
+        /// <param name="metricsDeclaration">A v1 version of the config.</param>
+        /// <returns>The serialized yaml.</returns>
+        public string Serialize(MetricsDeclarationV1 metricsDeclaration)
+        {
+            Guard.NotNull(metricsDeclaration, nameof(metricsDeclaration));
+
+            var serializer = YamlSerialization.CreateSerializer();
+            var rawMetricsDeclaration = serializer.Serialize(metricsDeclaration);
+            return rawMetricsDeclaration;
+        }
     }
 }

src/Promitor.Core.Scraping/Configuration/Serialization/v1/Core/AggregationDeserializer.cs

@@ -1,19 +1,19 @@
 using System;
 using Microsoft.Extensions.Logging;
-using Promitor.Core.Scraping.Configuration.Model;
+using Promitor.Core.Scraping.Configuration.Serialization.v1.Model;
 using YamlDotNet.RepresentationModel;
 
 namespace Promitor.Core.Scraping.Configuration.Serialization.v1.Core
 {
-    internal class AggregationDeserializer : Deserializer<Aggregation>
+    internal class AggregationDeserializer : Deserializer<AggregationV1>
     {
         internal AggregationDeserializer(ILogger logger) : base(logger)
         {
         }
 
-        internal override Aggregation Deserialize(YamlMappingNode node)
+        internal override AggregationV1 Deserialize(YamlMappingNode node)
         {
-            var aggregation = new Aggregation();
+            var aggregation = new AggregationV1();
 
             var interval = TimeSpan.FromMinutes(5);
             if (node.Children.ContainsKey("interval"))

src/Promitor.Core.Scraping/Configuration/Serialization/v1/Core/AzureMetadataDeserializer.cs

@@ -1,25 +1,25 @@
 using GuardNet;
 using Microsoft.Extensions.Logging;
-using Promitor.Core.Scraping.Configuration.Model;
+using Promitor.Core.Scraping.Configuration.Serialization.v1.Model;
 using YamlDotNet.RepresentationModel;
 
 namespace Promitor.Core.Scraping.Configuration.Serialization.v1.Core
 {
-    internal class AzureMetadataDeserializer : Deserializer<AzureMetadata>
+    internal class AzureMetadataDeserializer : Deserializer<AzureMetadataV1>
     {
         internal AzureMetadataDeserializer(ILogger logger) : base(logger)
         {
         }
 
-        internal override AzureMetadata Deserialize(YamlMappingNode node)
+        internal override AzureMetadataV1 Deserialize(YamlMappingNode node)
         {
             Guard.NotNull(node, nameof(node));
 
             var tenantId = node.Children[new YamlScalarNode("tenantId")];
             var subscriptionId = node.Children[new YamlScalarNode("subscriptionId")];
             var resourceGroupName = node.Children[new YamlScalarNode("resourceGroupName")];
 
-            var azureMetadata = new AzureMetadata
+            var azureMetadata = new AzureMetadataV1
             {
                 TenantId = tenantId?.ToString(),
                 SubscriptionId = subscriptionId?.ToString(),