add context to projectmappings to allow for small modifications (e.g.…

… name, namespace) without having to replace the entire template

Made resource_templates into a map, and made override merging of it (and context) be additive in nature so one can add a new mapping with a template without having to replace the existing ones

README.md

@@ -47,6 +47,7 @@ Parameterize the helm install with `--set appSettings.**` to control how kubetru
 | projectMappings.root.key_selector | A regexp to limit the keys acted against (client-side).  Supplies any named matches for template evaluation | string | "" | no |
 | projectMappings.root.skip | Skips the generation of resources for the selected projects | flag | false | no |
 | projectMappings.root.included_projects | Include the parameters from other projects into the selected ones.  This can be recursive in a depth first fashion, so if A imports B and B imports C, then A will get B's and C's parameters.  For key conflicts, if A includes B and B includes C, then the precendence is A overrides B overrides C.  If A includes \[B, C], then the precendence is A overrides C overrides B. | list | [] | no |
+| projectMappings.root.context | Additional variables made available to the resource templates.  Can also be templates | string | [default](helm/kubetruth/values.yaml#L93-L129) | no |
 | projectMappings.root.resource_templates | The templates to use in generating kubernetes resources (ConfigMap/Secrets/other) | string | [default](helm/kubetruth/values.yaml#L93-L129) | no |
 | projectMappings.<override_name>.* | Define override mappings to override settings from the root selector for specific projects. When doing this on the command-line (e.g. for `helm install`), it may be more convenient to use `--values <file>` instead of `--set` for large data sets | map | {} | no |
 
@@ -136,16 +137,19 @@ them wakes it up from a polling sleep.  This makes it quick and easy to test out
 configuration changes without having a short polling interval.
 
 To customize how the kubernetes resources are generated, edit the
-`resource_templatse` property in the ProjectMappings.  These templates are
+`resource_templates` property in the ProjectMappings.  These templates are
 processed using the [Liquid template
 language](https://shopify.github.io/liquid/), and can reference the following
 liquid variables:
 
  * `project` - The project name
  * `project_heirarchy` - The `included_projects` tree this project includes (useful to debug when using complex `included_projects`)
+ * `debug` - Indicates if kubetruth is operating in debug (logging) mode.
  * `parameters` - The CloudTruth parameters from the project
  * `parameter_origins` - The projects each parameter originates from (useful to debug when using complex `included_projects`)
- * `debug` - Indicates if kubetruth is operating in debug (logging) mode.
+ * `secrets` - The CloudTruth secrets from the project
+ * `secret_origins` - The projects each secret originates from (useful to debug when using complex `included_projects`)
+ * `context` - A hash of context variables supplied from ProjectMappings (useful to override portions of templates without having to replace them completely in an override)
 
 In addition to the built in liquid filters, kubetruth also define a few custom
 ones:
@@ -227,21 +231,9 @@ metadata:
 spec:
     scope: override
     project_selector: funkyProject
-    resource_templates:
-        - |
-            apiVersion: v1
-            kind: ConfigMap
-            metadata:
-                namespace: notSoFunkyNamespace
-                name: notSoFunkyConfigMap
-            <snipped>
-        - |
-            apiVersion: v1
-            kind: Secret
-            metadata:
-                namespace: notSoFunkyNamespace
-                name: notSoFunkySecret
-            <snipped>
+    context:
+        resource_name: notSoFunkyConfigMap
+        resource_namespace: notSoFunkyNamespace
 EOF
 ```
 
@@ -274,24 +266,7 @@ kubetruth-root                root       ^service         27m
 $ kubectl describe pm kubetruth-root
 Name:         kubetruth-root
 Namespace:    default
-Labels:       ...
-Annotations:  ...
-API Version:  kubetruth.cloudtruth.com/v1
-Kind:         ProjectMapping
-Metadata:
-  <snipped>
-Spec:
-  resource_templates:
-    - |
-        <snipped>
-    - |
-        <snipped>
-  included_projects:
-  key_selector:          
-  project_selector:      
-  scope:                 root
-  skip:                  false
-Events:                  <none>
+<snipped>
 ```
 
 ## Development

helm/helmv2/templates/projectmapping.yaml

@@ -32,9 +32,14 @@ spec:
           items:
             type: string
           description: Include the parameters from other projects into the selected ones.  This can be recursive in a depth first fashion, so if A imports B and B imports C, then A will get B's and C's parameters.  For key conflicts, if A includes B and B includes C, then the precendence is A overrides B overrides C.  If A includes [B, C], then the precendence is A overrides C overrides B.
+        context:
+          type: object
+          additionalProperties:
+            type: string
+          description: Context variables that can be used by templates.  The values can also be templates
         resource_templates:
-          type: array
-          items:
+          type: object
+          additionalProperties:
             type: string
           description: The templates to use in generating kubernetes resources
   additionalPrinterColumns:

helm/kubetruth/crds/projectmapping.yaml

@@ -34,9 +34,14 @@ spec:
                   items:
                     type: string
                   description: Include the parameters from other projects into the selected ones.  This can be recursive in a depth first fashion, so if A imports B and B imports C, then A will get B's and C's parameters.  For key conflicts, if A includes B and B includes C, then the precendence is A overrides B overrides C.  If A includes [B, C], then the precendence is A overrides C overrides B.
+                context:
+                  type: object
+                  additionalProperties:
+                    type: string
+                  description: Context variables that can be used by templates.  The values can also be templates
                 resource_templates:
-                  type: array
-                  items:
+                  type: object
+                  additionalProperties:
                     type: string
                   description: The templates to use in generating kubernetes resources
               required:

helm/kubetruth/values.yaml

@@ -89,13 +89,17 @@ projectMappings:
     key_selector: ""
     skip: false
     included_projects: []
+    context:
+      resource_name: "{{ project | dns_safe }}"
+      resource_namespace: ""
     resource_templates:
-      - |
+      configmap: |
         {%- if parameters.size > 0 %}
         apiVersion: v1
         kind: ConfigMap
         metadata:
-          name: "{{ project | dns_safe }}"
+          name: "{{ context.resource_name }}"
+          namespace: "{{ context.resource_namespace }}"
           labels:
             version: "{{ parameters | sort | to_json | sha256 | slice: 0, 7 }}"
           annotations:
@@ -109,12 +113,13 @@ projectMappings:
           {%- endfor %}
         {%- endif %}
 
-      - |
+      secret: |
         {%- if secrets.size > 0 %}
         apiVersion: v1
         kind: Secret
         metadata:
-          name: "{{ project | dns_safe }}"
+          name: "{{ context.resource_name }}"
+          namespace: "{{ context.resource_namespace }}"
           labels:
             version: "{{ secrets | sort | to_json | sha256 | slice: 0, 7 }}"
           annotations:

lib/kubetruth/config.rb

@@ -13,6 +13,7 @@ class DuplicateSelection < Kubetruth::Error; end
       :key_selector,
       :skip,
       :included_projects,
+      :context,
       :resource_templates,
       keyword_init: true
     ) do
@@ -23,17 +24,17 @@ def initialize(*args, **kwargs)
 
       def convert_types(hash)
         selector_key_pattern = /_selector$/
-        template_key_pattern = /_templates?$/
+        template_key_pattern = /_template$/
         hash.merge(hash) do |k, v|
           case k
             when selector_key_pattern
               Regexp.new(v)
             when template_key_pattern
-              if k.ends_with?('s')
-                v.collect {|t| Kubetruth::Template.new(t) }
-              else
-                Kubetruth::Template.new(v)
-              end
+              Kubetruth::Template.new(v)
+            when /^resource_templates$/
+              Hash[v.collect {|k, t| [k.to_s, Kubetruth::Template.new(t)] }]
+            when /^context$/
+              Kubetruth::Template::TemplateHashDrop.new(v)
             else
               v
           end
@@ -48,6 +49,7 @@ def convert_types(hash)
       key_selector: '',
       skip: false,
       included_projects: [],
+      context: {},
       resource_templates: []
     }.freeze
 
@@ -66,7 +68,12 @@ def load
 
         config = DEFAULT_SPEC.merge(root_mapping)
         @root_spec = ProjectSpec.new(**config)
-        @override_specs = overrides.collect { |o| ProjectSpec.new(**config.merge(o)) }
+        logger.debug { "ProjectSpec for root mapping: #{@root_spec.inspect}"}
+        @override_specs = overrides.collect do |o|
+          spec = ProjectSpec.new(**config.deep_merge(o))
+          logger.debug { "ProjectSpec for override mapping: #{spec.inspect}"}
+          spec
+        end
         config
       end
     end

lib/kubetruth/etl.rb

@@ -120,16 +120,18 @@ def apply
         config_origins = Hash[param_origins_parts[true] || []]
         secret_origins = Hash[param_origins_parts[false] || []]
 
-        project.spec.resource_templates.each_with_index do |template, i|
-          logger.debug { "Processing template #{i}/#{project.spec.resource_templates.size}" }
+        project.spec.resource_templates.each_with_index do |pair, i|
+          template_name, template = *pair
+          logger.debug { "Processing template '#{template_name}' (#{i}/#{project.spec.resource_templates.size})" }
           resource_yml = template.render(
             project: project.name,
             project_heirarchy: project.heirarchy,
             debug: logger.debug?,
             parameters: config_param_hash,
             parameter_origins: config_origins,
             secrets: secret_param_hash,
-            secret_origins: secret_origins
+            secret_origins: secret_origins,
+            context: project.spec.context
           )
           parsed_yml = YAML.safe_load(resource_yml)
           if parsed_yml
@@ -148,8 +150,13 @@ def params_to_hash(param_list)
 
     def kube_apply(parsed_yml)
       kind = parsed_yml["kind"]
-      namespace = parsed_yml["metadata"]["namespace"] || kubeapi.namespace
       name = parsed_yml["metadata"]["name"]
+      namespace = parsed_yml["metadata"]["namespace"]
+      if namespace.blank?
+        namespace = parsed_yml["metadata"]["namespace"] = kubeapi.namespace
+      end
+
+      kubeapi.set_managed(parsed_yml)
 
       ident = "'#{namespace}:#{kind}:#{name}'"
       logger.info("Applying kubernetes resource #{ident}")

lib/kubetruth/kubeapi.rb

@@ -15,7 +15,6 @@ def initialize(namespace: nil, token: nil, api_url: nil)
       token_path = '/var/run/secrets/kubernetes.io/serviceaccount/token'
 
       @namespace = namespace.present? ? namespace : File.read(namespace_path).chomp
-      @labels = {MANAGED_LABEL_KEY => MANAGED_LABEL_VALUE}
 
       @auth_options = {}
       if token
@@ -64,14 +63,14 @@ def ensure_namespace(ns = namespace)
     end
 
     def under_management?(resource)
-      labels = resource&.metadata&.labels
-      labels.nil? ? false : resource.metadata.labels[MANAGED_LABEL_KEY] == MANAGED_LABEL_VALUE
+      labels = resource&.[]("metadata")&.[]("labels")
+      labels.nil? ? false : resource["metadata"]["labels"][MANAGED_LABEL_KEY] == MANAGED_LABEL_VALUE
     end
 
     def set_managed(resource)
-      resource.metadata ||= {}
-      resource.metadata.labels ||= {}
-      resource.metadata.labels[MANAGED_LABEL_KEY] = MANAGED_LABEL_VALUE
+      resource["metadata"] ||= {}
+      resource["metadata"]["labels"] ||= {}
+      resource["metadata"]["labels"][MANAGED_LABEL_KEY] = MANAGED_LABEL_VALUE
     end
 
     def get_resource(resource_name, name, namespace=nil)
@@ -80,9 +79,7 @@ def get_resource(resource_name, name, namespace=nil)
 
     def apply_resource(resource)
       resource = Kubeclient::Resource.new(resource) if resource.is_a? Hash
-      set_managed(resource)
       resource_name = resource.kind.downcase.pluralize
-      resource.metadata.namespace ||= self.namespace
       client.apply_entity(resource_name, resource, field_manager: "kubetruth")
     end
 

lib/kubetruth/template.rb

@@ -10,6 +10,30 @@ class Template
     class Error < ::StandardError
     end
 
+    class TemplateHashDrop < Liquid::Drop
+
+      attr_reader :source
+
+      def initialize(template_hash)
+        @source = template_hash.stringify_keys
+        @parsed = {}
+      end
+
+      def liquid_method_missing(key)
+        if @source.has_key?(key)
+          @parsed[key] ||= Template.new(@source[key])
+          @parsed[key].render(@context)
+        else
+          super
+        end
+      end
+
+      def inspect
+        {source: @source, parsed: @parsed}.inspect
+      end
+
+    end
+
     module CustomLiquidFilters
 
       # From kubernetes error message
@@ -93,7 +117,7 @@ def initialize(template_source)
 
     INDENT = (" " * 2)
 
-    def render(**kwargs)
+    def render(*args, **kwargs)
       begin
 
         logger.debug do
@@ -104,7 +128,7 @@ def render(**kwargs)
           msg
         end
 
-        result = @liquid.render!(kwargs.stringify_keys, strict_variables: true, strict_filters: true)
+        result = @liquid.render!(*args, kwargs.stringify_keys, strict_variables: true, strict_filters: true)
 
         logger.debug do
           msg = "Rendered template:\n"
@@ -115,6 +139,16 @@ def render(**kwargs)
         result
 
       rescue Liquid::Error => e
+        indent = "  "
+        msg = "Template failed to render:\n"
+        @source.lines.each {|l| msg << (indent * 2) << l }
+        msg << indent << "with error message:\n" << (indent * 2) << "#{e.message}"
+        if e.is_a?(Liquid::UndefinedVariable)
+          msg << "\n" << indent << "and variable context:\n"
+          msg << (indent * 2) << kwargs.inspect
+        end
+        raise Error, msg
+
         raise Error, "Template failed to render: #{e.message}"
       end
     end