Simplify handling of inputs from files using emitters #1179 (#1830)

docs/CHANGELOG-v3.md

@@ -29,6 +29,14 @@ See [upgrade notes][1] for helpful information when upgrading from previous vers
 
 What's changed since pre-release v3.0.0-B0198:
 
+- New features:
+  - **Breaking change**: Simplify handling of inputs from files using emitters by @BernieWhite.
+    [#1179](https://github.com/microsoft/PSRule/issues/1179)
+    - Files are automatically read from input paths and emitted as objects to the pipeline.
+    - Emitter interface can be used to implement custom file readers and expansion of custom file types.
+    - The `File` and `Detect` input formats are no longer required and have been removed.
+    - Processing files and objects with rules is no longer recommended, and disabled by default.
+    - The `Input.FileObjects` can be set to `true` to enable processing of files as objects with rules.
 - Bug fixes:
   - Fixed reason reported for `startsWith` by @BernieWhite.
     [#1818](https://github.com/microsoft/PSRule/issues/1818)

docs/concepts/PSRule/en-US/about_PSRule_Options.md

@@ -27,6 +27,7 @@ The following workspace options are available for use:
 - [Execution.UnprocessedObject](#executionunprocessedobject)
 - [Include.Module](#includemodule)
 - [Include.Path](#includepath)
+- [Input.FileObjects](#inputfileobjects)
 - [Input.Format](#inputformat)
 - [Input.IgnoreGitPath](#inputignoregitpath)
 - [Input.IgnoreObjectSource](#inputignoreobjectsource)
@@ -1514,6 +1515,55 @@ variables:
   value: .ps-rule/;custom-rules/
 ```
 
+### Input.FileObjects
+
+Determines if file objects are processed by rules.
+This option is for backwards compatibility with PSRule v2.x in cases where file objects are used as input.
+
+By default, file are not processed by rules.
+Set to `$True` to enable processing of file objects by rules.
+
+This option can be specified using:
+
+```powershell
+# PowerShell: Using the InputFileObjects parameter
+$option = New-PSRuleOption -InputFileObjects $True;
+```
+
+```powershell
+# PowerShell: Using the Input.FileObjects hashtable key
+$option = New-PSRuleOption -Option @{ 'Input.FileObjects' = $True };
+```
+
+```powershell
+# PowerShell: Using the InputFileObjects parameter to set YAML
+Set-PSRuleOption -InputFileObjects $True;
+```
+
+```yaml
+# YAML: Using the input/fileObjects property
+input:
+  fileObjects: true
+```
+
+```bash
+# Bash: Using environment variable
+export PSRULE_INPUT_FILEOBJECTS=true
+```
+
+```yaml
+# GitHub Actions: Using environment variable
+env:
+  PSRULE_INPUT_FILEOBJECTS: true
+```
+
+```yaml
+# Azure Pipelines: Using environment variable
+variables:
+- name: PSRULE_INPUT_FILEOBJECTS
+  value: true
+```
+
 ### Input.Format
 
 Configures the input format for when a string is passed in as a target object.
@@ -1531,22 +1581,16 @@ The `-InputPath` parameter can be used with a file path or URL.
 
 The following formats are available:
 
-- None - Treat strings as plain text and do not deserialize files.
-- Yaml - Deserialize as one or more YAML objects.
-- Json - Deserialize as one or more JSON objects.
-- Markdown - Deserialize as a markdown object.
-- PowerShellData - Deserialize as a PowerShell data object.
-- File - Files are not deserialized.
-- Detect - Detect format based on file extension. This is the default.
-
-If the `Detect` format is used, the file extension will be used to automatically detect the format.
-When the file extension can not be determined `Detect` is the same as `None`.
+- `None` - Treat strings as plain text and do not deserialize files.
+  This is the default.
+- `Yaml` - Deserialize as one or more YAML objects.
+- `Json` - Deserialize as one or more JSON objects.
+- `Markdown` - Deserialize as a markdown object.
+- `PowerShellData` - Deserialize as a PowerShell data object.
 
 The `Markdown` format does not parse the whole markdown document.
 Specifically this format deserializes YAML front matter from the top of the document if any exists.
 
-The `File` format does not deserialize file contents.
-Each file is returned as an object.
 Files within `.git` sub-directories are ignored.
 Path specs specified in `.gitignore` directly in the current working path are ignored.
 A `RepositoryInfo` object is generated if the current working path if a `.git` sub-directory is present.

docs/specs/emitter-spec.md

@@ -0,0 +1,29 @@
+# Emitter specification
+
+A key component of PSRule design is the ability to read objects from file or PowerShell pipeline.
+PSRule provided built-in support for a limited number of file formats and object types.
+Conventions could be used to provide support for additional file formats and object types.
+
+Current challenges:
+
+- **Performance**: When processing a high volume of objects, reading objects from disk across multiple times can be slow.
+  Many repositories have a large number of files that need to be read and processed.
+  CI workflows often executed in container environments where disk access is slow or limited resources are available.
+- **Confusing output**: Files processed as objects often end up reporting as unprocessed, generating a default warning.
+  This is because they are processed by each available rule, but no rules are defined to process them.
+- **Expansion**: Handover of objects to conventions can be complex and does not expose strongly typed objects.
+  Expansion is a pattern used by PSRule for Azure to expand a file into multiple objects.
+  This is helpful when PSRule does not support a file format or object type natively.
+
+Goals with emitters:
+
+- **Single touch**: If a file needs to be read from disk, it should be read once to reduce IO.
+- **Extensible**: A interface should be provided to allow for custom emitters that can be used to read objects from any source.
+- **Multi-threaded**: Emitters should be thread safe and support parallel processing.
+- **Carried as needed**: Emitter should be only emit objects to be processed by rules.
+  File objects are not processed by default.
+
+The goal of emitters is to provide a high performance and extensible way to emit custom objects to the input stream.
+
+Emitters define a C# `IEmitter` interface for emitting objects to the input stream.
+The implementation of an emitter must be thread safe, as emitters can be run in parallel.

docs/specs/option-spec.md

@@ -4,7 +4,7 @@ This is a spec for implementing options in PSRule v2.
 
 ## Synopsis
 
-When executing resources options is often required to control the specific behaviour.
+When executing resources options is often required to control the specific behavior.
 In additional, many of these cases are scoped to the module being used.
 
 The following scopes exist:

pipeline.build.ps1

@@ -322,7 +322,7 @@ task Rules {
         As = 'Summary'
     }
     Import-Module (Join-Path -Path $PWD -ChildPath out/modules/PSRule) -Force;
-    Assert-PSRule @assertParams -OutputPath reports/ps-rule-file.xml -InputPath $PWD -Format File -ErrorAction Stop;
+    Assert-PSRule @assertParams -OutputPath reports/ps-rule-file.xml -InputPath $PWD -ErrorAction Stop;
 }
 
 task Benchmark {

ps-rule.yaml

@@ -33,6 +33,8 @@ input:
     - '.markdownlint.json'
     - '.github/dependabot.yml'
 
+    - '**/Resources/*.json'
+
     # Bug #1269: There is an issue preventing #1259 from being merged. Ignore this until bug is fixed.
     - 'docs/scenarios/containers/dockerfile'
 

schemas/PSRule-options.schema.json

@@ -301,6 +301,55 @@
       },
       "additionalProperties": false
     },
+    "emitter-option": {
+      "type": "object",
+      "title": "Emitter options",
+      "properties": {
+        "yaml": {
+          "type": "object",
+          "title": "YAML",
+          "description": "Support for the YAML format.",
+          "properties": {
+            "include": {
+              "type": "array",
+              "title": "Include",
+              "description": "Additional file types to process as YAML.",
+              "default": []
+            }
+          },
+          "additionalProperties": false
+        },
+        "json": {
+          "type": "object",
+          "title": "JSON",
+          "description": "Support for the JSON format including support for comments.",
+          "properties": {
+            "include": {
+              "type": "array",
+              "title": "Include",
+              "description": "Additional file types to process as JSON.",
+              "default": []
+            }
+          },
+          "additionalProperties": false
+        },
+        "markdown": {
+          "type": "object",
+          "title": "Markdown",
+          "description": "Support for the Markdown format.",
+          "properties": {
+            "include": {
+              "type": "array",
+              "title": "Include",
+              "description": "Additional file types to process as Markdown.",
+              "default": []
+            }
+          },
+          "additionalProperties": false
+        }
+      },
+      "additionalProperties": false
+    },
     "execution-option": {
       "type": "object",
       "title": "Execution options",
@@ -500,6 +549,13 @@
       "description": "Options that affect how input types are processed.",
       "markdownDescription": "Options that affect how input types are processed.",
       "properties": {
+        "fileObjects": {
+          "type": "boolean",
+          "title": "File objects",
+          "description": "Determines if file objects are processed by rules.",
+          "markdownDescription": "Determines if file objects are processed by rules. [See help](https://microsoft.github.io/PSRule/v3/concepts/PSRule/en-US/about_PSRule_Options/#fileobjects)",
+          "default": false
+        },
         "format": {
           "type": "string",
           "title": "Input format",
@@ -885,6 +941,10 @@
           "type": "object",
           "$ref": "#/definitions/convention-option"
         },
+        "emitter": {
+          "type": "object",
+          "$ref": "#/definitions/emitter-option"
+        },
         "execution": {
           "type": "object",
           "$ref": "#/definitions/execution-option"

src/PSRule.CommandLine/ClientContext.cs

@@ -65,7 +65,7 @@ private static PSRuleOption GetOption(ClientHost host, string? path)
         PSRuleOption.UseHostContext(host);
         var option = PSRuleOption.FromFileOrEmpty(path);
         option.Execution.InitialSessionState = Options.SessionState.Minimal;
-        option.Input.Format = InputFormat.File;
+        //option.Input.Format = InputFormat.File;
         option.Output.Style ??= OutputStyle.Client;
         return option;
     }

src/PSRule.Types/Data/ITargetObject.cs

@@ -27,4 +27,9 @@ public interface ITargetObject
     /// The path of the object.
     /// </summary>
     string? Path { get; }
+
+    /// <summary>
+    /// The value of the object.
+    /// </summary>
+    object Value { get; }
 }

src/PSRule.Types/Emitters/BaseEmitter.cs

@@ -0,0 +1,46 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+namespace PSRule.Emitters;
+
+/// <summary>
+/// A base class for implementing an emitter.
+/// </summary>
+public abstract class BaseEmitter : IEmitter
+{
+    private bool _Disposed;
+
+    /// <inheritdoc/>
+    public abstract bool Visit(IEmitterContext context, object o);
+
+    /// <inheritdoc/>
+    public abstract bool Accepts(IEmitterContext context, Type type);
+
+    #region IDisposable
+
+    /// <summary>
+    /// Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
+    /// </summary>
+    /// <param name="disposing">Determines if a dispose is occuring.</param>
+    protected virtual void Dispose(bool disposing)
+    {
+        if (!_Disposed)
+        {
+            if (disposing)
+            {
+                // TODO: dispose managed state (managed objects)
+            }
+            _Disposed = true;
+        }
+    }
+
+    /// <inheritdoc/>
+    public void Dispose()
+    {
+        // Do not change this code. Put cleanup code in 'Dispose(bool disposing)' method
+        Dispose(disposing: true);
+        GC.SuppressFinalize(this);
+    }
+
+    #endregion IDisposable
+}

src/PSRule.Types/Emitters/FileEmitter.cs

@@ -0,0 +1,67 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+using PSRule.Data;
+
+namespace PSRule.Emitters;
+
+/// <summary>
+/// An emitter that implements emitting objects from file streams.
+/// </summary>
+public abstract class FileEmitter : BaseEmitter
+{
+    /// <inheritdoc/>
+    public sealed override bool Accepts(IEmitterContext context, Type type)
+    {
+        return context != null && type != null &&
+            (typeof(IFileInfo).IsAssignableFrom(type) || type == typeof(string));
+    }
+
+    /// <inheritdoc/>
+    public sealed override bool Visit(IEmitterContext context, object o)
+    {
+        if (context == null || o == null) return false;
+
+        if (o is IFileInfo info && AcceptsFilePath(context, info))
+            return VisitFile(context, info.GetFileStream());
+
+        return o is string s && AcceptsString(context) ? VisitString(context, s) : false;
+    }
+
+    /// <summary>
+    /// Determines if the emitter accepts a file based on it's path.
+    /// </summary>
+    /// <param name="context"></param>
+    /// <param name="info"></param>
+    /// <returns></returns>
+    protected abstract bool AcceptsFilePath(IEmitterContext context, IFileInfo info);
+
+    /// <summary>
+    /// Determines if the emitter accepts an input string as content.
+    /// </summary>
+    /// <param name="context"></param>
+    /// <returns></returns>
+    protected virtual bool AcceptsString(IEmitterContext context)
+    {
+        return false;
+    }
+
+    /// <summary>
+    /// Visit a specific file.
+    /// </summary>
+    /// <param name="context"></param>
+    /// <param name="stream"></param>
+    /// <returns></returns>
+    protected abstract bool VisitFile(IEmitterContext context, IFileStream stream);
+
+    /// <summary>
+    /// Visit a string.
+    /// </summary>
+    /// <param name="context"></param>
+    /// <param name="content"></param>
+    /// <returns></returns>
+    protected virtual bool VisitString(IEmitterContext context, string content)
+    {
+        return false;
+    }
+}